Ivy: The Unified Machine Learning Framework

This interview is part of our OSS Spotlight series where we showcase founders of fast-growing open-source projects that are solving really unique problems and experiencing strong community adoption.

Sudip Chakrabarti spoke to Daniel Lenton, creator of the open-source project Ivy that is a unified Machine Learning framework enabling data scientists to automatically convert and use ML models across several popular ML frameworks like TensorFlow, PyTorch, JAX, etc.

Daniel shared with us his motivation behind creating Ivy and how he is keeping up with all the fast growth of the project.

Tell us about yourself. What did you do before, what has been your journey until you started working on Ivy?

Daniel: Funnily enough, for someone who created a project like Ivy that is all about Computer Science, I had started my academic life as a Mechanical Engineer at Imperial College where I studied engines and stuff. But, very soon I realized that I didn’t care much about any of that and really enjoyed programming. It is not so easy to switch majors in the UK; so, I had to learn on my own by watching a lot of online courses and building a ton of software on the side. Luckily, I was able to use that to leapfrog into the CS PhD program at Imperial College which then enabled me to dive much, much deeper into building software that I really enjoyed. And, that’s all I have been doing ever since.

Typically, open-source projects are started by folks who are frustrated by some problem. What was your motivation behind creating Ivy?

Daniel: I started my PhD in a lab that worked on Robotics and Machine Learning. Folks in the lab were using all kinds of ML frameworks - TensorFlow, PyTorch, JAX, etc. - and we all ended up with a very fragmented set of tools doing very similar work and wasting time re-implementing each other’s work in different frameworks. So, I saw an opportunity to build a tool that would enable people to write their models once - in their choice of framework - and allow them to share easily across other frameworks. While Ivy had nothing to do with my core PhD thesis, I figured that the broader ML world would also have the exact same need as my labmates. That led me to create what is now Ivy.

Tell us some of the history of the project - what were the early days like and what made you decide to open-source the project?

Daniel: The first version of Ivy was actually an embarrassingly long python file with a bunch of sort-of “if-then-else” code for different ML frameworks. But over time, as I became increasingly obsessed with software engineering principles, the architecture and the code base evolved to what Ivy is today. The decision to open source the project, however, was a no-brainer from day one for a couple of reasons. First, it was about achieving development scale. During the first few months of creating Ivy, I was adding up to ten functions every week and realized that there were a lot of low-hanging “implementation fruit” that lent itself very well to small but meaningful contributions from the community. The other reason was that, as a developer myself, I would very rarely use and trust a tool that wasn’t open source. So, it made complete sense to open source Ivy because I was building for my own self.

“It made complete sense to open source Ivy because I was building for my own self.”

What problem are you addressing with Ivy? Why would someone use it? How do their lives get better by using Ivy?

Daniel: The most intuitive use of Ivy is for someone in an R&D team who wants to use the latest and greatest models in their ML pipelines but has been frustrated because of differences between frameworks. For example, let’s say you are a PyTorch user who has just come across the latest awesome paper from DeepMind and want to use that new model in your ML pipeline. Today, either you would have to reimplement that model in PyTorch or wait for someone else from your team or the open-source community to do it for you. But for ML models, the devil is often in the details and it is easy to miss the nuances of the original model. So, if you had a tool that could automatically and accurately convert the original model to any framework of your choice in a matter of seconds it would be pretty magical! That’s exactly what Ivy does.

“If you had a tool that could automatically and accurately convert the original model to any framework of your choice in a matter of seconds it would be pretty magical! That’s exactly what Ivy does.”

Now that Ivy is out in the open, we are seeing other really interesting use cases pop up as well. For example, the developer of a new PyTorch library can now make it immediately available to users of other frameworks like TensorFlow or JAX with the click of a button and without investing any time or resources to reimplement.

In the long term, the grand vision here is that, as ML software (frameworks like TensorFlow, PyTorch, JAX, etc.) and hardware platforms (Graphcore, TPUs, FPGAs, etc.) continue to get fragmented, Ivy will become that center point to make different participants in the ecosystem work seamlessly with each other.

Tell us a bit about the progress so far. What metrics do you track and why? Why do those metrics matter to the project?

Daniel: We certainly track the “vanity metric” of Github stars - we are at over 5,000 stars in less than a year since launching - which has some correlation to how the user community is growing. The other much more important metric we track is the number of contributors, which is 330+ today and fairly high given how young the project is. We pay a lot of attention to the number of contributors because it is a direct measure of the community’s active participation in development.

So, tell us more about what you have done so far to build such a large contributor base.

Daniel: It is all about making it really easy for new contributors to contribute and we have invested heavily in that. For example, the last five months I have spent little time coding and mostly writing documents and creating youtube videos so that anyone can pick up the project and contribute easily. I myself test every video and doc we put out there. In addition, we have built the necessary testing infrastructure for people to contribute without breaking the code. All of this has allowed us to scale our community contributions and on some days, we are able to handle up to fifteen pull requests with a fairly small core team. Finally, we are a bit fortunate that the architecture of Ivy allows hundreds of functions to be developed using the exact same roadmap - so, if someone has already developed one function it is easier to add the next one. And of course, you have to be welcoming to your new contributors and be patient in answering questions from them as they get up to speed.

Any mistakes you have made in building the community and learnings you have had so far?

Daniel: Two so far, and hopefully not very many in the future. First, in the early days of Ivy we had some tasks that depended on several other subtasks (in some cases twenty of those) which made the merge process very slow and very painful. We were able to do a partial redesign of Ivy and remove some of those complex dependencies. The other mistake in the early days was our slow response to some of the pull requests. When you are building a fast-growing open-source project with PRs flying in from all different directions it is easy to fall behind. Thankfully, we course corrected and invested in infrastructure including a PR dashboard which now lets us track, prioritize and address PRs much faster than before.

Are there other projects - open-source or proprietary - that are solving a similar problem? Any quick thought on why you feel that you have the best approach out there?

Daniel: There are a few projects out there, which solve different parts of the same problem that Ivy is trying to solve. For example, EagerPy is a great project which provides a fully functional framework with a selectable backend, with a similar design to Ivy. However, it doesn’t include higher level classes, and it doesn’t enable transpilations. Thinc is also a great project which provides slightly higher level and easy-to-use abstractions, but again, it doesn’t include transpilations. In both cases, this means people need to learn the new tool and start implementing things using it, before any advantage can be seen. In contrast, Ivy doesn’t require anyone to learn the Ivy syntax and write Ivy code - just a single function decorator is needed and you can convert code between the various frameworks. ONNX is probably the most frequently used tool for performing inter-framework conversions such as this, but the focus there is very much on simplifying the deployment process, once training has been completed. In contrast, Ivy also enables code conversions at training time, and is able to unify legacy code from older framework versions which don’t ship with ONNX bindings. Ivy’s API coverage is also broader than ONNX, where we generally focus on the superset of all array based functions, whereas ONNX is much more focused on neural networks. Overall, as far as I can tell, there is nothing else out there that is looking to unify all ML frameworks to the same extent that Ivy is.

What have you learned since launching the project that you wish you knew when you had started? What would be one thing you would have done differently knowing what you know now.

Daniel: When we started, I think we could’ve certainly thought through the greenfield vs. brownfield use cases for Ivy a bit more. This is also something I notice open-source founders do not spend enough time on. For example, in the early days, Ivy did not have any transpiler to convert existing models from other frameworks into Ivy. We naively thought that just by making Ivy available people would write and share models directly on Ivy and then publish those models to their framework of choice. But, we quickly learned that people needed transpilers as they had a lot of existing investments into models built on those frameworks and were not going to throw those away. So, we had to build transpilers to bring in existing models into Ivy and that has greatly helped our adoption in brownfield use cases.

For you to build a business on top of the open-source Ivy project, what do you believe needs to happen?

Daniel: First, Ivy really needs to work 100% of the time for all permutations and combinations of ML frameworks and hardware platforms. We are close to getting there. Second, the ML space needs to have a high pace of innovation with people constantly building, sharing and trying out new models. I feel very good about that happening. In addition, fragmentation in the ML tool chain, both in frameworks and hardware platforms, makes Ivy a must-have tool for both practitioners as well as vendors who want to participate in the ML ecosystem.

What advice would you have for someone who is thinking of starting a new open-source project?

Daniel: First, do make sure that the problem you are solving is a really big pain point for enough people. Building an open-source project takes a lot of time, determination and patience; so, better make sure that you are in absolute love with the problem you are solving. Second, if your project is going to be part of a larger ecosystem - which typically is the case - make sure you figure out a way to work with the existing tool chain. Third, design the project in such a way that it is easy for others to contribute and also make sure that you create all the resources and infrastructure to make that happen. Finally, be comfortable with giving up some control of the project to the community. A successful open-source project needs a visionary creator but it can only succeed with a thriving and supporting community.

‍

‍