Python Package for ML deployments

Inspiration

Deploying machine learning models is hard. Especially dependencies, i.e. which numpy version goes with which sk-learn version. Docker fixes this problem but using a reusable images as long as the dockerfile is correctly populated. But populating the dockerfile can also be difficult sometimes, especially for data-scientists who primarily work within jupyter notebooks. They have to leave the jupyter notebooks, identify dependencies and then populate the docker files correctly.

What it does

To simplify this process, we created a python sdk that makes it easy to create a docker file and docker container from within the jupyter notebook. Our package is framework agnostic, which means that models can be built in any framework i.e. pytorch, hugging-face and sklearn, we can package them into a unified package, along with the relevant dependencies and create an endpoint around the inferences. This makes it easy to deploy the model to any cloud provider for inference.

How we built it

We took inspiration from packages that are trying to do similar things i.e. cog, and truss. We combined these two into a single package, with simple abstractions and the ability to save models built in any framework, with any python package.

Challenges we ran into

Creating a unified interface for different frameworks is hard as each framework has its own save mechanisms. Simple python pickling does not work here, because we also need to store the information of the third-party packages that the inference could depend upon. Similarly, inference endpoints are different for different packages. So the code requires a lot of if-elses.

Accomplishments that we're proud of

Getting two or three different ecosystem to work together.

What we learned

Python AST and dependencies are a complicated problem

What's next for Python Package for ML deployments

Simplify the code. Improve automatically inferring dependencies. Work on infrastructure such that models can be deployed easily. Create templates for famous LLMs to give users an idea of how to use the package.

Built With

• cog
• docker
• pipreq
• python
• truss