Notes from running machine learning in production inside a large, regulated enterprise: model registries, reproducible pipelines, the unglamorous plumbing that decides whether a model survives contact with real data, and why convention beats configuration when a team has to maintain what you built.
7 posts
Two years ago I inherited a small data science team with no handover, no documentation, and production models nobody could explain. This is the conversation I wish someone had with me before I started: the challenges, the costs, the thinking behind every move, and the one belief that carried it all. Build systems that outlive the people who built them.
The engine is now on version 0.5.0. This part shows how to actually use it: install it, write your first pipeline, run it on your laptop, then run the exact same folder on Docker, Kubernetes and Databricks. Every snippet here was run before it was written.
After all the unit tests and CI pipelines, the question remained: does it actually work on real data? The Kaggle Titanic dataset became the proving ground. This post covers what it proved, what it did not, and how Ubunye compares to existing frameworks.
Documentation, CI/CD, PyPI publishing, and the subtle bugs that live in single lines of code. The thesis is simple: finishing is rarer than starting, and the boring work is what separates a repository from a product.
The job hasn't disappeared. The mental model has changed. Here's what the roadmap looks like for data scientists and engineers entering (or navigating) the AI agent era, with a South African lens.
The model registry was where the project went from interesting framework to something a team could actually use in production. The design principle behind it turned out to be the most significant architectural decision in the entire project.
Every data team eventually hits the same wall. Notebooks work locally. Spark jobs work on the cluster. Nothing works together. This is the problem Ubunye Engine was built to solve.