Everything you build sits on engineering
Most of what an organisation builds with data and AI is gated by engineering, not by the idea, and the same plumbing gets rebuilt on every team and every platform. Ubunye Engine is one agreed way to describe a pipeline, wherever it ends up running.
The problem
You join a team, open the repo, and find five Spark projects, each structured differently, each with its own way of handling configs, credentials and deployment. One uses a JSON file, another hardcodes everything, a third has a 300-line bash script that someone wrote and that just works. Building a pipeline from scratch is mostly plumbing: wire up the connection, juggle credentials, learn a framework’s quirks, and write the same read, transform, write scaffold again. It is a lot of glue code standing between you and the three lines that actually matter.
Why it mattered
A lot of the intelligence an organisation builds, and most research that has to become a working system, is gated by engineering rather than by the idea. The individual tools are good at their own layer. The difficulty is that people work on laptops, on on-prem clusters and on different clouds, and connecting all of it is where the time goes. Engineering sits underneath everything else, so it is worth solving once rather than per team.
The context
The pattern showed up in every environment I worked in: real-time analytics on national telecoms infrastructure at Vodacom, enterprise ML at ABSA, geospatial models going into a product at IBM Research. The specifics changed, the pattern did not. Ubunye Engine is one artifact inside Ubunye AI Ecosystems, a broader open-source effort built on the idea that we should also build tools from the problems we keep hitting, not only consume them. Ubunye is isiZulu for unity, and the goal was never to add another tool to the stack, it was to agree on how the pieces fit.
What I did
I built the Engine around three ideas. Config over code: a pipeline is a YAML file declaring inputs, outputs and settings, not a program. Plugins for everything: a connector is a small class registered from outside, so a new data source needs no engine edits. Folders as architecture: every project is laid out as use case, pipeline, task, and the CLI uses that structure to scaffold, validate, plan and run. You write a transform() method; the engine handles connections, the Spark session, the read and write loop, model versioning and lineage.
What changed
The same task folder runs on a laptop, in Docker, on Kubernetes, against object storage, on Databricks and through a cloud submit path, with no code change. That is tested rather than claimed: a build job runs one pipeline across six environments and fails if a single byte of output differs. One rule makes it work, and it is deliberately strict: a task never chooses its own cluster, and the engine refuses a config that tries to, because a silent single-node run on paid compute is worse than an error.
Who benefited
Teams that have outgrown scattered scripts, at either end of the size range, and the next person who opens the repo and can run something in minutes instead of a week. It is also deliberately scoped, which matters as much as what it does: it is not an agent framework, not an orchestrator and not a compute engine. It is the standardisation layer between data sources and applications, and it makes the plumbing boring.
What remained
A published framework on PyPI with a documentation site and eleven worked examples that have all actually been run, seven built-in connectors, a model registry that writes to a local folder, a Databricks volume, S3 or GCS chosen purely by the path, lineage you can trace after the fact, and a plugin contract that lets someone add a source the engine has never seen. Claims that could not be executed were removed from the docs rather than left to mislead.
Technical context
Apache Spark, Python, config as YAML with Jinja2 templating validated by Pydantic, connectors for hive, jdbc, delta, s3, unity, binary and rest_api, a CLI (init, validate, plan, run, test, lineage, models), a Python API for Databricks, Delta Lake, Docker, Kubernetes, and submit paths for EMR Serverless and Dataproc Serverless. The package ships its types and a type checker guards every merge.
Related
Tfilterspy
Real sensors lie. GPS, radar, vehicle trackers and IoT devices all produce noisy readings, and you need an honest estimate of what is actually happening underneath the noise.
WritingHow I Learned to Build My Own Python Libraries (From Curiosity to Real Work)
creating from 1st principles
WorkNetwork intelligence and optimisation
Enormous volumes of noisy, weakly labelled operational data from a national network had to become decisions about where finite resources and intervention were needed most.
WorkBuilding the capability around insurance data science
Getting analytical products into production reliably depended on individual knowledge and one-off effort, in a function historically oriented more toward BI and analysis than production data-science engineering.