Open Source · Ubunye AI Ecosystems

Ubunye Engine: portable Spark pipelines for data and ML

The pipeline should outlive the platform it runs on

Data and ML teams rebuild the same infrastructure plumbing every time work moves between a laptop, a shared cluster and production. Ubunye Engine separates what a pipeline does from where it runs, so the logic stays still while the infrastructure underneath it is replaced.

What it is: Open source software
Role: Author and maintainer
Organisation: Ubunye AI Ecosystems
Period: 2025 to present (repository created September 2025; version 0.5.0)
Status: Active
Licence: MIT
Built with: Python
Apache Spark
Databricks
Kubernetes
Docker
Topics: Data engineering
MLOps
Extract, transform, load
Reproducibility
Open source software

Inspect itGitHub PyPI Documentation Examples Ubunye AI Ecosystems

Ubunye Engine: portable Spark pipelines for data and ML: The pipeline should outlive the platform it runs on

The problem

You join a team, open the repo, and find five Spark projects, each structured differently, each with its own way of handling configs, credentials and deployment. One uses a JSON file, another hardcodes everything, a third has a 300-line bash script that someone wrote and that just works. Building a pipeline from scratch is mostly plumbing: wire up the connection, juggle credentials, learn a framework’s quirks, and write the same read, transform, write scaffold again. It is a lot of glue code standing between you and the three lines that actually matter.

Why it mattered

A lot of the intelligence an organisation builds, and most research that has to become a working system, is gated by engineering rather than by the idea. The individual tools are good at their own layer. The difficulty is that people work on laptops, on on-prem clusters and on different clouds, and connecting all of it is where the time goes. Engineering sits underneath everything else, so it is worth solving once rather than per team.

The context

The pattern showed up in every environment I worked in: real-time analytics on national telecoms infrastructure at Vodacom, enterprise ML at Absa, geospatial models going into a product at IBM Research. The specifics changed, the pattern did not. Ubunye Engine is one artifact inside Ubunye AI Ecosystems, a broader open-source effort built on the idea that we should also build tools from the problems we keep hitting, not only consume them. Ubunye is isiZulu for unity, and the goal was never to add another tool to the stack, it was to agree on how the pieces fit.

What I did

I built the Engine around three ideas. Config over code: a pipeline is a YAML file declaring inputs, outputs and settings, not a program. Plugins for everything: a connector is a small class registered from outside, so a new data source needs no engine edits. Folders as architecture: every project is laid out as use case, pipeline, task, and the CLI uses that structure to scaffold, validate, plan and run. You write a transform() method; the engine handles connections, the Spark session, the read and write loop, model versioning and lineage.

What changed

The same task folder runs on a laptop, in Docker, on Kubernetes, against object storage, through a spark-submit path and on Databricks, with no code change. Most of that is tested rather than claimed: a build job runs one pipeline on local Spark, Docker, Kubernetes, an S3-compatible store and spark-submit, and fails unless all five produce byte-identical output. Databricks deploys and runs the same folder through its own job, where each notebook asserts its own output rather than being compared against the other five. One rule makes it work, and it is deliberately strict: a task never chooses its own cluster, and the engine refuses a config that tries to, because a silent single-node run on paid compute is worse than an error.

Who benefited

Teams that have outgrown scattered scripts, at either end of the size range, and the next person who opens the repo and can run something in minutes instead of a week. It is also deliberately scoped, which matters as much as what it does: it is not an agent framework, not an orchestrator and not a compute engine. It is the standardisation layer between data sources and applications, and it makes the plumbing boring.

What remained

A published framework on PyPI with a documentation site and eleven worked examples, each one executed rather than only reviewed, seven built-in connectors, a model registry that writes to a local folder, a Databricks volume, S3 or GCS chosen purely by the path, lineage you can trace after the fact, and a plugin contract that lets someone add a source the engine has never seen. Claims that could not be executed were removed from the docs rather than left to mislead.

Technical context

Apache Spark, Python, config as YAML with Jinja2 templating validated by Pydantic, connectors for hive, jdbc, delta, s3, unity, binary and rest_api, a CLI (init, validate, plan, run, test, lineage, models), a Python API for Databricks, Delta Lake, Docker, Kubernetes, and a spark-submit path, which is the shape EMR Serverless and Dataproc Serverless use. The setup and submit scripts for both exist and are deliberately marked as not yet executed, since neither service has a free tier. The package ships its types and a type checker guards every merge.

Lineage

Came before: Network intelligence at Vodacom. One of the environments where the same pipeline plumbing problem kept appearing.
Came before: Data science platform at Absa. The enterprise ML setting the engine was designed against.