Topic
Data engineering
Most of the data science on this site only worked because of the engineering underneath it: streaming pipelines for tens of millions of network events a day, a telematics workload modernised on Databricks, and decision support systems built over municipal data. Ubunye Engine is the open source result of meeting the same pipeline plumbing problem in every one of those places, and the case studies say what the data layer had to do in each.
4 work · 4 writing · Wikidata Q104659521
Work
- Data science platform at a bank: telematics and MLOpsAt Absa Group: architected the data engineering capability data science runs on, with Ubunye Engine at 100M+ events a day for 10+ products.
- Decision support systems for municipalities at the CSIRDjango based predictive analytics and decision support systems serving 17 municipalities, including the City of Cape Town, built at the CSIR.
- Generator optimisation and streaming at VodacomReal time analytics and optimisation for a national telecoms network: generator dispatch across 15,000+ sites and tens of millions of events a day.
- Ubunye Engine: portable Spark pipelines for data and MLUbunye Engine is an open source Python framework for config driven Spark pipelines. The same task folder runs on a laptop, Docker, Kubernetes or Databricks.
Writing
- Stage 2: Give Your Data One Safe HomeA model can only be as good as the data it learns from. How we brought scattered data into one checked and controlled home, and made security part of normal work.
- Stage 4: When People Push Back, Show ThemThe technology was not the hardest part. People were. How we got through pushback and red tape, by showing value in action, and how a crisis became our best argument.
- Who You Need on the TeamWho belongs on a data team, who to hire first, what to do when the budget runs out, why graduates matter, and where the team should report.
- How to Build a Data Science Function From Scratch: A RoadmapA five stage roadmap for building a data science, data engineering or AI function from scratch: what you will find, what to do first, what pushback to expect, and how to know when to move on.
Related topics
Subjects that share work with this one. Those with their own page are linked.
- Technical leadership
- Data science
- Python
- Apache Spark
- MLOps
- Databricks
- Decision support systems
- Docker
- Extract, transform, load
- Kubernetes
- Apache Flink
- Apache Kafka