Generator optimisation and streaming at Vodacom
Deciding where a national network needs attention first
Enormous volumes of imperfect operational data had to become decisions about where resources, infrastructure and intervention were needed most across a national telecommunications network.
- What it is
- Production system
- Role
- Senior Data Scientist, leading a team of ten
- Organisation
- Vodacom
- Period
- November 2021 to March 2024
- Status
- Completed
- Built with
- Apache Kafka
- Apache Flink
- Apache Spark
- Kubernetes
- Docker
- Python
- Topics
- Mathematical optimisation
- Stream processing
- Telecommunications
- Decision support systems
- Data engineering
- MLOps
- Technical leadership
The problem
A national mobile network generates continuous operational data from tens of thousands of sites, and much of it is noisy, weakly labelled and spread across separate operational systems. The recurring question was where to send finite resources next. During load-shedding that sharpened: sites fall back to generators and batteries, and fuel, logistics and field teams all run out before the problem does.
Why it mattered
A site that goes down is lost service for real people, and misdirected diesel and crews are real money at national scale. Answering it well needed the data turned into a decision quickly enough to act on, not a report produced after the fact.
The context
15,000+ national infrastructure sites and mobile generators, interdependent network effects, operational alarms, and telemetry arriving continuously. This was operational real-time scale rather than a static dataset: the systems had to keep making sense of live events across national infrastructure.
What I did
I led a team of 10 data scientists and ML engineers building real-time analytics, optimisation and decision-intelligence systems, combining machine learning, mathematical optimisation, streaming data and software engineering on generator optimisation, traffic forecasting, infrastructure planning, anomaly detection, resource allocation and site prioritisation. I led the Smart Generator Optimisation work across more than 15,000 sites, applying constrained optimisation over live telemetry to decide which generators to run, where and when. We built and designed streaming systems processing tens of millions of daily telemetry and alarm events using Kafka, PyFlink, PySpark and Kubernetes, enriching live operational data with telemetry and reference information for faster decisions. I established reusable engineering practices and technical standards across the team, and mentored practitioners while working closely with network, engineering, operations and business stakeholders.
What changed
Dispatch moved from reactive to informed, weighing cost, network impact and constraints across the whole estate in real time. The Smart Generator Optimisation work contributed approximately R1 billion in annual operational savings through better allocation and use of mobile power infrastructure. The engineering practices and streaming architecture made moving analytical work into reliable operational systems more repeatable.
Who benefited
Network operations teams, who gained continuously updated intelligence for infrastructure and resource decisions; the business, through lower operating cost and better capital allocation; and customers who stayed connected through power cuts.
What remained
Production optimisation and streaming systems, and reusable engineering practices and technical standards adopted across the team. Recognised with the Vodacom Star Award in 2022 for engineering contribution and impact.
Technical context
Kafka, PyFlink, PySpark, Kubernetes, Docker, GitLab CI, constrained optimisation (CVXPY), distributed processing, real-time IoT telemetry.
Related work
- Shares kubernetes, dockerUbunye Engine: portable Spark pipelines for data and MLUbunye Engine is an open source Python framework for config driven Spark pipelines. The same task folder runs on a laptop, Docker, Kubernetes or Databricks.
- Shares mlops, technical leadershipData science platform at a bank: telematics and MLOpsAt Absa Group: architected the data engineering capability data science runs on, with Ubunye Engine at 100M+ events a day for 10+ products.
- Shares decision support systems, data engineeringDecision support systems for municipalities at the CSIRDjango based predictive analytics and decision support systems serving 17 municipalities, including the City of Cape Town, built at the CSIR.
- Shares decision support systems, pythonStudent success analytics and recommendations at WitsAnalytics and a clustering based recommendation system for student success at Wits, with over R2 million a year in subsidy impact attributed to it.
Related writing
- Directly connectedWe Don't Have Rockets. We Have Towers. A Case for Building Africa's Data CentresGoogle, SpaceX and others are putting computers in orbit because power and cooling on Earth have run out of room. Africa has no space programme, but it has tens of thousands of cell towers that already have power, batteries, cooling, a fence and a link to the network. This post asks whether those towers can be a data centre spread across the land, and answers with the same graph and scoring method I used to send generators to towers during load shedding.
- Shares technical leadershipBuilding a Data Science Capability From the Ground Up: A Real Life RoadmapTwo years ago I inherited a small data science team with no handover, no documentation, and production models nobody could explain. This is the conversation I wish someone had with me before I started: the challenges, the costs, the thinking behind every move, and the one belief that carried it all. Build systems that outlive the people who built them.
- Shares telecommunicationsWhere the Towers Are. A Map of What Africa Already HasPart one argued that Africa's cell towers could be a data centre spread across the land. This part asks the obvious next question: where are they? Using open, crowdsourced data on 565,000 mobile cells, it maps South Africa and the continent, finds that the towers follow the people and the roads, and shows why a map of people and a map of land tell two very different stories.
- Shares pythonHow I Learned to Build My Own Python Libraries (From Curiosity to Real Work)creating from 1st principles