Part one of Building a Data Team. This is the whole road on one page. Then comes who you need, a part for each of the five stages, and a last part on what it makes possible.
The roadmap in one minute. A data function is not built by hiring clever people or buying a big platform. It is built in an order, and the order is mostly the same wherever you are. Stage 1: find out what is true. Stage 2: give the data one safe home. Stage 3: give the work a clear path. Stage 4: win people over by showing results. Stage 5: trust your people and let go.
Picture your first week in a new job. You ask for the handover and there is none. Computer models are running, and real customers are affected by them every day. Nobody can show you how they were built, what data feeds them, or whether they are right.
If you have been asked to build a data function, there is a good chance your start will look something like this. Mine did. I inherited a small and talented team at a large, heavily regulated company, and little around it that could help that team do its best work.
This is the roadmap I wish someone had given me. Use it to find where you are, what comes next, and what to expect on the way. My own experience is the worked example, so your details will differ. The road applies whether you are building a data science, data engineering or AI function, because all three stand on the same foundations.
Where you will probably start
These are the signs that you are at the start of the road. I found all of them. Nobody had done bad work on purpose. People were keeping things running with what they had.
- Systems built for a different job. Our computers were set up to run apps, not to process huge amounts of data. They could handle only about 100,000 records a day.
- Data that could not be trusted. The way data was collected caused errors and gaps long before anyone used it.
- Weak controls. Data sat in many separate databases. It was hard to see who had changed what, and code had no proper history.
- Work on personal machines. When someone went on leave, things could break. They did, more than once.
- No shared way of working. Nobody could easily answer a simple question: what are we busy with?
This is never only a technical problem. In my case, when work fell months behind, customers complained and some cancelled. Money was lost, and the complaints went all the way to the top.
Where you are going
A good data function is one whose systems outlive the people who built them.
If you leave, if anyone leaves, the knowledge, the data and the way of working must carry on. Keep that destination in view. When things get hard and every option looks reasonable, it tells you what to do next.
The road, stage by stage
When everything is broken, you will want to fix the loudest thing first. Do not. The loudest thing is rarely the thing holding the weight. Follow this order instead: find out what is true, give the data one safe home, give the work a clear path, show people instead of arguing, then trust your team and let go.
- 1Stage 1: Find out what is true
- 2Stage 2: One safe home for data
- 3Stage 3: A clear path for work
- 4Stage 4: Show, do not argue
- 5Stage 5: Trust, then let go
- 6Arrive: it outlives you
| Stage | The question it answers | You can move on when |
|---|---|---|
| 1. Find out what is true | What do we have, and is it right? | Everything you run is written down, and the team has a rhythm |
| 2. One safe home for data | Can we trust our data? | Your key data has one controlled home, with checks on the way in |
| 3. A clear path for work | How does an idea reach real use? | Someone else can run your work without your help |
| 4. Show, do not argue | How do we get support? | People outside your team ask for your way of working |
| 5. Trust, then let go | Does this need me? | You can take a week off and nothing important breaks |
Stage 4 is not really a step you finish and leave. Expect pushback from the first week. It is placed here because you need something real to show before you can win the argument.
Before you set off, read who you need on the team. At the end of the road, here is what it makes possible: knowing your customers, keeping them, pricing fairly, understanding big risks and working faster.
How long each stage took us
Stages overlap, and your times will depend on your size and your starting point. This is what it looked like for us.
- Stage 1, find out what is true: two to three months, and the documents keep improving.
- Stage 2, one safe home for data: about two months for the move.
- Stage 3, a clear path for work: about six months.
- Stage 4, show, do not argue: this runs the whole way through.
- Stage 5, trust, then let go: still going. This includes aligning with the teams that support us, such as technology.
What the road can get you
Here is what it did for my team, in less than two years. Treat it as an example, not a promise.
| Before | After | |
|---|---|---|
| Getting a model into use | About a year | Within a two week sprint |
| Clearing a big backlog | About three months | Under a day |
| Problems | Clients told us | We usually see them first |
| New people | Ask around and hope | Follow a written guide |
| Models | Trained on someone's machine, hard to trace | Registered, tested and promoted through automatic gates |
| Data | Hard to say who changed what | Controlled access, a traceable history and automatic checks |
| Other teams | Red tape and silos | Agreed ways of working |
| Revenue | Lost to errors and delays | Tens of millions of rands a month protected |
| Customer engagement | Hurt by delays and errors | Up by about 70% |
| Cancellations from operational problems | About 0.4% to 2% of policies a month | Down by about 85% |
The business figures are approximate. They are there to show the scale.
The team has become one of the most relied on in its department. The problems we uncovered have also led the business to rethink some products and redesign the processes that were creating bad data in the first place.
It is not finished, and yours will not be either. We are still aligning with the teams that support us. But the systems now run whether I am in the room or not. That was the goal the whole time.
How to use this roadmap
- Read the table above and find the first stage you cannot honestly tick off. That is where you are, even if you already have a big platform.
- Read that part. Each one covers what you will usually find, what to do, what it will cost, and how to know you are done.
- Do not skip ahead. If you are new, start with one question about everything your team runs: if the person who knows this left today, what would happen?
Write down the answers. That list is your starting point, and Stage 1 shows you what to do with it.