Turning environmental data into something people can use
Satellite, climate and environmental data is too large to treat as an ordinary dataset. The work was making it into information researchers, businesses and decision makers could act on.
The problem
Climate and environmental questions need satellite, geospatial and temporal data at a scale almost no organisation can process alone: large image collections, raster and vector formats, and varying spatial and temporal resolution. The analytical challenge was inseparable from the engineering one, because the data is too large to treat as an ordinary single-machine dataset.
Why it mattered
The point was never to build models in isolation. It was to turn enormous environmental datasets into reliable information that researchers, businesses and decision makers could actually use.
The context
IBM Research, working with distributed compute and IBM PAIRS Geoscope, an enterprise geospatial-temporal platform, described in plain terms, a system for analysing huge volumes of geospatial and time-dependent data, handling petabyte-scale data for clients globally.
What I did
I developed machine learning and geospatial analytics for environmental intelligence and climate-risk applications, and built predictive systems with TensorFlow on distributed data platforms. I deployed climate-forecasting models into the PAIRS Geoscope platform. I co-authored peer-reviewed research and worked with international research and engineering teams on large-scale AI initiatives. Separately, and during an emerging public-health crisis, I contributed to the Gauteng COVID-19 risk-index and prediction dashboard, a collaboration between IBM Research Africa, Wits University and the GCRO. It was used by the Gauteng Provincial Department of Health to identify infection hotspots and forecast hospital bed and ventilator demand for a province of more than 15 million residents, and was deployed publicly via IBM Cloud. It was a team effort, not mine alone.
What changed
Research became usable: forecasting models moved from experiment into an enterprise platform serving clients, and during the pandemic rapidly changing data was turned into information that could support provincial health planning rather than sitting in an analysis.
Who benefited
Enterprises needing environmental, climate and supply-chain risk intelligence through IBM’s platforms, the research community through published work, and provincial health planners making resourcing decisions under time pressure.
What remained
Models running inside an enterprise geospatial platform, a publicly deployed public-health dashboard, and peer-reviewed research, including "Long-Range Seasonal Forecasting of 2m-Temperature with Machine Learning", presented at the Tackling Climate Change with ML workshop at NeurIPS 2020 with colleagues from IBM Research.
Technical context
TensorFlow, distributed compute and distributed data platforms, IBM PAIRS Geoscope, IBM Cloud, geospatial raster and vector processing, large-scale experimentation, Python.
Related
Decision support for municipalities
Municipalities held data relevant to performance and service delivery, but not in a form that supported planning or operational decisions.
WorkUsing data to help students and a university decide
A university held a great deal of institutional data, but the questions that mattered for student success and planning were hard to answer from reporting alone.
WritingHow I Learned to Build My Own Python Libraries (From Curiosity to Real Work)
creating from 1st principles
WorkTfilterspy
Real sensors lie. GPS, radar, vehicle trackers and IoT devices all produce noisy readings, and you need an honest estimate of what is actually happening underneath the noise.