Climate forecasting and geospatial ML at IBM Research
Turning environmental data into something people can use
Satellite, climate and environmental data is too large to treat as an ordinary dataset. The work was making it into information researchers, businesses and decision makers could act on.
- What it is
- Programme of work
- Role
- Machine Learning Research Scientist
- Organisation
- IBM Research
- Period
- April 2020 to November 2021
- Status
- Completed
- Built with
- TensorFlow
- Python
- Topics
- Climate risk
- Geospatial machine learning
- Remote sensing
- Seasonal forecasting
- Deep learning
The problem
Climate and environmental questions need satellite, geospatial and temporal data at a scale almost no organisation can process alone: large image collections, raster and vector formats, and varying spatial and temporal resolution. The analytical challenge was inseparable from the engineering one, because the data is too large to treat as an ordinary single-machine dataset.
Why it mattered
The point was never to build models in isolation. It was to turn enormous environmental datasets into reliable information that researchers, businesses and decision makers could actually use.
The context
IBM Research, working with distributed compute and IBM PAIRS Geoscope, an enterprise geospatial-temporal platform, described in plain terms, a system for analysing huge volumes of geospatial and time-dependent data, handling petabyte-scale data and used for environmental and enterprise analytics.
What I did
I conducted applied machine-learning research in climate, environmental intelligence, remote sensing and geospatial analytics, combining scientific experimentation with production-oriented engineering. I worked with large geospatial, satellite and environmental datasets in distributed environments using TensorFlow and related tooling, developing forecasting and predictive models that could operate beyond single-machine research workflows, and deployed climate-forecasting models into the PAIRS Geoscope platform. I co-authored research with international scientists and engineers, contributing across experimentation, model development, evaluation, data pipelines and operationalisation. Separately, and during an emerging public-health crisis, I contributed to the Gauteng COVID-19 risk-index and prediction dashboard, a collaboration between IBM Research Africa, Wits University and the GCRO, supporting hotspot identification and healthcare-resource planning for the Gauteng Provincial Department of Health. It was a team effort, not mine alone.
What changed
Research became usable: forecasting models moved from experiment into an enterprise platform, and during the pandemic rapidly changing data was turned into information that could support provincial health planning rather than sitting in an analysis.
Who benefited
Enterprises needing environmental, climate and supply-chain risk intelligence through IBM’s platforms, the research community through published work, and provincial health planners making resourcing decisions under time pressure.
What remained
Models running inside an enterprise geospatial platform, a contribution to a provincial public-health planning dashboard, and co-authored research, including "Long-Range Seasonal Forecasting of 2m-Temperature with Machine Learning", presented at the Tackling Climate Change with ML workshop at NeurIPS 2020 with colleagues from IBM Research.
Technical context
TensorFlow, distributed compute and distributed data platforms, IBM PAIRS Geoscope, IBM Cloud, geospatial raster and vector processing, large-scale experimentation, Python.
Lineage
- Led to
- Flood and natural catastrophe risk in insurance. Geospatial climate risk carried into production insurance models.
Related research
- Directly connectedMachine learning for seasonal climate forecastingIBM Research work on forecasting 2m temperature and precipitation weeks to months ahead with machine learning, published at NeurIPS 2020 CCAI and EGU 2022.
- Directly connectedProposed: self supervised flood mapping from radarProposed doctoral research, not yet registered: physics informed self supervised learning for flood extent mapping from synthetic aperture radar.
- Shares deep learningEcho state networks for level set image segmentationMSc research at Wits: can recurrent models learn the iterative evolution of a level set segmentation? Echo state networks against trained RNN, LSTM and GRU.
Publications
- Shares seasonal forecasting, deep learningLong-Range Seasonal Forecasting of 2m-Temperature with Machine LearningTests whether two machine learning models, a convolutional and a recurrent neural network, can beat climatology when forecasting 2 m temperature up to 52 weeks ahead at six locations. The models improved on climatology up to 30 weeks lead time by correlation and up to 52 weeks by RMSE skill score, but only at some locations, and the authors note further work is needed for the models to add value in the tropics.
- Shares seasonal forecasting, climate riskML-based Probabilistic Prediction of 2m Temperature and Total PrecipitationProposes a daily probabilistic model that forecasts 2 m temperature and total precipitation globally, aimed at the skill gap between weather and seasonal forecasting. Physics based ensembles, climate modes and recent climatology are combined as inputs to Extreme Gradient Boosting, U-Net and Natural Gradient Boosting models. The authors report that it consistently outperforms both ECMWF 46 day forecasts and climatology. Co-authored across IBM Research in South Africa, Kenya, Brazil and the US.
Related work
- Shares geospatial machine learning, climate riskData science platform at a bank: telematics and MLOpsAt Absa Group: architected the data engineering capability data science runs on, with Ubunye Engine at 100M+ events a day for 10+ products.
- Shares pythonDecision support systems for municipalities at the CSIRDjango based predictive analytics and decision support systems serving 17 municipalities, including the City of Cape Town, built at the CSIR.
- Shares pythonGenerator optimisation and streaming at VodacomReal time analytics and optimisation for a national telecoms network: generator dispatch across 15,000+ sites and tens of millions of events a day.
- Shares pythonStudent success analytics and recommendations at WitsAnalytics and a clustering based recommendation system for student success at Wits, with over R2 million a year in subsidy impact attributed to it.