Making noisy sensor data easier to work with
Working with telemetry across telecoms and insurance, the same problem kept coming back, real sensors lie, so I built the filtering methods I kept re-implementing into one library behind an API people already know.
The problem
Real sensors lie. GPS drifts, radar is noisy, vehicle trackers and IoT devices produce readings you cannot use directly, and you need an honest estimate of the state underneath the noise.
Why it mattered
Bayesian filtering solves this and the mathematics is well established, but re-deriving Kalman, Particle or Ensemble filters from papers on each new project is slow and error prone. The bigger problem was that existing implementations tended to assume a state-estimation background, so a practitioner who simply needed a clean signal had to become a specialist first, or find one.
The context
The recurring context was telemetry: the same class of noisy time-series problem appearing across telecommunications and insurance, in telematics and IoT. The methods themselves are not specific to vehicles, which shaped the scope. The same five filters serve radar tracking, robot localisation, high-dimensional weather and ocean models, EEG, image denoising, and even smoothing signals in text.
What I did
I built Tfilterspy as an open-source Python library. The design decision that mattered was familiarity: every filter is a scikit-learn compatible estimator, so fit, predict, score, get_params and set_params behave the way a practitioner already expects, and a filter can slot into tooling built around that convention. I also wrote the part people actually get stuck on, a decision guide for choosing among the five: a Kalman filter for linear systems, an Extended Kalman filter where you can supply Jacobians, an Unscented Kalman filter where you cannot, an Ensemble Kalman filter for very high-dimensional states, and a Particle filter for non-Gaussian or multimodal problems. It is built to scale beyond one machine, with optional Dask parallelism for the ensemble and particle methods.
What changed
Filtering stopped being a re-derivation exercise and became fitting a familiar estimator. Beyond the forward pass there is RTS smoothing, forecasting a number of steps ahead, and a filter_step call for online use against a live stream. Particle degeneracy is visible rather than silent through effective sample size monitoring, and a memory mode drops covariance storage for roughly an 80 percent saving on very long series, so a million-step run is practical.
Who benefited
Engineers and scientists working with noisy time series who are not filtering specialists, and the production telematics state-estimation work it has been used in.
What remained
A published, documented library on PyPI: five filters behind one API, worked examples for GPS vehicle tracking, radar tracking and robot localisation, notebooks covering EEG, image denoising and benchmarks across all five, and a test suite that includes a ten thousand step dataset.
Technical context
Python, NumPy, SciPy, a scikit-learn BaseEstimator API, optional Dask for parallel ensemble and particle propagation, a parameter estimator utility, 37+ tests including a 10,000-step run, PyPI.
Related
Ubunye Engine
Every team rebuilds the same pipeline plumbing, structured differently each time, and the tooling is split across laptops, on-prem clusters and different clouds.
WritingHow I Learned to Build My Own Python Libraries (From Curiosity to Real Work)
creating from 1st principles
WorkDecision support for municipalities
Municipalities held data relevant to performance and service delivery, but not in a form that supported planning or operational decisions.
WorkTurning environmental data into something people can use
Climate and environmental questions need satellite and geospatial data at petabyte scale, too large to treat as an ordinary dataset, and almost no organisation can process it alone.