Python Sensor Fusion — Core Concepts

Why Fuse Sensors?

Every sensor has limitations:

Sensor	Strengths	Weaknesses
Camera	Color, texture, classification	No direct depth, fails in darkness
LiDAR	Precise 3D geometry, range	Expensive, no color, sparse at distance
Radar	Velocity measurement, works in weather	Low resolution, phantom reflections
GPS	Global position	2-5 meter error, indoor failure
IMU	Instant response, rotation measurement	Drifts over time
Wheel encoders	Reliable speed measurement	Slippage on ice or gravel

No single sensor covers all situations. Fusion combines complementary strengths: cameras provide what LiDAR cannot (color, classification), and LiDAR provides what cameras struggle with (precise distance). Redundancy also increases safety — if one sensor fails, others can compensate.

Fusion Levels

Sensor fusion operates at different abstraction levels:

Low-Level (Raw Data) Fusion

Combine raw measurements before any processing. Example: merging LiDAR point clouds with camera pixel colors to create colored 3D point clouds. This preserves the most information but requires precise calibration between sensors.

Mid-Level (Feature) Fusion

Each sensor independently extracts features (edges from cameras, clusters from LiDAR), then features are matched and combined. A camera detects a rectangular shape, LiDAR measures a cluster at the same location — together they confirm “car at 30 meters.”

High-Level (Decision) Fusion

Each sensor runs its own complete detection pipeline independently. The fusion system combines decisions: “Camera says 95% probability car, LiDAR says 90% probability car, radar says 85% probability moving object → confirmed car.” This is the simplest to implement and the most modular, but wastes information by processing sensors independently.

The Kalman Filter

The Kalman Filter is the cornerstone of sensor fusion. It optimally combines predictions from a motion model with noisy sensor measurements, assuming both are Gaussian (bell-curve) distributed.

The cycle repeats at every time step:

Predict: Use the motion model to estimate where the object should be now, based on where it was and how it was moving. This estimate has uncertainty that grows because the model is imperfect.

Update: When a sensor measurement arrives, combine it with the prediction. If the sensor is precise, trust it more. If the prediction is precise, trust it more. The result always has less uncertainty than either the prediction or the measurement alone.

This is the key insight: fusing two uncertain estimates always produces something more certain than either one alone. If your prediction says “the car is somewhere in this 3-meter zone” and the sensor says “the car is somewhere in this 2-meter zone,” the Kalman filter combines them and says “the car is in this 1.5-meter zone.”

The Extended Kalman Filter (EKF)

The basic Kalman Filter assumes linear systems. Real robotics problems are nonlinear — a turning car does not follow straight lines. The EKF handles this by linearizing the motion and measurement models at each step using Jacobian matrices. Most autonomous vehicle tracking systems use EKF or its variant, the Unscented Kalman Filter (UKF), which handles nonlinearity better without computing Jacobians.

Complementary Filters

A simpler alternative for specific sensor pairs. The complementary filter combines a sensor that is accurate in the short term (IMU) with one accurate in the long term (GPS or compass):

Fused estimate = α × (IMU-based estimate) + (1 - α) × (GPS-based estimate)

The constant α (between 0 and 1) determines the blending. High α trusts the fast IMU more; low α trusts the stable GPS more. This is widely used in drone attitude estimation, combining gyroscope data (fast but drifting) with accelerometer data (noisy but drift-free).

Real-World Architecture

A modern autonomous vehicle fusion pipeline typically works like this:

1. Each sensor runs its own preprocessing (camera detects objects, LiDAR clusters points, radar measures velocities)
2. A multi-object tracker maintains a state for each tracked object
3. When sensors produce detections, they are associated with existing tracks or create new ones
4. Each track’s state is updated using an EKF or UKF that incorporates measurements from whichever sensors detected that object
5. The tracker handles sensor-specific delays (camera processing takes longer than radar) by buffering and time-aligning measurements

Waymo’s system fuses 5 LiDAR sensors, 29 cameras, and 5 radar units. Each sensor updates object tracks independently, and the fusion layer resolves conflicts (when camera and LiDAR disagree about an object’s position, the fusion weights determine which to trust more based on the situation).

Common Misconception

“More sensors always means better performance.” Adding sensors increases system complexity, calibration difficulty, and computational load. Two poorly calibrated sensors produce worse results than one well-calibrated sensor. The fusion system is only as good as its calibration — if the camera and LiDAR disagree about where “forward” is by even 1 degree, objects will appear in the wrong place. Real systems spend enormous effort on precise inter-sensor calibration.

One thing to remember: The Kalman filter is the mathematical engine behind sensor fusion — it combines uncertain predictions with uncertain measurements to produce estimates that are always more accurate than either input alone.