Python Real-Time Dashboards — Deep Dive

Dash Real-Time Architecture

Dash’s standard dcc.Interval callback polls every N seconds. For true server-push, extend Dash with the dash-extensions library’s WebSocket support:

import dash
from dash import html, dcc, Output, Input
import plotly.graph_objects as go
from dash_extensions import WebSocket

app = dash.Dash(__name__)
app.layout = html.Div([
    WebSocket(id="ws", url="ws://localhost:8765/stream"),
    dcc.Graph(id="live-chart"),
])

@app.callback(Output("live-chart", "figure"), Input("ws", "message"))
def update_chart(msg):
    if msg is None:
        return dash.no_update
    data = json.loads(msg["data"])
    fig = go.Figure(go.Scatter(
        x=data["timestamps"],
        y=data["values"],
        mode="lines",
    ))
    fig.update_layout(transition_duration=300)
    return fig

A separate asyncio WebSocket server pushes data updates, and Dash callbacks react to incoming messages. This eliminates polling latency while keeping Dash’s declarative callback model.

Bokeh Server Application

Bokeh’s server maintains synchronized Python and JavaScript state through a WebSocket connection:

from bokeh.plotting import figure
from bokeh.models import ColumnDataSource
from bokeh.server.server import Server
from bokeh.application import Application
from bokeh.application.handlers import FunctionHandler
import numpy as np

def make_document(doc):
    source = ColumnDataSource(data={"x": [], "y": []})
    plot = figure(title="Live Sensor Data", width=800, height=400)
    plot.line("x", "y", source=source, line_width=2)

    counter = {"n": 0}

    def update():
        counter["n"] += 1
        new_data = {
            "x": [counter["n"]],
            "y": [np.random.normal(50, 5)],
        }
        source.stream(new_data, rollover=200)  # Keep last 200 points

    doc.add_periodic_callback(update, 500)  # Every 500ms

handler = FunctionHandler(make_document)
app = Application(handler)
server = Server({"/dashboard": app}, port=5006)
server.start()
server.io_loop.start()

The source.stream() method appends new data and pushes the delta to the browser — not the entire dataset. The rollover parameter prevents unbounded growth by discarding old points.

Each browser tab gets its own Document instance with its own Python callbacks. This means 100 viewers create 100 periodic callbacks. For high viewer counts, decouple data collection from per-client updates:

# Shared data collector
latest_data = {"x": [], "y": []}

async def collect_data():
    while True:
        reading = await sensor.read()
        latest_data["x"].append(reading.timestamp)
        latest_data["y"].append(reading.value)
        if len(latest_data["x"]) > 200:
            latest_data["x"] = latest_data["x"][-200:]
            latest_data["y"] = latest_data["y"][-200:]
        await asyncio.sleep(0.5)

def make_document(doc):
    source = ColumnDataSource(data={"x": [], "y": []})
    plot = figure(title="Sensor", width=800, height=400)
    plot.line("x", "y", source=source)

    def update():
        source.data = dict(latest_data)  # Copy shared state

    doc.add_periodic_callback(update, 1000)

Now data is collected once, and each viewer’s callback just copies the shared state.

Custom FastAPI + Plotly Pipeline

For maximum control, build the pipeline yourself:

from fastapi import FastAPI, WebSocket
from fastapi.staticfiles import StaticFiles
import json
import asyncio

app = FastAPI()

class DashboardState:
    def __init__(self):
        self.connections: set[WebSocket] = set()
        self.latest_metrics = {}

    async def connect(self, ws: WebSocket):
        await ws.accept()
        self.connections.add(ws)
        if self.latest_metrics:
            await ws.send_json(self.latest_metrics)

    def disconnect(self, ws: WebSocket):
        self.connections.discard(ws)

    async def broadcast(self, metrics: dict):
        self.latest_metrics = metrics
        dead = []
        for ws in self.connections:
            try:
                await ws.send_json(metrics)
            except Exception:
                dead.append(ws)
        for ws in dead:
            self.connections.discard(ws)

state = DashboardState()

@app.websocket("/ws/dashboard")
async def dashboard_ws(websocket: WebSocket):
    await state.connect(websocket)
    try:
        while True:
            await websocket.receive_text()  # Keep alive
    except Exception:
        state.disconnect(websocket)

The frontend uses Plotly.js directly. On receiving a WebSocket message, JavaScript calls Plotly.react() to update the chart efficiently without recreating it:

const ws = new WebSocket("ws://localhost:8000/ws/dashboard");
ws.onmessage = (event) => {
    const data = JSON.parse(event.data);
    Plotly.react("chart", data.traces, data.layout);
};

Plotly.react() diffs the new data against the existing chart and applies minimal DOM changes, making updates smooth even at high frequency.

Data Pipeline Integration

Real-time dashboards need a data source. Common patterns:

Redis as a Metrics Cache

import aioredis

async def metrics_collector():
    redis = aioredis.from_url("redis://localhost")
    while True:
        # Aggregate from source
        metrics = await compute_metrics()
        await redis.set("dashboard:metrics", json.dumps(metrics))
        await state.broadcast(metrics)
        await asyncio.sleep(1)

Redis provides two benefits: persistence across server restarts and a shared state when multiple dashboard instances run behind a load balancer.

Kafka Consumer to Dashboard

from confluent_kafka import Consumer

async def kafka_to_dashboard():
    consumer = Consumer({
        "bootstrap.servers": "broker:9092",
        "group.id": "dashboard-consumer",
        "auto.offset.reset": "latest",
    })
    consumer.subscribe(["metrics"])

    while True:
        msg = consumer.poll(timeout=0.1)
        if msg and not msg.error():
            metrics = json.loads(msg.value())
            await state.broadcast(metrics)
        await asyncio.sleep(0.01)

Consuming from Kafka ensures the dashboard shows the same data that the rest of the pipeline processes, eliminating consistency issues from separate data queries.

Performance Optimization

Chart Update Efficiency

Sending full datasets on every update wastes bandwidth. Use incremental updates:

# Instead of full data:
await ws.send_json({"traces": [{"x": all_x, "y": all_y}]})

# Send deltas:
await ws.send_json({
    "action": "extend",
    "traces": [{"x": [new_x], "y": [new_y]}],
    "max_points": 200,
})

The frontend applies Plotly.extendTraces() for appends and trims old points to maintain the window size.

Connection Limits

Each WebSocket connection costs ~20–50 KB. For 500 concurrent viewers:

• Memory: ~10–25 MB for connections alone
• CPU: broadcasting to 500 connections per update adds ~5–15ms
• Solution: batch broadcast using asyncio.gather() and limit update frequency to what humans perceive (10–20 fps is sufficient for smooth charts)

Pre-Aggregation

Do not query a database on every dashboard update. Pre-aggregate metrics:

• Use Redis sorted sets for time-windowed aggregations
• Maintain running averages in memory
• Push computed values, not raw data

Deployment Architecture

[Data Sources] → [Kafka/Redis] → [Dashboard Server] → [Load Balancer] → [Browsers]
                                       ↕
                                  [Redis Pub/Sub]
                                       ↕
                                 [Dashboard Server 2]

Multiple dashboard servers subscribe to the same Redis channel. Each server maintains its own set of WebSocket connections. A sticky-session load balancer ensures WebSocket upgrades route to the correct backend.

For Kubernetes deployments, use a headless service for WebSocket routing and Redis for cross-pod communication.

The one thing to remember: A production Python real-time dashboard separates data collection from client updates, uses incremental chart updates to minimize bandwidth, and scales viewer count through shared state (Redis) rather than per-client data computation.