Recipe: gRPC bidirectional streaming design

Overview

Bidirectional streaming keeps a single HTTP/2 channel open so the agent can push telemetry while the server pushes commands — no polling, no WebSocket handshake overhead.

Proto contract

service AgentChannel {
  rpc Connect(stream AgentMessage)
      returns (stream ServerMessage);
}

message AgentMessage {
  string agent_id = 1;
  oneof payload {
    Heartbeat heartbeat = 2;
    Telemetry telemetry = 3;
  }
}

message ServerMessage {
  oneof payload {
    Command command = 1;
    Ack ack = 2;
  }
}

Client loop

The agent opens one stream at startup and holds it for the lifetime of the process. A dedicated goroutine reads from the server; a separate goroutine writes heartbeats and telemetry. Reconnect uses exponential backoff with jitter capped at 30 seconds.

Server fan-out

The control plane maintains a registry of active streams keyed by agent ID. Inbound commands are dispatched via a channel per stream. Backpressure is handled with a small buffered channel and a deadline — if the agent does not ack within 5 seconds the command is retried or dropped based on priority.

Circuit breaker

If the server observes three consecutive stream failures from the same agent within 60 seconds, it stops accepting new streams from that agent for 120 seconds and logs a warning. This prevents reconnect storms from wedging the gRPC listener.