Recipe: Log shipping pipeline design
A reference architecture for shipping structured logs from a Windows-native loader to a centralized observability backend without compromising the loader's stealth profile.
Pipeline stages
Lock-free SPSC ring in shared memory. Writer (loader) pushes fixed-size entries with monotonic sequence numbers. No allocations after init.
Writer sets a named event when the ring crosses a high-water mark. The shipper process waits on this event with a 500ms timeout fallback.
A separate, unsigned companion process reads the ring, batches entries, and ships them over HTTPS to the ingestion endpoint. The loader never touches the network.
Vercel serverless function validates HMAC signatures, deduplicates by sequence number, and writes to Upstash KV with a 7-day TTL.
Meridian dashboard polls KV for recent entries and renders a live tail view with client-side filtering by severity and module.
Entry schema
{
"seq": u64,
"ts": i64,
"level": u8,
"module": u16,
"code": u16,
"payload": [u8; 48]
}Fixed 80-byte entries. No strings in the ring — the dashboard resolves module/code pairs against a static symbol table.
Resilience
- Ring buffer wraps gracefully — oldest entries are overwritten when the shipper is down.
- Shipper retries with exponential backoff (1s, 2s, 4s, max 30s).
- Ingestion endpoint returns 429 if KV rate limit is approached; shipper respects Retry-After.
- No log data touches disk — everything lives in the ring or in flight.