Ingest performance degredation in Prod EU cluster

Incident Report for Observe

Resolved

This incident has been resolved.

Posted Mar 16, 2026 - 12:12 PDT

Monitoring

A large increase in tracing data caused ingestion lag on the prod-eu-1 cluster, affecting multiple customers. Encoder replicas were scaled up and subsequently rolled to resolve unassigned partition workers. Memory pressure was identified as a contributing factor. Ingestion lag peaked at up to 45 minutes for some customers and has since recovered to near-normal levels. The team is continuing to monitor.

Posted Mar 16, 2026 - 11:55 PDT