Railway - We are investigating elevated error rates with our legacy edge network. This can be mitigated by unchecking the "Use Legacy Proxy" option within your service settings. – Incident details

We are investigating elevated error rates with our legacy edge network. This can be mitigated by unchecking the "Use Legacy Proxy" option within your service settings.

Resolved
Degraded performance
Started about 1 month agoLasted about 1 hour

Affected

Legacy Edge Network

Updates
  • Resolved
    Resolved
    We tracked down the error to a failure to validate config on the US-West Cluster in the Legacy Edge network. This config propagated throughout the edge network. The edge network uses incremental updates, except for when new nodes are booted. When these new nodes came online, they failed to start and thus blackholed a subset of traffic.
  • Investigating
    Update
    We are still investigating this incident, and the previously recommended mitigation steps to uncheck the "Use legacy proxy" within service settings are still recommended.
  • Investigating
    Investigating
    We are investigating elevated error rates with our legacy edge network. This can be mitigated by unchecking the "Use Legacy Proxy" option within your service settings.