Railway - Dashboard API service is timing out when responding to requests – Incident details

All systems operational

Dashboard API service is timing out when responding to requests

Resolved
Partial outage
Started about 1 month agoLasted about 7 hours

Affected

Dashboard

Partial outage from 10:04 PM to 4:35 AM

Updates
  • Resolved
    Resolved

    This incident has been resolved. We isolated the root cause to a spike in new users landing on our project creation page (https://railway.app/new) that exacerbated certain anomalous internal network conditions due to unusually high load in a short amount of time.

  • Monitoring
    Update
    We're monitoring the issue and tracked the continued latency down to a suspected issue with our internal networking cluster. We are in the process of mitigating the issue.
  • Monitoring
    Monitoring
    We found a few hot code paths that were contributing to our cache being overwhelmed. We've deployed a fix and the dashboard API service latencies are returning back to normal.
  • Investigating
    Update
    We are currently investigating this incident. Dashboard API service is delayed or timing out when responding to requests.
  • Investigating
    Investigating
    Dashboard API service is timing out when responding to requests