Railway - Notice history

We rolled out a temporary fix that did not work. We are in the process of bringing our EU edge proxy hosts back online for further investigations. This incident only applies to a subset of requests originating from or routed through the EU. Applications are still serving fine globally.

Build and deploy logs are fully operational and streaming to the CLI again. These logs were always streaming in the dashboard.

We've identified a firewall rule blocking CLI from establishing WebSocket connections preventing build and deploy logs from streaming. Please visit your dashboard to view them for now while we roll out fix to firewall.

The dashboard and all impacted Railway services is back up. We suffered from another DDoS attack and successfully mitigated it.

Our services have recovered and all requests are being served properly.

This incident has been resolved.

We had a series of targeted DDOS attacks causing heavy load on our systems and failures across the dashboard.

We have since implemented and tested tertiary-level network jails to block and ban these attackers. Initially the bans were for shorter durations of time while we tested our systems and now that we have confidence in them, these bans should keep attackers out permanently.

We've identified the cause and are working towards restoring services for all users.

This incident has been resolved.

The affected host is back up and running. All deploys have been restarted.

deployments on the affected hosts are starting to come back up, we expect full resolution within the next 10 minutes.

This incident is now resolved. Between 1:54am UTC and 2:06am UTC we saw a extremely high volume of traffic across our edge proxies in us-east and asia-southeast1 due to a DoS attack - our mitigations restricted this traffic, but a portion of edge requests originating from these regions failed / experienced higher latency during this time.

Between 1:54am UTC and 2:06am UTC we saw a extremely high volume of traffic across all our edge proxies due to a DoS attack - our mitigations restricted this traffic, but a portion of edge requests failed / had higher latency during this time. Traffic has now recovered, and we're continuing to monitor.

This incident has been resolved.

The issue seems to have been abated. During our discovery, we saw timeouts to GitHub and a few other external services. Given we don't use any egress products other than Cloud NAT from Google, we've escalated this to Google and are awaiting a response

Moving back to monitoring as we've observing it again

We've identified the issue as a bottleneck in our service mesh under heavier-than-average load which causes higher-than-normal latency.

Recovered. Initial response appears to be a flood of deployments with lots of variables put temporary lag on the database, which propagated to the dashboard. We will monitor as we continue to dig in

The issue has been resolved.

The issue has been resolved.

We have pushed a fix and are now monitoring.

This incident has been resolved.

We implemented a fix and are currently monitoring the result.

This incident has been resolved.

The machine is currently back online and serving workloads once again. The Railway team will monitor this machine as workloads fully restore. We appreciate your patience.

The team is currently restarting the affected machine and bringing on workloads back in a staged manner, some workloads may continue to be unavailable.

This incident has been resolved.

We implemented a fix and are currently monitoring the result.

We implemented a fix and are currently monitoring the result.

We are currently investigating this incident.

This incident has been resolved.

We had a spike in deploy requests and a small delay for some deploys as they got backlogged.

We implemented a fix and are currently monitoring the result.