Railway - Notice history

All applications on the impacted host have resumed.

The host recovered and applications on it are coming back online. It may take a few minutes for all applications on this box to be available.

This incident has been resolved. We found a bug in our scheduling algorithm which resulted in an instance in the US East region to schedule replica workloads on non-unique hosts, resulting in over subscription. To mitigate this, we have provisioned extra capacity in our non-primary regions (US East, Europe, and Southeast Asia)

Additionally, we will be digging into the scheduling issue to prevent this in the future.

The impacted host has been successfully restarted and all affected workloads are back online. We are investigating the root failure cause and monitoring the host closely.

We've begun a failover process to recover the impacted host.

Impacted regions have been rolled back and responses should be correct now.

We implemented a fix and are currently monitoring the result.

We are adding capacity to the us-west region.

Dashboard latency has recovered.

We've seen error rates drop since 22:30 UTC and docker have moved their incident status to monitoring. Resolving.

This has been confirmed as an upstream outage on DockerHub

This incident has been resolved.

Traffic is now back online. All impacted users should have their services restored. We are seeing high load on our US East proxy and are currently adding further mitigations to prevent this from happening again.

We've isolated this incident to workloads on our new Edge Proxy only.

Builds are becoming fully operational again.

All workloads on the affected host have been fully restored.

All workloads on the affected host have been restored. We're continuing to monitor.

We're continuing to recover the failed host; the replacement is provisioning.

This incident has been resolved. Our database was temporarily unreachable from a subset of our API backend hosts, resulting in slower load times and errors for some users. We have performed a rolling restart of those hosts and are seeing full recovery.

We have implemented a fix and are currently monitoring the result.