Railway - Disabled Builds for Hobby – Incident details

All systems operational

Disabled Builds for Hobby

Resolved
Operational
Started 3 months agoLasted about 10 hours

Affected

Builds

Degraded performance from 9:42 PM to 9:48 PM, Operational from 9:42 PM to 2:20 AM, Degraded performance from 2:20 AM to 7:17 AM

Builders

Operational from 9:42 PM to 2:20 AM, Degraded performance from 2:20 AM to 7:17 AM

Image Registry

Degraded performance from 9:42 PM to 9:48 PM, Operational from 9:48 PM to 2:20 AM, Degraded performance from 2:20 AM to 7:17 AM

Third-Party: npm Registry

Third-Party: Github Integrations

Third Party: Docker Hub

Updates
  • Resolved
    Update
    This incident has been resolved.
  • Resolved
    Resolved

    Turns out we never really disabled hobby workloads. Anonymous provisions remain disabled

  • Identified
    Update

    We have found a few hot code paths that have contributed to past instability. Hobby builds remain disabled at this time as the team implements a long term fix.

  • Identified
    Update

    We have restored staged changes to all users. Hobby builds remain disabled while we implement a core fix.

  • Identified
    Identified

    We have disabled builds from Hobby users once again to allow Pro organization builds to go through. While doing so, we are also going to disable the staged changes workflow to assist us in our investigation.

  • Monitoring
    Monitoring

    We believe we've resolved the issue. Re-enabling hobby and monitoring

  • Investigating
    Investigating

    We've seen a runup on hobby builds failing to push images and have disabled them. We're investigating