Railway - We're investigating reports of build failures on a subset of builder hosts – Incident details

We're investigating reports of build failures on a subset of builder hosts

Resolved
Partial outage
Started about 1 year agoLasted 13 minutes

Affected

Builds

Partial outage from 1:33 PM to 1:45 PM

Builders

Partial outage from 1:33 PM to 1:45 PM

Updates
  • Resolved
    Resolved

    The affected build node has been cycled. A critical daemon on the node crashed and did not correctly update the nodes healthcheck. Any work landing on that node was thus instantly failing. We're taking measures to fix the healthcheck to ensure only healthy build nodes take on work.

  • Identified
    Identified

    We've identified the affected builder instance and are cycling it out.

  • Investigating
    Investigating

    We're investigating reports of build failures on a subset of builder hosts