CI Disruption
CI jobs are getting assigned to worker but then never progress.
Timeline of Events
- 2025-06-06 00:02 UTC - A workaround has been added to re-queue up runs that appear to be stuck(no console.log content after 2 minutes).
- 2025-06-10 - The root cause was proven from production logs.
- 2025-06-13 13:25 UTC - A permanent fix was pushed to staging to observe
- 2025-06-17 18:51 UTC - The permanent fix was pushed to production