hub.foundries.io Disruption
Timeline of Events
- 20:00 UTC - CI jobs start to fail that are doing
docker push hub.foundries.io...
- 20:05 UTC - Our Docker registry(hub.foundries.io) has been running unchanged for 74 days.
- 21:00 UTC - A test server on home network was set up and pointed to the same Google Storage bucket. This did not suffer from the same blob upload errors we were seeing in production.
- 21:15 UTC - We conclude this has to be something internal to Google.
- 22:00 UTC - Things are magically working again.
What Happened
According to status.google.com, there have been no service issues with GCS. Clearly something was wrong and was fixed.