New factory creation process worked improperly

Timeline of Events

• 11:30 UTC - Post-check of functionality after maintenance failed for new factory
• 12:16 UTC - Issue confirmed, internal ticket with full description filed
• 16:30 UTC - Developed three different ways to fix the issue, started work in parallel with two of them
• 16:45 UTC - New factory creation disabled in UI
• 18:20 UTC - Hot fixes applied, applicaions deployed
• 18:28 UTC - New factory creation enabled in UI, checks started
• 18:35 UTC - Issue confirmed as resolved

What Happened

Software update for the main source-code service source.foundries.io brought changes in storage structure which weren’t tested propery. After the upgrade, factory creation process wasn’t able to complete repo creation - factories has ‘failed’ status. Number of affected factories: 1

As a result of internal discussions, we developed three different ways to fix the issue, started work in parallel with two of them immidiately. Better solution was approved and hotfix was developed. The fix introduces new application logic in part of factory creation. After the fix was deployed, we performed full tesing of functionality and confirmed issue as resolved.