The processor got disconnected and we were not alerted immediately after that happened, which lead to tests being stuck in the processing queue.

We've identified the root cause and ensuring this doesn't happen again in the future.

All processors are back online and the tests are now being processed again, however, it will take some time to clear up the whole clogged up queue.

No tests are currently being processed due to all processors being down

A volume reaching its maximum capacity prevented cache resolution, triggering a cascade of failures that left the region largely non-operational. Once disk space was restored, cache resolution resumed and the region became operational again.

Server is returning error 500 when users try to publish packages and the selected server is at us-west region

After monitoring the registry, we observed no further issues. It is important to note that, because some packages were rebuilt, users will need to clear their cached fingerprints, as those packages now have new checksums. We will publish a detailed RCA next week with more information about the incident and its root cause.

Even after fixing the failing packages, a global package cache was returning to users packages with divergent checksums. This issue is now fixed and we are monitoring the rollout

During a past incident, some packages were rebuilt causing their hashes/checksum to change. Those packages will fail when installing, to fix this issue, users must remove the fingerprints of those packages. The fingerprints are stored at ~/.swiftpm/security/fingerprints/<scope>.<name>.json.

The directory permission issue was caused by the permissions set when generating the zip file. The issue is fixed and the affected packages were re-published.

Users are experiencing directory permissions errors and mismatches in hashes when installing packages.