Power outage
All the compute nodes are now back in production, and all services have been restored.
Most services have been restored, you can now login and submit jobs again.
Some compute nodes and auxiliary services are still being worked on, they should come back up soon.
Earlier this morning, we had a breaker trip while facilities was going over a switching procedure. Power has been restored now. We’ve started working on bringing services back.
Sherlock is still mostly unavailable at this time.
It looks like we lost power in at least portions of the datacenter, investigations are underway.
2 Affected Services:
- Job scheduler / Public partitions
- Job scheduler / Owner partitions