The scheduler issue has been identified (a deadlock in the backfill scheduler) and we’re working with the Slurm developers on diagnosing and fixing the issue.
This issue was opened retrospectively.
The Slurm controller suffered from an issue last night, which prevented client commands (such as
sh_part) to operate normally. Scheduling of new jobs was paused during that time as well, but already running jobs continued to run normally.
Things are back to normal now. If you still encounter issues, please report them to firstname.lastname@example.org