/scratch metadata server outage

Resolved Sep 4, 2018 9:07 AM PDT

The metadata server issue has been resolved.

Monitoring Aug 22, 2018 1:16 PM PDT

The filesystem is back online.
We’re still looking into the root cause for those repeated crashes.

Investigating Aug 22, 2018 12:41 PM PDT

The metadata server just suffered another crash, we’re investigating.

Resolved Aug 22, 2018 8:21 AM PDT

The filesystem has been stable overnight, so we’re closing this incident.

Monitoring Aug 21, 2018 3:12 PM PDT

The filesystem is back online, we’ll continue monitoring it for new occurrences.

Investigating Aug 21, 2018 1:22 PM PDT

Another occurrence of this same issue happened, we’re looking into it.

Resolved Aug 15, 2018 8:08 AM PDT

The issue has been identified and resolved. Access to $SCRATCH and $GROUP_SCRATCH has been restored, and logins should go through normally now. We’ll continue to monitor the state of the storage system, and feel free to report any issue you may notice.

Investigating Aug 15, 2018 7:48 AM PDT

The metadata server for the /scratch filesystem is currently unavailable. The current effects include the impossibility to change directories to or list the contents of directories in $SCRATCH or $GROUP_SCRATCH, and possibly hanging logins.
We’re investigating and will post updates as we have them.