Some systems are experiencing issues

About This Site

This page is intended to provide a quick overview of the operational status of the Sepia lab. It doesn't try to provide many testing-related metrics.

For more detailed testing information, see the Grafana dashboard

Stickied Incidents

Tuesday 10th May 2022

Long Running Cluster Health Long Running Cluster Outage

While adding some new hosts to the Sepia Long Running Cluster, the cluster got into a state where all the MONs started locking up due to lack of system resources. Josh, Neha, Dan, and David have been working to restore the cluster service by service.

The following workloads are down:

  • teuthology runs
  • Ceph CI builds (Jenkins/shaman)
  • quay.ceph.io
  • telemetry.ceph.com / telemetry-public.ceph.com
  • chacra.ceph.com

All services relying on the LRC have been restored. I will be upgrading all the daemons to a version of Ceph that have fixes for the problems we ran into.

The LRC is back up and OSD recovery is still in progress. We're letting things settle before bringing up any clients.

Past Incidents

Sunday 24th April 2022

No incidents reported

Saturday 23rd April 2022

No incidents reported

Friday 22nd April 2022

No incidents reported

Thursday 21st April 2022

No incidents reported

Wednesday 20th April 2022

No incidents reported

Tuesday 19th April 2022

No incidents reported

Monday 18th April 2022

No incidents reported

Sunday 17th April 2022

No incidents reported

Saturday 16th April 2022

No incidents reported

Friday 15th April 2022

No incidents reported