We added several new disks to our storage cluster which results in quite some data being moved around. This results in temporarily impacted disk performance. We slowed down the data migration progress in order to lower the impact already. Usual disk performance is about to be restored soon.
Local mail delivery on longmore.uberspace.de was interrupted starting approximately at 20:00 hours (CEST / UTC+02:00) on 2022-09-18.
We have a network issue at one of our datacentres (95.143.172.0/24). We are investigating.
For an as of yet unknown reason, the uplink provider disabled both of our uplink connections on their side. They will write up a more detailed post-mortem report for us later.
The connection has been restored. We are still waiting for an explanation from our uplink provider.
It looks like both of our redundant switches simultaneously lost the connection to the data centre uplink. We still have a management connection open to them, they are both showing as transmitting but not receiving any signals on the uplink.
We are working with the data centre provider to restore connectivity.
Some host are having performance problems because of a disk pool issue.
A list of host and more details will follow shortly.
Recovery is done, and all hosts are up and running again.
The host are starting to come back up again.
amalthea.uberspace.de
aoede.uberspace.de
belinda.uberspace.de
ciffreo.uberspace.de
arche.uberspace.de
caliban.uberspace.de
callirrhoe.uberspace.de
charon.uberspace.de
bestla.uberspace.de
We are still looking into chaldene.uberspace.de; should be back up soon.
All listed hosts are cleanly shut down. They will be brought back up as soon as the storage layer problem is resolved.
Here's a list of the VMs that are affected:
amalthea.uberspace.de
aoede.uberspace.de
arche.uberspace.de
belinda.uberspace.de
bestla.uberspace.de
caliban.uberspace.de
callirrhoe.uberspace.de
chaldene.uberspace.de
charon.uberspace.de
ciffreo.uberspace.de
We need to power down the affected VMs to cleanly fix the issue in the storage layer.