All systems are operational

Past Incidents

16th February 2023

FRA3 Disk performance impacted

We added several new disks to our storage cluster which results in quite some data being moved around. This results in temporarily impacted disk performance. We slowed down the data migration progress in order to lower the impact already. Usual disk performance is about to be restored soon.

25th January 2023

mysql.uberspace.de MySQL/MariaDB Update

We are currently upgrading MySQL/MariaDB. The database as well as the mysql-binary is unavailable for a couple of minutes during the update.

  • all hosts updated 🎉

  • 4th January 2023

    musca.uberspace.de Performance Issues on musca.uberspace.de

    We are currently having performance issues on musca.uberspace.de because of a long running maintenance process.

  • The process has finished on 05.01.2023 2pm

  • 2nd January 2023

    eupheme.uberspace.de System crashed

    The host crashed and is in the process of rebooting right now.

  • Recovery is done, system is up and running.

  • 9th December 2022

    circinus.uberspace.de MariaDB repair running

    MariaDB repair is running, the system may face performance issues mainly due to database recovery.

  • successfully restarted mysqld

  • 30th November 2022

    vega.uberspace.de MariaDB crashed

    MariaDB repair is running, the system may face performance issues mainly due to database recovery.

  • successfully restarted mysqld

  • 2nd November 2022

    circinus.uberspace.de Reboot of host circinus.uberspace.de

    The system needs to be rebooted and will reappear in a few minutes.

  • reboot is done

  • 6th October 2022

    bus.uberspace.de System crashed

    The host crashed and is in the process of rebooting right now.

  • host was successfully rebooted at 11:55

  • 26th September 2022

    cordelia.uberspace.de System crashed

    The host crashed and is in the process of rebooting right now.

  • reboot is done

  • 18th September 2022

    longmore.uberspace.de mail delivery issues on longmore.uberspace.de

    Local mail delivery on longmore.uberspace.de was interrupted starting approximately at 20:00 hours (CEST / UTC+02:00) on 2022-09-18.

  • The issue was identified and fixed at 14:45 hours (CEST / UTC+02:00) on 2022-09-20. The mails that were queued during the incident have now been delivered.

  • 3rd September 2022

    multiple hosts down

    We have a network issue at one of our datacentres (95.143.172.0/24). We are investigating.

  • The connection has been restored again by our uplink provider.

  • 2nd September 2022

    multiple hosts down

    We have a network issue at one of our datacentres (95.143.172.0/24). We are investigating.

  • For an as of yet unknown reason, the uplink provider disabled both of our uplink connections on their side. They will write up a more detailed post-mortem report for us later.

  • The connection has been restored. We are still waiting for an explanation from our uplink provider.

  • It looks like both of our redundant switches simultaneously lost the connection to the data centre uplink. We still have a management connection open to them, they are both showing as transmitting but not receiving any signals on the uplink.

    We are working with the data centre provider to restore connectivity.

  • 27th July 2022

    vega.uberspace.de Mariadb on vega.uberspace.de crushed

    Mariadb hung since 4:55 as a result of an error and could not finish cleanly - exactly one hour later at 5:54 the instance ran again and finished the crash recovery process.

    9th July 2022

    holmes.uberspace.de System crashed

    The host crashed and is in the process of rebooting right now.

  • Recovery is done, system is up and running.

  • 22nd June 2022

    mcnaught.uberspace.de System crashed

    The host crashed and is in the process of rebooting right now.

  • Recovery is done, system is up and running.

  • 18th May 2022

    kohoutek.uberspace.de System crashed

    The host crashed and is in the process of rebooting right now.

  • Recovery is done, the system is up and running (since ~ 17:26 UTC).

  • 29th April 2022

    Performance problems on some hosts

    Some host are having performance problems because of a disk pool issue.

    A list of host and more details will follow shortly.

  • Recovery is done, and all hosts are up and running again.

  • The host are starting to come back up again.

    • amalthea.uberspace.de
    • aoede.uberspace.de
    • belinda.uberspace.de
    • ciffreo.uberspace.de
    • arche.uberspace.de
    • caliban.uberspace.de
    • callirrhoe.uberspace.de
    • charon.uberspace.de
    • bestla.uberspace.de

    We are still looking into chaldene.uberspace.de; should be back up soon.

  • All listed hosts are cleanly shut down. They will be brought back up as soon as the storage layer problem is resolved.

  • Here's a list of the VMs that are affected:

    • amalthea.uberspace.de
    • aoede.uberspace.de
    • arche.uberspace.de
    • belinda.uberspace.de
    • bestla.uberspace.de
    • caliban.uberspace.de
    • callirrhoe.uberspace.de
    • chaldene.uberspace.de
    • charon.uberspace.de
    • ciffreo.uberspace.de
  • We need to power down the affected VMs to cleanly fix the issue in the storage layer.

  • 21st April 2022

    mueller.uberspace.de MySQL storage migration

    To improve MySQL performance, we're moving the data onto a faster storage. This causes a MySQL downtime of 10-40 minutes within the next hours.

    20th April 2022

    brooks.uberspace.de MySQL storage migration

    To improve MySQL performance, we're moving the data onto a faster storage. This causes a MySQL downtime of 10-40 minutes within the next hours.

    hartley.uberspace.de MySQL storage migration

    To improve MySQL performance, we're moving the data onto a faster storage. This causes a MySQL downtime of 10-40 minutes within the next hours.

    halley.uberspace.de MySQL storage migration

    To improve MySQL performance, we're moving the data onto a faster storage. This causes a MySQL downtime of 10-40 minutes within the next hours.

    hergenrother.uberspace.de MySQL storage migration

    To improve MySQL performance, we're moving the data onto a faster storage. This causes a MySQL downtime of 10-40 minutes within the next hours.

    helin.uberspace.de MySQL storage migration

    To improve MySQL performance, we're moving the data onto a faster storage. This causes a MySQL downtime of 10-40 minutes within the next hours.

    faye.uberspace.de MySQL storage migration

    To improve MySQL performance, we're moving the data onto a faster storage. This causes a MySQL downtime of 10-40 minutes within the next hours.

    holmes.uberspace.de MySQL storage migration

    To improve MySQL performance, we're moving the data onto a faster storage. This causes a MySQL downtime of 10-40 minutes within the next hours.

    brewington.uberspace.de MySQL storage migration

    To improve MySQL performance, we're moving the data onto a faster storage. This causes a MySQL downtime of 10-40 minutes within the next hours.

    kushida.uberspace.de MySQL storage migration

    To improve MySQL performance, we're moving the data onto a faster storage. This causes a MySQL downtime of 10-40 minutes within the next hours.

    echeclus.uberspace.de MySQL storage migration

    To improve MySQL performance, we're moving the data onto a faster storage. This causes a MySQL downtime of 10-40 minutes within the next hours.

    christensen.uberspace.de MySQL storage migration

    To improve MySQL performance, we're moving the data onto a faster storage. This causes a MySQL downtime of 10-40 minutes within the next hours.

    encke.uberspace.de MySQL storage migration

    To improve MySQL performance, we're moving the data onto a faster storage. This causes a MySQL downtime of 10-40 minutes within the next hours.

    biela.uberspace.de MySQL storage migration

    To improve MySQL performance, we're moving the data onto a faster storage. This causes a MySQL downtime of 10-40 minutes within the next hours.

    borrelly.uberspace.de MySQL storage migration

    To improve MySQL performance, we're moving the data onto a faster storage. This causes a MySQL downtime of 10-40 minutes within the next hours.

    31st March 2022

    holmes.uberspace.de System crashed

    The host crashed and is in the process of rebooting right now.

  • Recovery is done, system is up and running.

  • 29th March 2022

    wirtanen.uberspace.de Mysql crashed on wirtanen.uberspace.de

    The MariaDB / Mysql crashed on wirtanen.uberspace.de

  • rebooted successfully, everything working again

  • we need to shutdown and reboot for fixing