All systems are operational

Past Incidents

21st July 2021

cassiopeia.uberspace.de Reboot with more RAM

We added some RAM and the host is in the process of rebooting right now.

  • Recovery is done, system is up and running.

  • crux.uberspace.de Reboot with more RAM

    We added some RAM and the host is in the process of rebooting right now.

  • Recovery is done, system is up and running.

  • 14th July 2021

    adrastea.uberspace.de Performance problems on host adrastea.uberspace.de

    we encounter episodic high load peaks on adrastea since yesterday 10pm

  • host is running stable now

  • 13th July 2021

    adrastea.uberspace.de Reboot with more RAM

    We added some RAM and the host is in the process of rebooting right now.

  • Recovery is done, system is up and running.

  • 9th July 2021

    schedar.uberspace.de System crashed

    The host crashed and is in the process of rebooting right now.

  • system is booting and will recover soon.

  • 6th July 2021

    High load due to storage deep scrubbing

    some hosts are currently experiencing a high load for a short time due to deep scrubbing the storage systems.

    5th July 2021

    High load due to storage deep scrubbing

    some hosts are currently experiencing a high load for a short time due to deep scrubbing the storage systems.

    2nd July 2021

    FRA3 various reboots

    The following systems need to be rebooted und and will reappear shortly after they go down.

    aoede.uberspace.de, cressida.uberspace.de, giacobini.uberspace.de, kopff.uberspace.de, vanness.uberspace.de, taylor.uberspace.de, neujmin.uberspace.de, ankaa.uberspace.de, cordelia.uberspace.de, alnilam.uberspace.de, johnson.uberspace.de, gale.uberspace.de, longmore.uberspace.de, hernmann.uberspace.de, wirtanen.uberspace.de, alpheca.uberspace.de, bestla.uberspace.de

  • Previous set of reboots finished. Next reboots: russel.uberspace.de, arche.uberspace.de, gibbs.uberspace.de, cyllene.uberspace.de, yeung.uberspace.de, brorsen.uberspace.de

  • Previous set of reboots finished. Next reboots: lovas.uberspace.de, caliban.uberspace.de, cupid.uberspace.de, harrington.uberspace.de, charon.uberspace.de, ciffreo.uberspace.de, sanguin.uberspace.de

  • Previous set of reboots finished. Next reboots: bernardi.uberspace.de, ananke.uberspace.de, belinda.uberspace.de, amalthea.uberspace.de, kushida.uberspace.de, carpo.uberspace.de, mcnaught.uberspace.de, giclas.uberspace.de, bus.uberspace.de, chiron.uberspace.de, kowal.uberspace.de

  • Previous set of reboots finished. Next reboots: aegaeon.uberspace.de, brewington.uberspace.de, mrkos.uberspace.de, helin.uberspace.de, howell.uberspace.de, callirrhoe.uberspace.de, bianca.uberspace.de, gacrux.uberspace.de, crommelin.uberspace.de, klemola.uberspace.de, shaula.uberspace.de, albiorix.uberspace.de, clark.uberspace.de, kojima.uberspace.de, bernardi.uberspace.de, ananke.uberspace.de, wolf.uberspace.de, belinda.uberspace.de, amalthea.uberspace.de

  • Previous set of reboots finished. Next reboots: maury.uberspace.de, jedicke.uberspace.de, whipple.uberspace.de, aegaeon.uberspace.de, brewington.uberspace.de, mrkos.uberspace.de, helin.uberspace.de

  • aegir.uberspace.de Reboot: aegir

    The system needs to be rebooted und and reappear in a few minutes.

  • completed.

  • adrastea.uberspace.de Reboot: adrastea

    The system needs to be rebooted und and reappear in a few minutes.

  • httpd fixed

  • httpd is crashed

  • completed.

  • 1st July 2021

    only IPv4 connectivity: aoede.uberspace.de, callirrhoe.uberspace.de, eirene.uberspace.de, erriapus.uberspace.de

    IPv6 not working, IPv4 functional

  • IPv6 connectivity on eirene.uberspace.de and erriapus.uberspace.de restored

  • IPv6 on aoede und callirrhoe are working again

  • machholz.uberspace.de Reboot

    The system needs to be rebooted und and reappear in a few minutes.

  • Reboot completed. Load and services might take a sort while to settle.

  • neujmin.uberspace.de Host NEUJMIN.uberspace.de recovering from XFS failures

    After a power outage at the data center the host was rebooted successfully and is operational. Because of some XFS errors there will be the need of a repair measures. Check out our blog post (in german) for more background information.

  • we fixed some dovecot file owner issues, mailsystem should work again now

  • we are encountering issues with the mail system

  • 30th June 2021

    clark.uberspace.de Host CLARK.uberspace.de recovering from XFS failures

    After a power outage at the data center the host is encountering XFS errors and is in process of recovering/repairing. Check out our blog post (in german) for more background information.

  • for some pathes we are currently syncing data from backup

  • the nginx https frontend and other services still need to be restored

  • we are just restoring SELinux Labels

  • cordelia.uberspace.de Host CORDELIA.uberspace.de needs to be repaired due to XFS errors

    After a power outage at the data center the host was rebooted successfully and is operational. Because of some XFS errors there will be the need of a repair measures. Check out our blog post (in german) for more background information.

  • webserver httpd is working again

  • webserver (httpd) not working currently

  • wirtanen.uberspace.de Host WIRTANEN.uberspace.de needs to be repaired due to XFS errors

    After a power outage at the data center the host was rebooted successfully and is operational. Because of some XFS errors there will be the need of a repair measures. Check out our blog post (in german) for more background information.

  • Recovery is done, system is up and running.

  • The system needs to be rebooted und and reappear in a few minutes.

  • We are recreating a pre-crash state, after that, the host should be fully up again.

  • Host will be down for a reboot.

  • repair finished, processing quotacheck now

  • The host is going down for xfs_repair, for about 5 hours (estimate).

  • Because of an unexpected crash we had to start the xfs repair already now, the host will be down for the time of processing the repairs.

  • repair process is rescheduled for monday 05.07.2021 10:30

  • the host will be down for xfs_repair on friday 02.07.2021 - 13:30 (about 5 hours)

  • vanness.uberspace.de Host VANNESS.uberspace.de needs to be repaired due to XFS errors

    After a power outage at the data center the host was rebooted successfully and is operational. Because of some XFS errors there will be the need of a repair measures. Check out our blog post (in german) for more background information.

  • Host is back up. We are still checking some things, but it runs for now.

  • Host will be down for a reboot.

  • Quotacheck done. Host is up again. We currlyt recreating a pristine state, after that, the host should be fully up again.

  • XFS repair done, host is running quotacheck.

  • The host is going down for xfs_repair, for about 5 hours (estimate).

  • repair process is rescheduled for monday 05.07.2021 10:30

  • the host will be down for xfs_repair on friday 02.07.2021 - 13:30 (about 5 hours)

  • kojima.uberspace.de Host KOJIMA.uberspace.de needs to be repaired due to XFS errors

    After a power outage at the data center the host was rebooted successfully and is operational. Because of some XFS errors there will be the need of a repair measures. Check out our blog post (in german) for more background information.

  • Redeployment done. Still needs a round of integrity checks.

  • Host is up again and (at least partially) working.

    Still needs some more work though and will be rebooted once more, but that probably will have to wait till morning.

  • The host still needs more fixes, after coming back from XFS repairs.

  • xfs repair is done, processing quota checks

  • host is going offline now

  • we will need to send the host in recovery mode for xfs_repair at 15:00 (estimated 1-2 hours)

  • giclas.uberspace.de Host GICLAS.uberspace.de needs to be repaired due to XFS errors

    After a power outage at the data center the host was rebooted successfully and is operational. Because of some XFS errors there will be the need of a repair measures. Check out our blog post (in german) for more background information.

  • Quotacheck and relabling are done. Host is up again.

    Still running checks, so things might not work as expected for now.

  • XFS repair finished, host is processing quotacheck now.

  • The host is going down for XFS repair.

  • the host will be down for xfs_repair on friday 02.07.2021 - 10:30 (about 5 hours)

  • bus.uberspace.de Host BUS.uberspace.de needs to be repaired due to XFS errors

    After a power outage at the data center the host was rebooted successfully and is operational. Because of some XFS errors there will be the need of a repair measures. Check out our blog post (in german) for more background information.

  • Quotacheck and relabling are done. Host is up again.

    Still running checks, so things might not work as expected for now.

  • repair finished, host is processing quotacheck now

  • host is offline now for xfs_repair

  • the host will be down for xfs_repair on friday 02.07.2021 - 10:30 (about 5 hours)

  • yeung.uberspace.de Host YEUNG.uberspace.de recovering from XFS failures

    After a power outage at the data center the host is encountering XFS errors and is in process of recovering/repairing. Check out our blog post (in german) for more background information.

  • Haraka is restored and accepting mails again.

  • the host is back online but we are still working on the haraka mailserver

  • host is now in process of quota check

  • bergelmir.uberspace.de Host BERGELMIR.uberspace.de recovering from XFS failures

    After a power outage at the data center the host is encountering XFS errors and is in process of recovering/repairing. Check out our blog post (in german) for more background information.

  • The host is up and running again :tada:

  • We are still syncing backups, so you might want to check if /var/www/virtual/username already contains what you expect.

  • We managed to fix a lot of issues with SE labels and PHP. Hopefully most things work for you now.

  • Just to clarify: the host is up, but still not fully functioning again. We are working on it, but it will take some more time (and reboots).

  • The host is back up again.

    We still have some repair task and testing to do though. Meaning there could be more reboots and performance issues.

  • Syncing completed, but the host won't boot due to SELinux errors. We to avoid mistakes made due to sleep deprivation, we'll continue the process tomorrow.

  • backup is still transferring to the new VM

  • The host is re-created with a fresh deployment. Backups are currently copyed over (this will take some time).

  • we had to give up the system recovery and will set up a new VM with data backup on monday, please note the update in our blogpost.

  • we are still transferring data to the new block device and doing some tests

  • we are mounting a new block device to replace the VM, the syncing is in progress now

  • adrastea.uberspace.de Host ADRASTEA.uberspace.de recovering from XFS failures

    After a power outage at the data center the host is encountering XFS errors and is in process of recovering/repairing. Check out our blog post (in german) for more background information.

  • the httpd web server is back online

  • the webserver is offline again

  • the httpd web server is back online

  • the host is back online but the web server (httpd) is still in error, we are working on it

  • 29th June 2021

    Recovering XFS failures on rebooted hosts

    After a power outage at the data centre and rebooting all hosts in FRA3 we encounter problems with XFS failures on some hosts. Please see the follow up incidents (30.06.2021) for status updates per host.

  • yeung.uberspace failed while encountering XFS errors, recovering now

  • arche.uberspace.de is recovered

  • neujmin.uberspace.de is recovered

  • amalthea.uberspace.de is recovered.

  • aegir.uberspace.de is recovered.

  • wolf.uberspace.de is recovered.

  • howell.uberspace.de is recovered.

  • ciffreo.uberspace.de is recovered.

  • mcnaught.uberspace.de is recovered.

  • neujmin.uberspace.de also needs XFS repair.

  • mrkos.uberspace.de is recovered.

  • FRA3 Major incident

    We need to reboot all hosts at FRA3 due to a power outage at the data centre.

  • Reboots are done, hosts are up again.

    Some host where affected by XFS errors during recovery, you can see the follow up incidents for the current state.

  • recovered:

    • dubhe.uberspace.de
    • neujmin.uberspace.de
    • russel.uberspace.de
  • recovered:

    • gibbs.uberspace.de
    • giclas.uberspace.de
  • recovered:

    • ananke.uberspace.de
    • aoede.uberspace.de
    • bestla.uberspace.de
    • callirhoe.uberspace.de
    • cressida.uberspace.de
    • crommelin.uberspace.de
    • cupid.uberspace.de
  • recovered

    • kopff.uberspace.de
    • kojima.uberspace.de
    • iocaste.uberspace.de
  • recovered:

    • hernmann.uberspace.de
  • recovered:

    • ferdinand.uberspace.de
    • finlay.uberspace.de
    • gale.uberspace.de
    • giacobini.uberspace.de
    • jedicke.uberspace.de
  • recovered:

    • euporie.uberspace.de
    • europa.uberspace.de
    • eurydome.uberspace.de
    • johnson.uberspace.de
    • lovas.uberspace.de
  • recovered:

    • eukelade.uberspace.de
    • eupheme.uberspace.de
  • recovered:

    • erinome.uberspace.de
    • erriapus.uberspace.de
    • euanthe.uberspace.de
    • kowal.uberspace.de
    • longmore.uberspace.de
  • recovered:

    • halimede.uberspace.de
    • halley.uberspace.de
    • harpalyke.uberspace.de
  • recovered:

    • encke.uberspace.de
    • epimetheus.uberspace.de
    • farbauti.uberspace.de
    • faye.uberspace.de
    • fenrir.uberspace.de
    • fornjot.uberspace.de
    • francisco.uberspace.de
    • galatea.uberspace.de
    • ganymed.uberspace.de
    • gehrels.uberspace.de
    • greip.uberspace.de
  • recovered:

    • echeclus.uberspace.de
    • eirene.uberspace.de
    • elst.uberspace.de
    • hartley.uberspace.de
    • hegemone.uberspace.de
    • helene.uberspace.de
    • helike.uberspace.de
    • hergenrother.uberspace.de
    • hermippe.uberspace.de
  • recovered:

    • dysnomia.uberspace.de
    • herse.uberspace.de
    • himalia.uberspace.de
    • hippocamp.uberspace.de
    • holmes.uberspace.de
    • hyrrokkin.uberspace.de
    • iapetus.uberspace.de
    • ijiraq.uberspace.de
    • maury.uberspace.de
  • recovered:

    • chiron.uberspace.de
    • christensen.uberspace.de
    • deimos.uberspace.de
    • desdemonia.uberspace.de
    • despina.uberspace.de
    • dia.uberspace.de
    • dione.uberspace.de
    • klemola.uberspace.de
    • kohoutek.uberspace.de
  • recovered:

    • caliban.uberspace.de
    • charon.uberspace.de
    • daphnis.uberspace.de
    • hati.uberspace.de
    • mueller.uberspace.de
    • olbers.uberspace.de
    • sanguin.uberspace.de
    • schuster.uberspace.de
  • recovered:

    • carpo.uberspace.de
    • chaldene.uberspace.de
    • pizarro.uberspace.de
    • read.uberspace.de
    • reinmuth.uberspace.de
    • scotti.uberspace.de
    • shoemaker.uberspace.de
    • skiff.uberspace.de
  • recovered:

    • borrelly.uberspace.de
    • brewington.uberspace.de
    • brooks.uberspace.de
    • brorsen.uberspace.de
    • bus.uberspace.de
    • spahr.uberspace.de
    • spitaler.uberspace.de
    • taylor.uberspace.de
    • tempel.uberspace.de
    • tritton.uberspace.de
    • tuttle.uberspace.de
  • recovered:

    • bianca.uberspace.de
    • biela.uberspace.de
    • boethin.uberspace.de
    • vanness.uberspace.de
    • westphal.uberspace.de
    • whipple.uberspace.de
    • yeung.uberspace.de
  • recovered:

    • bernardi.uberspace.de
    • wild.uberspace.de
  • recovered:

    • bebhionn.uberspace.de
    • belinda.uberspace.de
    • wirtanen.uberspace.de
  • Here's a list of the host in the FRA3 datacenter, that are affected:

    • adrastea.uberspace.de
    • aegir.uberspace.de
    • amalthea.uberspace.de
    • arche.uberspace.de
    • bergelmir.uberspace.de
    • ciffreo.uberspace.de
    • clark.uberspace.de
    • cordelia.uberspace.de
    • howell.uberspace.de
    • mcnaught.uberspace.de
    • mrkos.uberspace.de
    • wolf.uberspace.de

    Already recovered or not affected:

    • aegaeon.uberspace.de
    • aitne.uberspace.de
    • albiorix.uberspace.de
    • alnilam.uberspace.de
    • alpheca.uberspace.de
    • ananke.uberspace.de
    • ankaa.uberspace.de
    • aoede.uberspace.de
    • arend.uberspace.de
    • ariel.uberspace.de
    • atlas.uberspace.de
    • autonoe.uberspace.de
    • barnard.uberspace.de
    • bebhionn.uberspace.de
    • belinda.uberspace.de
    • bernardi.uberspace.de
    • bestla.uberspace.de
    • bianca.uberspace.de
    • biela.uberspace.de
    • boethin.uberspace.de
    • borrelly.uberspace.de
    • brewington.uberspace.de
    • brooks.uberspace.de
    • brorsen.uberspace.de
    • bus.uberspace.de
    • caliban.uberspace.de
    • callirhoe.uberspace.de
    • carpo.uberspace.de
    • chaldene.uberspace.de
    • charon.uberspace.de
    • chiron.uberspace.de
    • christensen.uberspace.de
    • cressida.uberspace.de
    • crommelin.uberspace.de
    • cupid.uberspace.de
    • cyllene.uberspace.de
    • daphnis.uberspace.de
    • deimos.uberspace.de
    • desdemonia.uberspace.de
    • despina.uberspace.de
    • devico.uberspace.de
    • dia.uberspace.de
    • dione.uberspace.de
    • dubhe.uberspace.de
    • dysnomia.uberspace.de
    • echeclus.uberspace.de
    • eirene.uberspace.de
    • elara.uberspace.de
    • elst.uberspace.de
    • encke.uberspace.de
    • epimetheus.uberspace.de
    • erinome.uberspace.de
    • erriapus.uberspace.de
    • euanthe.uberspace.de
    • eukelade.uberspace.de
    • eupheme.uberspace.de
    • euporie.uberspace.de
    • europa.uberspace.de
    • eurydome.uberspace.de
    • farbauti.uberspace.de
    • faye.uberspace.de
    • fenrir.uberspace.de
    • ferdinand.uberspace.de
    • finlay.uberspace.de
    • fornjot.uberspace.de
    • francisco.uberspace.de
    • gacrux.uberspace.de
    • galatea.uberspace.de
    • gale.uberspace.de
    • ganymed.uberspace.de
    • gehrels.uberspace.de
    • giacobini.uberspace.de
    • gibbs.uberspace.de
    • giclas.uberspace.de
    • greip.uberspace.de
    • halimede.uberspace.de
    • halley.uberspace.de
    • harpalyke.uberspace.de
    • harrington.uberspace.de
    • hartley.uberspace.de
    • hati.uberspace.de
    • hegemone.uberspace.de
    • helene.uberspace.de
    • helike.uberspace.de
    • helin.uberspace.de
    • hergenrother.uberspace.de
    • hermippe.uberspace.de
    • hernmann.uberspace.de
    • herse.uberspace.de
    • himalia.uberspace.de
    • hippocamp.uberspace.de
    • holmes.uberspace.de
    • hyrrokkin.uberspace.de
    • iapetus.uberspace.de
    • ijiraq.uberspace.de
    • iocaste.uberspace.de
    • jedicke.uberspace.de
    • johnson.uberspace.de
    • klemola.uberspace.de
    • kohoutek.uberspace.de
    • kojima.uberspace.de
    • kopff.uberspace.de
    • kowal.uberspace.de
    • kushida.uberspace.de
    • longmore.uberspace.de
    • lovas.uberspace.de
    • machholz.uberspace.de
    • maury.uberspace.de
    • mueller.uberspace.de
    • neujmin.uberspace.de
    • olbers.uberspace.de
    • pizarro.uberspace.de
    • rasalhague.uberspace.de
    • read.uberspace.de
    • reinmuth.uberspace.de
    • russel.uberspace.de
    • sanguin.uberspace.de
    • schuster.uberspace.de
    • scotti.uberspace.de
    • shaula.uberspace.de
    • shoemaker.uberspace.de
    • skiff.uberspace.de
    • spahr.uberspace.de
    • spitaler.uberspace.de
    • taylor.uberspace.de
    • tempel.uberspace.de
    • tritton.uberspace.de
    • tuttle.uberspace.de
    • vanness.uberspace.de
    • westphal.uberspace.de
    • whipple.uberspace.de
    • wild.uberspace.de
    • wirtanen.uberspace.de
    • yeung.uberspace.de
  • Possible site-wide power outage at FRA3 (185.26.156.0/24, 2a00:d0c0:200::/48), the data centre operator is looking into it.

  • 28th June 2021

    dubhe.uberspace.de Host dubhe is being migrated to Uberspace 7

    This host will move to U7, causing a downtime of multiple hours today.

  • Migration successfully completed

  • 26th June 2021

    FRA3 Degraded performance in FRA3 (185.26.156.0/24, 2a00:d0c0:200::/48)

    FRA3 (185.26.156.0/24, 2a00:d0c0:200::/48) is currently experiencing performance issues and intermittent availability. We're looking into it.

  • All hosts have been restored.

  • Most of the hosts are back online. We are working to restore all services.

  • We're booting the uberspace hosts. Progress: 85% up.

  • We're booting the uberspace hosts. Progress: 70% up.

  • We're booting the uberspace hosts. Progress: 40% up.

  • We're booting the uberspace hosts. Progress: 20% up.

  • The cluster needs to be rebooted, causing a restart of all hosts in FRA3 / 185.26.156.0/24.

  • One of our ceph nodes has unusually high load and needs to be rebooted. This will restart the following hosts: kohutek, adrastea, hati, herse, christensen.uberspace.de.