Past Incidents

Recovery is done, system is up and running.

completed.

httpd fixed

httpd is crashed

completed.

Reboot completed. Load and services might take a sort while to settle.

we fixed some dovecot file owner issues, mailsystem should work again now

we are encountering issues with the mail system

webserver httpd is working again

webserver (httpd) not working currently

Recovery is done, system is up and running.

The system needs to be rebooted und and reappear in a few minutes.

We are recreating a pre-crash state, after that, the host should be fully up again.

Host will be down for a reboot.

repair finished, processing quotacheck now

The host is going down for xfs_repair, for about 5 hours (estimate).

Because of an unexpected crash we had to start the xfs repair already now, the host will be down for the time of processing the repairs.

repair process is rescheduled for monday 05.07.2021 10:30

the host will be down for xfs_repair on friday 02.07.2021 - 13:30 (about 5 hours)

Host is back up. We are still checking some things, but it runs for now.

Host will be down for a reboot.

Quotacheck done. Host is up again. We currlyt recreating a pristine state, after that, the host should be fully up again.

XFS repair done, host is running quotacheck.

The host is going down for xfs_repair, for about 5 hours (estimate).

repair process is rescheduled for monday 05.07.2021 10:30

the host will be down for xfs_repair on friday 02.07.2021 - 13:30 (about 5 hours)

Redeployment done. Still needs a round of integrity checks.

Host is up again and (at least partially) working.

Still needs some more work though and will be rebooted once more, but that probably will have to wait till morning.

The host still needs more fixes, after coming back from XFS repairs.

xfs repair is done, processing quota checks

host is going offline now

we will need to send the host in recovery mode for xfs_repair at 15:00 (estimated 1-2 hours)

Quotacheck and relabling are done. Host is up again.

Still running checks, so things might not work as expected for now.

XFS repair finished, host is processing quotacheck now.

The host is going down for XFS repair.

the host will be down for xfs_repair on friday 02.07.2021 - 10:30 (about 5 hours)

Quotacheck and relabling are done. Host is up again.

Still running checks, so things might not work as expected for now.

repair finished, host is processing quotacheck now

host is offline now for xfs_repair

the host will be down for xfs_repair on friday 02.07.2021 - 10:30 (about 5 hours)

Haraka is restored and accepting mails again.

the host is back online but we are still working on the haraka mailserver

host is now in process of quota check

The host is up and running again :tada:

We are still syncing backups, so you might want to check if /var/www/virtual/username already contains what you expect.

We managed to fix a lot of issues with SE labels and PHP. Hopefully most things work for you now.

Just to clarify: the host is up, but still not fully functioning again. We are working on it, but it will take some more time (and reboots).

The host is back up again.

We still have some repair task and testing to do though. Meaning there could be more reboots and performance issues.

Syncing completed, but the host won't boot due to SELinux errors. We to avoid mistakes made due to sleep deprivation, we'll continue the process tomorrow.

backup is still transferring to the new VM

The host is re-created with a fresh deployment. Backups are currently copyed over (this will take some time).

we had to give up the system recovery and will set up a new VM with data backup on monday, please note the update in our blogpost.

we are still transferring data to the new block device and doing some tests

we are mounting a new block device to replace the VM, the syncing is in progress now

the httpd web server is back online

the webserver is offline again

the httpd web server is back online

the host is back online but the web server (httpd) is still in error, we are working on it

Reboots are done, hosts are up again.

Some host where affected by XFS errors during recovery, you can see the follow up incidents for the current state.

recovered:

• dubhe.uberspace.de
• neujmin.uberspace.de

• russel.uberspace.de

recovered:

• gibbs.uberspace.de
• giclas.uberspace.de

recovered:

• ananke.uberspace.de
• aoede.uberspace.de
• bestla.uberspace.de
• callirhoe.uberspace.de
• cressida.uberspace.de
• crommelin.uberspace.de
• cupid.uberspace.de

recovered

• kopff.uberspace.de
• kojima.uberspace.de
• iocaste.uberspace.de

recovered:

• hernmann.uberspace.de

recovered:

• ferdinand.uberspace.de
• finlay.uberspace.de
• gale.uberspace.de
• giacobini.uberspace.de
• jedicke.uberspace.de

recovered:

• euporie.uberspace.de
• europa.uberspace.de
• eurydome.uberspace.de
• johnson.uberspace.de
• lovas.uberspace.de

recovered:

• eukelade.uberspace.de
• eupheme.uberspace.de

recovered:

• erinome.uberspace.de
• erriapus.uberspace.de
• euanthe.uberspace.de
• kowal.uberspace.de
• longmore.uberspace.de

recovered:

• halimede.uberspace.de
• halley.uberspace.de
• harpalyke.uberspace.de

recovered:

• encke.uberspace.de
• epimetheus.uberspace.de
• farbauti.uberspace.de
• faye.uberspace.de
• fenrir.uberspace.de
• fornjot.uberspace.de
• francisco.uberspace.de
• galatea.uberspace.de
• ganymed.uberspace.de
• gehrels.uberspace.de
• greip.uberspace.de

recovered:

• echeclus.uberspace.de
• eirene.uberspace.de
• elst.uberspace.de
• hartley.uberspace.de
• hegemone.uberspace.de
• helene.uberspace.de
• helike.uberspace.de
• hergenrother.uberspace.de
• hermippe.uberspace.de

recovered:

• dysnomia.uberspace.de
• herse.uberspace.de
• himalia.uberspace.de
• hippocamp.uberspace.de
• holmes.uberspace.de
• hyrrokkin.uberspace.de
• iapetus.uberspace.de
• ijiraq.uberspace.de
• maury.uberspace.de

recovered:

• chiron.uberspace.de
• christensen.uberspace.de
• deimos.uberspace.de
• desdemonia.uberspace.de
• despina.uberspace.de
• dia.uberspace.de
• dione.uberspace.de
• klemola.uberspace.de
• kohoutek.uberspace.de

recovered:

• caliban.uberspace.de
• charon.uberspace.de
• daphnis.uberspace.de
• hati.uberspace.de
• mueller.uberspace.de
• olbers.uberspace.de
• sanguin.uberspace.de
• schuster.uberspace.de

recovered:

• carpo.uberspace.de
• chaldene.uberspace.de
• pizarro.uberspace.de
• read.uberspace.de
• reinmuth.uberspace.de
• scotti.uberspace.de
• shoemaker.uberspace.de
• skiff.uberspace.de

recovered:

• borrelly.uberspace.de
• brewington.uberspace.de
• brooks.uberspace.de
• brorsen.uberspace.de
• bus.uberspace.de
• spahr.uberspace.de
• spitaler.uberspace.de
• taylor.uberspace.de
• tempel.uberspace.de
• tritton.uberspace.de
• tuttle.uberspace.de

recovered:

• bianca.uberspace.de
• biela.uberspace.de
• boethin.uberspace.de
• vanness.uberspace.de
• westphal.uberspace.de
• whipple.uberspace.de
• yeung.uberspace.de

recovered:

• bernardi.uberspace.de
• wild.uberspace.de

recovered:

• bebhionn.uberspace.de
• belinda.uberspace.de
• wirtanen.uberspace.de

Here's a list of the host in the FRA3 datacenter, that are affected:

• adrastea.uberspace.de
• aegir.uberspace.de
• amalthea.uberspace.de
• arche.uberspace.de
• bergelmir.uberspace.de
• ciffreo.uberspace.de
• clark.uberspace.de
• cordelia.uberspace.de
• howell.uberspace.de
• mcnaught.uberspace.de
• mrkos.uberspace.de
• wolf.uberspace.de

Already recovered or not affected:

• aegaeon.uberspace.de
• aitne.uberspace.de
• albiorix.uberspace.de
• alnilam.uberspace.de
• alpheca.uberspace.de
• ananke.uberspace.de
• ankaa.uberspace.de
• aoede.uberspace.de
• arend.uberspace.de
• ariel.uberspace.de
• atlas.uberspace.de
• autonoe.uberspace.de
• barnard.uberspace.de
• bebhionn.uberspace.de
• belinda.uberspace.de
• bernardi.uberspace.de
• bestla.uberspace.de
• bianca.uberspace.de
• biela.uberspace.de
• boethin.uberspace.de
• borrelly.uberspace.de
• brewington.uberspace.de
• brooks.uberspace.de
• brorsen.uberspace.de
• bus.uberspace.de
• caliban.uberspace.de
• callirhoe.uberspace.de
• carpo.uberspace.de
• chaldene.uberspace.de
• charon.uberspace.de
• chiron.uberspace.de
• christensen.uberspace.de
• cressida.uberspace.de
• crommelin.uberspace.de
• cupid.uberspace.de
• cyllene.uberspace.de
• daphnis.uberspace.de
• deimos.uberspace.de
• desdemonia.uberspace.de
• despina.uberspace.de
• devico.uberspace.de
• dia.uberspace.de
• dione.uberspace.de
• dubhe.uberspace.de
• dysnomia.uberspace.de
• echeclus.uberspace.de
• eirene.uberspace.de
• elara.uberspace.de
• elst.uberspace.de
• encke.uberspace.de
• epimetheus.uberspace.de
• erinome.uberspace.de
• erriapus.uberspace.de
• euanthe.uberspace.de
• eukelade.uberspace.de
• eupheme.uberspace.de
• euporie.uberspace.de
• europa.uberspace.de
• eurydome.uberspace.de
• farbauti.uberspace.de
• faye.uberspace.de
• fenrir.uberspace.de
• ferdinand.uberspace.de
• finlay.uberspace.de
• fornjot.uberspace.de
• francisco.uberspace.de
• gacrux.uberspace.de
• galatea.uberspace.de
• gale.uberspace.de
• ganymed.uberspace.de
• gehrels.uberspace.de
• giacobini.uberspace.de
• gibbs.uberspace.de
• giclas.uberspace.de
• greip.uberspace.de
• halimede.uberspace.de
• halley.uberspace.de
• harpalyke.uberspace.de
• harrington.uberspace.de
• hartley.uberspace.de
• hati.uberspace.de
• hegemone.uberspace.de
• helene.uberspace.de
• helike.uberspace.de
• helin.uberspace.de
• hergenrother.uberspace.de
• hermippe.uberspace.de
• hernmann.uberspace.de
• herse.uberspace.de
• himalia.uberspace.de
• hippocamp.uberspace.de
• holmes.uberspace.de
• hyrrokkin.uberspace.de
• iapetus.uberspace.de
• ijiraq.uberspace.de
• iocaste.uberspace.de
• jedicke.uberspace.de
• johnson.uberspace.de
• klemola.uberspace.de
• kohoutek.uberspace.de
• kojima.uberspace.de
• kopff.uberspace.de
• kowal.uberspace.de
• kushida.uberspace.de
• longmore.uberspace.de
• lovas.uberspace.de
• machholz.uberspace.de
• maury.uberspace.de
• mueller.uberspace.de
• neujmin.uberspace.de
• olbers.uberspace.de
• pizarro.uberspace.de
• rasalhague.uberspace.de
• read.uberspace.de
• reinmuth.uberspace.de
• russel.uberspace.de
• sanguin.uberspace.de
• schuster.uberspace.de
• scotti.uberspace.de
• shaula.uberspace.de
• shoemaker.uberspace.de
• skiff.uberspace.de
• spahr.uberspace.de
• spitaler.uberspace.de
• taylor.uberspace.de
• tempel.uberspace.de
• tritton.uberspace.de
• tuttle.uberspace.de
• vanness.uberspace.de
• westphal.uberspace.de
• whipple.uberspace.de
• wild.uberspace.de
• wirtanen.uberspace.de
• yeung.uberspace.de

Possible site-wide power outage at FRA3 (185.26.156.0/24, 2a00:d0c0:200::/48), the data centre operator is looking into it.