We observe that IPv4 connectivity is currently impacted. IPv6 connectivity is unaffected.
After an unexpected core router reboot, the standby router took over. While in theory both routers have exactly the same routing configuration, we discovered a bug in the IPv4 routing configuration which only affected the standby router. This bug is now fixed and IPv4 connectivity fully restored since 14:08 CEST.
Today from 15:30 (GMT+2) we will be doing maintenance works on our routers and uplink connections in our FRA4 datacenter (95.143.172.0/24 network), after some instabilities observed in the past which we now want to put to an end my making some major changes. The maintenance works will be carried out in person in the datacenter and in collaboration with the datacenter provider. While we try our best to limit impact on the connectivity, there might be shorter periods of network outages.
The network 95.143.172.0/24 at our datacenter FRA4 is currently experiencing an outage. We're investigating.
IPv6 connectivity at our FRA4 datacenter is currently impacted and showing packet loss. IPv4 is working fine. The root cause has been identified already and our operations team is working on a fix.
Due to maintenance work, it is possible that the network connections to some of our hosts might be temporarily disrupted (a list of hosts will follow).
Maintainance finished without major disruptions.
A list of hosts, that might suffer from short disruptions of network connectivity:
acamar.uberspace.de
achernar.uberspace.de
aldebaran.uberspace.de
antares.uberspace.de
antila.uberspace.de
apus.uberspace.de
aquila.uberspace.de
ara.uberspace.de
aries.uberspace.de
auriga.uberspace.de
bootes.uberspace.de
caelum.uberspace.de
canis.uberspace.de
canopus.uberspace.de
capella.uberspace.de
carina.uberspace.de
cassiopeia.uberspace.de
centaurus.uberspace.de
cepheus.uberspace.de
cetus.uberspace.de
circinus.uberspace.de
columba.uberspace.de
corvus.uberspace.de
crater.uberspace.de
crux.uberspace.de
cygnus.uberspace.de
delphinus.uberspace.de
dorado.uberspace.de
draco.uberspace.de
fomalhaut.uberspace.de
fulu.uberspace.de
grus.uberspace.de
hamal.uberspace.de
hercules.uberspace.de
horologium.uberspace.de
hydrus.uberspace.de
indus.uberspace.de
jarnsaxa.uberspace.de
juliet.uberspace.de
lacerta.uberspace.de
leo.uberspace.de
lepus.uberspace.de
libra.uberspace.de
lupus.uberspace.de
lynx.uberspace.de
menkar.uberspace.de
mensa.uberspace.de
monoceres.uberspace.de
musca.uberspace.de
norma.uberspace.de
octans.uberspace.de
pavo.uberspace.de
perseus.uberspace.de
phoenix.uberspace.de
pictor.uberspace.de
puppis.uberspace.de
rigel.uberspace.de
sagitta.uberspace.de
sculptor.uberspace.de
serpens.uberspace.de
sirius.uberspace.de
triangulum.uberspace.de
tucana.uberspace.de
ursa.uberspace.de
vega.uberspace.de
vela.uberspace.de
volans.uberspace.de
vulpecula.uberspace.de
Multiple host are currently unreachable, we are investigating.
A full list will follow ASAP.
The underlying issue was a switch config problem causing the Ceph storage network to be unresponsive, so no disk I/O was possible for the - still running - U7 VMs. The problem has been solved for now and according to our service monitoring, all VMs with all of their services are running fine again.
All host seem to be operational again.
All host - except crater.uberspace.de - are reachable again. Some might still need time to recover before normal operation.
We forgot to reload the mail services when we recently renewed their TLS certificates. If your mail client complains about an expired IMAP/POP3/SMTP certificate, please restart the mail client and potentially your computersystem. We restarted the services with the new certificate on 23 Apr 2023.
We are investigating a network outage on FRA4, the following hosts are affected:
acamar.uberspace.de
achernar.uberspace.de
aldebaran.uberspace.de
antares.uberspace.de
antila.uberspace.de
apus.uberspace.de
aquila.uberspace.de
ara.uberspace.de
aries.uberspace.de
auriga.uberspace.de
bootes.uberspace.de
caelum.uberspace.de
canis.uberspace.de
canopus.uberspace.de
capella.uberspace.de
carina.uberspace.de
cassiopeia.uberspace.de
centaurus.uberspace.de
cepheus.uberspace.de
cetus.uberspace.de
circinus.uberspace.de
columba.uberspace.de
corvus.uberspace.de
crater.uberspace.de
crux.uberspace.de
cygnus.uberspace.de
delphinus.uberspace.de
dorado.uberspace.de
draco.uberspace.de
fomalhaut.uberspace.de
fulu.uberspace.de
grus.uberspace.de
hamal.uberspace.de
hercules.uberspace.de
horologium.uberspace.de
hydrus.uberspace.de
indus.uberspace.de
jarnsaxa.uberspace.de
juliet.uberspace.de
lacerta.uberspace.de
leo.uberspace.de
lepus.uberspace.de
libra.uberspace.de
lupus.uberspace.de
lynx.uberspace.de
menkar.uberspace.de
mensa.uberspace.de
monoceres.uberspace.de
musca.uberspace.de
norma.uberspace.de
octans.uberspace.de
pavo.uberspace.de
perseus.uberspace.de
phoenix.uberspace.de
pictor.uberspace.de
puppis.uberspace.de
rigel.uberspace.de
sagitta.uberspace.de
sculptor.uberspace.de
serpens.uberspace.de
sirius.uberspace.de
triangulum.uberspace.de
tucana.uberspace.de
ursa.uberspace.de
vega.uberspace.de
vela.uberspace.de
volans.uberspace.de
vulpecula.uberspace.de
On friday april 7 one of our backup raid controllers lost two of its four disks. We were unable to repair the damage with the remaining 2 disks in a sufficient way and decided to start over with the backups for the affected hosts. Due to the crash there are no backups available from april 7 and earlier for the following hosts:
bestla
bergelmir
bianca
caliban
callirrhoe
carpo
eirene
fairydust
johnson
pizarro
sanguin
The original data on these hosts and the backups on other hosts were not affected by this. New backups for the affected hosts are running since april 8.
We are investigating an outage of our cluster on FRA4, the following hosts are affected:
acamar.uberspace.de
achernar.uberspace.de
aldebaran.uberspace.de
antares.uberspace.de
antila.uberspace.de
apus.uberspace.de
aquila.uberspace.de
ara.uberspace.de
aries.uberspace.de
auriga.uberspace.de
bootes.uberspace.de
caelum.uberspace.de
canis.uberspace.de
canopus.uberspace.de
capella.uberspace.de
carina.uberspace.de
cassiopeia.uberspace.de
centaurus.uberspace.de
cepheus.uberspace.de
cetus.uberspace.de
circinus.uberspace.de
columba.uberspace.de
corvus.uberspace.de
crater.uberspace.de
crux.uberspace.de
cygnus.uberspace.de
delphinus.uberspace.de
dorado.uberspace.de
draco.uberspace.de
fomalhaut.uberspace.de
fulu.uberspace.de
grus.uberspace.de
hamal.uberspace.de
hercules.uberspace.de
horologium.uberspace.de
hydrus.uberspace.de
indus.uberspace.de
jarnsaxa.uberspace.de
juliet.uberspace.de
lacerta.uberspace.de
leo.uberspace.de
lepus.uberspace.de
libra.uberspace.de
lupus.uberspace.de
lynx.uberspace.de
menkar.uberspace.de
mensa.uberspace.de
monoceres.uberspace.de
musca.uberspace.de
norma.uberspace.de
octans.uberspace.de
pavo.uberspace.de
perseus.uberspace.de
phoenix.uberspace.de
pictor.uberspace.de
puppis.uberspace.de
rigel.uberspace.de
sagitta.uberspace.de
sculptor.uberspace.de
serpens.uberspace.de
sirius.uberspace.de
triangulum.uberspace.de
tucana.uberspace.de
ursa.uberspace.de
vega.uberspace.de
vela.uberspace.de
volans.uberspace.de
vulpecula.uberspace.de
We are currently encountering a Network Error on our Cluster in FRA4, we are investigating.
the network has been fixed by the site operator
we are in contact with the network operators at the cluster site, they are working on the issue
This afffects the following Hosts:
acamar.uberspace.de
achernar.uberspace.de
aldebaran.uberspace.de
antares.uberspace.de
antila.uberspace.de
apus.uberspace.de
aquila.uberspace.de
ara.uberspace.de
aries.uberspace.de
auriga.uberspace.de
bootes.uberspace.de
caelum.uberspace.de
canis.uberspace.de
canopus.uberspace.de
capella.uberspace.de
carina.uberspace.de
cassiopeia.uberspace.de
centaurus.uberspace.de
cepheus.uberspace.de
cetus.uberspace.de
circinus.uberspace.de
columba.uberspace.de
corvus.uberspace.de
crater.uberspace.de
crux.uberspace.de
cygnus.uberspace.de
delphinus.uberspace.de
dorado.uberspace.de
draco.uberspace.de
fomalhaut.uberspace.de
fulu.uberspace.de
grus.uberspace.de
hamal.uberspace.de
hercules.uberspace.de
horologium.uberspace.de
hydrus.uberspace.de
indus.uberspace.de
jarnsaxa.uberspace.de
juliet.uberspace.de
lacerta.uberspace.de
leo.uberspace.de
lepus.uberspace.de
libra.uberspace.de
lupus.uberspace.de
lynx.uberspace.de
menkar.uberspace.de
mensa.uberspace.de
monoceres.uberspace.de
musca.uberspace.de
norma.uberspace.de
octans.uberspace.de
pavo.uberspace.de
perseus.uberspace.de
phoenix.uberspace.de
pictor.uberspace.de
puppis.uberspace.de
rigel.uberspace.de
sagitta.uberspace.de
sculptor.uberspace.de
serpens.uberspace.de
sirius.uberspace.de
triangulum.uberspace.de
tucana.uberspace.de
ursa.uberspace.de
vega.uberspace.de
vela.uberspace.de
volans.uberspace.de
vulpecula.uberspace.de
We added several new disks to our storage cluster which results in quite some data being moved around. This results in temporarily impacted disk performance. We slowed down the data migration progress in order to lower the impact already. Usual disk performance is about to be restored soon.