All systems are operational

Past Incidents

26th September 2023

auriga.uberspace.de Reboot with more RAM

The host was reported unreachable by monitoring at 08:19. The vm had run out of RAM. We added some RAM and the host finished the process of rebooting now and everything seems to be running operationally again.

25th September 2023

FRA3 IPv4 connectivity impacted

We observe that IPv4 connectivity is currently impacted. IPv6 connectivity is unaffected.

  • After an unexpected core router reboot, the standby router took over. While in theory both routers have exactly the same routing configuration, we discovered a bug in the IPv4 routing configuration which only affected the standby router. This bug is now fixed and IPv4 connectivity fully restored since 14:08 CEST.

  • 1st September 2023

    aegaeon.uberspace.de web server down on aegaeon.uberspace.de

    The web server on aegaeon.uberspace.de fails to start. We're investigating.

  • we identified a faulty config file and fixed it. everything is up and running again.

  • 17th August 2023

    MariaDB crashed

    MariaDB on helin.uberspace.de crashed. It was restarted successfully at 11:50 CEST.

    31st July 2023

    aries.uberspace.de One hosting cluster node crashed

    The cluster node hosting aries.uberspace.de is crashed. We're migrating the VM to another cluster node.

  • The VM is booted on a functional cluster node.

  • puppis.uberspace.de One hosting cluster node crashed

    The cluster node hosting puppis.uberspace.de is crashed. We're migrating the VM to another cluster node.

  • The VM is booted on a functional cluster node.

  • FRA4 Maintenance work at FRA4, might affect uplinks

    Today from 15:30 (GMT+2) we will be doing maintenance works on our routers and uplink connections in our FRA4 datacenter (95.143.172.0/24 network), after some instabilities observed in the past which we now want to put to an end my making some major changes. The maintenance works will be carried out in person in the datacenter and in collaboration with the datacenter provider. While we try our best to limit impact on the connectivity, there might be shorter periods of network outages.

  • All network config and router upgrades have been finished successfully.

  • 25th July 2023

    FRA4 FRA4 network down

    The network 95.143.172.0/24 at our datacenter FRA4 is currently experiencing an outage. We're investigating.

  • A multicast storm had temporarily blocked both of our uplink ports. The situation has been resolved, network connectivity is fully restored.

  • We're working with the datacenter technicians to find the root cause of the outage.

  • 18th July 2023

    FRA4 IPv6 connectivity impacted

    IPv6 connectivity at our FRA4 datacenter is currently impacted and showing packet loss. IPv4 is working fine. The root cause has been identified already and our operations team is working on a fix.

  • The root cause have been irregular router advertisements being sent. This has been fixed, IPv6 connectivity is stable again.

  • 21st June 2023

    possibillity for temporarily disruption of network connectivity to some host

    Due to maintenance work, it is possible that the network connections to some of our hosts might be temporarily disrupted (a list of hosts will follow).

  • Maintainance finished without major disruptions.

  • A list of hosts, that might suffer from short disruptions of network connectivity:

    • acamar.uberspace.de
    • achernar.uberspace.de
    • aldebaran.uberspace.de
    • antares.uberspace.de
    • antila.uberspace.de
    • apus.uberspace.de
    • aquila.uberspace.de
    • ara.uberspace.de
    • aries.uberspace.de
    • auriga.uberspace.de
    • bootes.uberspace.de
    • caelum.uberspace.de
    • canis.uberspace.de
    • canopus.uberspace.de
    • capella.uberspace.de
    • carina.uberspace.de
    • cassiopeia.uberspace.de
    • centaurus.uberspace.de
    • cepheus.uberspace.de
    • cetus.uberspace.de
    • circinus.uberspace.de
    • columba.uberspace.de
    • corvus.uberspace.de
    • crater.uberspace.de
    • crux.uberspace.de
    • cygnus.uberspace.de
    • delphinus.uberspace.de
    • dorado.uberspace.de
    • draco.uberspace.de
    • fomalhaut.uberspace.de
    • fulu.uberspace.de
    • grus.uberspace.de
    • hamal.uberspace.de
    • hercules.uberspace.de
    • horologium.uberspace.de
    • hydrus.uberspace.de
    • indus.uberspace.de
    • jarnsaxa.uberspace.de
    • juliet.uberspace.de
    • lacerta.uberspace.de
    • leo.uberspace.de
    • lepus.uberspace.de
    • libra.uberspace.de
    • lupus.uberspace.de
    • lynx.uberspace.de
    • menkar.uberspace.de
    • mensa.uberspace.de
    • monoceres.uberspace.de
    • musca.uberspace.de
    • norma.uberspace.de
    • octans.uberspace.de
    • pavo.uberspace.de
    • perseus.uberspace.de
    • phoenix.uberspace.de
    • pictor.uberspace.de
    • puppis.uberspace.de
    • rigel.uberspace.de
    • sagitta.uberspace.de
    • sculptor.uberspace.de
    • serpens.uberspace.de
    • sirius.uberspace.de
    • triangulum.uberspace.de
    • tucana.uberspace.de
    • ursa.uberspace.de
    • vega.uberspace.de
    • vela.uberspace.de
    • volans.uberspace.de
    • vulpecula.uberspace.de
  • 20th June 2023

    multiple host are unreachable

    Multiple host are currently unreachable, we are investigating.

    A full list will follow ASAP.

  • The underlying issue was a switch config problem causing the Ceph storage network to be unresponsive, so no disk I/O was possible for the - still running - U7 VMs. The problem has been solved for now and according to our service monitoring, all VMs with all of their services are running fine again.

  • All host seem to be operational again.

  • All host - except crater.uberspace.de - are reachable again. Some might still need time to recover before normal operation.

  • 14th June 2023

    centaurus.uberspace.de nginx offline on centaurus.uberspace.de

    the nginx on centaurus has stopped, web access is not available, we are investigating

  • the incident has been resolved

  • 23rd April 2023

    mail server certificate expired

    We forgot to reload the mail services when we recently renewed their TLS certificates. If your mail client complains about an expired IMAP/POP3/SMTP certificate, please restart the mail client and potentially your computersystem. We restarted the services with the new certificate on 23 Apr 2023.

    12th April 2023

    FRA4 Network on FRA4 is down

    We are investigating a network outage on FRA4, the following hosts are affected:

    acamar.uberspace.de
    achernar.uberspace.de
    aldebaran.uberspace.de
    antares.uberspace.de
    antila.uberspace.de
    apus.uberspace.de
    aquila.uberspace.de
    ara.uberspace.de
    aries.uberspace.de
    auriga.uberspace.de
    bootes.uberspace.de
    caelum.uberspace.de
    canis.uberspace.de
    canopus.uberspace.de
    capella.uberspace.de
    carina.uberspace.de
    cassiopeia.uberspace.de
    centaurus.uberspace.de
    cepheus.uberspace.de
    cetus.uberspace.de
    circinus.uberspace.de
    columba.uberspace.de
    corvus.uberspace.de
    crater.uberspace.de
    crux.uberspace.de
    cygnus.uberspace.de
    delphinus.uberspace.de
    dorado.uberspace.de
    draco.uberspace.de
    fomalhaut.uberspace.de
    fulu.uberspace.de
    grus.uberspace.de
    hamal.uberspace.de
    hercules.uberspace.de
    horologium.uberspace.de
    hydrus.uberspace.de
    indus.uberspace.de
    jarnsaxa.uberspace.de
    juliet.uberspace.de
    lacerta.uberspace.de
    leo.uberspace.de
    lepus.uberspace.de
    libra.uberspace.de
    lupus.uberspace.de
    lynx.uberspace.de
    menkar.uberspace.de
    mensa.uberspace.de
    monoceres.uberspace.de
    musca.uberspace.de
    norma.uberspace.de
    octans.uberspace.de
    pavo.uberspace.de
    perseus.uberspace.de
    phoenix.uberspace.de
    pictor.uberspace.de
    puppis.uberspace.de
    rigel.uberspace.de
    sagitta.uberspace.de
    sculptor.uberspace.de
    serpens.uberspace.de
    sirius.uberspace.de
    triangulum.uberspace.de
    tucana.uberspace.de
    ursa.uberspace.de
    vega.uberspace.de
    vela.uberspace.de
    volans.uberspace.de
    vulpecula.uberspace.de
    
  • Network connectivity is fully restored.

  • The datacenter operator is working on it, we just got feedback on the phone. Connectivity should be back within a few minutes.

  • 11th April 2023

    webmail.uberspace.de Webmailer issue

    There are currently problems logging in to our webmailer (webmail.uberspace.de) with a @username.uber.space or @uber.space domain, we are working on it.

  • the issue has been fixed

  • 7th April 2023

    Backup crashed for several hosts

    On friday april 7 one of our backup raid controllers lost two of its four disks. We were unable to repair the damage with the remaining 2 disks in a sufficient way and decided to start over with the backups for the affected hosts. Due to the crash there are no backups available from april 7 and earlier for the following hosts:

    bestla
    bergelmir
    bianca
    caliban
    callirrhoe
    carpo
    eirene
    fairydust
    johnson
    pizarro
    sanguin
    

    The original data on these hosts and the backups on other hosts were not affected by this. New backups for the affected hosts are running since april 8.

    2nd April 2023

    Cluster FRA4 is down

    We are investigating an outage of our cluster on FRA4, the following hosts are affected:

    acamar.uberspace.de
    achernar.uberspace.de
    aldebaran.uberspace.de
    antares.uberspace.de
    antila.uberspace.de
    apus.uberspace.de
    aquila.uberspace.de
    ara.uberspace.de
    aries.uberspace.de
    auriga.uberspace.de
    bootes.uberspace.de
    caelum.uberspace.de
    canis.uberspace.de
    canopus.uberspace.de
    capella.uberspace.de
    carina.uberspace.de
    cassiopeia.uberspace.de
    centaurus.uberspace.de
    cepheus.uberspace.de
    cetus.uberspace.de
    circinus.uberspace.de
    columba.uberspace.de
    corvus.uberspace.de
    crater.uberspace.de
    crux.uberspace.de
    cygnus.uberspace.de
    delphinus.uberspace.de
    dorado.uberspace.de
    draco.uberspace.de
    fomalhaut.uberspace.de
    fulu.uberspace.de
    grus.uberspace.de
    hamal.uberspace.de
    hercules.uberspace.de
    horologium.uberspace.de
    hydrus.uberspace.de
    indus.uberspace.de
    jarnsaxa.uberspace.de
    juliet.uberspace.de
    lacerta.uberspace.de
    leo.uberspace.de
    lepus.uberspace.de
    libra.uberspace.de
    lupus.uberspace.de
    lynx.uberspace.de
    menkar.uberspace.de
    mensa.uberspace.de
    monoceres.uberspace.de
    musca.uberspace.de
    norma.uberspace.de
    octans.uberspace.de
    pavo.uberspace.de
    perseus.uberspace.de
    phoenix.uberspace.de
    pictor.uberspace.de
    puppis.uberspace.de
    rigel.uberspace.de
    sagitta.uberspace.de
    sculptor.uberspace.de
    serpens.uberspace.de
    sirius.uberspace.de
    triangulum.uberspace.de
    tucana.uberspace.de
    ursa.uberspace.de
    vega.uberspace.de
    vela.uberspace.de
    volans.uberspace.de
    vulpecula.uberspace.de
    
  • the network issue have been fixed by the site operators

  • 31st March 2023

    manual and lab are down

    Our Manual and the Lab on out Netflify instance are down at the moment.

  • The issue has been resolved.

  • As our netlify instance is not working at the moment, we rerouted the manual and the lab to another location. The DNS changes have to settle but both domains should be working soon again for you:

  • 25th March 2023

    Network on FRA4 is down

    We are currently encountering a Network Error on our Cluster in FRA4, we are investigating.

  • the network has been fixed by the site operator

  • we are in contact with the network operators at the cluster site, they are working on the issue

  • This afffects the following Hosts:

    acamar.uberspace.de
    achernar.uberspace.de
    aldebaran.uberspace.de
    antares.uberspace.de
    antila.uberspace.de
    apus.uberspace.de
    aquila.uberspace.de
    ara.uberspace.de
    aries.uberspace.de
    auriga.uberspace.de
    bootes.uberspace.de
    caelum.uberspace.de
    canis.uberspace.de
    canopus.uberspace.de
    capella.uberspace.de
    carina.uberspace.de
    cassiopeia.uberspace.de
    centaurus.uberspace.de
    cepheus.uberspace.de
    cetus.uberspace.de
    circinus.uberspace.de
    columba.uberspace.de
    corvus.uberspace.de
    crater.uberspace.de
    crux.uberspace.de
    cygnus.uberspace.de
    delphinus.uberspace.de
    dorado.uberspace.de
    draco.uberspace.de
    fomalhaut.uberspace.de
    fulu.uberspace.de
    grus.uberspace.de
    hamal.uberspace.de
    hercules.uberspace.de
    horologium.uberspace.de
    hydrus.uberspace.de
    indus.uberspace.de
    jarnsaxa.uberspace.de
    juliet.uberspace.de
    lacerta.uberspace.de
    leo.uberspace.de
    lepus.uberspace.de
    libra.uberspace.de
    lupus.uberspace.de
    lynx.uberspace.de
    menkar.uberspace.de
    mensa.uberspace.de
    monoceres.uberspace.de
    musca.uberspace.de
    norma.uberspace.de
    octans.uberspace.de
    pavo.uberspace.de
    perseus.uberspace.de
    phoenix.uberspace.de
    pictor.uberspace.de
    puppis.uberspace.de
    rigel.uberspace.de
    sagitta.uberspace.de
    sculptor.uberspace.de
    serpens.uberspace.de
    sirius.uberspace.de
    triangulum.uberspace.de
    tucana.uberspace.de
    ursa.uberspace.de
    vega.uberspace.de
    vela.uberspace.de
    volans.uberspace.de
    vulpecula.uberspace.de
    
  • 16th February 2023

    FRA3 Disk performance impacted

    We added several new disks to our storage cluster which results in quite some data being moved around. This results in temporarily impacted disk performance. We slowed down the data migration progress in order to lower the impact already. Usual disk performance is about to be restored soon.

    25th January 2023

    mysql.uberspace.de MySQL/MariaDB Update

    We are currently upgrading MySQL/MariaDB. The database as well as the mysql-binary is unavailable for a couple of minutes during the update.

  • all hosts updated 🎉

  • 4th January 2023

    musca.uberspace.de Performance Issues on musca.uberspace.de

    We are currently having performance issues on musca.uberspace.de because of a long running maintenance process.

  • The process has finished on 05.01.2023 2pm

  • 2nd January 2023

    eupheme.uberspace.de System crashed

    The host crashed and is in the process of rebooting right now.

  • Recovery is done, system is up and running.

  • 9th December 2022

    circinus.uberspace.de MariaDB repair running

    MariaDB repair is running, the system may face performance issues mainly due to database recovery.

  • successfully restarted mysqld

  • 30th November 2022

    vega.uberspace.de MariaDB crashed

    MariaDB repair is running, the system may face performance issues mainly due to database recovery.

  • successfully restarted mysqld

  • 2nd November 2022

    circinus.uberspace.de Reboot of host circinus.uberspace.de

    The system needs to be rebooted and will reappear in a few minutes.

  • reboot is done

  • 6th October 2022

    bus.uberspace.de System crashed

    The host crashed and is in the process of rebooting right now.

  • host was successfully rebooted at 11:55