All systems are operational

Stickied Incidents

18th September 2024

Problems sending to Google Mail accounts without SPF and/or DKIM

Google Mail (googlemail.com; gmail.com) no longer accepts mails when the sender domain does not have valid SPF and/or DKIM set up. To set up SPF and/or DKIM for your domain hosted at Uberspace.de, please refer to our manual.

Past Incidents

8th October 2024

FRA4 Network problems

We encounter network problems due to a failed switch at our site FRA4

  • the problem has been fixed

  • 25th September 2024

    Mails to Microsoft Mail accounts get rejected

    Mails to Microsoft accounts are rejected, we are not given a reason and our request to unblock our mailgates is not responded to. We change the mail gateways as far as we can, but unfortunately the block seems to take effect after a few hours on the gateway we changed.

    18th September 2024

    Problems sending to Google Mail accounts without SPF and/or DKIM

    Google Mail (googlemail.com; gmail.com) no longer accepts mails when the sender domain does not have valid SPF and/or DKIM set up. To set up SPF and/or DKIM for your domain hosted at Uberspace.de, please refer to our manual.

    27th August 2024

    no mail to any t-online hosts

    Currently, we can not deliver mail to any t-online servers.

  • We can connect to t-online's MX servers again.

  • While we have no detailed information at this time, we suspect an error with t-online's infrastructure, as reported elsewhere on the net over the last hour. We keep investigating.

    E.g. https://www.ifun.de/telekom-kunden-berichten-von-flaechendeckenden-e-mail-stoerungen-238410/

  • Partial problems with mail delivery from all *.uberspace.de hosts

    Since the end of last week we have been experiencing increasing problems with mail delivery to certain destinations from all uberspace.de hosts.

    The exact cause is still is not yet known, but we are investigating and working on workarounds.

  • Mail delivery was rerouted from the individual hosts to our mail gateway. Delivery problems should be resolved.

  • We're checking our logs on a daily basis and identify domains, which are affected. We work around these domains on the next business day. So if you get a bounce for a domain you haven't sent a mail within the last two weeks today please try again the next day. We'll have that domain on our list by then.

  • Since we still haven’t received any helpful response from the reputation service provider, we have now implemented a permanent workaround for all domains with mail servers behind IronPort that we are aware of.

  • After a brief recovery, a large proportion of our hosts are now blocked again. While we continue to wait for answers as to why, we have now placed an smtproute over working hosts for all blocked hosts.

  • Apparently, a major provider of e-mail security solutions had classified us as “poor” - contrary to its publicly available information and initial responses to our inquiries - and blocked us on its IronPort firewall.

    Unfortunately, despite several attempts, we still have no information about which patterns might have been responsible, but at least the block seems to have been lifted. Nevertheless, we will continue to monitor this closely.

  • 24th July 2024

    dashboard.uberspace.de dashboard.uberspace.de is offline

    We're looking into an outage of dashboard.uberspace.de

  • we identified an error caused by a software update of the MariaDB database server and implemented a fix.

  • 16th July 2024

    FRA3 Packet loss at FRA3

    We are observing some packet loss at FRA3

  • Everything works again.

  • 26th June 2024

    ananke.uberspace.de MariaDB on ananke impacted

    While ananke itself is working fine, its MariaDB instance seems to hang. We're checking things out.

  • Recovery is done, system is up and running. MariaDB works fine again. As it didn't even react to a SIGKILL and completely blocked even a shutdown, we had to hard-reset ananke. Afterwards, MariaDB was running fine; tables are undergoing a consistency check. There's no indication of circumstances that could have led to this situation, so we suspect a rare bug in MariaDB itself. No other hosts were affected, and the underlying storage system is completely healthy with no errors.

  • The system needs to be rebooted and will reappear in a few minutes.

  • 25th June 2024

    FRA3 Network instabilities at FRA3

    A planned routing failover had unexpected side effects. We're already in debug mode, stable operation should be back in a moment.

  • Shifting traffic from our soon-to-be-remove router to our new replacement router is finished and network connectivity has been fully reestablished.

  • 12th June 2024

    FRA3 Network problems at FRA3

    We are encountering network problems on our cluster at FRA3, some services might be affected, we are investigating

  • One of our routers unexpectedly rebooted, causing temporary problems for a few minutes.

  • 4th June 2024

    pavo.uberspace.de Reboot

    The system needs to be rebooted and will reappear in a few minutes.

  • back up since 19:38

  • 27th May 2024

    FRA3 Network Outage

    FRA3 is down because of an issue with one of our switches. We are working on it.

  • The incident has been resolved.

  • 26th May 2024

    schuster.uberspace.de Reboot schuster.uberspace.de

    The system needs to be rebooted and will reappear in a few minutes.

  • Recovery is done, system is up and running.

  • 22nd May 2024

    FRA3 Several Hosts down

    One of our VM nodes in FRA3 is down, this affects the following U7 hosts:

    • farbauti.uberspace.de
    • europa.uberspace.de
    • eurydome.uberspace.de
    • bernardi.uberspace.de

    we are investigating the problem

  • All U7 hosts are up and running again

  • 21st May 2024

    FRA3 network connectivity in FRA3 impacted

    A core router is unexpectedly rebooting. Connectivity should be restored soon.

  • Core router reboot finished. Connectivity is restored.

  • 14th May 2024

    FRA4 network outage

    We noticed a network outage at our FRA4 datacentre. Based on information of our provider, this is reportedly caused by a power outage at a major data carrier.

  • Our datacenter has immediately switched to backup connections, but it took a few minutes until the routing tables have been adapted accordingly. During this time, we observed partial connectivity in varying ways (some peerings worked faster than others). The majority of connections was back on 12:06 GMT+2, three minutes after the incident. A few peerings needed up to ten more minutes to reconnect cleanly. Between 12:21 GMT+2 and 12:23 GMT+2 there was another short outage due to works needed to stabilize the situation, which resolved quickly.

  • 12th May 2024

    olbers.uberspace.de Reboot with more RAM

    We added some RAM and the host is in the process of rebooting right now.

  • Recovery is done, system is up and running.

  • 8th May 2024

    IPv6 network problems

    We are currently encountering problems with the IPv6 network, we are investigating.

  • One of our core routers seems to be broken and causes only partially working IPv6 connections. We have moved connections to the replacement router and scheduled the broken one for repair.

  • 13th February 2024

    epimetheus.uberspace.de System crashed

    The host crashed and is in the process of rebooting right now.

  • Recovery is done, system is up and running.

  • 29th January 2024

    kohoutek.uberspace.de Reboot with more RAM

    We added some RAM and the host is in the process of rebooting right now.

  • Recovery is done, system is up and running.

  • 26th January 2024

    ananke.uberspace.de Reboot of ananke due to MariaDB failure

    After MariaDB being unresponsive to signals, we decided to give the whole system a reboot. We will check for integrity afterwards.

  • Recovery is done, system is up and running. MariaDB data does not show any losses. All table repairs have finished without errors.

  • 21st January 2024

    System crashed

    One of our nodes crashed, we are rebooting these servers:

    • schuster.uberspace.de
    • scotti.uberspace.de
    • skiff.uberspace.de
    • whipple.uberspace.de
    • epimetheus.uberspace.de
  • All services are fully back up and running.

  • All hosts are up again. Some services might need a couple of minutes to be fully usable again.

  • 16th January 2024

    menkent.uberspace.de Reboot with more RAM

    We added some RAM and the host is in the process of rebooting right now.

  • Recovery is done, system is up and running.

  • 10th January 2024

    mail server unavailable on wolf, yeung, acamar, achernar and aldebaran

    we're investigating an SMTP relay outage on wolf, yeung, acamar, achernar and aldebaran

  • the issue has been fixed.

  • 5th January 2024

    All Uberspace 7 SSH Terrapin mitigations

    We have enabled mitigations for the Terrapin SSH vulnerability. If you are experiencing trouble connecting to Uberspace hosts via SSH, please make sure your SSH client is up to date and supports more secure ciphers.

    2nd January 2024

    kaus.uberspace.de Reboot with more RAM

    We added some RAM and the host is in the process of rebooting right now.

  • Recovery is done, system is up and running.

  • 1st January 2024

    belinda.uberspace.de Reboot with more RAM

    We added some RAM and the host is in the process of rebooting right now.

  • Recovery is done, system is up and running.

  • 31st December 2023

    schedar.uberspace.de Reboot with more RAM

    We added some RAM and the host is in the process of rebooting right now.

    30th December 2023

    sabic.uberspace.de Reboot with more RAM

    We added some RAM and the host is in the process of rebooting right now.

    6th December 2023

    ferdinand.uberspace.de Zeitweilige Performance Einschränkungen bei ferdinand.uberspace.de

    Aktuell kann es auf dem Host ferdinand zu einer Einschränkung der Performance kommen. Dort läuft aktuell eine Kampagne von postmitherz.org über die wir uns sehr freuen, die aber teilweise zu hohen Lastspitzen führt. Wir haben für den Host daher bereits einen eigenen Server Node freigemacht um für den Zeitraum (bis 18.12.) der Kampagne möglichst viel Spielraum zu geben.

    Solltest du dennoch ein Problem mit deiner Seite feststellen, schreib bitte unserem Support und wir suchen gemeinsam nach einer Lösung.

  • Die Aktion ist beendet und es sind keine weiteren Einschränkungen zu erwarten.

  • 30th November 2023

    hergenrother.uberspace.de System crashed

    The host crashed due to the storage problems and is in the process of rebooting right now.

  • The host rebooted successfully

  • FRA3 We have encountered a problem with our storage, some hosts encounter performance problems

    The issue has already been resolved, the load on the hosts is going down

    27th November 2023

    FRA3 Partial downtime/failures FRA3

    we observe partial failures on fr3

  • Everything is up and running, and the last host is also showing normal performance again.

    The root cause was likely an unintentionally installed package on the Ceph nodes during an OS update, which caused the IPs in Ceph to become disoriented.

  • All hosts are reachable again, and their services are running. However, we still observe high load on some of them. We are addressing this issue, but overall, the situation has already significantly eased.

  • After a network/configuration problem of our ceph-cluster at FRA3, we have the situation under control again, but still have some follow-up errors to fix. some hosts and services are still down. We are working on it

  • Hardware down

    We have received down notifications for the hosts neujmin howell callirrhoe eirene bernardi epimetheus and chaldene from monitoring, suspect underlying hardware as the cause and are currently investigating the incident.

    We are in process of rebooting the crashed hosts right now.

    17th November 2023

    System crashed on europa, eurydome, farbauti, euporie, dione, halimede, caliban

    One node with the following Hosts crashed, we are in the process of restoring them right now:

    • europa
    • eurydome
    • farbauti
    • euporie
    • dione
    • halimed
    • caliban
  • All hosts and and services are up and smooth running again.

  • All VMs have been booted on other nodes and are accessible again.

  • 16th November 2023

    FRA3 IPv6 connectivity impacted

    We have received reports that our IPv6 network 2a00:d0c0:200::/48 (FRA3) is unreachable from some networks. We are investigating the issue. IPv4 connectivity should be unaffected.

  • Our upstream provider changed the routing, it looks like this fixed the issue. If you still experience IPv6 timeouts, please contact us at hallo@uberspace.de.