I want to assure you that we are doing everything possible to get the environment up and running again. None of our server team
will sleep until that is the case. So far, we have learned that the issue was caused by the breakdown of a switch, which is the hardware in the middle of the network that connects all servers and storages together. It was not caused by an attack of some sort, neither by a human error. Unfortunately a hardware failure like this cannot possibly be prevented for 100%. The failing switch has caused problems in the software and the team is working hard at the moment to restore those. Obviously for all downtime your subscription will be extended accordingly although we appreciate that this will not, in many cases, fully compensate for the inconvenience. We still hope that the problems can be resolved shortly, though we cannot give you a valid estimate as the situation at hand is far some standard. Meanwhile, please accept our sincerest apologies.
In a normal case of hardware failure we have a contingency plan. All databases are backed up and mirrored on separate sites. This addresses the very occasional possibility of a hardware failure of a certain device. In such a (rare) instance we would always be online again within an hour or so. However, in this case we are dealing with an entirely unprecedented software issue, all data is available but the server software cannot access it fully. It is almost impossible to prepare for, or anticipate.
Our team has been working with various external expert teams in the field - so far without success. To give you a better understanding of what is going on, some extra information about the platform: our websites run on a cloud that is built from 7 vast servers. Those servers connect to 4 storages for all their data, the websites themselves, the search engines and the images. Storage 1 and 2 are currently not accessible, storage 3 and 4 are. However, storage 1 holds all databases, the backups of which are on storage 2. We need at least 1 of the 2 back online. Both storages appear to be running and all data is intact. However, when connecting the cloud to the storage, they cannot communicate with one another.
We are working with teams from the manufacturers. Although it seems a software issue, also hardware replacements (of network cards) are considered. There are still lots of options to proceed. I really regret that I have to say this, but we may be facing a situation that is going to take longer. Worst case the storages are taken to the manufacturers so they can get them running in their lab and get the data from it, so we can put it all up on new storages.
I would very much like to give you a good estimate of when all will be resolved, but I cannot with certainty. We aim for today but it may be longer. Again I really cannot apologise enough for the the inconvenience that this is undoubtedly likely to cause you.