Outage in response to Brute Force Attack

17-01-2019 Update Disruption 16:00

At this time we can report that the announcements page on our website has also been restored. There too we will now inform customers of this outage. Furthermore, a message will be sent by email to all parties involved. Our own backend is now also up and running again so we can finally do this.

https://www.mijnhostingpartner.nl/client/announcements.php

Regarding the affected servers that are still down and/or where an old image has been restored, we are still working with various parties to crack the encryption code. This is a process that takes a very long time and we cannot say when it will be finished. It is an ongoing process that is picked up every day by specialists in this area.

Again we ask people to check if they do not have a (recent) local backup of their entire website. Then your website (with old data) is in any case online and we can later, when the servers are released, migrate the databases again. By the way, we found out that a number of customers had set up a backup on their space without knowing it. I would like to ask everyone to check their root (via filemanager in the controlpanel) to see if there is a backup there after all.

If it is not there you have the following options, besides waiting for the servers to come back online:

- A redirect to another page (Facebook?)

- Post a temporary page (Under construction?)

- Build/place a new website

If you want us to create an extra database set for you so you can start all over again with a clean database, please let us know. The old database will of course remain intact and will be available again once the affected servers are released. Please let us know what we can do for you.

---------------------------------------

16-01-2019 Update Failure 14:45

At this moment we are working hard to restore customers that use (were using) the MySQL 3 and MySQL 4 server and make the website/data available. These are manual actions that are being taken up one by one. Have you reported your website via chat and/or the ticket system then you are on the list to be handled. You will automatically be notified when this has happened and / or from your side actions are performed.

Regarding the MySQL 5, we still have not succeeded to make the old image operational. We are still working on this. In the meantime, we are working on a different way to secure the data of this old image and to make it available, just like we did with MySQL 3 and 4.

The MySQL 7 is not yet fully up and running, but the end is in sight here. For the affected clients on this server, progress can be seen, but there is still some work to be done by us.

As for the MSSQL 8 server, several actions are running simultaneously. In addition to restoring, attempts are being made to crack the encryption. Several companies are involved in this. At this stage it is not possible to say when this will be finished and when a decryption key will be available. If the key is available then other (MySQL) servers from which an old backup has been restored will also become operational again.

The best and fastest way is to restore a backup of the database that you have locally. If you do not have one, you have the following options at this stage, in addition to waiting for the servers in question to come back online:

- A redirect to another page (Facebook?)

- Place a temporary page (Under construction?)

- Build/place a new website

If you want us to create an extra database set for you so you can start all over again with a clean database, please let us know. The old database will of course remain intact and will be available again once the affected servers are released. Any new data can be migrated with the old. Please let us know what we can do for you via a chat message or a ticket.

Because this post is getting very long now I decided to make a new post. We have also been busy making our website fully operational again. We will restore the announcements page and provide information there as well. Also, one of these days a mail will go out with information.

--------------------------------------

15-01-2019 Update Malfunction 11:00

Compared to yesterday, there is no news regarding the recovery of the affected servers. The process is still ongoing and we are still in the process of restoring the servers. In the meantime all our time is spent restoring and making backups work. We are taking care of customers one by one and that is a lot of work. Besides this work we try to answer as many questions as possible via ticket, mail and chat. We understand the frustration when it takes a long time for a response to a question. However, to be able to help people further we need to focus and actually finish something. This means that questions are left unanswered. There is no other way. It is currently a viscous circle that we are trying to break. I hope to come up with something positive today.

---------------------------------------

14-01-2019 Update Disruption 13:00

At the moment we are still restoring / decrypting the affected SQL servers and backup servers. Aggrieved customers are asking how long it will take, but unfortunately I cannot give a clear answer. Of course I would like to, but people should realize that the process is ongoing and we just have to sit this one out. We feel incredibly sorry for the affected customers (all the way on the MSSQL8 server). We too are victims in this .

If we can help the customers in any other way we would like to hear from you. We can temporarily forwarder the page (Facebook?), place a temporary (under construction) page, but we are certainly willing to make an extra database addon active in the space so that they can set it up again. Please create a ticket for this and/or come to the chat with this request.

We are still working on MySQL 5 server. Unfortunately, the old image will not start up yet. MySQL 7 will be back on the air today. There will be an update about this but it will not take long now.

--------------------------------------

13-01-2019 Update Malfunction 15:15

Regarding the MSSQL 8 server I can inform victims that today we are working on restoring the server and decrypting the ransomware. By ruling things out we are getting a little further in the right direction. It is just a very time-consuming process. That is why we have called in extra help to speed things up. We are definitely going to get the server live again. Data is not lost. We can't give a time frame yet, unfortunately.

As for the MySQL 3 server, we were able to restore all the data. That's good news for customers who use this server. However, we are not able to get the server live at this time. For customers who are using this server I would like to ask you to open a chat with the domain name, control panel login name and database name that is used for the website. We will then make the affected website operational again. The messages / websites will be handled in order.

Regarding the MySQL 5, we have not yet succeeded to get the old image working. We are still working on it. As soon as we know more we will let you know.

--------------------------------------

12-01-2019 Update Disruption 17:00

At this moment all MySQLWebadmin pages are accessible again for the servers that are operational.

https://www.mysqlwebadmin.nl/

Regarding the MSSQL 8 server, I can inform victims that two outside companies that specialize in decrypting this type of ransomware have been brought in to expedite the process. Today we have again made steps in the right direction but it remains unclear at this time what stage of the overall process we are in. As soon as more information is available we will let you know.

--------------------------------------

12-01-2019 Update Malfunction 13:00

In order to create some clarity, I have indicated below which servers are active and what their status is:

MySQL 1: Operational, fully restored

MySQL 2: Operational, completely restored

MySQL 3: Offline, but at the end of the afternoon probably something positive to report

MySQL 4: Operational, fully restored

MySQL 5: Offline, will become operational any moment with old backup

MySQL 6: Operational, fully restored

MySQL 7: Offline, no view on recovery yet

MySQL 8: Operational, but with old backup

MySQL 9: Operational, fully restored

MySQL 10: Operational, fully restored

MySQL 11: Operational, but with old backup

MySQL 12: Operational, fully restored

MySQL 13: Operational, fully restored

MySQL 14: Operational, completely restored

MySQL 15: Operational, fully restored

MySQL 16: Operational, fully restored

MSSQL 1: Operational, fully restored

MSSQL 2: Operational, fully restored

MSSQL 3: Operational, fully restored

MSSQL 4: Operational, fully restored

MSSQL 8: Offline, no view on recovery yet

MSSQL2014: Operational, fully restored

MSSQL2014-1: Operational, fully restored

How do you know if you are indeed using one of these servers? See this article: https://www.mijnhostingpartner.nl/client/index.phpÃ

--------------------------------------

11-01-2019 Update Failure 18:30

We have recovered an old backup of the MySQL 8 server. This will bring some of the customers who were using this back online. This is probably old data but at least it is something. Once the backup server on MYSQL 8 is fully restored we will overwrite this backup.

It could be that there is still a connection problem causing the page not to be displayed but it could also be that it is a backup from before you were a customer. In that case it is best to send us a chat message. We will then check whether you are in the backup file.

As for the other MySQL servers and remaining MSSQL server and their progress, unfortunately there is nothing new to report.

We are also looking at these servers to see if we can restore an old backup.

--------------------------------------

11-01-2019 Update Disruption 15:00

We have currently recovered an old backup of the MySQL 11 server. This will bring some of the customers who were using it back online. This is probably old data but at least it is something. Once the backup server on MYSQL 11 is fully restored we will overwrite this backup.

It could be that there is still a connection problem causing the page not to be displayed but it could also be that it is a backup from before you were a customer. In that case it is best to send us a chat message. We will then check whether you are in the backup file.

As for the other MySQL servers and remaining MSSQL server and their progress, unfortunately there is nothing new to report.

We are also looking at these servers to see if we can restore an old backup.

--------------------------------------

11-01-2019 Update Malfunction 10:30

Dear affected people, I am asked to give a status of the current sitiuation. At the moment there is little progress in decrypting and restoring the affected servers. We are making small steps in the background, but as stated earlier, it is very slow.

In the meantime more and more websites are up and running again and we are trying to help our customers with restoring old backups so they can continue their business. There is also an exchange issue that needs to be addressed.

We have quite a backlog of tickets, but we will try to answer/resolve them as much as possible today.

--------------------------------------

10-01-2019 Update Malfunction 17:30

Finally after a few days there is some good news to report! We can report that the MYSQL 4 server is fully operational. We have been able to restore all data. Now we have done some test actions with customers and we conclude that probably some actions are still needed in the WP-ADMIN. This includes updating plugins. If you are using the MySQL 4 server and your website is not online yet, we ask you to start a chat via facebook and/or join the chat via our website.

How do you know if you are indeed using one of these servers? See this article: https://www.mijnhostingpartner.nl/client/index.phpÃ

Furthermore I received a message that for some customers it is still unclear what exactly is going on. So here is a short explanation: Last Sunday, January 6, we were attacked by means of a brute force / DDOS attack in which malicious people have seen an opportunity to inject malware / ransomware into the core of our network. This ultimately paralyzed several MySQL and MSSQL servers and a number of exchange databases. To make matters worse, they also had a chance to inject our backup servers. A quick recovery of our services was/is therefore enormously hampered. Since then we have been working non stop to get the situation under control and to restore the servers. We are progressing very slowly, but are confident that we can get everything up and running again. For more information you can read the past status updates. If there are still questions I will try to answer them as best as I can. It might be best to send me a chat message.

As for the other servers, we are still in the process of repairing them and cannot give much more information at this stage. As soon as there is an update I will let you know.

--------------------------------------

10-01-2019 Update Malfunction 10:00

Dear customers, affected people, we have not heard from you for a while and understand that many people think that we are not working in the background to solve the problems. This is certainly not true. Our entire team has been working almost nonstop since the attack. Many customers may feel let down because we are not responding or are responding late but I promise everyone that all questions, tickets and mail will be responded to. One can probably imagine that there are relatively few of us compared to the number of customers we serve. We try to help everyone as much as possible in the shortest possible time. All I can say is: hang in there and keep having faith that things will work out. We certainly have that! Thanks again for your understanding and we really appreciate all the supportive responses we receive. It gives us energy to continue! Thank you for that!

Now an update: If you are using the following MySQL servers: 3,4 (partially),5, 8 and 11 and MSSQL8 your website will not be visible. How do you know if you are indeed using one of these servers? See this article: https://www.mijnhostingpartner.nl/client/index.phpÃ

All above mentioned servers AND backupservers are infected with malware. This means that we cannot access them. So restoring a backup is not possible at this stage. However, we are working hard in the background to restore these servers. And this is succeeding, only it is going agonizingly slow. We are making small steps but we really feel that we are going in the right direction. When asked if we can give a time indication, I must unfortunately say that this is very difficult because we cannot properly estimate how far we are in the overall process. However, there is an end to it!

Regarding Exchange we are in the process of migrating our customers. Those who are not yet (properly) working we will try to help as best we can. Please be patient, these customers will be helped as well!

If there are still questions you can always send me a chat message. It will take a while, but I answer them all!

--------------------------------------

09-01-2019 Update Disruption 16:15

At this time, the scans of the affected servers are still in progress. However, it is proceeding agonizingly slowly and it is becoming increasingly clear that it may take a very long time to recover. This is of course terrible and we would like to be able to report other news at this stage. Many people have asked in a private message for another solution but this is only possible if you still have a local version or backup of the website in question. We will then set up a new environment and make that backup work in that new space. This is very laborious but it will probably help you.

We continue to work very hard to solve the problems in the background. We are really trying and making progress but it is very, very slow. We understand that people can curse us by now but also hope that people realize that this is an impotence situation for us. We are trying to stay positive and help customers one by one. I hope you guys can stay positive and constructive as well.

-------------------------------------

09-01-2019 Update outage 09:15

Unfortunately, we have to note that there are still certain exchange accounts not working despite the scans completing successfully. Anyone whose account is not working I would like to receive a private message from. We will then recreate the mailbox so at least the mail will work again. At a later stage we will restore the data.

Regarding the MSSQL and MySQL servers that are still down, it is unfortunately not yet possible to indicate when they will start working. All scans have been performed and each time inconsistencies remain so that we cannot proceed. We remain positive and continue to work to get it working.

For anyone who uses these servers and has a local backup I ask you to create a ticket through our site and / or report it in the online chat. The quickest way to get the website working again is to recreate it and restore the backup.

I am sorry to report this at this stage. We are doing everything we can since the attack to get everything up and running again, but unfortunately the malware has been injected so deep into our system that it is taking an incredibly long time to fix. I am of course willing to answer any questions you might have via messenger.

-------------------------------------

09-01-2019 Update failure 07:30

At this moment the exchange service is working again. However, it may take some time before the mail is loaded.

An update on the MSSQL 8 server will follow within the hour.

--------------------------------------

Update 21:45

Exchange is currently at 93%. If this continues then the Exchange service will be working again before 12 hours.

The following MySQL servers are also active: MySQL 9, MySQL 10 and MySQL 12.

The MSSQL 6 server is also active.

---------------------------------------

08-01-2019 Update failure 19:15

The online chat has been placed back on the website. Any questions can and will be answered by our online helpdesk tomorrow from 09:00.

Exchange is currently almost at 90%.

MySQL 13 is also active again.

---------------------------------------

08-01-2019 Update malfunction 17:15

At this moment the following MS SQL servers are active again: SQL1, SQL2 and SQL2014MHP. A scan on SQL8 is still running at the moment and as soon as it is finished it will also be back up and running.

In cooperation with Microsoft we have installed a tool that shows the status of the last check of Exchange and at this moment it is 83%. So we are really approaching the end here.

---------------------------------------

08-01-2019 Update outage 14:45

Also MySQL 4 server is online and also our online chat through our website will be available again. Exchange is unfortunately still in the process of final verification. This is frustrating also for us because we have no progress indicator at this stage and therefore cannot estimate how long this will take approximately. MySQL 5 and 11 are now in the completion phase.

For your information: We are getting questions as to what is known at this time about how this was able to occur. the answer to this is that last Sunday we suffered an incredibly severe attack where malicious people had the opportunity to break into the core of our network and servers. There they did a lot of damage. How they did this is still under investigation. The priority now is to bring all our services back online. Because we don't know what is 'infected', we have to check, scan, verify and secure everything before we can bring it back online. We will provide more information about this at a later stage. There are many processes going on at the moment and one by one the servers are coming back up. We see light at the end of the tunnel but we still have some way to go. The last mile is, also in this case, the hardest...

Thanks again for your understanding and patience. It is an incredibly frustrating situation for us as well.

---------------------------------------

08-01-2019 Update Malfunction 10:30

Meanwhile, the MySQL 1 and MySQL 6 servers are up again. We are now working on the MySQL 5 and MySQL 11. Exchange is now connected to the database again and that is a good sign. He is still working on the last checks but then also this should work again.

--------------------------------------

08-01-2019 Update Malfunction 08:00

In the meantime the MySQL9 is also operational again and the webmail should also work properly again. Exchange is working on my last checks now.

--------------------------------------

08-01-2019 Update Failure 06:00

The scans are pretty much done and Exchange should also be working again soon. The focus now is first on the MS SQL and MySQL servers and Exchange. It is expected that these will all be fully operational again in the course of the morning. In the meantime, several web servers have gone live again and most websites will function again.

--------------------------------------

07-01-2019 Update Disruption 22:00

At this time, MySQL6 is again approachable. The other servers are all being scanned and checked for inconsistencies at this time. This will continue throughout the night. When a scan is completed a web server will be put live again. Because it is not clear when which server will be ready it is impossible for us to say exactly when which website will be active again. Regarding the exchange boxes a check is still being performed on them. Therefore it is still not possible to send or receive mail with an Exchange account. The scan is currently running at 70% and it will take several hours before it is finished. After this the mailboxes will be available again. As soon as there is a new update we will report this. For now we will just have to sit out the scans.

----------------------------------------

07-01-2019 Update Malfunction 17:30

Herewith the announcement that the control panel is active again. Any changes/adjustments can now be made there again.

---------------------------------------

07-01-2019 Update Failure 15:50

At this moment we are working hard to get the controlpanel working again. When this is done we will try to get the online chat and our knowledge base up and running again. Together with the controlpanel we will also get the websites up and running again.

----------------------------------------

07-01-2019 Update Malfunction 12:40

As it turned out there were still some errors in the exchange mail. This is now being fixed. The VPSes are also largely back on the air. After that we will start with the web servers.

A note to all impatient people: We understand your frustration and irritation. You will undoubtedly understand that we are not waiting for this either and that we will do everything in our power to get the services up and running again as soon as possible. Posting negative comments here really doesn't contribute to this. This is not productive and feels like a stab to the people who have been working on this non-stop since yesterday 17:30. It is an outage / attack that we have not experienced before and we are doing everything possible to get the services working again as soon as possible. We kindly ask for your understanding and patience. Thank you very much in advance.

-------------------------------------

07-01-2019 Update Malfunction 09:30

Fortunately, more and more services are coming back up. The mail traffic is working again, the managed wordpress environments are working again and also our own page is (partially) visible and working again. We hope to have the rest of our services up and running again soon. Just hang in there! Thanks for your understanding!

--------------------------------------

07-01-2019 Update Malfunction 06:30

Unfortunately, we have to announce that it will take longer for our services to be back online. Several services are already back up and running but it is still unclear when everything will be up and running again. We are doing our best to solve the problems but it is more complicated than first thought.

--------------------------------------

06-01-2019 Update outage 23:30

We have now found out what is causing the problem and we are working hard to fix it. Our services will slowly come back online over the next few hours but our technicians indicate that it may take all night. We apologize for the inconvenience. We will try to keep the downtime as short as possible.

06-01-2019 17:30 - There is an outage going on at this time. We apologize for the inconvenience. Our technicians are working hard to figure out what is going on. As soon as there is more information we will share it here.