Less than two hours later, the affected systems were isolated, our emergency plan had been activated and the forensic analysis was already under way. While the actual attack only lasted a short time, its consequences kept us busy for almost three weeks.

The most important good news was that our productive customer systems continued to run throughout the entire incident. After the forensic investigation was completed, there was no evidence that customer data, source code or production databases had been affected. Nevertheless, our company was in crisis mode for almost three weeks, during which technical, organisational and legal challenges had to be dealt with at the same time.

Today we are deliberately sharing this incident with the public. Not as a flawless success story, but as an honest review of an attack that showed us that even companies with extensive security expertise have vulnerabilities and can make mistakes.

What matters, therefore, is not being flawless, but being prepared for an emergency. Good preparation can limit the spread of an attack, reduce the damage and help a company remain operational even during a crisis. It is just as important to review the incident honestly afterwards and learn from the insights gained.

The first 120 minutes

While the attackers were still trying to put pressure on our employees by phone, our first immediate measures were already under way. All relevant employees were brought together in an online conference, the existing emergency plan was activated and responsibilities for forensics, technical recovery, legal clarification and communication were assigned.

At the same time, our technical teams began shutting down parts of the infrastructure in a controlled manner and consistently disconnecting affected systems from the network. The initial forensic analysis was launched while the next steps to contain the attack were coordinated in parallel.

In such a situation, there is hardly any time for lengthy debates on principles. Within a very short time, decisions have to be made about which systems to shut down, who documents the technical measures and who takes over communication with employees, customers and external bodies. At the same time, possible reporting and notification obligations have to be checked and cooperation with forensic and legal specialists organised.

Many of these questions seem self-evident in normal day-to-day business, until they have to be answered under considerable time pressure. The fact that the relevant roles and responsibilities had already been clarified did not prevent the attack, but it did ensure that the technical incident was not accompanied by organisational chaos.

Monday: Customer services are running, but much is blocked internally

The most important relief became apparent the following Monday: our productive customer systems were still running stably. The SaaS platforms were working and, based on what we knew at the time, there was no evidence that the production environment had been affected by the attack.

The fact that customer services remained available was no coincidence, but the result of a deliberately chosen security architecture. Our productive and administrative systems are strictly separated from each other. Internally, we call this approach our “submarine principle”, in which technical bulkheads are designed to prevent a security incident from spreading unhindered from one area to other parts of the infrastructure.

This separation proved particularly valuable during the attack. Nevertheless, the impact on our internal day-to-day work was considerable. Administrative systems were only available to a limited extent, various internal processes had to be organised on a provisional basis, and at the same time the technical analysis, legal clarifications and communication with customers and employees were running at full speed.

The incident clearly showed us that a cyber attack can put a considerable strain on a large part of a company even when the actual customer services continue to function. The availability of productive systems by no means implies that the rest of the business can continue undisturbed.

Prevention remains the goal. Resilience is what counts when prevention fails at some point. A well-thought-out security architecture must therefore be designed not only to prevent attacks, but also to limit their impact as effectively as possible in an emergency.

The hardest question: Which systems can we still trust?

An affected system can, under certain circumstances, be restored relatively quickly from an existing backup. The much more difficult question, however, is whether that system can then actually be used securely again.

During the recovery, we therefore had to check not only which data and systems had been affected, but also whether user accounts had been compromised, additional access points had been set up or permissions had been changed unnoticed. Only once such questions have been sufficiently clarified is it possible to decide responsibly which systems can be reconnected to the network and released for regular operation.

This is precisely an often underestimated challenge after a cyber attack. Successful data recovery does not mean that the original security of the IT infrastructure has been restored. That requires a thorough technical review and, where necessary, the complete rebuild of individual systems.

The entry point was an inconspicuous peripheral device

Our forensic analysis showed that a vulnerability in a peripheral network device served as the attackers’ initial entry point. From there, they managed to move further within the administrative environment and gain additional privileges.

This finding was particularly important for us because it made clear how large a company’s actual attack surface can be. When securing IT infrastructure, the focus is often on servers, workstations and firewalls. It is easy to overlook the fact that less conspicuous network devices also have their own software, user accounts and network access.

If such devices are not regularly updated, sufficiently segmented and consistently monitored, they can become the starting point of a much larger security incident. Our attack thus clearly showed that a security architecture must always take into account the supposedly insignificant components as well.

Data exfiltration in the administrative area, customer systems kept running

In the course of the attack, data was exfiltrated from our administrative area. Internal company and employee data was affected, which is why the incident was by no means without consequences, even though our productive customer systems remained available.

After the forensic investigation was completed, there was no evidence that the production environment or customer data had been compromised. This distinction was of great importance both for our internal assessment and for external communication.

From the outside, all that often remains after a cyber attack is the simplified statement that a company has been hacked. In reality, however, the technical and business impact can vary greatly. It makes a considerable difference whether administrative systems are affected or whether attackers gain access to productive customer environments.

We therefore decided to inform customers and employees about the incident early and transparently. We communicated which areas were affected according to the current state of knowledge, for which systems there was no evidence of compromise and which measures for containment and investigation had already been initiated.

This openness was not always comfortable, but it was necessary. Instead of further increasing uncertainty through incomplete information, we wanted to create as factual and comprehensible a basis as possible on which our customers and employees could assess the situation.

Why the recovery took almost three weeks

Our backups worked and the affected data was stored encrypted as well as logically and physically separated. The actual data recovery was therefore not the greatest challenge.

The far more time-consuming part was thoroughly cleaning up the affected infrastructure and ensuring that the systems could subsequently be operated in a trustworthy manner again. To this end, affected systems were rebuilt or restored from verified backups and additionally hardened before being put back into operation.

In parallel, we reviewed critical authentication and access structures, some parts of which had to be rebuilt. In addition, all passwords were reset as a precaution to reduce possible risks from compromised credentials.

This password change in particular turned into an organisational challenge that is often underestimated in normal day-to-day business. For a company to be able to reliably replace all relevant credentials after an attack, it must know which user accounts exist, who is responsible for them and which systems depend on each other.

Forgotten administrator accounts or service accounts in particular can make such a clean-up unnecessarily difficult. Without a complete overview of these access points, not only does the effort increase, but so does the risk that individual compromised permissions remain in place unnoticed.

The almost three-week recovery phase made it clear to us that working backups are an indispensable prerequisite, but on their own they do not guarantee a quick and secure return to normal operations.

Six decisions that limited the damage

In retrospect, it was above all six decisions and prepared measures that helped us limit the impact during the attack and gradually restore operations.

1. Consistent segmentation

The strict separation between administrative and productive systems limited the spread of the attack and prevented its effects from extending unhindered to our entire infrastructure. The forensic investigation found no evidence of compromise of the productive customer systems.

2. Separate and working backups

Our backups were logically and physically separated from the affected systems, encrypted and actually restorable. This gave us a reliable basis for rebuilding the infrastructure without being dependent on a possible ransom payment.

3. Clear responsibilities

Thanks to the responsibilities already defined, technical recovery, forensic investigations, data protection clarifications and communication could be organised in parallel. This preparation saved valuable time and reduced the risk of important decisions being left undone in the rush of the first few hours.

4. Proactive communication

Instead of waiting for all investigation results first, we informed our customers and employees early on about the known effects of the incident. It was important to us to communicate only verified findings and to clearly distinguish between confirmed information and questions that were still open.

5. Ability to act without paying a ransom

The intact backups and the consistent separation of the production environment enabled us to drive the recovery forward independently and without paying a ransom. The decisive factor was that we were able to rely on existing technical and organisational measures.

6. Support from external specialists

For the forensic analysis and the data protection assessment, we brought in external specialists and data protection advisors. Their experience helped us to work through the incident in a structured way and to make important decisions on a sound basis. Especially in an acute crisis, a company should not have to start developing basic procedures through “learning by doing”.

What we had underestimated and do differently today

The attack showed us not only which security measures had worked, but also revealed existing weaknesses. In particular, we had underestimated the importance of securing peripheral network devices and monitoring administrative areas.

We have drawn concrete consequences from these findings. Since the incident, we have been reviewing the segmentation of our infrastructure, existing permissions, patch levels and monitoring processes even more rigorously. We now also use AI-supported methods to further develop our security monitoring.

However, these changes are not a one-off project that can be fully completed at some point. An IT infrastructure changes constantly, new vulnerabilities emerge and attackers continuously refine their methods. Security concepts must therefore also be reviewed regularly and adapted to new requirements.

Above all, our incident confirmed to us that security vulnerabilities and human error can never be completely ruled out. This makes it all the more important to put in place, in advance, the technical and organisational prerequisites for containing an attack and dealing with its consequences.

These include a well-thought-out security architecture, clearly defined responsibilities, working protection mechanisms and the willingness to critically question existing processes. It is just as crucial to review an incident openly afterwards and to consistently incorporate the insights gained into the further security strategy.

Preparation makes the difference before an emergency occurs

A cyber attack does not automatically end as soon as a compromised server has been shut down or a backup has been successfully restored. The actual recovery is only complete when the affected systems are functioning in a trustworthy manner again, critical access points have been checked and cleaned up, and employees and customers have received reliable answers to their questions.

Our experience has shown that cybersecurity goes far beyond the purely technical protection of systems. It affects a company’s operational security as well as organisational responsibilities, transparent communication and the ability of management to make sound decisions even under considerable pressure.

For companies, one crucial question therefore arises: Would you be able today to isolate part of your IT infrastructure at short notice, restore it from secure backups and keep your business running at the same time, without first having to clarify who makes which decisions and which measures are required?

We had to answer this question under real conditions and learned how valuable good preparation is.

I hope every company finds the answer to this question before it needs it under real pressure.