Many organisations have backups and have made arrangements for recovering their IT environment. This provides an important foundation for business continuity. Yet one question often remains unanswered: will recovery work when a serious disruption actually occurs?
A backup may have completed successfully while restoring the data takes unexpectedly long. A server may be running again while the associated application remains inaccessible. And a recovery plan may have been carefully prepared, only to turn out to depend on an employee who is unavailable at the crucial moment. Only when recovery is actually carried out does it become clear whether the arrangements and technical provisions are sufficient.
A successful backup does not mean successful recovery
Backup software may report every day that all jobs have completed successfully. Primarily, this means that data has been stored according to the configured procedure. It does not yet tell you whether that data can be restored within the required time, whether its contents are usable or whether the associated applications will then function correctly.
Consider, for example, an ERP system whose database is backed up daily. Restoring that database may succeed technically, while the application subsequently fails to start because of a missing configuration or an incompatible software version. Integrations with other systems may also be missing, leaving employees still unable to process orders. From a technical perspective, the data has been recovered, but from a business perspective, recovery has not yet succeeded.
A recovery test must therefore go beyond checking whether files can be read back. Its purpose is to establish whether a business process is usable again, within an acceptable time and without unacceptable data loss. The recovery test thus provides a practical assessment of the RPO and RTO objectives established earlier.
What should a recovery test demonstrate?
The scope of a recovery test depends on the systems and processes that matter to your organisation. For a relatively simple application, restoring data in a separate environment and checking that it works may be sufficient. A complex business process may involve multiple servers, databases, network services, user accounts and external services.
A good test reveals which components are needed and the sequence in which they must be recovered. It examines not only the technical work, but also the availability of documentation, the access rights required and the arrangements with suppliers. These organisational dependencies in particular can determine the eventual recovery time during an incident.
The trustworthiness of the recovered environment also deserves attention. Following a security incident, it cannot be assumed that all systems are safe to bring back into use immediately. A recovery procedure must therefore allow for checks on data integrity, the state of systems and the risks of reinfection. A rapid return to production has little value if it allows the same disruption to occur again.
What a recovery test reveals in a production environment
A manufacturing company makes daily backups of its ERP environment. The backup reports show nothing unusual, and the organisation assumes that its administrative system can be available within a few hours of a disruption. To test that expectation, it decides to carry out a recovery test in an isolated environment.
The database can be restored without any problems, but when the ERP application is started, a specific configuration turns out to be missing. An integration with the warehouse system then proves to depend on an external supplier. That supplier must first restore an additional setting before the complete order processing workflow can be tested.
As a result, the technical recovery work takes considerably longer than had been assumed. Certain actions also turn out not to be documented in the existing recovery plan. The test does not disrupt day-to-day production, but it does reveal that the organisation would probably face longer downtime than expected during a real incident.
Based on the findings, the missing configurations are backed up, supplier arrangements are tightened and the recovery procedure is updated. A subsequent test can then establish whether these measures deliver the intended result.
Key insight
The value of the first test therefore lies not only in what went well, but above all in the shortcomings that were identified in time.
Four components of a useful recovery test
A recovery test delivers the most value when it is clear in advance what is being examined, how the result will be assessed and what will happen to the findings. Four components provide the foundation.
Test objective and scenario:
Which business process needs to be recovered, what disruption is being simulated and which systems fall within the scope of the test?Execution and dependencies:
Can data, applications and connections be recovered in the correct sequence, and are the necessary people and suppliers available?Functional and safe return to use:
Can employees carry out the necessary work, and has the recovered environment been established to be trustworthy?Evaluation and improvement:
What recovery time and data loss were actually established, what problems occurred and how will they be resolved?
Not every test needs to cover the entire IT environment. It is often sensible to start with the most important business processes and then investigate different scenarios. A distinction can be made between restoring individual data, recovering a complete application and resuming a business process with all its associated dependencies.
The results must, however, be documented in concrete terms. Merely concluding that a test has succeeded provides insufficient insight into actual recovery capabilities. It is precisely the measured recovery time, how up to date the restored data is and any manual interventions that reveal whether the organisation can meet its objectives.
Testing without disrupting day-to-day business operations
One understandable reason for postponing recovery tests is the concern that they could affect normal work. Particularly for manufacturing companies or organisations with business-critical applications, taking systems out of service unnecessarily is undesirable. Yet a recovery test does not have to mean actually shutting down the production environment.
Much of the recovery work can be carried out in a separate test environment. Backups can be restored, applications started and integrations checked there without employees having to interrupt their day-to-day work. However, the test environment must be prevented from inadvertently connecting to production systems or carrying out real transactions.
An isolated test also has limitations. Not all the dependencies and circumstances of a real incident can be fully simulated. It is therefore important to establish in advance what a test does and does not demonstrate. Where necessary, technical recovery tests can be supplemented with exercises that also assess decision-making, communication and supplier arrangements.
Recovery capability changes with your IT environment
A recovery plan that was successfully tested last year may no longer be complete today. New applications, changed network connections, software updates and different suppliers can affect how systems need to be recovered. Changes in business operations may also mean that earlier recovery objectives no longer match actual needs.
Recovery testing should therefore not be confined to implementing a new backup provision or preparing a recovery plan. It must form part of routine IT management, with significant changes potentially prompting a reassessment of existing recovery procedures. How often and how extensively testing takes place depends on the business risk and the consequences of a possible disruption.
Malenco approaches recovery tests with these connections in mind. We look not only at whether data is available, but also at whether applications, infrastructure and dependencies enable business operations to resume. We use the findings to make targeted improvements to recovery procedures, technical provisions and organisational arrangements.
Do you know what your recovery plan is worth in practice?
Until recovery has been tested, important assumptions remain about the time required, the availability of data and the functioning of dependent systems. A recovery test makes these assumptions testable and helps prevent shortcomings from coming to light only during a serious disruption.
Testing is therefore more than a one-off technical check: it is a way to assess your organisation's actual recovery capability and improve it step by step.
Would you like to know whether your recovery plan can be carried out in practice? Malenco helps you assess, carry out and improve recovery tests, so that it becomes clear what your organisation actually needs to resume work following an IT disruption.
