Virtualisation platforms tend not to attract much attention while they’re working. Applications run, virtual machines start when asked and dashboards display reassuring quantities of green. The platform becomes part of the scenery: important, certainly, but no more likely to feature in a board discussion than the building’s electrical distribution system.
That changes abruptly when it stops working.
By then, virtualisation may be supporting customer-facing services, research systems, internal applications, databases, development environments and various other workloads whose owners all consider them uniquely important. What appeared to be an infrastructure problem becomes an operational problem very quickly.
The choice of virtualisation platform is therefore not only a technical decision. It’s also a decision about business continuity.
Start With the Service, Not the Virtual Machine
Infrastructure teams naturally think in terms of hosts, clusters, virtual machines, storage pools and networks. The business is usually less interested in these things.
A business user does not particularly care whether an application runs on virtual machine vm-app-037, which lives on host node-04 and stores its disks on a replicated storage pool. They care that the application they need is available.
This distinction matters when considering resilience.
A highly available virtual machine does not necessarily create a highly available business service. The service may also depend on:
- A database running elsewhere
- DNS
- Authentication
- Network connectivity
- Shared storage
- A licence server
- An external API
- A particularly old system that everybody is afraid to restart
The virtual machine could recover perfectly while the service remains unusable.
Before deciding how resilient a virtualisation platform needs to be, identify the business services it supports. For each service, establish its dependencies, importance and acceptable interruption. This produces a much more useful starting point than declaring that every virtual machine is critical. If everything is critical, the word “critical” no longer has any meaning.
Understand the Effect of Platform Failure
Virtualisation consolidates workloads onto fewer physical systems. This is one of its principal advantages: better hardware utilisation, easier management and greater flexibility. It also concentrates risk.
A failed physical server may affect several virtual machines. A storage problem may affect an entire cluster. A network configuration error can have unusually democratic consequences, disrupting workloads without favour.
This doesn’t make virtualisation inherently fragile. A well-designed virtual environment can be considerably more resilient than a collection of independent physical servers. Clustering, replication, high availability and live migration all provide useful ways to maintain or restore services.
However, those capabilities need to be designed around credible failure scenarios.
Consider what would happen if you lost:
- A physical host
- A storage system
- A network path
- A complete rack or location
- Administrative access
- The management interface
- Several hosts at once
- The person who understands why the cluster was configured that way
The last scenario is not normally included in product datasheets, but it occurs rather more often than catastrophic data-centre fires.
High Availability Isn’t a Complete Continuity Plan
High availability is valuable. If a host fails, workloads may be restarted automatically on another node. With suitable architecture, maintenance can also be performed without unnecessarily interrupting services.
But high availability doesn’t protect against every kind of failure.
If a file is deleted accidentally, high availability may preserve the application’s ability to report that the file is missing. If data’s corrupted, replication may distribute that corruption efficiently. If an attacker gains sufficient administrative access, several highly available systems can be damaged in quick succession.
High availability, replication and backups address different problems:
- High availability helps services continue or restart following the failure of infrastructure components.
- Replication maintains additional copies of current data or workloads.
- Backups provide recoverable historical copies following deletion, corruption, compromise or other unwanted change.
None replaces the others.
The National Cyber Security Centre recommends using virtualisation capabilities such as clustering, replication and snapshots to support rapid recovery, alongside backups held in a known good state. That is a useful combination, provided the word “known” is doing some real work.
A backup is known to be good because it has been restored and checked, not because the backup system sent an email containing a green tick.
Define Recovery Expectations Before an Incident
Two measures are particularly useful when discussing recovery:
- Recovery Time Objective (RTO): the maximum acceptable time for a service to remain unavailable.
- Recovery Point Objective (RPO): the maximum acceptable amount of data loss, measured in time.
A service with an RTO of four hours should be recoverable within four hours. A service with an RPO of one hour should not lose more than one hour’s data.
These requirements influence architecture and cost.
A service that must survive component failure without interruption needs a different design from one that can remain unavailable until the following working day. A database where losing five minutes of transactions would be serious needs a different backup and replication strategy from an archive that changes once a month.
Shorter recovery objectives generally require more infrastructure, automation, testing and operational attention. They should therefore be agreed according to business impact rather than technical enthusiasm.
This also allows workloads to be prioritised. During a serious incident, restoring everything simultaneously is unlikely to be possible. Recovery should follow a documented order based on which services matter most and which dependencies must be available first. Restoring an application before its identity service, database and network are working may provide a welcome sense of activity, but not much else.
Test the Recovery Process
A recovery plan that has never been tested is a theory. The test doesn’t always need to involve switching off an entire data centre on a Friday afternoon. In fact, this may be an unnecessarily exciting place to start.
Testing can be introduced progressively:
- Restore individual files and application data.
- Restore a complete virtual machine into an isolated environment.
- Confirm that the restored workload starts correctly.
- Test the dependencies between several workloads.
- Simulate the loss of a cluster node.
- Test recovery using people other than those who designed the system.
- Record the actual recovery time and compare it with the agreed objective.
The results should update the documentation and, where necessary, the architecture.
Testing often uncovers small but important omissions: missing credentials, undocumented firewall rules, incomplete backups, dependencies nobody remembered or recovery instructions that make perfect sense to their author and nobody else. Finding these during a planned exercise is mildly inconvenient. Finding them during an incident is more unfortunate.
Give the Platform an Owner
Virtualisation environments often sit between several areas of responsibility. One team manages the hosts. Another looks after storage. Networking belongs to someone else. Application teams own the workloads, security controls administrative access and a supplier provides support for selected components.
This arrangement can work, but only if the boundaries are clear.
Someone needs responsibility for the platform as a whole. That includes ensuring that:
- Monitoring is working and alerts lead to action
- Capacity is reviewed
- Patches and upgrades are planned
- Hardware and software remain supported
- Backups and restores are tested
- Documentation is maintained
- Security controls are reviewed
- Known risks have owners
- Incidents have a clear escalation route
Without that ownership, necessary work is easily deferred because no individual failure appears urgent. A platform may continue operating with outdated software, limited capacity, untested recovery and incomplete documentation for a surprisingly long time.
Reduce Key Person Dependency
Complex infrastructure develops history. An unusual network configuration may exist because of an application requirement from five years ago. A storage setting may compensate for a hardware limitation that no longer applies. A virtual machine may not be migrated automatically because somebody once discovered that doing so caused a problem.
If this knowledge exists only in the memory of one engineer, the organisation has a continuity risk. The answer is not to make experienced people less important. It is to ensure the platform can be understood and operated when they are unavailable.
Useful controls include:
- Current architecture diagrams
- Documented build and recovery procedures
- Configuration management
- Change records
- Shared administrative access under suitable controls
- Cross-training
- External specialist support
- Recovery tests performed by different people
Documentation should explain why important decisions were made, not simply record the resulting settings. Knowing that a value is set to 0 is fine. Knowing what might happen if it becomes 1 is better.
Maintain the Platform Purposely
In many organisations maintenance gets deferred in small increments. An upgrade is postponed because an application needs testing. A storage warning is acknowledged because capacity will be added next quarter. Documentation falls behind because the engineer who made the change is busy resolving the next problem. Each decision may be reasonable on its own. Together, they create drift.
A business-critical platform needs a lifecycle:
- Supported software versions
- Regular security updates
- Planned platform upgrades
- Capacity forecasting
- Hardware lifecycle management
- Periodic architecture review
- Backup and recovery testing
- Review of administrative access
- Defined support arrangements
The purpose is not to change the platform constantly. It is to ensure that the platform’s condition is known, its risks are visible and necessary work happens before circumstances choose the timetable.
Questions for Your Next Virtualisation Review
A useful review should be able to answer the following:
- Which business services depend on the virtualisation platform?
- What are their agreed recovery time and recovery point objectives?
- Are all important service dependencies documented?
- Which infrastructure failures can the current design tolerate?
- When were representative workloads last restored successfully?
- How long did recovery actually take?
- Who owns platform health, capacity, patching and upgrades?
- Is specialist knowledge shared and documented?
- Are software, hardware and support arrangements current?
- What risks are known, and who has accepted them?
- What happens if a critical problem occurs outside normal working hours?
- Does the current platform still provide the right operational and commercial fit?
The answers don’t all need to be perfect. Few real environments are. What matters is that the organisation understands its position and makes deliberate decisions about the gaps.
Virtualisation Should Make Services More Resilient
Virtualisation provides powerful tools for improving availability, recovery and operational flexibility. It also places many important workloads onto shared infrastructure, increasing the consequences when the platform is poorly designed or insufficiently managed.
The important question is not simply, “Is the virtualisation platform working?”
It is whether the organisation can rely on it, recover it and operate it confidently when the usual people, systems or circumstances are unavailable.
That is a business-continuity question – albeit one involving rather more storage configuration than most business continuity managers would ideally choose.
Review your virtualisation environment
Tiger Computing helps organisations assess virtualisation environments, evaluate platform options and plan the architecture, migration and ongoing support needed for business-critical workloads.
Learn more about our Proxmox VE Services and book a call to discuss whether your current virtualisation operating model is keeping pace with the services it supports.
