Backup systems failed to activate
Google has released details regarding a service disruption at its London data centres during October. The incident occurred while the facility was undergoing planned maintenance on its electrical grid connection. During this period, the site was operating with reduced redundancy, meaning there were fewer backup systems available than usual.
When a power fluctuation occurred, the Programmable Logic Controllers (PLCs) failed to trigger the backup diesel generators. PLCs are the industrial computers that manage hardware processes. Because these controllers were not configured correctly for the specific maintenance state of the building, they did not bridge the gap between the mains grid and the emergency power supply.
The cause of the downtime
The investigation found that the software instructions within the PLCs did not account for the temporary electrical layout used during maintenance. This led to a complete loss of power for a portion of the data centre.
For small and medium businesses, this resulted in the unavailability of hosted websites, databases, and Google Workspace tools. While cloud providers typically have multiple layers of protection, this event shows that physical infrastructure maintenance remains a point of vulnerability.
Lessons for UK businesses
Google has stated they are updating their maintenance protocols and PLC configurations to prevent a recurrence. However, this incident serves as a reminder that no single data centre is immune to technical failure.
If your business operations depend on constant uptime, you may wish to consider a multi-region configuration. This involves hosting copies of your data or services in two different geographical locations, such as London and Belgium. If one location suffers a power failure, traffic can be redirected to the other.
We recommend reviewing your current cloud setup to understand which 'region' your data resides in and whether your business continuity plan accounts for a total provider outage in that area.