Google Cloud stopped operating in Paris early on Wednesday morning local time due to “water intrusion,” said the off-prem biz, which a day earlier reported profitability for the first time.
Coincidentally, the Paris metro area experienced light rain today. Google declined to clarify whether the “water intrusion” was related to the weather, stating that details will be provided in its incident report next week. But the flood appears to have been linked to a fire.
The Google Cloud Status page records service trouble affecting multiple Cloud services in the europe-west9-a zone, starting at 1900 PDT on April 25, which is 0400 on April 26 in Paris. By 2051 PDT, or 0551 CEST, Google acknowledged that the outage was due to water.
“Water intrusion in europe-west9-a led to an emergency shutdown of some hardware in that zone,” the company’s status page explains. “There is no current ETA for recovery of operations in europe-west9-a, but it is expected to be an extended outage. Customers are advised to fail over to other zones in europe-west9 if they are impacted.”
A short while later, the incident description changed to “a multi-cluster failure and has led to an emergency shutdown of multiple zones.”
The outage has affected more than 90 Google Cloud services for europe-west9-a customers and was ongoing at the time this article was filed.
“We apologize to all who are affected by the disruption,” the Google Cloud status notifications state. Google declined to comment beyond its status page statements.
A fire incident has occurred in a room at one of the two data centers in our Paris campus this morning. The Fire Brigade has been in attendance and the fire is now contained,” said Global Switch, which operates the Paris data center where Google Cloud colocates.
“The fire response systems in the building have performed as designed and no one has been injured. A number of customers have been temporarily affected and our site team is working to restore services to those customers as soon as possible.”
A post to the French Network Operators Group mailing list suggests the outage arose from a cooling system water pump failure that caused water to accumulate and leak. That in turn is said to have flooded the battery room and caused the data center fire.
It’s not immediately clear whether data center fire suppression equipment or the actions of firefighters to contain the blaze further contributed to problems with Google Cloud hardware.
When it rains, it pours: Google Cloud reported a separate disruption on Wednesday. Its Google Cloud Networking and Cloud Load Balancing services had problems for about an hour and a half, starting 0336 PDT.
Though more brief, the load balancing problems were far broader, affecting not just the europe-west9 zone but multiple zones in Asia, Australia, Europe, North America, and South America.
