Google Cloud Service Health Up... Note

Google Cloud Service Health Updates

status.cloud.google.com is the official status dashboard for Google Cloud services. It provides real-time information about the current status of various Google Cloud services, including Compute Engine, App Engine, Cloud Storage, and more. The website displays a list of all Google Cloud services, along with their current status, which can be one of the following: - Green: The service is operating normally. - Yellow: The service is experiencing a partial outage or degradation. - Red: The service is experiencing a full outage. For each service, the website also displays a brief description of the issue, if any, and the time the issue started. Additionally, the website provides a history of past incidents and outages for each service. The website is updated in real-time, allowing users to quickly check the status of Google Cloud services and plan accordingly. It is a useful resource for developers, administrators, and users who rely on Google Cloud services for their applications and workloads. The website also provides links to incident reports and root cause analyses for past outages, which can be helpful for understanding what went wrong and how Google Cloud is working to prevent similar issues in the future. Overall, the Google Cloud Status Dashboard is a transparent and informative resource that helps users stay up-to-date with the status of Google Cloud services.

Thread Of Notes

UPDATE: Network traffic to Google Cloud originating from Delhi, Chennai, Mumbai and surrounding areas is experiencing intermittent periods of elevated latency and possible packet loss.

Network traffic to Google Cloud from India, specifically Delhi, Chennai, Mumbai, and surrounding areas, is experiencing intermittent disruptions. These disruptions manifest as elevated latency and potential packet loss for affected customers. The issue stems from traffic rerouting due to an incident at the Delhi facility, overwhelming regional capacity. Consequently, a subset of Hybrid Connectivity and Virtual Private Cloud customers are facing latency spikes. Media CDN customers might also observe increased latencies. Initial mitigation efforts have restored some internet edge peering capacity in Delhi. Augmentation of out-of-region internet edge peering capacity in Chennai is also nearing completion. Network backbones have been optimized, and Delhi backbone capacity has been increased. Additional capacity has been restored between Delhi-Chennai and Delhi-Mumbai. Customers may experience slightly elevated latency until the affected facility is fully restored, with no current workaround available.

UPDATE: Network traffic to Google Cloud originating from Delhi, Chennai, Mumbai and surrounding areas is experiencing intermittent periods of elevated latency and possible packet loss.

Network traffic to Google Cloud from India, specifically Delhi, Chennai, and Mumbai, is experiencing increased latency and potential packet loss. This issue stems from a fire at a third-party data center in Delhi, which caused an emergency shutdown of networking equipment. This event isolated a local Point of Presence and reduced overall network capacity in the region. Google Cloud rerouted traffic from the affected Delhi facility to compensate for the reduced local serving capabilities. Consequently, some Hybrid Connectivity and Virtual Private Cloud customers may experience intermittent latency spikes due to capacity constraints across Indian metros and ISPs. Media CDN customers might also have noticed higher latencies than usual. Initial mitigation efforts have shown positive results, including restoring some Internet Edge peering capacity in Delhi. Further actions are being taken, such as augmenting out-of-region Internet Edge peering capacity in Chennai and optimizing network backbones. These measures aim to increase available headroom and provide additional load balancing. Google Cloud will continue to monitor network performance closely and provide the next update by Wednesday, June 17, 2026, at 17:00 PDT.

UPDATE: Network traffic to Google Cloud originating from Delhi, Chennai, Mumbai and surrounding areas is experiencing intermittent periods of elevated latency and possible packet loss.

Network traffic to Google Cloud from India is experiencing intermittent latency and packet loss. This is due to a fire at a third-party data center in Delhi that caused an emergency power shutdown. The incident isolated a local network point of presence and reduced overall capacity in the metro area. Rerouting significant traffic from the affected Delhi facility has impacted some Hybrid Connectivity and Virtual Private Cloud customers. These customers may experience latency spikes as demand exceeds available capacity across Indian regions. Initial traffic mitigations have shown some positive results for certain customers. Google is actively pursuing additional Internet Edge peering capacity in Delhi to improve local latency. They are also augmenting regional peering capacity in Chennai. Further enhancements to Delhi backbone capacity are expected by mid-June. Google will continue to monitor network performance and provide the next update by June 15th.

UPDATE: Network traffic to Google Cloud originating from Delhi, Chennai, Mumbai and surrounding areas is experiencing intermittent periods of elevated latency and possible packet loss.

A fire at a third-party data center in Delhi triggered an emergency power shutdown, impacting networking equipment. This event led to intermittent elevated latency and potential packet loss for Google Cloud traffic originating from Delhi, Chennai, Mumbai, and surrounding areas. The incident isolated a local Point of Presence in Delhi and reduced overall network capacity in the metro. Google Cloud rerouted significant traffic from the affected Delhi facility to compensate for reduced local serving capabilities. Consequently, some Hybrid Connectivity and Virtual Private Cloud customers may experience latency spikes due to increased demand on existing capacity across Indian metros and ISPs. Initial mitigation efforts have shown some positive results for customers. Further actions include pursuing additional Internet Edge peering capacity to enhance network resilience. Google Cloud is also optimizing capacity across its network backbones to provide more headroom for VPC customers in the affected region. Plans are in place to augment the local Delhi POP and migrate select peering partners to boost regional capacity. Customers may experience temporarily elevated latency and suboptimal routing until the affected facility is fully restored.

UPDATE: Network traffic to Google Cloud originating from Delhi, Chennai, Mumbai and surrounding areas is experiencing intermittent periods of elevated latency and possible packet loss.

Google Cloud is experiencing network issues affecting traffic from Delhi, Chennai, Mumbai, and surrounding areas. A fire at a third-party data center in Delhi caused an emergency power shutdown of networking equipment. This incident isolated a local Point of Presence and reduced overall network capacity in the region. As a result, some Hybrid Connectivity and Virtual Private Cloud customers may see intermittent latency spikes. The rerouting of traffic to compensate for reduced local serving capabilities has led to this issue. Demand currently exceeds the available capacity across Indian metros and regional ISPs. Google is actively investigating further traffic mitigations and augmenting Internet Edge peering. They are also collaborating with local ISP partners to address the impact of unplanned failures. Affected customers might experience slightly elevated latency and non-optimal routing until the facility is restored. There is currently no workaround available for these network disruptions.

UPDATE: Network traffic to Google Cloud originating from Delhi, Chennai, Mumbai and surrounding areas is experiencing intermittent periods of elevated latency and possible packet loss.

A fire at a third-party data center in Delhi initiated an emergency power shutdown. This event isolated a local networking point of presence and reduced overall network capacity in the Delhi metro area. Consequently, Google Cloud experienced intermittent elevated latency and potential packet loss for traffic originating from Delhi, Chennai, Mumbai, and surrounding regions. The issue began at 11:22 AM US/Pacific on June 9, 2026. Google rerouted significant traffic from the affected Delhi facility to compensate for reduced local serving capabilities. This rerouting has impacted a subset of Hybrid Connectivity and Virtual Private Cloud customers. These customers may observe intermittent latency spikes as demand surpasses available capacity across Indian metros and regional ISPs. Google is actively investigating further traffic mitigations and Internet Edge peering augmentations. Currently, there is no workaround available for the affected customers. The next update regarding this incident is scheduled for June 10, 2026, at 5:00 PM PDT.

RESOLVED: Vertex AI Gemini API customers experienced increased error rates when accessing the global endpoint.

On February 27, 2026, Vertex AI Gemini API experienced an outage impacting multiple services. The incident began at 04:37 and lasted nearly two hours, affecting users in US regions and globally. The main issue involved increased error rates for customers using various Gemini models. The primary root cause was a problematic configuration change to a safety filtering service. This change caused service disruptions and capacity loss, resulting in 429 and 503 errors. Google engineers were alerted at 04:54 and quickly investigated the issue. They rolled back the configuration change and added more capacity to fully restore service by 06:35. Affected products included Agent Assist, Dialogflow CX, and Google Cloud Support. Google is implementing stricter rollout validation and improved alerting systems. PayGo customers primarily saw 429 errors, while Provisioned Throughput customers saw 503 errors. The issue affected the global endpoint and several US locations.

RESOLVED: Vertex AI Gemini API customers experienced increased error rates when accessing the global endpoint.

On February 27, 2026, from 04:37 to 06:35 US/Pacific, the Vertex AI Gemini API models experienced an outage. This incident caused increased error rates for users of various Gemini versions, including previews. The affected services included Google Cloud Support, Agent Assist, Vertex Gemini API, and Dialogflow CX in US regions and globally. The root cause was a problematic configuration change to a safety filtering service supporting all Gemini models. This change caused interactions that resulted in service disruptions and errors like 429 and 503. Google engineers were alerted at 04:54 and began investigating the issue immediately. The solution involved a rollback to the prior configuration and adding additional service capacity. Full service restoration was achieved by 06:35 when the rollback had fully propagated. PayGo customers primarily saw 429 errors, while Provisioned Throughput customers received 503 errors. The outage affected multiple US regions and locations within North and South America. Google is conducting a full root cause analysis and will publish a final incident report with preventative measures. Google apologizes for the inconvenience and is taking steps to improve platform reliability.

RESOLVED: Vertex AI Gemini API customers experienced increased error rates when accessing the global endpoint.

A service incident occurred on February 27, 2026, impacting the Vertex Gemini API Global endpoint. This caused higher error rates for customers using multiple Gemini models through the global endpoint. The incident began at 4:36 AM PST and lasted until 6:45 AM PST. Several Google services, including Dialogflow CX and Agent Assist, were negatively affected due to their reliance on the Gemini APIs. The root cause appears to be a recent configuration change. The issue was resolved by rolling back the problematic configuration change. Service restoration led to full resolution of the problem, allowing systems to function correctly. Customers in multiple geographic locations, including North America and South America, experienced the issue. Google expresses gratitude for customer patience during the incident. Affected products included Agent Assist, Dialogflow CX, Google Cloud Support, and the Vertex Gemini API itself.

RESOLVED: We are investigating elevated error rates with multiple products in us-east1

A Google Cloud Platform (GCP) and Google Workspace (GWS) service disruption occurred on July 18, 2025, in the us-east1 region, lasting up to 1 hour and 57 minutes. The incident began at 7:42 AM and concluded at 9:47 AM Pacific Time. A procedural error during planned hardware replacement caused an incorrect network switch disconnection. This led to a partitioned network control plane, resulting in stale topology information and packet loss. The issue was identified at 7:06 AM, with remediation efforts starting immediately. Traffic was rerouted to mitigate the impact, with Workspace services restored by 8:40 AM. Full service restoration was achieved at 9:47 AM. The incident affected multiple GCP and GWS products, primarily in the Southeast US. Google is implementing safety controls and preventative measures to avoid future occurrences. A small percentage of persistent disks were briefly unavailable, but no data loss occurred.

RESOLVED: We are investigating elevated error rates with multiple products in us-east1

A service disruption/outage occurred on July 18, 2025, between 07:42 and 09:47 US/Pacific time. The incident affected multiple Google Cloud Platform (GCP) and Google Workspace (GWS) products in the us-east1 region. The GCP impact lasted for 1 hour and 37 minutes, while the GWS impact lasted for 30 minutes. The issue caused elevated latencies and error rates in the affected region. The root cause of the issue was a procedural error during a planned hardware maintenance in one of Google's data centers in the us-east1 region. The engineering team mitigated the issue by draining traffic away from the clusters and restoring the affected hardware. A full incident report will be completed in the following days, providing a full root cause and preventive actions. The affected products experienced elevated latencies and error rates in the us-east1 region. The incident affected a wide range of GCP and GWS products, including AlloyDB for PostgreSQL, Apigee, Gmail, Google Meet, and many others. The affected location was South Carolina (us-east1).

RESOLVED: We are investigating elevated error rates with multiple products in us-east1

An incident occurred on July 18, 2025, between 7:42 am and 9:47 am Pacific Time. The issue was resolved for all affected products by 9:47 am Pacific Time. Preliminary analysis suggests that elevated packet loss during routine network maintenance in the us-east1-b zone caused the service disruption. A full Incident Report with the root cause will be published once internal investigations are complete. The company thanks users for their patience during the resolution process. The incident affected multiple products, including AlloyDB for PostgreSQL, Apigee, Artifact Registry, and many others. These products were impacted in the South Carolina (us-east1) location. The incident did not affect other locations. The company is still investigating the root cause of the issue. A detailed report will be published once the investigation is complete.

UPDATE: We are investigating elevated error rates with multiple products in us-east1

An incident occurred on July 18, 2025, starting at 07:42 US/Pacific time and ending at 09:47 US/Pacific time. The issue affected the network control plane in the us-east1 zones, but engineers have successfully recovered it. Multiple services are now reporting full recovery, and product engineers are continuing to validate the remaining services. The affected products include a wide range of Google Cloud services, such as AlloyDB for PostgreSQL, Apigee, and Google Cloud Storage. Other affected products include Cloud Billing, Cloud Build, Cloud External Key Manager, and Cloud Firestore, among many others. The incident also impacted various Google Cloud services like Google App Engine, Google BigQuery, and Google Cloud SQL. The affected location was South Carolina, specifically the us-east1 zones. Engineers are still working to ensure all services are fully operational, and an update with more details is expected by 11:00 AM US/Pacific on July 18, 2025. The recovery efforts are ongoing, and the team is working to prevent similar incidents in the future. The incident has been resolved, but the team will continue to monitor the situation to ensure that all services are stable and functioning as expected.

UPDATE: We are investigating elevated error rates with multiple products in us-east1

The incident occurred on July 18, 2025, between 7:42 am and 9:47 am US/Pacific time. Engineers have successfully recovered the network control plane in the affected us-east1 zones. Multiple services are reporting full recovery, and product engineers are validating the remaining services. Another update with more details will be provided by 10:30 am US/Pacific on July 18, 2025. The incident affected various Google Cloud products, including Apigee, Artifact Registry, Cloud Build, and many others. The affected products also include Google App Engine, Google BigQuery, Google Cloud Bigtable, and Google Cloud Console, among others. The incident impacted locations in South Carolina, specifically the us-east1 zones. The recovery efforts are ongoing, with engineers working to validate the remaining services. The incident has been resolved, and services are returning to normal. Further updates will be provided to ensure complete transparency and resolution.

UPDATE: We are investigating elevated error rates with multiple products in us-east1

Incident began at 2025-07-18 07:42 and ended at 2025-07-18 09:47 (all times are US/Pacific). Our engineers have confirmed that us-east1-b is partially affected. All other zones in us-east1 are currently operating normally. Our engineers have recovered the failed hardware and are currently recovering the network control plane in the affected zones. We'll provide another update by 10:00 AM US/Pacific, July 18, 2025. Affected products: Cloud External Key Manager, Cloud HSM, Cloud Key Management Service, Cloud Run, Dialogflow CX, Google App Engine, Google Cloud Bigtable, Google Cloud Dataproc, Google Cloud SQL, Google Cloud Storage, Google Cloud Support, Google Compute Engine, Google Kubernetes Engine, Hybrid Connectivity, Identity and Access Management, Secret Manager, Virtual Private Cloud (VPC) Affected locations: South Carolina (us-east1)

UPDATE: We are investigating elevated error rates with multiple products in us-east1

Incident began at 2025-07-18 07:06 (all times are US/Pacific). We're currently experiencing elevated latency and error rates for several Cloud services in the us-east1 region, beginning at 7:06 AM PDT today, July 18, 2025. Our initial investigation points to a hardware infrastructure failure as the likely cause. We apologize for any disruption this may be causing. We'll provide an update with more details by 9:15 AM PDT today. Affected products: Cloud Run, Dialogflow CX, Google App Engine, Google Cloud Bigtable, Google Cloud Dataproc, Google Cloud SQL, Google Cloud Storage, Google Cloud Support, Google Compute Engine, Google Kubernetes Engine, Hybrid Connectivity, Virtual Private Cloud (VPC) Affected locations: South Carolina (us-east1)

RESOLVED: Multiple GCP products are experiencing Service issues

On June 12, 2025, Google Cloud, Workspace, and Security Operations experienced increased 503 errors in external API requests due to a faulty code change in Service Control. A new feature added on May 29, 2025, lacked proper error handling and feature flag protection. A policy change on June 12 triggered a null pointer, causing the Service Control binary to crash regionally, then globally. The Site Reliability Engineering team identified the root cause within 10 minutes and implemented a red-button to disable the problematic serving path. Recovery started within 40 minutes, but larger regions like us-central-1 experienced overload due to a "herd effect." Full resolution in us-central-1 took roughly 2 hours and 40 minutes, with traffic rerouted and task creation throttled. Google has frozen changes to the Service Control stack and will focus on modularizing the architecture. They will audit systems using globally replicated data, enforce feature flags for critical binaries, and improve error handling and testing. Google also plans to enhance external communications and ensure monitoring infrastructure remains operational during outages. Many Google Cloud products across numerous regions were impacted by this incident.

RESOLVED: Multiple GCP products are experiencing Service issues

A Google Cloud and Workspace incident occurred on June 12, 2025, from 10:49 to 13:49 US/Pacific, lasting 3 hours. The incident resulted in increased 503 errors for external API requests, globally impacting customer access. An invalid automated quota update to the API management system caused the rejections. Recovery involved bypassing the problematic quota check, restoring most regions within two hours. The us-central1 region experienced a longer recovery due to database overload. Some products had residual impact for up to an hour, while a few others recovered later. Google is preparing a detailed incident report outlining the root cause and remediation steps. Measures will be implemented to prevent API management failures, improve metadata propagation, and enhance error handling. The incident affected numerous Google Cloud and Workspace products, including Compute Engine, BigQuery, Gmail, and Google Calendar. The impact spanned across various global locations.

UPDATE: Multiple GCP products are experiencing Service issues

An incident impacting Vertex AI Online Prediction and other Google Cloud services began on June 12, 2025, at 10:51 PDT. The incident caused elevated 5xx errors with some Model Garden models. The primary issue was resolved by 17:05 PDT, restoring normal Vertex AI serving in most regions. However, europe-west1 and asia-southeast1 were still experiencing capacity issues. Engineers are actively working to restore normal serving capacity in these two remaining regions. The estimated time for full resolution in europe-west1 and asia-southeast1 is 19:45 PDT. An update on the situation will be provided by 19:45 PDT on the same day. A wide range of Google Cloud products and services were affected by this incident. The incident impacted a large number of locations across multiple regions globally.

UPDATE: Multiple GCP products are experiencing Service issues

An incident impacting multiple Google Cloud products and locations began on June 12, 2025, at 10:51 PDT. Personalized Service Health is now resolved and reflecting accurate updates. The issue affecting Google Cloud Dataflow was fully resolved by 17:10 PDT. However, Vertex AI Online Prediction continues to experience elevated 5xx errors, specifically with models in the Model Garden. Engineers are working to mitigate these errors, and a gradual decrease in error rates is being observed. The estimated time for full resolution of the Vertex AI Online Prediction issue is 22:00 PDT. A wide range of Google Cloud services were affected, including API Gateway, AlloyDB, Cloud Run, BigQuery, and more. The incident impacted various geographic locations across Asia, Europe, North America, South America, and other regions. An update with the latest details will be provided by 22:00 PDT on June 12, 2025. Customers should expect continued error rates on Vertex AI until the ETA.

UPDATE: Multiple GCP products are experiencing Service issues

An incident impacting multiple Google Cloud products began on June 12, 2025, at 10:51 PDT and ended at 18:18 PDT. Residual impacts persist for Google Cloud Dataflow, especially in us-central1, where backlog clearance is ongoing. Customers may experience delays with Dataflow operations in us-central1, with no ETA for full recovery. Vertex AI Online Prediction users may still encounter elevated 5xx errors with certain Model Garden models. Engineers are working to mitigate these errors, aiming for full resolution by 22:00 PDT. Personalized Service Health updates are delayed; users should refer to the Cloud Service Health dashboard. A wide range of Google Cloud products were affected, including API Gateway, BigQuery, Compute Engine, and Cloud Storage. The incident impacted various locations globally, spanning Asia, Europe, and North and South America. The next update will be provided on June 12, 2025, at 17:45 PDT.

UPDATE: Multiple GCP products are experiencing Service issues

An incident began on June 12, 2025, at 10:51 US/Pacific time. Google Cloud products are still experiencing residual impact, including delays with Dataflow operations and elevated 5xx errors with some Vertex AI Online Prediction models. Personalized Service Health updates are delayed, and customers are advised to use the Cloud Service Health dashboard for updates. There is currently no estimated time for full mitigation of these services. An update will be provided by 17:00 PDT with current details. The incident affected multiple products, including API Gateway, Cloud Data Fusion, Cloud Firestore, and many others. The affected regions include Johannesburg, Taiwan, Hong Kong, Tokyo, and many others across the globe. The incident impacted multiple locations, including asia-south1, asia-northeast1, europe-central2, and others. The affected products and locations are listed in detail. The incident is still ongoing, and customers are advised to monitor the situation for updates.

UPDATE: Multiple GCP products are experiencing Service issues

A Google Cloud incident began on June 12, 2025, at 10:51 AM Pacific Time. Most Google Cloud products have fully recovered as of 1:45 PM Pacific Time. However, some products are still affected, and users should continue to monitor the services and dashboard for individual product recoveries. An update will be provided by 4:00 PM Pacific Time with current details. The incident affected numerous Google Cloud products, including API Gateway, Apigee, Cloud Data Fusion, Cloud Firestore, and many more. The affected locations include Johannesburg, multiple regions in Asia, Europe, Australia, South America, and the United States. The incident impacted a wide range of regions, including africa-south1, asia-east1, europe-central2, and us-west4, among many others. The affected regions are spread across the globe, with multiple regions in Asia, Europe, and the Americas. Users are advised to continue monitoring the services and individual product recoveries. The incident is still being addressed, and users will be updated with current details by 4:00 PM Pacific Time on June 12, 2025.

UPDATE: Multiple GCP products are experiencing Service issues

A Google Cloud incident began on June 12, 2025, at 10:51 US/Pacific time. Most Google Cloud products have confirmed full service recovery. However, a few services are still experiencing residual impact, and their respective engineering teams are actively working on recovery. The recovery is expected to complete in less than an hour. An update will be provided by 15:00 PDT with current details. The incident affected numerous Google Cloud products, including API Gateway, Cloud Data Fusion, Cloud Firestore, and many others. The affected locations include Johannesburg, Multi-region: asia, Taiwan, and many other regions across the globe. The affected regions span across Africa, Asia, Australia, Europe, North and South America. The incident impacted multiple regions, including asia, eu, nam, and us. The recovery efforts are ongoing, and an update will be provided soon.

UPDATE: Multiple GCP products are experiencing Service issues

An incident began on June 12, 2025, at 10:51 US/Pacific time. Mitigation efforts have been implemented in us-central1 and multi-region/us, and signs of recovery are being seen. Confirmation from internal monitoring and customers indicates recovery in multiple regions, including us-central1 and multi-region/us. Full recovery is expected within an hour. An update will be provided by 14:30 PDT with current details. The incident affected numerous Google Cloud products, including API Gateway, Cloud Data Fusion, and Google Cloud Storage. The affected locations include Johannesburg, Taiwan, Tokyo, and many other regions across the globe. The incident impacted multiple regions, including asia, eu, nam, and us. The affected regions include us-central1, asia-east1, europe-north1, and many others. The incident is being closely monitored, and updates will be provided as soon as possible.

UPDATE: Multiple GCP products are experiencing Service issues

An incident began on June 12, 2025, at 10:51 am Pacific Time. The root cause has been identified, and mitigations have been applied. The infrastructure has recovered in all regions except us-central1. Google Cloud products that rely on the affected infrastructure are seeing recovery in multiple locations. Engineers are aware of customers still experiencing issues on us-central1 and multi-region/us and are actively working on full recovery. There is no ETA for full recovery. An update will be provided by Thursday, June 12, 2025, at 14:00 PDT with current details. The incident affects numerous Google Cloud products, including API Gateway, Cloud Data Fusion, and Cloud Spanner, among others. The incident also affects multiple locations worldwide, including Johannesburg, Tokyo, Mumbai, and Sydney, among others.

UPDATE: Multiple GCP products are experiencing Service issues

The incident began on June 12, 2025, at 10:51 US/Pacific time. Engineers have found the root cause and applied mitigations, but customers are still experiencing varying degrees of impact on individual Google Cloud products. The underlying dependency has recovered in all locations except us-central1. All respective engineering teams are working on service recovery, but there is no estimated time for full service recovery. An update will be provided by 13:30 PDT on June 12, 2025, with current details. The affected products include Agent Assist, AlloyDB for PostgreSQL, Apigee, and many others. The affected locations include Johannesburg, Multi-region: asia, Taiwan, and many others. The incident has a global impact, affecting multiple regions and locations. Engineers are actively engaged in service recovery, but the incident is ongoing. The full list of affected products and locations is provided in the original message.

UPDATE: Multiple GCP products are experiencing Service issues

An incident began on June 12, 2025, at 10:51 am US/Pacific time. All locations except us-central1 have fully recovered, while us-central1 is mostly recovered. There is no estimated time for full recovery in us-central1. An update will be provided by June 12, 2025, at 1:00 pm PDT. The incident affected multiple products, including Agent Assist, Cloud Data Fusion, Cloud Firestore, and many others. The affected locations include Johannesburg, Taiwan, Hong Kong, Tokyo, and many others across Asia, Europe, North America, and South America. The incident affected multiple regions, including asia, eu, nam5, us, and more. The affected locations in the US include Iowa, South Carolina, Northern Virginia, and others. The incident is still ongoing, with an update expected soon. The affected products and locations are numerous, and the incident is still being resolved.

UPDATE: Multiple GCP products are experiencing Service issues

An incident began on June 12, 2025, at 10:51 US/Pacific time. Engineers are working to mitigate the issue, and some locations have already recovered. However, there is no estimated time yet. An update will be provided by 12:45 PDT with the latest details. The incident affects multiple Google Cloud services, including Cloud Data Fusion, Cloud Memorystore, and Cloud Workstations. The affected locations include Johannesburg, Taiwan, Hong Kong, Tokyo, and many others across Asia, Europe, North America, and South America. The incident is global, with multiple regions affected in the US, Asia, and other areas. The affected products include Google Cloud Bigtable, Google Cloud Console, and Identity Platform, among others. The incident is ongoing, and users should expect disruptions to the affected services until further notice.

UPDATE: Multiple GCP products are experiencing Service issues

A service issue began on June 12, 2025, at 10:51 PDT, affecting multiple Google Cloud Platform (GCP) products. The engineering team is investigating the problem, and an update will be provided by 12:15 PDT. The issue is causing varying levels of service impacts with API requests. No workaround is currently available. The affected products include Cloud Data Fusion, Cloud Memorystore, Cloud Workstations, and several others. The issue is affecting multiple locations worldwide, including Johannesburg, Taiwan, Tokyo, Mumbai, and many others. The affected locations include multiple regions in Asia, Europe, South America, and North America. The service disruption is causing inconvenience to all affected users. The engineering team is working to resolve the issue as soon as possible.

UPDATE: Multiple GCP products are experiencing Service issues

A service issue began affecting multiple Google Cloud Platform (GCP) products on Thursday, June 12, 2025, at 10:51 PDT. The engineering team is investigating the issue and will provide an update by 12:15 PDT with current details. The affected products include Google Cloud Bigtable, Google Cloud Console, Google Cloud Dataproc, Google Cloud Storage, Identity Platform, and Identity and Access Management. The issue is impacting multiple regions, including Johannesburg, Multi-region: asia, Taiwan, Hong Kong, Tokyo, Osaka, Seoul, Mumbai, Delhi, Singapore, Jakarta, and many others. The symptoms include varying levels of service impacts on the affected products. There is currently no available workaround for the issue. The incident is causing disruptions to users, and the team apologizes for the inconvenience. The affected locations include Johannesburg, Multi-region: asia, Taiwan, Hong Kong, Tokyo, Osaka, Seoul, Mumbai, Delhi, Singapore, Jakarta, and many others. The team will continue to investigate and provide updates on the issue.

RESOLVED: Google Compute Engine (GCE) issue impacting multiple dependent GCP services across zones

On May 19, 2025, Google Compute Engine (GCE) experienced issues with Spot VM termination, lasting approximately 8 hours and 42 minutes. This led to performance degradation and timeouts in specific regions like us-central1 and us-east4. The root cause was a configuration change that disabled a feature flag, causing Spot VMs to get stuck during termination. This resulted in a backlog, impacting performance for all VM types in certain areas. Internal monitoring alerted Google engineers, and they initiated a rollback to address the problem. The rollback, completed on May 20, 2025, at 03:55 US/Pacific, mitigated the impact of the issue. Customers experienced increased latency for VM control plane operations, specifically impacting VM creation, modification, and deletion. The issue affected various Google Cloud Platform (GCP) products that rely on GCE, including GKE and Dataflow. Google is taking steps to prevent future incidents, including auditing safety checks and addressing scalability concerns. Google is reviewing its phased rollout approach to prevent similar disruptions. The company apologizes for the impact and is committed to improving platform reliability.

RESOLVED: Google Compute Engine (GCE) issue impacting multiple dependent GCP services across zones

A service disruption occurred from May 19, 2025, 20:23 to May 20, 2025, 05:05 (US/Pacific time zone). The incident affected multiple Google Cloud services, including Compute Engine, Kubernetes Engine, Cloud Dataflow, Cloud SQL, and others. The issue was caused by a recent configuration change that negatively affected Google Compute Engine's handling of spot virtual machine terminations, leading to overloaded control plane services. This caused disruptions to VM instance creation, termination, and reservation consumption. The problem was mitigated by reverting to the previous configuration, resolving the impact on all affected services. Customers experienced elevated latency or timeouts in various operations, including VM instance creation, and termination, as well as issues with Cloud Dataproc, Cloud Dataflow, Cloud Build, and other services. A full incident report will be completed in the coming days. The affected regions and zones included us-central1, us-east4, and multiple other zones in Asia, Europe, Australia, and South America. The incident resulted in various errors, timeouts, and failures in different Google Cloud services.

RESOLVED: Google Compute Engine (GCE) issue impacting multiple dependent GCP services across zones

Incident began at 2025-05-19 20:23 and ended at 2025-05-20 05:05 (all times are US/Pacific). The issue with multiple dependent GCP services has been resolved for all affected users as of Tuesday, 2025-05-20 05:05 US/Pacific. We thank you for your patience while we worked on resolving the issue. Affected products: AlloyDB for PostgreSQL, Cloud Build, Cloud Filestore, Colab Enterprise, Google App Engine, Google Cloud Composer, Google Cloud Dataflow, Google Cloud Dataproc, Google Cloud Deploy, Google Cloud SQL, Google Compute Engine, Google Kubernetes Engine, Managed Service for Apache Kafka, Migrate to Virtual Machines Affected locations: Belgium (europe-west1), Iowa (us-central1), South Carolina (us-east1), Northern Virginia (us-east4), Oregon (us-west1)

UPDATE: Google Compute Engine (GCE) issue impacting multiple dependent GCP services across zones

A multi-service incident affecting Google Cloud Platform began on May 19, 2025, at 20:23 US/Pacific. Multiple dependent GCP services experienced issues such as elevated latency and failures. The engineering team deployed a mitigation, observing improvements across affected zones. Impacted services included Dataproc, Compute Engine, Kubernetes Engine, Dataflow, Filestore, and Cloud Build. Cloud SQL, Composer, AlloyDB for PostgreSQL, App Engine, Migrate to Virtual Machines, Vertex GenAI, Cloud Deploy, and Memorystore for Redis were also affected. Many of the impacted products have already been mitigated. Customers experienced issues with instance creation, scaling, and operation latency. The workaround suggested using alternate zones to mitigate the impact. Affected locations included Belgium, Iowa, South Carolina, Northern Virginia, and Oregon. Further information was scheduled for release on May 20, 2025, at 05:30 US/Pacific with the incident ending at 05:05.

UPDATE: Google Compute Engine (GCE) issue impacting multiple dependent GCP services across zones

Starting on May 19, 2025, at 20:23 US/Pacific, multiple Google Cloud services experienced issues. The incident impacted Google Compute Engine, Kubernetes Engine, Dataflow, SQL, AlloyDB, Composer, Build, Dataproc, Filestore, and App Engine Flexible. Users may have encountered increased latency, timeouts, or failures in various operations like creating, deleting, resizing, or updating instances and clusters. The problem affected zones in Belgium, Iowa, South Carolina, Northern Virginia, and Oregon. Google's engineering team deployed a mitigation, and improvements were observed across all affected zones. As a workaround, customers were advised to utilize alternate zones to avoid impact. Cloud Build customers experienced elevated latency in build completion or sporadic build failures. Cloud SQL users encountering failures in create operations were advised to retry after deleting failed instances. Cloud Composer users saw failures in environment creation, upgrades, and delays in scaling Airflow workers. An update with more information was scheduled for May 20, 2025, at 05:00 US/Pacific, and the incident ended then.

UPDATE: Google Compute Engine (GCE) issue impacting multiple dependent GCP services across zones

A multi-service incident affecting Google Cloud products began on May 19, 2025, at 20:23 US/Pacific. The incident impacts services like Compute Engine, Kubernetes Engine, Dataflow, SQL, AlloyDB, Composer, Build, Dataproc, Filestore, and App Engine. Users may encounter increased latency, timeouts, or failures in various operations across these services. The engineering team is actively working on mitigation without a current ETA. Affected operations include instance creation, resizing, scaling, and updates. A potential workaround involves using alternate zones to avoid the impacted regions. The affected locations are Belgium, Iowa, South Carolina, Northern Virginia, and Oregon. The incident also affects both default and private pools in Cloud Build. Customers experiencing failures in create operations for Cloud SQL can retry after deleting failed instances. An update is expected on May 20, 2025, at 04:30 US/Pacific.

RESOLVED: Customers are experiencing connectivity issues with multiple Google Cloud services in zone us-east5-c

On March 29, 2025, Google Cloud Services in the us-east5-c zone experienced service degradation for over six hours due to a power outage. The root cause was a failure of the uninterruptible power supply (UPS) system when utility power was lost. A critical battery failure within the UPS prevented it from providing continuous power, leading to virtual machine downtime and packet loss. Google engineers were alerted and diverted traffic, restoring power via generator after nearly two hours. Some services required manual actions for full recovery, extending restoration times. Google is working with the UPS vendor and hardening cluster power recovery to prevent recurrence. Impacted services included AlloyDB, BigQuery, Bigtable, Composer, Dataflow, Filestore, and others, with varying levels of disruption. Customers experienced unavailability, increased latency, and errors depending on the service and zonal dependencies. The incident affected the Columbus (us-east5) location, impacting various products relying on the affected zone. Google apologized for the disruption and is committed to improving platform reliability.

RESOLVED: Customers are experiencing connectivity issues with multiple Google Cloud services in zone us-east5-c

A Google Cloud incident occurred on March 29, 2025, between 12:53 and 19:15 US/Pacific time, lasting 6 hours and 19 minutes. The incident affected multiple Google Cloud products in the us-east5-c region, causing some zonal resources to be unavailable. The root cause was a utility power outage and a failure of batteries in the uninterruptible power supply (UPS) system. This prevented a power source transfer to generators, causing downtime for some Compute Engine instances and packet loss within the zone. Capacity constraints were also experienced for Google Kubernetes Engine in other zones of us-east5. A small number of Persistent Disks were unavailable during the outage. Engineers mitigated the impact by diverting traffic away from the affected location and bypassing the failed UPS to restore power via generator. The incident was fully mitigated by 19:12 US/Pacific on March 29, 2025. A full Incident Report will be completed in the following days, providing a detailed root cause analysis. The us-east5-c zone has since transitioned back to primary power without further impact as of March 30, 2025, at 17:30 US/Pacific.

RESOLVED: Customers are experiencing connectivity issues with multiple Google Cloud services in zone us-east5-c

The incident occurred on March 29, 2025, between 12:53 and 19:15 US/Pacific time. The us-east5-c zone is currently stable and running on an alternate power source. All previously impacted products have been mitigated as of 19:15 US/Pacific. However, a small number of Persistent Disks are still in recovery and being actively worked on. Customers experiencing issues attaching Persistent Disks should open a support case. Engineers are monitoring service stability before transitioning back to primary power. Updates will be provided via PSH by Sunday, March 30, 2025, at 01:30 US/Pacific. The incident affected multiple products, including AlloyDB for PostgreSQL, Cloud Firestore, and Google Compute Engine, among others. The affected location was Columbus (us-east5). The company apologizes to those affected by the disruption.

UPDATE: Customers are experiencing connectivity issues with multiple Google Cloud services in zone us-east5-c

Incident began at 2025-03-29 12:53 and ended at 2025-03-29 19:15 (all times are US/Pacific). Our engineers are actively working on recovery following a power event in the affected zone. Full recovery is currently expected to take several hours. The impacted services include Cloud Interconnect, Virtual Private Cloud (VPC), Google Compute Engine, Persistent Disk , AlloyDB, Cloud Dataproc, Cloud Dataflow, Cloud Filestore, Identity and Access Management, Cloud SQL , Google Kubernetes Engine, Cloud Composer, BigQuery, Cloud Bigtable and more. We have determined that no other zones (a, b) in the us-east5 region are impacted. We will provide an update by Saturday, 2025-03-29 20:00 US/Pacific with current details. We apologize to all who are affected by the disruption. Affected products: Google Compute Engine Affected locations: Columbus (us-east5)

RESOLVED: Apigee customers may experience unable to login to Admin UI portal.

An incident occurred on March 4, 2025, starting at 12:04 US/Pacific time and ending at 13:40 US/Pacific time. The issue affected Apigee Hybrid, Apigee Edge Public Cloud, and Apigee, and was resolved for all users by 13:30 US/Pacific time. A preliminary analysis suggests that the root cause of the issue was a certificate expiration. The affected products included Apigee, Apigee Edge Public Cloud, and Apigee Hybrid. The issue impacted various locations across the globe, including multiple cities in Asia, Australia, Europe, North America, and South America. The affected locations in Asia included Taiwan, Hong Kong, Tokyo, Osaka, Seoul, Mumbai, Delhi, Singapore, Jakarta, and others. In Europe, the affected locations included Warsaw, Finland, Madrid, Belgium, and several other cities. The issue also impacted locations in North America, including Montréal, Toronto, Mexico, and several cities in the United States. The resolution of the issue was completed by 13:30 US/Pacific time, and users can now access the affected products without any issues. The company has thanked users for their patience while the issue was being resolved, and the incident has been fully addressed.

UPDATE: Apigee customers may experience unable to login to Admin UI portal.

An incident occurred on 2025-03-04, starting at 12:04 and ending at 13:40 US/Pacific time, affecting Apigee customers who were unable to log in to the Admin UI portal. The issue began earlier, at 01:25 US/Pacific, and the engineering team was investigating the cause. The team planned to provide an update by 14:15 US/Pacific with the latest details. The disruption affected various locations worldwide, including multiple cities in Asia, Europe, North America, and South America. The affected products included Apigee, Apigee Edge Public Cloud, and Apigee Hybrid. The issue was specifically related to the Portals Admin UI, which was unable to log in to portals. At the time of the report, there was no available workaround for the issue. The affected locations were widespread, covering numerous cities across different regions. The incident caused inconvenience to all who were affected by the disruption, and an apology was issued to those impacted. The engineering team's investigation and update were expected to provide further clarification on the issue and its resolution.

RESOLVED: Cloud Asset Inventory customers' queries may not return result

Incident began at 2025-02-19 01:14 and ended at 2025-02-19 23:56 (all times are US/Pacific). The issue with Cloud Asset Inventory has been resolved for all affected users as of Wednesday, 2025-02-19 23:20 US/Pacific. We thank you for your patience while we worked on resolving the issue. Affected products: Cloud Asset Inventory Affected locations: Global

UPDATE: Cloud Asset Inventory customers' queries may not return result

Incident began at 2025-02-19 01:14 and ended at 2025-02-19 23:56 (all times are US/Pacific). Summary: Cloud Asset Inventory customers' queries may not return result Description: Mitigation work is currently underway by our engineering team. We do not have an ETA for mitigation at this point. We will provide more information by Thursday, 2025-02-20 03:00 US/Pacific. Diagnosis: Cloud Asset Inventory customers impacted by this issue may see empty result from their QueryAssets queries. Workaround: None at this time Affected products: Cloud Asset Inventory Affected locations: Global

UPDATE: Cloud Asset Inventory customers' queries may not return result

Incident began at 2025-02-19 01:14 (all times are US/Pacific). Summary: Cloud Asset Inventory customers' queries may not return result Description: Mitigation work is currently underway by our engineering team. We do not have an ETA for mitigation at this point. We will provide more information by Wednesday, 2025-02-19 22:00 US/Pacific. Diagnosis: Cloud Asset Inventory customers impacted by this issue may see empty result from their QueryAssets queries. Workaround: None at this time Affected products: Cloud Asset Inventory Affected locations: Global

UPDATE: Cloud Asset Inventory customers' queries may not return result

Incident began at 2025-02-19 01:14 (all times are US/Pacific). Summary: Cloud Asset Inventory customers' queries may not return result Description: Mitigation work is currently underway by our engineering team. We do not have an ETA for mitigation at this point. We will provide more information by Wednesday, 2025-02-19 14:30 US/Pacific. Diagnosis: Cloud Asset Inventory customers impacted by this issue may see empty result from their QueryAssets queries. Workaround: None at this time Affected products: Cloud Asset Inventory Affected locations: Global

UPDATE: Cloud Asset Inventory customers' queries may not return result

Incident began at 2025-02-19 01:14 (all times are US/Pacific). Summary: Cloud Asset Inventory customers' queries may not return result Description: We are experiencing an issue with Cloud Asset Inventory beginning at Wednesday, 2025-02-19 01:14 US/Pacific. Our engineering team continues to investigate the issue. We will provide an update by Wednesday, 2025-02-19 12:00 US/Pacific with current details. We apologize to all who are affected by the disruption. Diagnosis: Cloud Asset Inventory customers impacted by this issue may see empty result from their QueryAssets queries. Workaround: None at this time Affected products: Cloud Asset Inventory Affected locations: Global

RESOLVED: SIEM Dashboards and SOAR Advanced Dashboards for Google Security Operations (SecOps) were unavailable.

On February 18, 2025, between 2:42 PM and 6:40 PM PST, Chronicle Security and Chronicle SOAR experienced an issue that prevented users from accessing dashboards, displaying the error message "An error occurred while loading dashboards." The outage lasted for 3 hours and 58 minutes, affecting customers globally. The root cause was traced to the revocation of associated internal Looker licenses, which were subsequently restored to resolve the issue. Impacted products included Chronicle SOAR and Chronicle Security, while affected locations included Tokyo, Mumbai, Singapore, Sydney, Europe, Belgium, Turin, London, Frankfurt, Netherlands, Zurich, Paris, Doha, Dammam, Tel Aviv, Toronto, São Paulo, Iowa, and South Carolina.

RESOLVED: SIEM Dashboards and SOAR Advanced Dashboards for Google Security Operations (SecOps) were unavailable.

Incident began at 2025-02-18 14:42 and ended at 2025-02-18 18:41 (all times are US/Pacific). The issue with Chronicle Security, Chronicle SOAR has been resolved for all affected users as of Tuesday, 2025-02-18 18:35 US/Pacific. We thank you for your patience while we worked on resolving the issue. Affected products: Chronicle SOAR, Chronicle Security Affected locations: Tokyo (asia-northeast1), Mumbai (asia-south1), Singapore (asia-southeast1), Sydney (australia-southeast1), Multi-region: europe, Belgium (europe-west1), Turin (europe-west12), London (europe-west2), Frankfurt (europe-west3), Netherlands (europe-west4), Zurich (europe-west6), Paris (europe-west9), Doha (me-central1), Dammam (me-central2), Tel Aviv (me-west1), Toronto (northamerica-northeast2), São Paulo (southamerica-east1), Multi-region: us, Iowa (us-central1), South Carolina (us-east1)