Intermittent errors running queries

Incident Report for Starburst Galaxy

Resolved

The incident has now been resolved.

Posted Oct 27, 2023 - 21:27 UTC

Monitoring

We are currently monitoring the situation, but all systems should be back to operational now.

Posted Oct 27, 2023 - 18:28 UTC

Identified

Root cause has been identified and a fix is being implemented. This was a result of spiking load on our query dispatcher which caused it to restart several times (which resulted in query errors during submission). The query dispatcher has been given more resources

Posted Oct 27, 2023 - 18:26 UTC

Investigating

Some users are experiencing intermittent query failures (with 502 error codes). We are currently investigating the issue to find the root cause

Posted Oct 27, 2023 - 15:34 UTC

This incident affected: Clusters (AWS) (AWS ap-northeast1, AWS ap-southeast1, AWS ap-southeast2, AWS ca-central1, AWS eu-central1, AWS eu-west1, AWS eu-west2, AWS eu-west3, AWS us-east1, AWS us-east2, AWS us-west1, AWS us-west2), Clusters (Azure) (Azure centralindia, Azure eastus, Azure eastus2, azure francecentral, Azure southcentralus, Azure westeurope), and Clusters (GCP) (GCP asia-south1, GCP asia-southeast1, GCP europe-west2, GCP europe-west6, GCP southamerica-east1, GCP us-central1, GCP us-east1).