tag:status.galaxy.starburst.io,2005:/historyStarburst Galaxy Status - Incident History2024-03-29T05:23:50ZStarburst Galaxytag:status.galaxy.starburst.io,2005:Incident/203858552024-03-28T12:10:15Z2024-03-28T12:10:15ZIntermittent errors running queries<p><small>Mar <var data-var='date'>28</var>, <var data-var='time'>12:10</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved. We will begin to enable auto-suspend in a gradual manner.</p><p><small>Mar <var data-var='date'>28</var>, <var data-var='time'>09:06</var> UTC</small><br><strong>Monitoring</strong> - Cloudflare has reportedly mitigated the issues. We are seeing queries executed successfully from the newly started clusters.</p><p><small>Mar <var data-var='date'>28</var>, <var data-var='time'>08:55</var> UTC</small><br><strong>Update</strong> - We’ve temporarily disabled auto-suspend as a way to mitigate the Cloudflare issue. We apologize for any inconvenience. For real-time updates, you can see Cloudflare status page: https://www.cloudflarestatus.com/</p><p><small>Mar <var data-var='date'>28</var>, <var data-var='time'>08:04</var> UTC</small><br><strong>Update</strong> - We have verified that existing running clusters are not impacted. We apologize for any inconvenience. For real-time updates, you can see Cloudflare status page: https://www.cloudflarestatus.com/</p><p><small>Mar <var data-var='date'>28</var>, <var data-var='time'>07:41</var> UTC</small><br><strong>Identified</strong> - We have identified an issue in the Cloudflare tunnels impacting users’ ability to execute queries. We are working closely with Cloudflare to resolve the issue.</p>tag:status.galaxy.starburst.io,2005:Incident/203658842024-03-26T22:19:48Z2024-03-26T22:19:48ZAWS us-east1 limited functionality impacted<p><small>Mar <var data-var='date'>26</var>, <var data-var='time'>22:19</var> UTC</small><br><strong>Resolved</strong> - The AWS outage leading to these cluster issues was resolved as of 2:45 PM PDT. Errors in Galaxy stopped at the same time as AWS indicates their outage was resolved. We will continue to monitor.</p><p><small>Mar <var data-var='date'>26</var>, <var data-var='time'>21:37</var> UTC</small><br><strong>Identified</strong> - We have identified the root cause of the ongoing issue with starting and scaling Galaxy clusters in the AWS us-east1 region. It is due to a current outage in AWS services. See https://health.aws.amazon.com/health/status for more details.</p><p><small>Mar <var data-var='date'>26</var>, <var data-var='time'>21:02</var> UTC</small><br><strong>Investigating</strong> - We have identified an issue in the AWS us-east1 region impacting users' ability to start or scale Galaxy clusters. We are working closely with AWS to resolve the issue.</p>tag:status.galaxy.starburst.io,2005:Incident/203529812024-03-25T20:30:02Z2024-03-25T20:30:02ZElevated error rates in query execution<p><small>Mar <var data-var='date'>25</var>, <var data-var='time'>20:30</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Mar <var data-var='date'>25</var>, <var data-var='time'>19:55</var> UTC</small><br><strong>Monitoring</strong> - A fix has been deployed. We are continuing to monitor the service.</p><p><small>Mar <var data-var='date'>25</var>, <var data-var='time'>19:08</var> UTC</small><br><strong>Identified</strong> - We identified the root cause and are actively working to implement a fix.</p><p><small>Mar <var data-var='date'>25</var>, <var data-var='time'>17:44</var> UTC</small><br><strong>Investigating</strong> - We are investigating elevated error rates which may impact the ability to run queries.</p>tag:status.galaxy.starburst.io,2005:Incident/200999442024-02-28T20:37:16Z2024-02-28T20:37:16ZGalaxy UI outage<p><small>Feb <var data-var='date'>28</var>, <var data-var='time'>20:37</var> UTC</small><br><strong>Resolved</strong> - The incident has been resolved. We updated configuration in the backend to be able to handle higher traffic.</p><p><small>Feb <var data-var='date'>28</var>, <var data-var='time'>19:27</var> UTC</small><br><strong>Monitoring</strong> - A mitigation has been put in place and we are continuing to monitor</p><p><small>Feb <var data-var='date'>28</var>, <var data-var='time'>19:09</var> UTC</small><br><strong>Update</strong> - Galaxy UI is back up - we are continuing to investigate to find the root cause and monitoring the situation closely.</p><p><small>Feb <var data-var='date'>28</var>, <var data-var='time'>19:04</var> UTC</small><br><strong>Update</strong> - We are continuing to investigate this issue.</p><p><small>Feb <var data-var='date'>28</var>, <var data-var='time'>18:56</var> UTC</small><br><strong>Investigating</strong> - We are currently investigating an issue causing the Galaxy UI to not load</p>tag:status.galaxy.starburst.io,2005:Incident/200897072024-02-27T22:02:20Z2024-02-27T22:02:20ZIntermittent errors running queries<p><small>Feb <var data-var='date'>27</var>, <var data-var='time'>22:02</var> UTC</small><br><strong>Resolved</strong> - The issue has been resolved. We will continue to monitor the system</p><p><small>Feb <var data-var='date'>27</var>, <var data-var='time'>18:46</var> UTC</small><br><strong>Monitoring</strong> - We have implemented a fix and are monitoring the situation for any further issues</p><p><small>Feb <var data-var='date'>27</var>, <var data-var='time'>17:52</var> UTC</small><br><strong>Identified</strong> - We have identified the root cause as a rollback failing in us-east-1. We're now implementing a fix</p><p><small>Feb <var data-var='date'>27</var>, <var data-var='time'>17:29</var> UTC</small><br><strong>Investigating</strong> - We are currently investigating an issue that's causing intermittent errors running queries for some customers.</p>tag:status.galaxy.starburst.io,2005:Incident/200874242024-02-27T15:12:22Z2024-02-27T15:12:22ZGalaxy platform service disruption<p><small>Feb <var data-var='date'>27</var>, <var data-var='time'>15:12</var> UTC</small><br><strong>Resolved</strong> - Normal service has been restored</p><p><small>Feb <var data-var='date'>27</var>, <var data-var='time'>15:04</var> UTC</small><br><strong>Monitoring</strong> - Services are slowly recovering</p><p><small>Feb <var data-var='date'>27</var>, <var data-var='time'>14:30</var> UTC</small><br><strong>Identified</strong> - We are still working to restore service. We hope to have a solution with 1 hour.</p><p><small>Feb <var data-var='date'>27</var>, <var data-var='time'>13:01</var> UTC</small><br><strong>Update</strong> - We are implementing a fix to restore service in the next 1-2 hours</p><p><small>Feb <var data-var='date'>27</var>, <var data-var='time'>12:29</var> UTC</small><br><strong>Update</strong> - We are continuing to investigate this issue.</p><p><small>Feb <var data-var='date'>27</var>, <var data-var='time'>12:26</var> UTC</small><br><strong>Investigating</strong> - We are currently investigating a platform service disruption. You may be unable to execute any operations. We are investigating and will provide a status update as soon as available</p>tag:status.galaxy.starburst.io,2005:Incident/200525682024-02-26T17:38:39Z2024-02-26T17:38:39ZLooker client timeouts<p><small>Feb <var data-var='date'>26</var>, <var data-var='time'>17:38</var> UTC</small><br><strong>Identified</strong> - We have identified the cause and are working on a fix.</p><p><small>Feb <var data-var='date'>22</var>, <var data-var='time'>22:57</var> UTC</small><br><strong>Investigating</strong> - We are experiencing sporadic timeouts for heavily loaded Looker clients. We are actively investigating this incident and will update once we have more details.</p>tag:status.galaxy.starburst.io,2005:Incident/199525492024-02-12T20:20:21Z2024-02-12T20:20:21ZJobs feature is not visible in the UI<p><small>Feb <var data-var='date'>12</var>, <var data-var='time'>20:20</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Feb <var data-var='date'> 9</var>, <var data-var='time'>23:36</var> UTC</small><br><strong>Investigating</strong> - We are currently experiencing an issue that prevents users from seeing their jobs through the UI. This is only an issue with the monitor and does not affect existing or new jobs. A fix has been identified and it is expected that this issue will be addressed on Monday.</p>tag:status.galaxy.starburst.io,2005:Incident/198977132024-02-06T12:48:56Z2024-02-06T12:48:56ZErrors creating catalogs using Hive metastores<p><small>Feb <var data-var='date'> 6</var>, <var data-var='time'>12:48</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Feb <var data-var='date'> 6</var>, <var data-var='time'>00:18</var> UTC</small><br><strong>Monitoring</strong> - A fix has been deployed and we are monitoring the results.</p><p><small>Feb <var data-var='date'> 3</var>, <var data-var='time'>01:09</var> UTC</small><br><strong>Identified</strong> - We have identified an issue that affects the creation of catalogs using Hive metastores in all clouds. We have found a fix and working on it.</p>tag:status.galaxy.starburst.io,2005:Incident/194312472023-12-13T21:43:17Z2023-12-13T21:43:17ZWe are investigating an issue with clusters failing to start<p><small>Dec <var data-var='date'>13</var>, <var data-var='time'>21:43</var> UTC</small><br><strong>Resolved</strong> - The issue was identified as a misconfiguration in specific accounts. Due to the way we do updates, no customer workloads were impacted.</p><p><small>Dec <var data-var='date'>13</var>, <var data-var='time'>20:23</var> UTC</small><br><strong>Investigating</strong> - We're currently investigating an issue where some Galaxy clusters may fail to start. The issue is not widespread, but we'll post an update as soon as we know more.</p>tag:status.galaxy.starburst.io,2005:Incident/193775442023-12-08T19:26:14Z2023-12-08T19:26:14ZClusters failing to initialize<p><small>Dec <var data-var='date'> 8</var>, <var data-var='time'>19:26</var> UTC</small><br><strong>Resolved</strong> - This incident is resolved.</p><p><small>Dec <var data-var='date'> 8</var>, <var data-var='time'>15:51</var> UTC</small><br><strong>Monitoring</strong> - We have provisioned additional capacity and are seeing improvements to clusters stuck in the starting state. We will continue to monitor our fix.</p><p><small>Dec <var data-var='date'> 8</var>, <var data-var='time'>15:27</var> UTC</small><br><strong>Identified</strong> - We have identified an issue with clusters failing to initialize. We are provisioning more capacity to resolve the issue.</p>tag:status.galaxy.starburst.io,2005:Incident/192517222023-11-30T19:29:52Z2023-11-30T19:29:52ZCloudflare in degraded state<p><small>Nov <var data-var='date'>30</var>, <var data-var='time'>19:29</var> UTC</small><br><strong>Resolved</strong> - Cloudflare has resolved their incident and we have removed the workaround and are running as normal.</p><p><small>Nov <var data-var='date'>30</var>, <var data-var='time'>19:27</var> UTC</small><br><strong>Update</strong> - We are continuing to monitor for any further issues.</p><p><small>Nov <var data-var='date'>30</var>, <var data-var='time'>17:32</var> UTC</small><br><strong>Monitoring</strong> - At 16:31 UTC we identified an issue with starting Galaxy clusters. We determined that the root cause was that Cloudflare was in a degraded state due to DNS propagation. We are implementing a workaround to mitigate the issue and monitoring the situation.</p>tag:status.galaxy.starburst.io,2005:Incident/191953752023-11-23T01:47:25Z2023-11-23T01:47:25ZDegraded Performance<p><small>Nov <var data-var='date'>23</var>, <var data-var='time'>01:47</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Nov <var data-var='date'>23</var>, <var data-var='time'>00:13</var> UTC</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Nov <var data-var='date'>22</var>, <var data-var='time'>22:03</var> UTC</small><br><strong>Identified</strong> - A fix is being tested and if successful will be rolled out to customer environments.</p><p><small>Nov <var data-var='date'>22</var>, <var data-var='time'>16:23</var> UTC</small><br><strong>Update</strong> - We’re currently experiencing degraded performance issues with Galaxy. We’ve potentially identified the root cause and are actively working to restore normal performance levels. We apologize for any inconvenience.</p><p><small>Nov <var data-var='date'>22</var>, <var data-var='time'>14:45</var> UTC</small><br><strong>Investigating</strong> - We’re currently experiencing degraded performance issues with Galaxy. We are currently working to restore normal performance levels. We apologize for any inconvenience.</p>tag:status.galaxy.starburst.io,2005:Incident/190000152023-11-05T20:26:51Z2023-11-05T20:26:51ZIntermittent issues running queries<p><small>Nov <var data-var='date'> 5</var>, <var data-var='time'>20:26</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved. If you still encounter problems, please contact Starburst Galaxy Support.</p><p><small>Nov <var data-var='date'> 4</var>, <var data-var='time'>22:01</var> UTC</small><br><strong>Update</strong> - Cloudflare has reportedly mitigated the issues. We have restored auto suspend for all clusters and are actively monitoring. We expect no disruption to Galaxy. If you encounter problems, please contact Starburst Galaxy Support.</p><p><small>Nov <var data-var='date'> 4</var>, <var data-var='time'>19:49</var> UTC</small><br><strong>Update</strong> - Cloudflare has reportedly mitigated the issues. We are beginning to enable auto suspend at this time in a gradual manner. We expect no disruption to Galaxy. If you encounter problems, please contact Starburst Galaxy Support.</p><p><small>Nov <var data-var='date'> 3</var>, <var data-var='time'>19:24</var> UTC</small><br><strong>Update</strong> - We have identified and implemented a mitigation for the current Cloudflare issues. If you still encounter problems, please contact Starburst Galaxy Support.</p><p><small>Nov <var data-var='date'> 3</var>, <var data-var='time'>18:16</var> UTC</small><br><strong>Update</strong> - We have an issue running queries or starting clusters due to an incident with Cloudflare. Most of running clusters should not be impacted. We've temporarily disabled auto-suspend as a way to mitigate the Cloudflare issue. Clients will be issued credits for excess costs.</p><p><small>Nov <var data-var='date'> 3</var>, <var data-var='time'>18:00</var> UTC</small><br><strong>Identified</strong> - We continue to investigate reports of inability to run queries.</p><p><small>Nov <var data-var='date'> 3</var>, <var data-var='time'>02:53</var> UTC</small><br><strong>Monitoring</strong> - As of now, creating and resuming clusters are working as expected. However we continue to monitor. As a precaution, we've temporarily disabled auto-suspend for running clusters as a way to mitigate the Cloudflare issue. Clients will be issued credits for excess costs.</p><p><small>Nov <var data-var='date'> 2</var>, <var data-var='time'>21:48</var> UTC</small><br><strong>Update</strong> - We have an issue starting or resuming clusters due to an incident with Cloudflare. Currently running clusters however are not affected. We've temporarily disabled auto-suspend as a way to mitigate the Cloudflare issue. Clients will be issued credits for excess costs.</p><p><small>Nov <var data-var='date'> 2</var>, <var data-var='time'>16:12</var> UTC</small><br><strong>Update</strong> - Due to an incident with Cloudflare, we are still experiencing an outage starting or resuming clusters. Currently running clusters are not affected and will not auto-suspend as a way to mitigate the Cloudflare issue. Clients will be issued credits for excess costs.</p><p><small>Nov <var data-var='date'> 2</var>, <var data-var='time'>13:22</var> UTC</small><br><strong>Update</strong> - Due to an incident with Cloudflare, we are experiencing an outage. We apologize for any inconvenience. For real-time updates, you can see their status page: https://www.cloudflarestatus.com/</p><p><small>Nov <var data-var='date'> 2</var>, <var data-var='time'>13:14</var> UTC</small><br><strong>Identified</strong> - Due to an incident in our upstream cloud provider, we are experiencing an outage. We apologize for any inconvenience.</p>tag:status.galaxy.starburst.io,2005:Incident/189846312023-11-01T02:56:03Z2023-11-01T02:56:03ZErrors running queries and starting/resuming Accelerated clusters<p><small>Nov <var data-var='date'> 1</var>, <var data-var='time'>02:56</var> UTC</small><br><strong>Resolved</strong> - The incident has now been resolved for all cluster types.</p><p><small>Nov <var data-var='date'> 1</var>, <var data-var='time'>02:19</var> UTC</small><br><strong>Update</strong> - We have mitigated the issue for Standard and Fault Tolerant clusters. Engineers are working on mitigating the issue for Accelerated clusters.</p><p><small>Nov <var data-var='date'> 1</var>, <var data-var='time'>01:36</var> UTC</small><br><strong>Identified</strong> - The root cause has been identified and the engineers are working on mitigating the issue.</p><p><small>Nov <var data-var='date'> 1</var>, <var data-var='time'>01:11</var> UTC</small><br><strong>Update</strong> - We are continuing to investigate this issue.</p><p><small>Nov <var data-var='date'> 1</var>, <var data-var='time'>01:09</var> UTC</small><br><strong>Investigating</strong> - There have been several reports of errors running queries and of clusters failing to start or resume. We are aware of the issue and currently investigating.</p>tag:status.galaxy.starburst.io,2005:Incident/189465792023-10-27T21:27:09Z2023-10-27T21:27:09ZIntermittent errors running queries<p><small>Oct <var data-var='date'>27</var>, <var data-var='time'>21:27</var> UTC</small><br><strong>Resolved</strong> - The incident has now been resolved.</p><p><small>Oct <var data-var='date'>27</var>, <var data-var='time'>18:28</var> UTC</small><br><strong>Monitoring</strong> - We are currently monitoring the situation, but all systems should be back to operational now.</p><p><small>Oct <var data-var='date'>27</var>, <var data-var='time'>18:26</var> UTC</small><br><strong>Identified</strong> - Root cause has been identified and a fix is being implemented. This was a result of spiking load on our query dispatcher which caused it to restart several times (which resulted in query errors during submission). The query dispatcher has been given more resources</p><p><small>Oct <var data-var='date'>27</var>, <var data-var='time'>15:34</var> UTC</small><br><strong>Investigating</strong> - Some users are experiencing intermittent query failures (with 502 error codes). We are currently investigating the issue to find the root cause</p>tag:status.galaxy.starburst.io,2005:Incident/189156702023-10-26T16:16:44Z2023-10-26T16:16:44ZErrors running queries and starting/resuming clusters<p><small>Oct <var data-var='date'>26</var>, <var data-var='time'>16:16</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Oct <var data-var='date'>26</var>, <var data-var='time'>16:06</var> UTC</small><br><strong>Update</strong> - All Galaxy systems continue to be operational. We are still monitoring this incident until we get an RCA from the cloud provider.</p><p><small>Oct <var data-var='date'>25</var>, <var data-var='time'>22:48</var> UTC</small><br><strong>Update</strong> - We are continuing to monitor for further issues and continuing to engage with the cloud provider to get an RCA and driving to resolution.</p><p><small>Oct <var data-var='date'>25</var>, <var data-var='time'>19:09</var> UTC</small><br><strong>Monitoring</strong> - Our cloud provider has mitigated the incident - we are continuing to monitor the situation, but our services should be back to operational now</p><p><small>Oct <var data-var='date'>25</var>, <var data-var='time'>16:57</var> UTC</small><br><strong>Update</strong> - Our cloud provider has identified the issue and is working on a fix - we are continuing to monitor things on our end and will post updates as systems get operational again</p><p><small>Oct <var data-var='date'>25</var>, <var data-var='time'>15:30</var> UTC</small><br><strong>Identified</strong> - Due to an incident in our upstream cloud provider, we are experiencing a partial outage. Any new or resumed Galaxy clusters may be inaccessible from clients. We are engaged with the cloud provider and are working on resolution with them. Existing running clusters should continue to function.</p><p><small>Oct <var data-var='date'>25</var>, <var data-var='time'>02:54</var> UTC</small><br><strong>Monitoring</strong> - The issue has now been mitigated - we have implemented a workaround by restarting cloudflared in the affected Trino planes. We are continuing to monitor the situation.</p><p><small>Oct <var data-var='date'>25</var>, <var data-var='time'>01:14</var> UTC</small><br><strong>Identified</strong> - The root cause has been identified and the engineers are working on mitigating the issue. This was caused by one of our providers not updating its configuration from the upstream. We are engaging with the provider to resolve the issue at the source</p><p><small>Oct <var data-var='date'>24</var>, <var data-var='time'>19:45</var> UTC</small><br><strong>Update</strong> - We are continuing to investigate this issue.</p><p><small>Oct <var data-var='date'>24</var>, <var data-var='time'>18:16</var> UTC</small><br><strong>Investigating</strong> - There have been several reports of errors running queries, which are failing intermittently and sometimes taking a long time. We are aware of the issue and currently investigating. When a root cause is identified, we'll post more updates.</p>tag:status.galaxy.starburst.io,2005:Incident/186356682023-09-28T22:24:28Z2023-09-28T22:24:28ZDegraded performance when starting or scaling Galaxy clusters in AWS us-east-1<p><small>Sep <var data-var='date'>28</var>, <var data-var='time'>22:24</var> UTC</small><br><strong>Resolved</strong> - We’ve restored Galaxy us-east-1 to full availability.</p><p><small>Sep <var data-var='date'>28</var>, <var data-var='time'>20:30</var> UTC</small><br><strong>Monitoring</strong> - We have restored normal operation in the AWS us-east-1 region through workarounds to mitigate the ongoing AWS issue. We will continue to monitor.</p><p><small>Sep <var data-var='date'>28</var>, <var data-var='time'>20:23</var> UTC</small><br><strong>Update</strong> - We are continuing to investigate this issue.</p><p><small>Sep <var data-var='date'>28</var>, <var data-var='time'>20:05</var> UTC</small><br><strong>Investigating</strong> - We’re currently experiencing degraded performance when starting or scaling Galaxy clusters in the AWS us-east-1 region. The root cause is AWS experiencing errors and increased delays in network propagation for newly launched EC2 instances in that region.</p>tag:status.galaxy.starburst.io,2005:Incident/178353942023-07-12T16:52:26Z2023-07-12T16:52:26ZDegraded performance when deploying Galaxy clusters in us-east-1<p><small>Jul <var data-var='date'>12</var>, <var data-var='time'>16:52</var> UTC</small><br><strong>Resolved</strong> - We’re restored Galaxy us-east-1 to full availability.</p><p><small>Jul <var data-var='date'>12</var>, <var data-var='time'>16:16</var> UTC</small><br><strong>Monitoring</strong> - We have added additional instance types to cover the availability shortage. We are seeing instance availability improve. We will continue to monitor the situation.</p><p><small>Jul <var data-var='date'>12</var>, <var data-var='time'>15:43</var> UTC</small><br><strong>Identified</strong> - We’re currently experiencing degraded performance when deploying Galaxy clusters in us-east-1. The root cause is that the instance types that are used to run Galaxy clusters cannot be acquired from AWS. We are currently assessing options to restore performance. We apologize for any inconvenience. Next update in one hour.</p>tag:status.galaxy.starburst.io,2005:Incident/177748872023-07-05T23:19:54Z2023-07-05T23:19:54ZGalaxy outage<p><small>Jul <var data-var='date'> 5</var>, <var data-var='time'>23:19</var> UTC</small><br><strong>Resolved</strong> - We have restored Galaxy to full availability. We experienced a service outage between 17:13 and 17:29 US Eastern time. During that time, queries would not execute and the Galaxy user interface was unavailable. We apologize for any inconvenience it might have caused.</p><p><small>Jul <var data-var='date'> 5</var>, <var data-var='time'>22:40</var> UTC</small><br><strong>Update</strong> - We have restored Galaxy to full availability. We experienced a service outage between 17:13 and 17:29 US Eastern time. During that time, queries would not execute and the Galaxy user interface was unavailable. We apologize for any inconvenience it might have caused.</p><p><small>Jul <var data-var='date'> 5</var>, <var data-var='time'>22:11</var> UTC</small><br><strong>Monitoring</strong> - Galaxy was unavailable for approximately 15 minutes. A fix has been implemented and we are monitoring the results.</p>tag:status.galaxy.starburst.io,2005:Incident/175865332023-06-15T22:49:25Z2023-06-15T22:49:25ZCustomers are unable to browse catalogs in the Galaxy UI<p><small>Jun <var data-var='date'>15</var>, <var data-var='time'>22:49</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Jun <var data-var='date'>15</var>, <var data-var='time'>21:40</var> UTC</small><br><strong>Monitoring</strong> - A fix has been deployed and we are monitoring results.</p><p><small>Jun <var data-var='date'>15</var>, <var data-var='time'>20:37</var> UTC</small><br><strong>Identified</strong> - We’re experiencing a partial outage with the Galaxy UI. Browsing catalogs are impacted, but querying data is unaffected. We’ve identified the root cause and actively working to restore the service within an hour. We apologize for any inconvenience.</p>tag:status.galaxy.starburst.io,2005:Incident/175639442023-06-13T21:03:18Z2023-06-13T21:03:18ZIntermitent outages due to AWS outage<p><small>Jun <var data-var='date'>13</var>, <var data-var='time'>21:03</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Jun <var data-var='date'>13</var>, <var data-var='time'>21:01</var> UTC</small><br><strong>Update</strong> - We are continuing to monitor for any further issues.</p><p><small>Jun <var data-var='date'>13</var>, <var data-var='time'>21:01</var> UTC</small><br><strong>Update</strong> - We are continuing to monitor for any further issues.</p><p><small>Jun <var data-var='date'>13</var>, <var data-var='time'>20:17</var> UTC</small><br><strong>Update</strong> - We are continuing to monitor for any further issues.</p><p><small>Jun <var data-var='date'>13</var>, <var data-var='time'>20:17</var> UTC</small><br><strong>Update</strong> - We’re experiencing a service outage with Galaxy. The root cause is due to an AWS outage. We apologize for any inconvenience. Customers may be unable to submit queries to their clusters or create new clusters on AWS. The next update will be provided in 1 hour.</p><p><small>Jun <var data-var='date'>13</var>, <var data-var='time'>19:57</var> UTC</small><br><strong>Monitoring</strong> - We are currently experiencing intermittent outages and performance degradation due to an AWS outage. We will continue to monitor the situation and provide updates when the situation meaningfully changes.</p>tag:status.galaxy.starburst.io,2005:Incident/173785352023-05-27T01:14:22Z2023-05-27T01:14:22ZNew Tables Created by Hive Connector After May 25 17:00 UTC With Double Slashes in Location Will Not Be Readable<p><small>May <var data-var='date'>27</var>, <var data-var='time'>01:14</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved and we did not see any current customer impact.</p><p><small>May <var data-var='date'>26</var>, <var data-var='time'>22:12</var> UTC</small><br><strong>Monitoring</strong> - The fix has been deployed to all regions and the engineering team is monitoring the release.</p><p><small>May <var data-var='date'>26</var>, <var data-var='time'>21:04</var> UTC</small><br><strong>Update</strong> - We expect the issue to be fixed by 22:00 UTC.</p><p><small>May <var data-var='date'>26</var>, <var data-var='time'>21:00</var> UTC</small><br><strong>Update</strong> - We are continuing to work on a fix for this issue.</p><p><small>May <var data-var='date'>26</var>, <var data-var='time'>17:31</var> UTC</small><br><strong>Identified</strong> - We have identified the fix which should be rolled out by 22:00 UTC. The corruption should only impact newly created hive tables on AWS s3 with a double slash within the table location when creating a catalog.</p><p><small>May <var data-var='date'>26</var>, <var data-var='time'>16:45</var> UTC</small><br><strong>Update</strong> - We are continuing to investigate this issue.</p><p><small>May <var data-var='date'>26</var>, <var data-var='time'>14:21</var> UTC</small><br><strong>Investigating</strong> - We have identified an issue with Galaxy where new hive tables created can be corrupted on read. This would impact any tables created since May 25th, 17:00 UTC. We are working on a fix and will be applying once its been verified.</p>tag:status.galaxy.starburst.io,2005:Incident/171545322023-05-05T20:15:28Z2023-05-05T20:15:28ZWe are experiencing higher than usual latencies in AWS ap-southeast2 region<p><small>May <var data-var='date'> 5</var>, <var data-var='time'>20:15</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>May <var data-var='date'> 5</var>, <var data-var='time'>14:13</var> UTC</small><br><strong>Identified</strong> - We are working on a fix that should mitigate the problem. The tentative ETA for the fix is 2PM eastern time.</p>tag:status.galaxy.starburst.io,2005:Incident/171532482023-05-05T12:36:42Z2023-05-05T12:36:42ZFree tier clusters aren't available in eu-central1<p><small>May <var data-var='date'> 5</var>, <var data-var='time'>12:36</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>May <var data-var='date'> 5</var>, <var data-var='time'>11:47</var> UTC</small><br><strong>Identified</strong> - The free tier cluster is not able to start in the eu-central1 region.</p>