tag:status.zdscada.com,2005:/historyzdSCADA Status - Incident History2024-03-29T00:23:19-05:00zdSCADAtag:status.zdscada.com,2005:Incident/200465762024-02-22T13:39:11-06:002024-02-22T13:39:11-06:00Nationwide partial AT&T outage<p><small>Feb <var data-var='date'>22</var>, <var data-var='time'>13:39</var> CST</small><br><strong>Resolved</strong> - The AT&T outage appears to be resolved or mostly resolved. Please check https://about.att.com/pages/network-update for future updates. <br /><br />Closing this zdSCADA incident thread.</p><p><small>Feb <var data-var='date'>22</var>, <var data-var='time'>10:27</var> CST</small><br><strong>Update</strong> - We continue to monitor the outage. AT&T has issued the following statement:<br /><br />“Some of our customers are experiencing wireless service interruptions this morning. We are working urgently to restore service to them. We encourage the use of Wi-Fi calling until service is restored”<br /><br />About 95% of the AT&T cellular devices on zdSCADA have resumed communicating. <br /><br />If your cell phone is AT&T and you do not have service, you can still use the "My Alarms" page from the app (if you have Wifi) to check the status of your zdSCADA alarms.</p><p><small>Feb <var data-var='date'>22</var>, <var data-var='time'>05:39</var> CST</small><br><strong>Monitoring</strong> - We are seeing AT&T locations slowly coming back online. Approximately 70% are online. This appears to be a nationwide issue with the AT&T service.</p><p><small>Feb <var data-var='date'>22</var>, <var data-var='time'>04:21</var> CST</small><br><strong>Investigating</strong> - Many locations in zdSCADA that are connecting via AT&T cellular modem are experiencing an outage. <br /><br />It appears to be AT&T issue and not limited to zdSCADA. Users with AT&T cell phones may not be able to receive text and phone call alarms. <br /><br />We are continuing to investigate the issue.</p>tag:status.zdscada.com,2005:Incident/189397772023-10-26T21:06:07-05:002023-10-26T21:06:07-05:00SMS Delivery Delays<p><small>Oct <var data-var='date'>26</var>, <var data-var='time'>21:06</var> CDT</small><br><strong>Resolved</strong> - SMS delivery has been stable since the last update. The downstream providers report the issue has been resolved.</p><p><small>Oct <var data-var='date'>26</var>, <var data-var='time'>18:54</var> CDT</small><br><strong>Monitoring</strong> - We are seeing that SMS messages are resuming. We are continuing to monitor our downstream SMS providers.</p><p><small>Oct <var data-var='date'>26</var>, <var data-var='time'>18:39</var> CDT</small><br><strong>Investigating</strong> - We are currently experiencing an issue with delivery of text messages alarms. Our SMS vendor is aware of the issue. Phone call and email alerts are working normally. If you have urgent alarms that are only configured for text messaging, Please periodically check your "My Alarms" page in the zdSCADA app until the issue is resolved.<br />Twilio status page: https://status.twilio.com/<br />Bandwidth status page: https://status.bandwidth.com/</p>tag:status.zdscada.com,2005:Incident/182982142023-08-31T09:49:56-05:002023-08-31T09:49:56-05:00Partial Cellular Outage<p><small>Aug <var data-var='date'>31</var>, <var data-var='time'>09:49</var> CDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Aug <var data-var='date'>28</var>, <var data-var='time'>22:55</var> CDT</small><br><strong>Monitoring</strong> - We continues to observe service outage on ~180 modems on the PODs cellular network.<br /><br />Since this issue only affects a subset of zdSCADA customers, future updates will only be sent to the users that are affected.<br /><br />Here is the latest update from the cellular service provider:<br />We have been advised by the network engineers that the solution put in place did not have the desired outcome. They are still working to resolve this issue.</p><p><small>Aug <var data-var='date'>28</var>, <var data-var='time'>20:39</var> CDT</small><br><strong>Update</strong> - zdSCADA continues to see service outage on ~180 modems on the PODs cellular network.<br /><br />Here is the latest update from the cellular service provider:<br /> Engineers have found the root cause of the issue and together with the Signalling and Core Team are working to find a solution to fix the issue. We will provide another update as soon as we receive it from the network.</p><p><small>Aug <var data-var='date'>28</var>, <var data-var='time'>17:29</var> CDT</small><br><strong>Update</strong> - We are starting to see recovery on the affected modems. Approximately 1/3rd of them have come back online over the last two hours. The cellular provider is still working on resolving the issue completely.</p><p><small>Aug <var data-var='date'>28</var>, <var data-var='time'>15:42</var> CDT</small><br><strong>Identified</strong> - We are experiencing a partial service outage with our PODS cellular service provider. <br /><br />The provider is aware of the outage and is working on restoring service.<br /><br />This is limited to about 5% of all zdSCADA modems, all with IP addresses in the range 10.49.0.0 - 10.49.255.255. <br /><br />We are monitoring the situation and will provide updates when they become available.</p>tag:status.zdscada.com,2005:Incident/174928222023-06-07T06:00:23-05:002023-06-07T06:00:23-05:00zdSCADA Scheduled Maintenance<p><small>Jun <var data-var='date'> 7</var>, <var data-var='time'>06:00</var> CDT</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Jun <var data-var='date'> 7</var>, <var data-var='time'>05:31</var> CDT</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Jun <var data-var='date'> 6</var>, <var data-var='time'>14:13</var> CDT</small><br><strong>Scheduled</strong> - We will be undergoing scheduled maintenance during this time. There will be a brief period where the site will be inaccessible around 5:40AM Central Time, planned for less than 5 minutes.</p>tag:status.zdscada.com,2005:Incident/174239612023-05-30T19:31:14-05:002023-05-30T19:31:14-05:00zdSCADA Database Capacity Increase<p><small>May <var data-var='date'>30</var>, <var data-var='time'>19:31</var> CDT</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>May <var data-var='date'>30</var>, <var data-var='time'>18:30</var> CDT</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>May <var data-var='date'>30</var>, <var data-var='time'>13:28</var> CDT</small><br><strong>Scheduled</strong> - We will be undergoing scheduled maintenance during this time. There will be a brief period where the site will be inaccessible around 6:40PM Central Time, planned for less than 5 minutes.</p>tag:status.zdscada.com,2005:Incident/167523682023-04-04T16:17:29-05:002023-04-04T16:17:29-05:00SMS Alarm delivery issue to Verizon handsets<p><small>Apr <var data-var='date'> 4</var>, <var data-var='time'>16:17</var> CDT</small><br><strong>Resolved</strong> - Our downstream partner is reporting the incident is resolved, and we are seeing normal delivery times for Verizon SMS.</p><p><small>Apr <var data-var='date'> 4</var>, <var data-var='time'>15:30</var> CDT</small><br><strong>Monitoring</strong> - SMS messages are starting to be delivered to Verizon handsets. It may take some time for all messages to be delivered that were queued during the outage. We will continue to monitor and update when SMS traffic is normal again.</p><p><small>Apr <var data-var='date'> 4</var>, <var data-var='time'>13:46</var> CDT</small><br><strong>Investigating</strong> - A downstream carrier issue is preventing Toll-Free SMS delivery to some Verizon handsets. Voice alarms are unaffected. We are continuing the monitor the situation and will provide updates as the situation resolves.</p>