This issue has been resolved. No need to restart controllers at this stage. We'll continue to monitor as controllers continue to check-in on their own over the next 2 hours.
Posted Apr 21, 2020 - 18:20 EDT
Update
We continue to monitor and resolve connection issues with the mobile app. Dealers (Pros) and end-Customers who are effected by the outage will experience one or more of the following: 1. Control4 Mobile App Connections will result in a 401 error code, “Unable to Connect”. This is resolved through re-authenticating via the Mobile App login screen. 2. Control4 Mobile App Connects successfully. However, Intercom Anywhere returns a ‘failed wait state’ error message. SnapAV/Control4 engineers will resolve this shortly (within an hour of this update) by pushing a minor Controller update. Controller restarts will resolve this issue but are not required.
Posted Apr 21, 2020 - 15:32 EDT
Update
After the rollback of the authentication token, we had to update some records in the database. Customers will need to re-login to their mobile applications. If customers are still experiencing issues connecting with their projects after around noon MST on Wednesday, we recommend a reboot of their controller to force a renew of their authentication token. If need be, we can force a reboot of controllers remotely. Please let me know if you have any questions. Thank you for your patience as we've worked to resolve this as quickly as possible.
Posted Apr 21, 2020 - 03:47 EDT
Update
We attempted another upgrade this evening, but continued to see issues around authentication with the mobile apps and MyDot. We've rolled back the release again and continue to monitor.
Posted Apr 20, 2020 - 20:47 EDT
Monitoring
Rollback is complete. All devices will need to reathenticate again. Therefore, on mobile devices, customers will need to "Remove System" in Settings and re-enter their credentials. This should only impact users who have tried to authenticate over the last hour since the upgraded was attempted.
Posted Apr 20, 2020 - 14:25 EDT
Identified
Unfortunately, degraded performance especially with remote mobile connections has continued with the upgrade. Therefore, we've decided to rollback the Token Authentication upgrade. This rollback will require all customers to reauthenticate after the rollback is complete--including mobile app users.
Posted Apr 20, 2020 - 14:04 EDT
Investigating
The Cloud Services team is currently upgrading the Token authentication service. All 8 API services have been updated. We are monitoring all affected services for errors. There are some initial slowdowns with all authentication attempts through mobile apps, ComposerPro and other products that utilize this authentication methods. We'll continue to closely monitor and if degraded performance continues, we'll roll back the update.
Posted Apr 20, 2020 - 13:25 EDT
This incident affected: End-Customer Services Health (Service: Control4 Access Anywhere, Service: Control4 APIs).