Clui Incident Report - Clui Partial Outage - November 9, 2020

The following is the incident report for the Clui access disruption that occurred on November 9, 2020. We understand this service issue has impacted our valued customers and users and we apologise to everyone who was affected. 

Issue Summary

For approximately x minutes, from x to y, certain Clui users were unable to access the Clui platform, viewing "Your connection is not private" messages with "NET::ERR_CERT_COMMON_NAME_INVALID" error. Approximately 60% of users were affected by the outage. The root cause of this service disruption was due to a mixup in configuration of servers which caused a custom SSL certificate to be served for all learning spaces. 

Background and Actions

SSL certificates are small data files that cryptographically establish an encrypted link between a web server and a browser. This link ensures that all data passed between the web server and browser remain private.

Around 5:45pm AEDT, a new learning space with a custom domain was created in Clui. However, settings were not set correctly for the binding of the new learning space and the result of that was that a certain set of learning spaces were served with a custom SSL certificate which did not match our clui domain. As a result all affected learning spaces were shown an “unsecure” warning in the browser. 

The issue was also reported to Clui by customers who were impacted not long after and the servers were configured so they served the correct SSL certificates after which, the learning spaces behaved as expected. This was resolved within 2 hours of the issue being reported.  Furthermore, we would like to reassure Clui customers that their data was not at risk at any point during the incident.

The Clui Engineering team conducted an internal review and analysis of the November 9 event and are taking the following actions to address the underlying causes of the issue to help prevent recurrence: 

• Updating the process for creating new custom learning spaces to ensure the step to ensure the correct settings are applied for the binding of the new learning space.
• Reviewing monitoring systems to ensure any outages are correctly recorded 
• Reviewing processes for tasks that impact production environment 

Clui is committed to continually and quickly improving our technology and operational processes to prevent service disruptions. We appreciate your patience and apologise for the impact to your organisation. We thank you for your business and continued support.