System Outage
Incident Report for x.ai
Postmortem

Public root cause analysis of domain failure on 12.04.17

On the morning of December 4, 2017 our site and email processing systems went down. After some investigation, we realized that another party had taken our domain, though our account was up to date, and we had initiated no changes. After contacting our DNS registrar, we discovered that the .ai NIC (which is based in Anguilla) had mistakenly assigned our domain to a Chinese nameserver. We also learned that Anguilla updates the top level registry manually and that recent hurricanes have severely damaged their infrastructure.

Fortunately, after contacting our registrar and the errant owner of our domain, we recovered it and put our systems back online. Once we were able to get the domain owner to reverse the error, we experienced residual delays as we waited for our DNS change to propagate through the internet (a process that can take up to 24 hours).

Posted 9 months ago. Dec 12, 2017 - 13:51 EST

Resolved
System outage has been resolved and is slowly coming back to full functionality.
Posted 10 months ago. Dec 04, 2017 - 17:51 EST
Monitoring
The root cause of the outage (which was external to x.ai) has been identified and fixed. We are now waiting for all systems to become fully operational.
Posted 10 months ago. Dec 04, 2017 - 15:46 EST
Investigating
We're experiencing a system outage that’s affecting emails to and from Amy & Andrew as well as my.x.ai. We’re on it and working to fix ASAP.
Posted 10 months ago. Dec 04, 2017 - 13:34 EST