We apologize for any inconvenience caused by the temporary shutdown of the Tendermint core node, as it may lead to some degradation on the mainnet. We are working closely with them to ensure a seamless restoration of services. In the meantime, developers can make use of alternative networks.
Hi! Do you have a time of resolution? Thanks!!
We’re navigating to the folks at Tendermint Core as they’ve gone through several org restructures.
We hope to resolve this soon, but can’t give a final estimate just yet.
Hey Everyone,
The Cosmos Node is fixed, and key reconstruction on the ‘mainnet’ network is working as expected.
Thanks a lot for your patience. We hope to provide uninterrupted services moving forward.
Post Mortem Report
Summary
On 23rd May, starting at 5:32am GMT +8, our Cosmos Node started returning 404 errors across the board, which resulted in degradation of our infrastructure services. This issue lasted approximately 43 hours 43 minutes, until 25th May 1:15 AM, when the affected node was restored to full operation.
Timeline
- 23rd May 5:32 AM GMT+8: The first instance of the 404 errors was detected on Cosmos Node.
- 23rd May 6:10 AM GMT+8: We began to reach out via multiple communication channels to understand the issue.
- 25th May 1:15 AM GMT+8: The Cosmos Node was confirmed to be fully restored and operational.
Root Cause Analysis
Following an in-depth investigation, it was determined that the downtime was due to cosmos.network domain shifting ownership. Consequently, this interrupted the DNS resolution and resulted in the node being unable to serve requests, thus leading to the 404 errors. It is important to note that despite this, Tendermint continued to sync throughout this period.
Impact
As Cosmos Node was one of nine nodes in operation in our infrastructure, the incident created a significant disruption in our services.
Mitigation and Prevention
We restored the service by resolving the DNS issue and ensuring the Cosmos Node could successfully respond to requests.
We deeply regret the inconvenience caused during this period and would like to assure our users that we are taking strong measures to prevent such incidents in the future. We appreciate your patience and understanding.