We sincerely apologize for the service disruptions which occurred on the MainOne network between Wednesday January 22nd and Saturday January 25th last week. The impact of the outage varied from just a few hours for some of our customers to over two days for others. This disruption in services were caused by circumstances beyond our reasonable control; primarily damage from inclement weather of Storm Gloria in Spain and a submarine cable outage which had occurred on January 17th on the backup route between Seixal, Portugal and London..
While we have provided details of the circumstances below, we want to reassure our customers that such compounded network failures are indeed a rare occurrence and that MainOne is committed to continuing to build diversity into our network so that impact of similar outages in the future can be minimized.
We thank you for your patronage and apologize once again for disruption to your business.
Detailed chronology and root cause of Network Outage:
JAN 22, 6.25PM – Services from the MainOne cable Station in Seixal, Portugal northbound to London were observed down on various monitoring tools. Investigation revealed a partial outage/inability to reach the internet through our major providers i.e. TATA Communications and Century Link in London, with access available through only internet exchange points e.g. DE-CIX, AMIX, GINX, WAFIX, GGC and IXPN, thus making it difficult for IP Access and IP Transit to access the internet. This also affected all our IPLC customers with services terminated or routed through London.
JAN 22, 8.38PM – In response to the outage, we activated additional IP transit capacity in Lisbon where we have a POP and all IP Access services were fully restored. We also initiated restoration of IP Transit customers, however we were capacity constrained. This is because the primary route from Seixal to London had experienced a failure the previous week (Jan 17, 2020 which also affected the WACS cable system), and so that link was not available for failover or partial re-routing of traffic when the Jan 22nd failure occurred.
JAN 23 – We were made to understand that because of severe weather in Spain where the failure occurred, restoration of the damaged links could take some time even though the repair crew was on ground to fix. In addition, we previously had information that the primary link into the UK through Highbridge would not be restored until the week of Feb 4th. Hence we had to find new alternatives. MainOne established a new route from Seixal to enable us source additional IP Transit capacity in Lisbon and this was activated at 10.02PM. With the availability of this capacity, all our IP transit customers were partially restored. In addition, we were able to source STM16 IPLC to London, but this could only partially restore a handful of IPLC customers
JAN 24- At 11.09AM: MainOne IP Transit links were fully restored and IP connectivity to the internet and all IP customers were restored to their full capacity. At 4.08PM, additional restoration IPLC circuits were made available and our IPLC customers were restored to either partial or full capacity.
JAN 25- At 1.05PM: we obtained some alternate capacity and some customers initially restored to partial capacity were restored their full capacity.
JAN 25 – At 5.49PM: the fiber cut on the power line in Spain leading to the outage was resolved and all services thereafter restored to their full capacity.
JAN 27 – 31: While all our customers now have access to the full capacity subscribed from MainOne, some customers remain either fully or partially on restoration links, some of which are longer routes thus resulting in higher latency than expected. Customers are being contacted to arrange for maintenance windows to restore services to their original links which we expect to complete no later than Jan 31, 2020.
Root Cause: Outage was due to fiber cut fiber cut at 37 kilometers from Arañuelo towards Malpartida, Spain as a result of severe weather experienced with Storm Gloria. https://www.dailymail.co.uk/news/article-7920031/Spain-digs-snow-aftermath-Storm-Gloria-killed-eight-people.html and at a time when the alternate route between Seixal, Portugal and London, England was down since January 17, 2020 due to a submarine cable cut which is scheduled for repair the week of February 3, 2020.
Affected fiber cores were finally spliced to restore services at 5.49PM, Saturday January 25, 2020 and network services to MainOne restored.