IP Transit

Unscheduled Outage - Megaport Sydney IX - 19-10-17

Gelöst

Incident Resolved

  • Second Megaport connection has been setup and will remain in place while root cause with primary megaport connection worked on.
Beobachtung

Update

  • Work has been completed to balance inbound traffic and levels are looking more normal.
  • Issue will now be monitored for a period to confirm work around is functional

Estimated Time of Repair

  • N/A

Next Update

  • N/A
Beobachtung

Update

  • We've made changes and seeing stability across the Megaport Sydney IX
  • Due to the new port setup our bilat peerings need adjustment
  • Next step is to achieve a better of balance of traffic to help with congestion

Estimated Time of Repair

  • N/A

Next Update

  • 3 hours
Untersuchung

Update

  • Some issues were detected on the new link this morning so we have pulled down the new peering (approx 8am) to mitigate problems
  • Currently investigating

Estimated Time of Repair

  • N/A

Next Update

  • 1 hours
Beobachtung

Update

  • The work around Sydney IX connection has been implemented
  • AKL IX peering still to be setup
  • Currently monitoring traffic levels to see if anything else is required however this change should elevate the symptoms

Estimated Time of Repair

  • N/A

Next Update

  • N/A
Aktualisiert

Update

  • IXs will be getting deployed now at 5pm NZ time

Estimated Time of Repair

  • 4 hours

Next Update

  • 3 hours
Aktualisiert

Update

  • New IX connections will be getting setup at 3pm today
  • Customers will see traffic from Megaport AKL IX also go via different path as well during the setup phase

Estimated Time of Repair

  • 3 hours

Next Update

  • 2 hours
Problem identifiziert

Update

  • Routing has been adjusted to push all traffic away from Megaport Sydney IX onto other paths
  • Provisioning of new port is in progress

Estimated Time of Repair

  • N/A

Next Update

  • 1 hour
Bewertet

Summary

  • Over the last couple of days we have seen some intermittent issues with traffic being delivered to us across the Megaport Sydney IX fabric.
  • After working with Megaport the issue has been traced to and ARP \ MAC learning issue inside their environment.
  • As part of steps to resolve the issue their vendor Brocade has been involved
  • As a work around we are going to establish a new megaport connection at a different location
  • To prepare for this we are going to adjust routing so the Sydney IX is not used. Traffic will fail over to the Vibe international path.

Date \ Time

  • 19/10/17

Service Impact

  • IP Transit

Customer Impact

  • Customers may see slower than normal international speeds while we are in a degraded state (i.e. not all international capacity available)
  • When we migrate to the new megaport customers will also see the Mega Auckland IX path disappear for a short period - traffic will re-route from AKL-IX

Workaround

  • N/A

Estimated Time of Repair

  • The goal is to have the new megaport connection stood up tonight. The main dependency is physical cross connects - work is underway to confirm this.

Next Update

  • 1 hour