Hazard Notice - Office365 Delivery - 19/11/19 3:45pm ongoing

Resolved

Incident Resolved

  • Antispam mail queues are now clear, and all mail is flowing normally
Updated

Update

  • We can confirm that this incident was caused by a change by Microsoft to their Office 365 email platform
  • Microsoft incident ID EX196121, Exchange Online
  • Text of their notice
    Users that were receiving emails from external users may have experienced delays. Additionally, we’ve also confirmed that delivery of emails with attachments sent to internal endpoints may have been delayed. During this time, mail will have been queued by the send and is expected to be delivered normally once the situation is resolved.
    Final status: We’ve rolled back the change and can confirm that service is restored.
    Scope of impact: This issue could affect any mail user capable of receiving mail from external users, or internal mail with included attachments.
    Start time: Tuesday, November 19, 2019, 2:40 PM (1:40 AM UTC)
    End time: Wednesday, November 20, 2019, 12:50 AM (11/19/2019, 11:50 AM UTC)
    Preliminary root cause: A configuration change to our infrastructure caused unexpected impact to mail flow.

Next Update

  • Later today we will update this notice when all queued mail is confirmed to be delivered
Updated

Update

  • Mail delivery to Office365 now appears to be flowing again with no delay. However there is still a large number of emails queued on our antispam cluster destined for domain hosted on 3rd party mail system, which we are working to clear.
    This does not appear have any noticeable impact on regular mailflow, and we are keeping it under observation

Estimated Time of Repair

  • This is a Hazard Notice only

Next Update

  • N/A
Assessed

Summary

  • It appears that there a delay in mail delivery to Office365 mailboxes. This is being seen across our 3 outbound email servers, as well as the Antispam cluster

Notice Period

  • 19 November 2019 3:45pm - ongoing

Service Impact

  • Mail delivery to Office365 mailboxes

Customer Impact

  • Users with mailboxes hosted on Office365 may see delays in receiving email from external parties
2 Affected Services: