Data service degradation in the Americas region
Incident Report for emnify
Postmortem

For a period of 4 hours (08:05 - 12:06 UTC), around a dozen network operators lost network connectivity to emnify. As a result, devices attached to these networks could not establish data connectivity. The affected operators were mostly located in the Americas region and, based on our analysis, share a common carrier.

Since the source of the problem was outside of emnify’s network, we are working with our two contracted carriers to understand the root cause. One of our carriers has acknowledged problems on a router in Dallas around the time of the incident, which might be the handover point for the affected traffic. We are conducting ongoing analysis with our carriers to determine why the failover of traffic to another network path did not occur. Since emnify, like the affected operators, has redundant connectivity in two US locations with each of our two carriers, such cases are not anticipated.

We will update this document with additional information when available.

Posted Sep 06, 2024 - 13:52 UTC

Resolved
This incident has been resolved and affected operators are operational again.

In case you are still experiencing issues, please submit a request to our Customer Support team.
Posted Sep 05, 2024 - 13:19 UTC
Monitoring
We are now starting to see a recovery and affected devices are now reconnecting since 12:06 UTC.

We will continue to closely monitor the situation.
Posted Sep 05, 2024 - 12:31 UTC
Update
We are continuing to investigate the issue. We are communicating with our carriers and pushing them for an update.

We will post an update shortly.
Posted Sep 05, 2024 - 11:54 UTC
Update
We are continuing to investigate the problem in the Americas which seems to be caused by the signaling carrier of the affected networks.
Meanwhile, some devices attaching to our network via LTE from other countries could have experienced connectivity issues.

We have meanwhile made some adjustments from our side which should help with the new connections outside the Americas region.

We will post a new update shortly.
Posted Sep 05, 2024 - 09:56 UTC
Investigating
Dear customers,

We are currently investigating a possible service degradation in some American countries. We will publish a new update shortly.
Posted Sep 05, 2024 - 08:57 UTC
This incident affected: Network Connectivity (Network Connectivity - North America, Network Connectivity - Central and South America & Caribbean).