Description of outage and impact:

Gradwell were alerted to an incident that a number of customers were experiencing total loss of service on broadband circuits and hosted servers for 3CX instances. It was identified the servers, and the broadband circuits were provided via the same upstream supplier. Gradwell Engineers were fully engaged on the incident which was given a priority one status

Cause & Resolution

Our upstream provider identified FPC (Flexible PIC Concentrator) errors on the providers edge router at one data centre. Traffic was re-routed to an alternate link, bypassing the erroring node. Emergency planned engineering works were scheduled for 3 Oct 00:00 to 4 Oct 04:00, to replace the erroring FPC component.

Root Cause

Erroring FPC component.

Prevention of recurrence

Our upstream provider is providing additional training and feedback to their relevant internal teams in respect of network monitoring platforms and associated alarms and a reviewing their provider Edge traffic threshold alarms.

Please accept our apologies for the loss of services and the impact it has had to your business.

Posted Oct 23, 2024 - 11:11 BST

Resolved

The upstream providers monitoring shows all services have remained stable since restoration earlier.

If you are still experiencing any issues, please call our support team for further investigation.

Please accept our apologies for the inconvenience these issue have caused.

Posted Oct 02, 2024 - 14:12 BST

Monitoring

The upstream provider has advised they have identified a fault condition on a network node which has been resolved and services are restoring and will continue to be closely monitored.

Please accept our apologies for the inconvenience these issue have caused.

Posted Oct 02, 2024 - 12:30 BST

Update

The upstream providers architecture engineers are continuing to investigating the fault and have advised a further update will be available within the next hour.

We would like to apologise for the inconvenience this is causing and appreciate your understanding

Posted Oct 02, 2024 - 12:06 BST

Investigating

Hello,

We're currently investigating an issue with hosted 3CX services and broadband services supplied by one upstream provider.

Please accept our sincere apologies for the inconvenience caused.

We will provide a further update by 12:00

Posted Oct 02, 2024 - 11:30 BST

This incident affected: Connectivity (Fibre line, EFM, ADSL Broadband, FTTC, Phone lines) and Voice & Calls Services (Hosted 3CX Server).