Brixly - Network Status https://network.brixly.uk/ 2018-06-26T08:03:32+00:00 Brixly - Network Status <![CDATA[Alfa - Hardware Upgrade - Completion]]> https://network.brixly.uk/incident/34/ The server will be rebooted to complete the migration - this will take sites offline for a short period.

Downtime was kept within the estimated period, although delayed slightly due to the scale of the migration.

The work has been ongoing since 11pm last night, however there were some delays unfortunately caused by Cloudlinux licences not being activated immediately and some delayed responses from their support department, which delayed the migration start until 2am.

As such, the migration task overran, however didn't justify a restart of the entire process.

Sites will be online shortly, and services resumed (with the increased performance).

Please avoid contacting support / live chat whilst the process is completed.

]]> 2018-06-20T10:49:00+00:00 Brixly - Network Status <![CDATA[Alfa - Hardware Upgrade - Update]]> https://network.brixly.uk/incident/33/ The hardware upgrade is almost complete.

To finalise the process, we will be switching the IP Addresses over, and re-configuring the network devices. As such, downtime may be intermittent.

Many thanks,

Brixly

]]> 2018-06-20T09:56:00+00:00 Brixly - Network Status <![CDATA[Alfa - Hardware Upgrade]]> https://network.brixly.uk/incident/32/ The migration between hardware is underway - the process should take a few hours longer than expected, however, we will ensure downtime is kept to a minimum on the final switch between the machines.

Further updates will be provided here only, so please avoid raising support enquiries during this phase of the migration.

Many thanks,

Brixly L3 Team

]]> 2018-06-20T01:27:00+00:00 Brixly - Network Status <![CDATA[Server Replacement]]> https://network.brixly.uk/incident/31/ We have worked throughout the night to build a replacement server, whereby we have replaced RAID1 disks for a RAID10 SSD array to improve the IOwait performance.

This is in progress - on completion of the account restorations, we will need to switch IP addresses therefore some indeterminacy should be expected.

UPDATE: This has now completed - the Hotel server is now running on new hardware. Instead of the anticipated 15 - 20 minutes downtime, we managed to get this to around 5 minutes total.

All sites are now online and fully operational.

]]> 2018-05-31T03:35:00+00:00 Brixly - Network Status <![CDATA[Intermittent Downtime - Hotel]]> https://network.brixly.uk/incident/30/ The Hotel server will be rebooted and services restarted at intermittent intervals throughout today

We noticed an iowait spike, which appears to be caused by kswapd, whilst ram / memory is available.

This seems to be a kernel bug, however to attempt to combat this for stability we are adding more ram to the machine.

Update:

We have changed the status of this work to a major outage - we apologise for the inconvenience

Our L3 tech has been despatched to the DC to investigate the issue with the server not booting.

Service will be restored as soon as possible.

Please keep checking here for an update as opposed to raising Support enquiries / chat requests. You as clients have the same amount of information here as our staff on front line support.

Update:

Our team have now booted the machine, and have added a custom script to temporarily resolve an issue until Cloudlinux can check Monday

This has been set to performance issue, as the server is online and mostly stable.

Update:

The server is now stable. We have had our situation prioritised by the Cloudlinux Engineers - they have made some tweaks which have helped stabilising things further from our ram increase.

Changing this to resolved.

]]> 2018-05-26T08:54:00+00:00 Brixly - Network Status <![CDATA[Scheduled Maintenance - Charlie]]> https://network.brixly.uk/incident/29/ Network / Uplink changes and DDoS Routing

We will be making some changes to the Charlie server in order to better prevent any further DDoS attacks on the Charlie server.

Work will be carried out at 3am, 24th May 2018 and is estimated to take approx 30 minutes.

]]> 2018-05-23T11:09:26+00:00 Brixly - Network Status <![CDATA[Kernel Patches / Network Patches Required - Charlie]]> https://network.brixly.uk/incident/28/ IMPORTANT:

We will be running some changes via the BIOS, which will be done at 3am tomorrow morning.

Once complete, we expect all service to be resumed.

]]> 2018-05-23T11:07:00+00:00 Brixly - Network Status <![CDATA[DDoS Attack]]> https://network.brixly.uk/incident/27/ Please note that the team are working tirelessly to combat a DDoS attack on the Charlie server.

Please note connectivity will be intermittent until the mitigation is in place.

]]> 2018-05-23T06:47:00+00:00 Brixly - Network Status <![CDATA[Bravo - Status Update]]> https://network.brixly.uk/incident/25/ The server is back online and all services restored.

]]> 2018-04-30T18:58:00+00:00 Brixly - Network Status <![CDATA[Bravo - Boot Update]]> https://network.brixly.uk/incident/24/ The server is in part back online - we are just running some final changes to bring the server to a stable position.

]]> 2018-04-30T18:15:00+00:00 Brixly - Network Status <![CDATA[Boot Issue - Bravo]]> https://network.brixly.uk/incident/23/ Please note that following a CPU / process issue on the Bravo server, the machine was rebooted. On reboot, the server has entered maintenance mode.

The team are working to resolve this as the highest priority.

Services will be resumed as soon as possible.

]]> 2018-04-30T17:53:00+00:00 Brixly - Network Status <![CDATA[Disk Issues - Alfa]]> https://network.brixly.uk/incident/22/ The server will be rebooting for around 10 minutes due to some required disk / BIOS changes.

** Update - The server reboot has completed, and the service is now fully operational **

]]> 2018-04-30T11:54:00+00:00 Brixly - Network Status <![CDATA[MySQL / Golf - Resolved]]> https://network.brixly.uk/incident/21/ The issue with InnoDB tables has now been resolved

]]> 2018-04-26T01:11:00+00:00 Brixly - Network Status <![CDATA[MySQL Issues - Golf]]> https://network.brixly.uk/incident/20/ The MySQL service is currently up, but isn't responding to requests.

We are investigating this currently and will resume services as soon as possible.

]]> 2018-04-25T23:03:00+00:00 Brixly - Network Status <![CDATA[Bravo - Update]]> https://network.brixly.uk/incident/19/ The server has been restored to a fully functioning state.

]]> 2018-04-25T10:30:00+00:00 Brixly - Network Status <![CDATA[BRAVO - Current connectivity issues]]> https://network.brixly.uk/incident/18/ Currently we are experiencing connectivity issues - the team are investigating.

This issue is currently being caused by a DDoS attack from China.

We have temporarily placed a block on all connections coming from China.

]]> 2018-04-25T09:35:00+00:00 Brixly - Network Status <![CDATA[Urgent Reboot - Bravo]]> https://network.brixly.uk/incident/14/ The Bravo server is currently being rebooted as part of emergency maintenance. Service will be restored within a few minutes.

]]> 2018-04-03T10:05:00+00:00 Brixly - Network Status <![CDATA[Bravo - Reboot Complete - Remounting /home3]]> https://network.brixly.uk/incident/15/ We have rebooted the bravo server successfully, the service is now online and performing as normal.

]]> 2018-04-03T09:44:00+00:00 Brixly - Network Status <![CDATA[Foxtrot - Migration Complete]]> https://network.brixly.uk/incident/13/ Thanks for your patience during the migration process. We have now completed the move from foxtrot to bravo.cloudns.io

If you were using our nameservers, or custom nameservers then you would already be pointing to the new IP address.

If you are using Cloudflare please update your IP to the one displayed in whm -> List Accounts

I'd you have any issues with a particular site following the move then please raise a support ticket so can check in detail for you.

]]> 2018-03-27T01:54:00+00:00 Brixly - Network Status <![CDATA[Foxtrot - Update]]> https://network.brixly.uk/incident/12/ We are pleased to confirm the foxtrot server is back online. We will however be continuing to move the contents of this server to alternative hardware - as such, the previously mentioned details regarding nameserver updates still apply.

The migration process is ongoing, however the server is online and serving content for the duration. This will vastly speed up the migration progress.

Please note that contents of some sites may be partial, or missing data until the process has completed.

Many thanks for your patience.

]]> 2018-03-26T20:06:00+00:00 Brixly - Network Status <![CDATA[Foxtrot - Restore in Progress]]> https://network.brixly.uk/incident/11/ We have began migration from the foxtrot.cloudns.io server to its replacement - an email notification has been sent to all clients affected by the outage with more information.

Please continue to check back here for updates.

As mentioned on the email, migration will be done in two parts - first, the account data (this is to ensure mail services are restored first of all) - then site data secondly.

Whilst we don't have an ETA, we do expect the process will take a few hours to complete due to the amount of data required to be restored.

]]> 2018-03-26T18:23:00+00:00 Brixly - Network Status <![CDATA[Foxtrot - Update]]> https://network.brixly.uk/incident/10/ Apologies for the delay in getting the server back online - there is an issue in booting the machine up which appears to be related to the storage layer / SAN. Of course, we are working on this as high priority.

The team are working on a recovery process currently, whereby all accounts MAY be restored to a separate physical machine. We strongly encourage you set the nameservers for any domains on the foxtrot.cloudns.io server to point to our own nameservers (ns1.cloudns.io / ns2.cloudns.io) or your own custom nameservers resolving to those same IP addresses - this will allow us to migrate accounts fairly easily from our end.

Further updates will be provided here.

]]> 2018-03-26T16:40:00+00:00 Brixly - Network Status <![CDATA[Foxtrot - Outage - Update]]> https://network.brixly.uk/incident/9/ We apologise for the delays in getting the server back online - there seems to be an issue with the reboot, which we are working on currently.

The team would appreciate that tickets were kept to a minimum during this time and questions regarding the downtime will be referred here.

]]> 2018-03-26T15:01:00+00:00 Brixly - Network Status <![CDATA[Outage - Foxtrot]]> https://network.brixly.uk/incident/8/ The foxtrot server became unavailable due to a sudden spike in DDoS traffic - this is being mitigated currently and the service will be resumed as soon as possible.

]]> 2018-03-26T14:15:00+00:00 Brixly - Network Status <![CDATA[Upgrades - Zulu - Complete]]> https://network.brixly.uk/incident/7/ The upgrade process has now completed.

]]> 2018-03-22T15:18:00+00:00 Brixly - Network Status <![CDATA[Upgrades - Zulu]]> https://network.brixly.uk/incident/6/ We are currently running some upgrades on the Zulu server - whilst this is ongoing, you may notice some intermittency in site connectivity.

The process should be complete in around 30 minutes.

]]> 2018-03-22T15:12:00+00:00 Brixly - Network Status <![CDATA[Disk issue -- Resolved]]> https://network.brixly.uk/incident/5/ The issue was resolved with a reboot. The system is now operating normally.

]]> 2018-03-17T15:47:00+00:00 Brixly - Network Status <![CDATA[Disk Issues]]> https://network.brixly.uk/incident/4/ The golf server has had an IO spike caused by a mistake within a script on our end. We are rebooting the machine for a quick resolution.

]]> 2018-03-17T15:36:00+00:00 Brixly - Network Status <![CDATA[Attack mitigated - network stabalised]]> https://network.brixly.uk/incident/2/ The previous reported attack has now been mitigated

]]> 2018-03-08T02:24:21+00:00 Brixly - Network Status <![CDATA[Attack mitigation / network stabalisation]]> https://network.brixly.uk/incident/1/ There seems to be some intermittency with our Albany / USA - cPanel server. From analysis of the logs, it appears to relate to a select few accounts, which have now been suspended.

We will continue to monitor this.

]]> 2018-03-08T01:13:00+00:00