Spotlight on...Sohonet: Trial by fire

3 April 2017

Sohonet is well-known as the leading global expert in connectivity and data management services for the media and entertainment industry. It's probably safe to say that the majority of UK facilities, especially in Soho, would grind to a halt should they be without them. So it's incredible that when every company's worst nightmare, a massive fire, wiped out their office space, Sohonet were up and running at full capacity in 48 hours. Some of their clients didn't even notice.

 

Sohonet's Chief Operating Officer, Damien Carroll (below), tells us more about it and how Sohonet recovered.

Damien Carroll - Sohonet


So what was the sequence of events for this fairly devastating fire? 

Well it took place on 12th September, so actually most of the executive team were still over at IBC in Amsterdam. It was the Monday morning and I'd just got back for the working week on Sunday night, so thankfully I was in the country. It started at around 5am. We monitor all of our customer equipment remotely and so we do the same with our own.

Our alarm system started seeing equipment drop sequentially, one-by-one, as the fire spread. It's an unusual pattern which obviously wasn't a power failure. So we dispatched an engineer to see what was happening and when he arrived there at 6.30am and called me, there were actually 5 fire engines and 27 fire fighters outside trying to tame the blaze on the 4th floor. Obviously, massive amounts of smoke billowing out of the building.

The fire chief actually said they had one near miss with one of their fire fighters, having got into trouble inside and everyone was absolutely covered in black soot. It was probably the worst scenario for any company to see.

I arrived around 7.30am and the priority was obviously to do a recce of whether anyone was hurt or likely to have been in the building. Thankfully that wasn't the case. I then contacted our execs in Amsterdam - actually, at first they thought I was taking the mick and didn't believe me! A couple of responses came through the group text, "Nice try, Damien!". So I just sent a picture of all the fire fighters, which was much more effective!

I was finally allowed in at midday and confronted with just a black shell. Structurally sound still, but the entire air-con system had to be ripped out and the office innards completely cleared. The entire building, not just Sohonet, was kicked out for 2 months due to the smoke damage everywhere.

 

Sohonet 5


What was the plan of action after that?

We enacted our disaster recovery immediately. The first step, which is the most-practised in case of not being able to use the building, was to notify staff about where to work - we had engineers go to Pinewood Studios and we can run the network from an office we have there. The second most important step after ascertaining the safety of our staff is to make sure we can maintain service to our clients, so that hopefully no one notices the problem. Especially first thing on a Monday, which is a busy time! Other staff were told to work from home and we drafted others into a cafe in Soho where we took up some tables and worked remotely, to monitor operations.

Thankfully as we've grown over time, we made some excellent architectural decisions about the company, our network and our protocols. In the early days when we took over Pinewood's network, we knew trying to get engineers out there in good time is very difficult from London, so it was really important to be able to work remotely and it's now inherent in what we do, luckily. If this had happened 10 years ago, when remote networking wasn't common, this would have been much more problematic for maintaining normal operations.

It really shows the importance of having those protocols in place, because the fire chief said we were going to be out of the building for a long time.  That's the point we enacted our full DR (disaster recovery). We started immediately on our internal communication and our staff, thankfully, don't panic. They fight "fires" every day! Our LA staff were brought into the loop asap and they actually went into their office straight away in the early AM and started supporting our London network. The LA office even went into their stores and put together overnight parcels of parts and spares that we may need for customers etc. because we couldn't be certain how long reordering would take.

Then we had to tell our customers. I think it's indicative of the effectiveness of our DR plan that no-one noticed immediately! The only ones that did were the ones in our building, until people started wandering by because they all work in Soho. We put out a very low-key statement on twitter, just because people were starting to see the building and we didn't want to hide anything.

 

After hearing that this would be a long-term issue for the company, what did you do next?

The thing is, with DR for something of this magnitude, you can never fully practise it. It's not just a fire drill. The fundamental part of what a DR for us is that we have systems already in place, that we use every day which can still be used in extenuating circumstances. For example for telephones our systems not only route to internal extensions but also to personal mobile phones at exactly the same time. So decisions we'd made over time such as everyone having a company phone and being given a company laptop may have been more expensive, but were absolutely ideal for this situation.

We also already operate in 27 cities around the world, even if we don't have a physical office in those cities, so the capability to work on problems remotely was also there. 

Almost all of our systems are hosted externally as well; our email, our CRM  database (Customer Relationship Management) is all in the Cloud or in data centres. Anybody sitting at home with a secure private connection could access the company systems just like they would had they been sat at their desks.

 

Sohonet3

So with the success of these systems and this 'trial by fire', has DR planning become something you aid other companies with?

Absolutely. The important thing about all those systems on the DR side and the training for that is to make it part of your daily operations. If you're a small company you don't have the time or the overhead to enact DR planning, so you 'operationalise' it and make it normal. We're really proud to say that we got praise for doing a good job after this fire we were able to say 'it just kicked in, no problems'. Everybody knew exactly what they needed to do and we could expand that over 6 months. We were actually back working at full capacity within 48 hours - by 4.30pm on the same Monday as the fire we had signed off on three offices spaces, for business as usual on Wednesday morning.

Making sure you have logs, you monitor your equipment and working simultaneously with a sympathetic insurance company is vital. We have an advantage as a tech-based company that we're open to adopting and actually being aware of new technologies which can help in this situation, so this is also something we advise clients on now.

Sohonet - New office


So what's next for Sohonet?

Well, not only are we excited to have moved back into our fully refurbished building, but we're currently expanding our workforce quite a bit, with 22 vacancies across the entire company - twelve of which are in the UK. We're doing very well at the moment and we want to maintain our R&D development and customer service.

What's important is that these new recruits and our current staff have the right environment to come in and do their jobs - the fire obviously interrupted this! But as a company it has given us an opportunity for a clean slate and to redesign the office layout and the way we work together as a team. While we do maintain flexibility with how people can work (whether mobile or at home) moving into temporary office space has highlighted the importance of being altogether, being able to collaborate and having a company culture.

 

More on Sohonet