The big news in bad IT this week is Virgin Blue, who was forced to cancel flights after their airport check-in system failed for 21 hours – as reported on Monday here.
The question we don’t see answered yet is how much this glitch will actually cost the company. Bear in mind this is a large public company that is constantly under scrutiny by press and media. There are, of course, the immediate costs of fixing the glitch, the cost of not getting people onto flights, the cost of hotels for stranded passengers, and then – if we look deeper – the ancillary costs such as stressed employees and damage to the brand.
No doubt the cost of this glitch will be significant; possibly higher than the cost of implementing a more robust IT system in the first place. Unfortunately, hindsight is not a replacement for failover systems. To make it worse, the system was outsourced to a third party who should have invested in robust systems.
Sure, in our SME businesses we can have our systems down, and hardly expect it to make the local paper; let alone the national media. But, this was just short of a day of downtime for Virgin Blue, and the fallout from it has been extensive. I’m sure they had a disaster recovery path planned and tested for this system, hence why the outage was only measured in hours – not days or weeks.
So, how would your business cope should your computer systems fail? Do you know where your recovery systems are, or how long it would take to bring them online? Do you have people who know what to do in the event of a major glitch? Have you checked the quality and capability of third party providers?
Unfortunately, most SME businesses do not have a monitored backup system – if they even back up at all. Even fewer have a structured path to recovery.
Most IT providers that work with SME clients sell backup tools and methods, but very few discuss recovery. Several years ago, I would have argued that this was dangerous, but probably cost justified in small business, as the cost of building a recovery option was high and the manual processes of rebuilding over a few days could still be used. However, in the almost nine years that I’ve been in business, computers have penetrated further and further into businesses and have become entrenched in our communication and production cycles. Today, we are so dependent on our PCs that business simply comes to a standstill when computers fail. This means that we now need a structured recovery solution, despite the cost of building and maintaining it.
The good news is that, in recent months, the cost of having offsite recovery options has come down in price to a point where any sized business can find an affordable solution to the recovery issue.
Examples of the technology options vary enormously, and it’s important to understand why the cheapest offsite backup options may not be right for your business, in terms of recovery cycles. Just having files offsite is no longer good enough. It’s time SME businesses started demanding a recovery time as part of the proposition, including a commitment that data of a certain age will be up and running within a certain amount of time.
Many of the online do-it-yourself repositories take days to get your data back to you. But, there’s currently a revolution in this area of technology, and it’s about to go another step in just a few months. Many large hosting organisation such as PacNet and other ISPs are about to offer businesses the opportunity to store backup solutions in their data centres, and have failover servers on standby for almost instant recovery. At the moment, these services are available from smaller providers, but the quality of the service is less than top notch and the prices are relatively high.
Soon, the quality will improve and the price will drop, as large companies bring competition to this lucrative market. It’s already a huge business in America where broadband is faster and cheaper than it is here (don’t get me started on why we need some form of NBN soon!).
Ultimately, make sure you’re investing not just in backup, but also in a recovery solution that will get your complete systems back up and running in hours – not days or weeks. Then ensure you have them tested, and keep testing to ensure all your systems can be recovered and scripts are updated regularly.
In summary, what we can all learn from Virgin Blue this week is:
1. Have offsite backup that is a full system snapshot.
2. Ensure there are disaster recovery capabilities.
3. Run a test of the recovery.
4. Script the process.
5. Keep testing and updating.
As always, there are multiple levels of solutions and a huge array of price points. If you’re not sure of the level of cover you need, seek a knowledgeable advisor.
elephant in the corner, and get on with building your successful business. Without the risk of $20,000 fines.
Click here to read more IT Systems expert advice.
David Markus is the founder of Combo – the IT services company that ensures IT is never an impediment to growth.
Smart Company is the leading online publication in Australia for free news, information and resources catering to Australia’s entrepreneurs, small and medium business owners and business managers.
