In a recent e-book on Enterprise VoIP Security, I included some explanation of process and why they matter. Here's a story I just saw on VoIPNews:

NASA Blunder Leads To VoIP Shutdown
Human error disables all networking and phone service for several hours

Hunter Holcombe

Several hundred NASA employees were forced to rely on personal cell phones and PDAs Wednesday when a new VoIP system shut down at their Washington D.C. headquarters.

The system outage, which occurred at approximately 1:30 p.m., April 12, disabled the phone system and all computer networking for several hours. While technicians were able to reinstate network connectivity by 3 p.m., VoIP phone service did not return until 7:30 p.m.

"The technicians were doing standard configurations and adding new users when the system shut down," said Sonja Alexander, spokesperson for NASA. "But, during the outage, key NASA employees were able to continue working by using their cell phones."

According to an internal NASA memo circulated the following day, the system shutdown occurred when a contracted technician inadvertently deleted the entire NASA Headquarters VoIP user database while adding 19 new VoIP user accounts to the system. Although a safety net is built into the system, and a warning question appeared advising against the action, "the technician answered the question incorrectly," the memo said.

NASA headquarters began installing the VoIP system in January, a project that is expected to be finished sometime this month. Spokesperson Alexander said that, in response to the shutdown, a series of safety measures are being put into place to ensure against future malfunctions, as well as to reduce recovery time should another one take place.

"I know that we are creating additional procedural changes so that, if something happened again, the system would be restored more quickly," she said.

This is a perfect textbook case example of why change control processes need to be developed, implemented. documented and tested.

I did not copy the whole story here to flog NASA. There's no reason to. I also didn't post it to fuel any misconceptions people might have about the resilience of VoIP. VoIP is as resilient and as secure as the IP network it runs on. This isn't about resilience or reliability. It's about people. People can make mistakes. We minimize mistakes by creating rigorous checnge control processes and following them. 

Just as many companies risk not performing a comprehensive readiness assessment prior to implementing VoIP, many companies shortcut process. There's a danger in thinking your existing work habits are good enough. Solid procedures need to be rigorously adhered to. Do the work and review your change controls processes when implementing VoIP so you don't wind up with a needless outage due to human error.


Technorati Tags: Change Control, process, VoIP pain point