[Previous entry: "Annies Macaroni and Cheese"] [Main Index] [Next entry: "Type 55 Polaroid"]

08/18/2003 Archived Entry: "Blackout"

I was amused a couple of days ago when I read in a local paper that experts say that a blackout like the US had in the Northeast is unlikely to happen in the Pacific Northwest. My guess is that if anyone were to ask one of these "experts" if the blackout which occurred last week could happen they would have said that it was so unlikely as to not be worth worrying about.

This whole event stinks of being a software problem to me. Things were OK, then got weird. My guess is that there is lots of software used to control the distribution of power. Here is something to ponder. Lets says you develop a system of circuits and software to control electrical power distribution. Can you test it? Can you test it throughly? Are you sure it can run for years and not get weird?

Software of any complexity is difficult to make reliable. As a matter of fact, embedded programmers who make the software which run such things as your microwave oven and dishwasher often use a feature called a "watchdog timer". The concept of a watchdog timer is to avoid having your microwave oven software "lock up". Basically what happens is that if the software appears to go into some sort of endless loop, rather than cooking your burrito at high for two hours, it restarts the microcontroller. Better a cold burrito than a burned-out oven, or burned down house. Very likely the software in the machines around you often malfunction, but you never realize it.

But really complex systems of software can't be handled so simply. There might be thousands of programs interacting in various ways, directly and indirectly, and for something like a power grid, they need to work 24/7. You can't have someone calling people and saying, "Hello, we're going to be shutting off the power of a couple of hours to upgrade our system" without having lots of upset taxpayers.

This is not because programmers are stupid or sloppy, but because bulletproof programming is hard, and making systems work together is even harder. NASA has lost well tested space probes because of things like this.

The sad thing is that if this weird blackout was due to a software bug, perhaps from an odd interaction between systems of software, it will be very difficult to pin down. As a matter of fact, maybe it will never be really understood.

I hope the failure of the Northeast's power grid is not just attributed to aging technology. Over zealous upgrading will likely make things worse. Long turn R&D into the design of very reliable programming methodologies might be a better idea.

Entries and Images Copywrite M. Stewart