Software bug contributed to blackout kevin poulsen, securityfocus 20040211. Every so often i teach an electrical engineering class, and towards the end we discuss engineering ethics. Information primarily taken from nercs technical analysis of the system collapse. For more public eye satellite imagery, see the picture of the week gallery of satellite and other imagery of places in the news. System blackout causes and cures by damir novosel 106. The great northeast blackout of november 9, 1965 began at 5. Investigators found several factorsalarm system issues, software bugs, and an overloaded. Canada power system outage task force g august 14th blackout. The blackout that exposed the flaws in the grid the new. Ten years to the day after the great northeast blackout of 2003 caused new york city to go dark.
A softwarerelated failure mode taxonomy bin li,1 ming li,1 ken chen,2 and carol smidts1. The safety relay was set to trip if other protective equipment deeper within the ontario hydro system failed to operate properly. Cause, software bug in the alarm system in the control room of firstenergy. What should have been a manageable local blackout cascaded into the collapse of the entire northeast region. The northeast blackout of 1965 was a significant disruption in the supply of electricity on tuesday, november 9, 1965, affecting parts of ontario in canada and connecticut, massachusetts, new hampshire, new jersey, new york, rhode island, pennsylvania, and vermont in the united states. Similarly, firstenergy fe had a number of software failures in their energy. This causes a transient flashover event as current carried by the line is. The software should have given one system precedent. Most did not get their power back until two days later. One of them was buried in a massive piece of software compiled from. Final report on the august 14, 2003 blackout in the united.
The beckwith electric center for learning offers a variety of classroom seminars covering diverse and complex subjects relating to the electric power field, ranging from industry theory to product specific applications and operation. The northeast blackout struck 15 years ago today extremetech. Feb 11, 2004 turns out after all the various speculation on reasons, the big northeast blackout from last summer was the result of a software bug that was only just discovered and not as many had speculated. The blackout was a telling example of how vulnerable modern infrastructure can be to disruption and a reminder that our sophisticated failure prevention systems are only as effective as the. The multiple failures make the problem worse and worse, and a large area ends up in the dark. Results provided by the risk assessment methodology are used to make decisions. The blackout that occurred on august 14, 2003 in the midwest regions of the united states and the province of ontario in canada was the worst in u. Take a look at the major power outages over many years, and see that the problems we face are not because we arent building enough power plants. The failures occurred when multiple systems trying to access the same information at once got the equivalent of busy signals, he said. A previouslyunknown software flaw in a widelydeployed general electric energy management system contributed to the devastating scope of the august 14th northeastern u. Taking after are 6 famous software disasters in as beneath. After an extensive analysis, we submitted a report to the task force that.
Nov 06, 2011 the blackout was determined to have been caused by a faulty relay at sir adam beck station on the ontario side of niagara falls. Northeast blackout of 2003 your expert root cause analysis. Is our power grid more reliable one year after the blackout. Even if the tripped lines and firstenergys failure to detect and report them prove to be the exclusive initiating cause of the 2003 blackout, it still will be necessary to account for the failure. By thinkreliability staff on august 14, 2003, over 50 million people in the u. Northeast blackout caused by a software bug techdirt. Aug, 2008 the 2003 northeast blackout five years later. The cascading event, which started shortly after 4. Software failure failure modes in medical device software. Many past blackouts, such as northeast blackouts in 1965 and 2003, new york city blackout in 1977, two wecc blackouts in 1996, were caused by cascading failures 5, 6table 1 lists some of the. Software failure synonyms, software failure antonyms.
From electronic voting to online shopping, a significant part of our daily life is mediated by software. Blackout threat unmitigated a decade after the northeast. The northeast blackout of 2003 was a widespread power outage that occurred throughout parts of the northeastern and midwestern united states and the. A narration of the events that lead to the great northeast blackout on august 14th, 2003. Et, 21 power plants shut down in just three minutes. The northeast blackout of 2003 was a widespread power outage throughout parts of the. Prevention remains difficult without methods to calculate the risk of cascading power failures. The massive blackout prompts some companies to turn to providers of disaster recovery services, but it operations at. The northeast blackout of 2003 was a widespread power outage that occurred throughout parts of the northeastern and midwestern united states and the canadian province of ontario on thursday. In this page, i collect a list of wellknown software failures. I was the first journalist to report on a bug in a ge energy management system that resulted in an alarm system failure at firstenergys.
This loss of electricity was the largest blackout in north american history. Cascading blackouts are, by their nature, highly correlated events. Breakthrough computer system reliability and nuclear war. As has been taught in ce370, engineering disasters may often be underestimated due to the fact that the different components of the failure seem initially to be orthogonal and thus, uncorrelated. A major outage knocked out power across the eastern united states and parts of canada on august 14, 2003. The 2003 northeast blackout five years later tougher regulatory measures are in place, but were still a long way from a smart power grid by jr minkel on august, 2008. We examined the public health effects of the northeast blackout of august 2003 and the emergency response to the blackout by the new york city department of health and mental hygiene dohmh. One piece of the system fails, and then the pieces near it cannot handle the increased load caused by the failure, so they fail.
Todd myers, technical advisor, epri transmission research, epri tim mcjunkin, power systems engineer project manager, idaho national laboratory energy systems lab tim vice, transmission maintenance analyst, westar energy eileen lockhart, uas program lead, xcel energy. As the august 14th blackout began, a flaw buried in widelydeployed. Its the system, not a shortage of power plants that is the problem. Great northeast power blackout of 2003 satellite imagery.
I will start with a study of economic cost of software bugs. What will be key challenges for the resilience of the electricity subsector in the future. In addition to status monitoring, they also incorporate sophisticated models to predict the proper response to fault conditions. Ultimately, this blinded miso to the real time failures the grid was experiencing. The reason it took so long to get everything back up again is. The northeast blackout of 2003 in august 2003, a massive power outage occurred in northeastern us and eastern canada. Bornings work with computer professionals for social responsibility resulted in a definitive analysis of the role of computer failure in accidental nuclear war. Fifteen years ago, a software bug resulted in over 50 million people losing. The cause of the power failure is explained in depth by the suburban emergency management project. Computer system reliability and nuclear war alan borning. Pataki reluctantly recalled one of the two major blackouts of the last 40 years in the northeast the 1965 power failure, which left an 80,000squaremile stretch of the united states and. Much larger than the northeast blackout of 1965, in america alone, this blackout affected 45 million people in 8 states. A case study of the 2003 north american power outage key questions what were the main vulnerabilities and threats related to the electricity subsector of the energy sector at the time of the blackout. Technical analysis of the august 14, 2003, blackout nerc.
In nearly every major blackout, the situation is the same. According to the official analysis of the blackout prepared by the us and. The cause of the failure was the setting of a protective relay on one of the transmission lines from the sir adam beck hydroelectric power station no. A software bug at firstenergy corporation in ohio caused this power outage. Notice any opinions, findings, conclusions, or recommendations expressed in this publication do not necessarily reflect the views of the applied technology council atc, the department of homeland security. A single transmission line from the niagara generating station tripped opened. A software bug stalled firstenergys alarm system for more than an hour, leaving. Last saturdays power outage in new york city affected fewer than 100,000 people in a city of 8. A collection of wellknown software failures software systems are pervasive in all aspects of society. The northeast blackout of 2003 was a widespread power outage throughout parts of the northeastern and midwestern united states, and the canadian province of ontario on august 1428, 2003, beginning just after 4. Power system reliability analysis considering protection failures.
System blackout causes and cures by damir novosel 1062003 northeast blackout energy pulse energy central media coverage index library geni global energy network institute. The report makes clear that this blackout could have been prevented and that immediate actions must be taken in both the united states and canada to ensure that our electric system is more reliable. The startup circuits at most of the power plants were designed to be powered from the functioning power grid. When overloaded transmission lines hit untrimmed trees, the alarm did not sound to warn maintenance workers. Software failure cited in august blackout investigation computerworld. It sent a surge of power south, causing power overloads and automatic system shutdowns. New york news coverage of the northeast blackout of 2003. Final report on the august 14th blackout in the united states and canada.
The event started at the ontario new york border, near niagara falls. As a matter of fact, programming bugs can irritate, however, the defective programming can likewise be costly, humiliating, ruinous and savage. Software bug contributed to blackout securityfocus. Ten years later, we are still grappling with concerns over the vulnerability of the power grid.
Probabilistic risk assessment is a methodology to assess the probability of failure or success of a mission. Transmission operations and maintenance conference instructed by. The august 14 blackout affected the northeastern portion of the eastern. A study of the impact of peak demand on increasing. View notes lecture 0 from computer s etc at university of malaysia, sarawak. Software failure cited in august blackout investigation. The 2003 northeast blackoutfive years later scientific american. The blackouts proximate cause was a software bug in the alarm system at the control room of firstenergy, an akron, ohiobased company, which rendered operators unaware of the need to redistribute load after overloaded transmission lines drooped into foliage.
The northeastern blackout 2003 the blackout report. The whole system may have gone down for the reason given in. I bring up the usual examples three mile island, challenger, and the hyatt regency walkway collapse but id like to use an example or two thats more specific to ee rather than a mechanical general design problem. Aug 14, 20 ten years ago today, large portions of the midwest and northeast united states and into canada went dark. Fifty million people were affected, including residents of new york, cleveland and detroit, as well as toronto and ottawa, canada. Associate professor of computer science, university of washington, seattle. The 2003 northeast blackoutfive years later scientific.