Thursday, July 16, 2015

What can a natural disaster teach us about incident response?

During the overnight hours of Saturday night and Sunday morning May 23-24, heavy rain in the Texas Hill Country triggered a flash flood of near-Biblical proportions in Wimberley and San Marcos. This is the story of a disaster response program executed exceptionally well.

During the overnight hours of Saturday night and Sunday morning May 23-24, heavy rain in the Texas Hill Country triggered a flash flood of near-Biblical proportions in Wimberley and San Marcos. This is the story of a disaster response program executed exceptionally well.

Flooding rains are not uncommon in Central Texas. The region has long been known as "Flash Flood Alley" due to its hilly terrain, shallow soils, and proximity to the moisture-laden Gulf of Mexico. When rain falls, it essentially has two options: soak into the soil, or flow downhill; the shallow and rocky soils of the Hill Country limit the former, so even a moderate rain causes runoff.

The weekend of Memorial Day 2015, however, was something else entirely. Over a period of a few hours, between nine and 12.5 inches of rain fell over a wide range of the Hill Country - much of which fell within the watersheds of the Blanco and San Marcos rivers. A foot of rainfall - a third of a typical year's total - inundated the region in just a few hours, and had to go somewhere.


This article first appeared in
CSOonline
The result was a catastrophic flash flood. The Blanco River rose 17 feet in a half hour, and 33 feet in a three-hour span, peaking far higher than had ever been recorded or even thought possible. Towering cypress trees that had survived six centuries of drought and flood were no match for this monstrous storm. In the lead photo to this story, massive cypress trees have been stripped of bark and branches 30 feet above the normally-placid river's surface.

Ultimately, a 40-foot wall of water rushed downstream, scouring away everything in its path: trees, vehicles, homes, bridges, and unfortunately, people.

Shortly afterward I wrote of the amazing resilience embedded in Texas culture. Texas has a long history of neighbors helping neighbors, and that culture showed itself after this disaster. From the many volunteers searching for the missing, to local businesses offering to replace vital necessities, to an impromptu clearinghouse to lend and borrow heavy equipment, watching the community set about the business of recovering has been an inspiration.

Individual resilience is not enough in the face of a disaster of this magnitude though. Rebuilding hundreds of homes, roadway infrastructure, communications, and the other essentials of modern life requires a coordinated effort. Whether cyber or physical, some lessons apply in any disaster.



Lesson 1: Preparedness


In the midst of a crisis an organization falls back on its planning and preparedness. When an incident occurs, it is too late to put together a response plan. Flash floods are a known threat in Central Texas, and the region has several initiatives to address this threat.

The City of Austin Flood Early Warning System reports the current state of over 1,000 "low water crossings" - often little more than a roadway with a culvert to allow a creek to pass beneath; during heavy rain, these crossings will frequently be temporarily impassable.

The counties have spent years building awareness: 18 to 24 inches of moving water is enough to sweep most vehicles off the road. The saying "Turn Around, Don't Drown" is ingrained in the minds of residents.

Ten counties in the greater Austin area participate in a Regional Notification System, whereby residents and interested parties can register their landlines and cell phones to receive notification of threats to life or property.

Hays County has a long-standing volunteer Community Emergency Response Team trained to respond to wildfires, tornadoes, car wrecks, and flash floods.

Hays County has set up haysinformed.com as a central location for authoritative information during an emergency.

All of these steps required time to plan and to implement - and all were in place long before this crisis arose.



Lesson 2: Damage control


During an incident, damage control is the first rule. In a cyber event, you may be able to isolate the compromised environment to prevent further damage. You don't contain 10 billion gallons of rushing water though, so you do the next best thing: get out of the way.

Amazingly, in a flash flood that destroyed 320 homes during the middle of the night, only 12 individuals were swept away. One was rescued the following morning and is alive today. Nine have been recovered deceased. Tragically, the two remaining unaccounted for are a 6-year-old boy, and the 4-year-old daughter of the rescued survivor.

In this event, "damage control" took a multi-pronged approach. The National Weather Service had issued a flash flood watch early Saturday, but the situation did not become critical until around midnight Saturday night/Sunday morning. The NWS archive records the increasingly dire warnings as the reality of this event unfolded. This leads to lesson three...



Lesson 3: Communication is vital


During a crisis, it is critical to be able to communicate with those in harm's way, as well as with the response team. Depending on the nature of the emergency, traditional communication channels may not be available. A cyber event can interfere with email. An IT infrastructure failure can render VoIP telephone systems useless. A natural disaster may disrupt landline telephones or cell phones. It is important therefore that the disaster response plan include alternate communication channels.

In this case, the county utilized "geo-fencing" to send evacuation notices to every cell phone within a defined danger zone. In addition county emergency responders traveled door-to-door waking up residents and warning them to evacuate. In addition, many Central Texas residents have weather radios - radios tuned into dedicated frequencies operated by the National Oceanic and Atmospheric Administration to issue alerts for weather and other hazards.

After the immediate evacuation, radio became an important communication vehicle as cell phone service was temporarily disrupted. Local radio station Wimberley Valley Radio is an Internet radio station planning to add a local transmitter next summer; they obtained an FCC license to broadcast over the air immediately for 30 days to communicate to the area.



Lesson 4: Response and recovery have different characteristics


Communication during the crisis differs from communication during recovery. During the crisis, the emphasis is on reaching everyone as quickly as possible, through as many channels as possible, and informing them of the urgency to get out now. Recovery is a very long process during which communication can get tedious. This phase requires discipline and a plan for the long haul.

Two bridges were completely demolished. Asphalt was stripped from other roads. Several low water crossings have been damaged - or are still submerged and cannot be evaluated yet. Hundreds of homes were heavily damaged or destroyed. Vast amounts of trees, debris, personal belongings, and sediment have relocated.



The Wimberley Flash Flood completely destroyed the Fisher Store Road bridge


The City of San Marcos, Hays County Sheriff's Office, and Hays County Emergency Operations Center, have set the bar high for future disaster efforts. These organizations have been very communicative about road closures and openings, alternate routes, repair plans, and instructions to residents. Some of the instructions have been stark: public safety curfews; debris removal instructions addressingappliances, concrete foundation debris, and cars; requests to search private land for missing persons and vehicles, and requests to not burn debris until it has been searched for valuables.

Under normal conditions, a careful permitting and inspection process is appropriate to ensure homes are built to code and follow local zoning restrictions. When hundreds of homes are destroyed, and hundreds more are affected, bureaucracy just gets in the way. Hays County changed the process, waiving permit fees and expediting the permitting process for those affected.



Lesson 5: A culture of self-preservation / self-reliance is invaluable


Residents of Wimberley and San Marcos didn't wait for FEMA or the county to come to their rescue. They set about recovering as soon as the sun came up. Not only that, but the surrounding community stepped in to offer assistance. Hundreds of people from the surrounding area volunteered for the search and rescue effort.

Numerous businesses donated essentials - water, food, sunblock, replacement prescription glasses or contacts, diapers, and other items we take for granted. The county organized a "lender library" of supplies for digging out: wheelbarrows, shovels, Bobcats, backhoes, dump trucks. A grass-roots laundry service sprung up with the support of a local laundromat, offering clean clothing (and in some cases, stuffed animals) to people who had lost everything except the clothes on their back.

Recovering from a disaster of this magnitude takes authoritative coordination. That leadership only goes so far though: the recovery is immeasurably more effective when residents or employees know in advance what they should do - and when they are empowered to begin in their immediate sphere of influence without waiting for direction.

Wimberley and San Marcos have a long road ahead of them, but they will recover thanks to outstanding preparedness, quick and decisive communication, a well-rehearsed disaster response plan, and a resilient spirit.

Do you have something to add? A question you'd like answered? Think I'm out of my mind? Join the conversation below, reach out by email at david (at) securityforrealpeople.com, or hit me up on Twitter at @dnlongen

Whois David?

My photo

I have spent the better part of two decades in information technology and security, with roots in application developer support, system administration, and network security. My specialty is cyber threat intelligence - software vulnerabilities and patching, malware, social networking risks, etc. In particular, I strive to write about complex cyber topics in a way that can be understood by those outside the infosec industry.

Why do I do this? A common comment I get from friends and family is that complex security topics give them headaches. They want to know in simple terms how to stay safe in a connected world. Folks like me and my peers have chosen to make a profession out of hacking and defending. I've been doing this for the better part of two decades, and so have a high degree of knowledge in the field. Others have chosen different paths - paths where I would be lost. This is my effort to share my knowledge with those that are experts in something else.

When not in front of a digital screen, I spend my time raising five rambunctious teens and pre-teens - including two sets of twins. Our family enjoys archery, raising show and meat rabbits, and simply enjoying life in the Texas hill country.

For a decade I served as either Commander or a division leader for the Awana Club in Dripping Springs, Texas; while I have retired from that role I continue to have a passion for children's ministry. At the moment I teach 1st through 3rd grade Sunday School. Follow FBC Dripping Springs Kids to see what is going on in our children's ministries.