Meet the new boss. Same as the old boss.

Image credit: https://flic.kr/p/8Wu3PU

Image credit: https://flic.kr/p/8Wu3PU

It’s confirmed. As mentioned by Kevin Perry, Tom Alrich and Peter Behr, the Federal Energy Regulatory Commission (FERC) will be performing their own audits of the North American Electric Reliability Corporation (NERC) Critical Infrastructure Protection Standards (CIPS) in 2016. It sounds like they will be choosing entities that are already on the 2016 audit cycle and will likely continue this practice into 2017, possibly beyond. Given the FERC staffing level, I would expect something on the order of 7-8 audits per year.

 

I’ve been able to get a little more information from my contacts at FERC and ex-FERC staffers with knowledge of the approach. I’m hearing that the strongest focus will be on the application of CIP-002-5 and the resulting Bulk Electric System (BES) Cyber Systems (and/or BES Cyber Assets). I’m not getting any indications that FERC will be overly strict or dig any deeper than the NERC Regions currently do in their audits.

 

I do expect FERC to sample all types of utilities ranging from the big Investor Owned Utilities, to the municipals/publics, the cooperatives and even the independent power producers. FERC has shown a tendency to gather information from all interested parties and not just focus on a single “demographic” under their purview.

 

No, FERC is not taking over the audit process. CIP Compliance Audits are still a NERC/Regional thing and no changes to the current model are expected for the foreseeable future. FERC has always had the authority to perform their own audits so this is well within their scope. However, historically, when FERC participated in an audit they were (ostensibly) there to audit NERC and/or the Region performing the audit of any given utility company – and not auditing the utility directly. This “new” approach is an independent audit of the utility, not necessarily involving NERC or any Region.

 

What could go wrong? I see (at least) a couple of interesting areas for potential unintended consequences of this activity…

 

First, FERC may have a different perspective (interpretation) on what the language in the CIP standards means to them. I hesitate to use the word “interpretation” because that has a special regulatory meaning but it still applies. The industry is already struggling with the NERC and Regional interpretive and guidance processes. Adding FERC to this mess may muddy the water even more. Another view on this is that it may actually normalize the interpretations and guidance because we will finally know how FERC understands the language.

 

The second area of interest is the Compliance Monitoring and Enforcement Program (CMEP). Will FERC follow the NERC CMEP or will they do something different. And if so, how far from NERC’s process will it be? If/when FERC finds a violation, what will the process look like? Will the penalties be steeper?

 

Why is FERC doing this? Some speculate that FERC is flexing their muscle and trying to send a message that they are still in control. Some say that FERC doesn’t have high confidence in what they are getting from NERC so they want to go see for themselves where the industry stands with respect to CIP compliance. Some say that FERC is positioning to increase the scope of the standards to cover more of the industry. In short, none of the speculation is positive and there is a lot of fear, uncertainty and doubt (FUD) about this whole thing. FUD makes people do crazy things, so expect some wackiness as this starts to gain traction.

 

I, for one, think this is a good thing. Having FERC actively engaged may be challenging at first, but the long term effect will mean a more informed and involved regulator. I am hopeful that this will increase the likelihood of better regulation.

More to the Story

image credit Luigi Torreggiani

image credit Luigi Torreggiani

Sincere thanks to David Perera (@daveperera) for the interview and article on Politico "Highway bill would give feds new power over electric grid".

If I had one ask to David, it would be to include more of our conversation. Particularly the part where I stated that DOE has some very bright people with deep and relevant experience in emergency response. The agency should definitely be at the table (as a peer). They are the best source for context and can provide the most effective solutions. The power grid is a complex beast and the defensive/response actions should have the least impact to the system (keep the lights on, and keep the problem from getting worse).

Special of the Day: Elephant

First, thanks to Joseph Menn of Reuters for the article. Joseph is a great reporter. He asks intelligent questions, and clearly "gets it" in my opinion. I just wanted to add a little more to my part of the story.

The quotes provided… "We've got this cancer that is growing inside our critical infrastructure. When are we going to go under the knife instead of letting this fester?" and "We need to restructure some regulations and incentives." …were part of a larger conversation where many things were discussed. Everyone should know by now that Industrial Control Systems, in particular the legacy devices, are prone to security vulnerabilities. For example, in the article two of the researchers (Eric Forner and Brian Meixell) took less than 16 hours to find something worthy of a BlackHat talk.

This is not an issue for any one country or company. We all buy the same gear from the same vendors.

Everyone asks if regulation would make the situation better. The simple answer is no. Regulation - by itself - won't solve this problem. A good example would be the NERC CIP Standards. They are often credited with getting the security ball rolling for the electric sector. They may or may not ultimately lead to a more secure grid, but I can say for certain that we have seen some very real "unintended consequences" from the existing regulatory model for the electric sector in North America. Other countries and continents are watching what is happening here and basing their future choices on our lessons learned.

For those that don't fully appreciate our current regulatory construct, I'll keep it simple. With the existing Powers That Be, there is no way to regulate cybersecurity for the power grid end-to-end. There's no department, agency, bureau, commission or corporation with jurisdiction to cover all of it. So, we're left with a patchwork of rules from authorities that work together through a model that more closely resembles a bar-fight than a choir singing from the same hymnal. Sometimes this friction is actually beneficial. Sometimes it isn't. Either way, the utilities are left with the bill. And like most situations, money matters.

Which brings me to incentives. Somehow, we found a way to incentivize things like Smart Grid. To the tune of billions of dollars. We haven't done the same thing for cybersecurity. We need incentives for R&D so new security technologies can make their way from idea to product. We need incentives to upgrade to these new security technologies and get rid of the excuse that it costs too much to take an outage and replace hardware. We need incentives and assistance to train the next wave of cybersecurity professionals for the industry. We need incentives to share information between utilities so they can gain situational awareness from their interconnected peers (and the government, if they so desire) - in a way that won't put them in a legal pinch for doing so.

Through all of this, electric utilities are not just sitting on their hands when it comes to cybersecurity. There are dedicated, responsible, and profoundly smart people keeping the juice flowing. They do it every day. They fix it fast when it breaks. And they work just as hard to keep it secure. This is serious business for them. They are using the equipment they have, and yes, some of it is vulnerable to the types of attack you will see at BlackHat next week. Yet, they get the job done safely and at the lowest cost possible. So, the next time you flip the switch on the wall, remember that your local utility is harnessing one of the most powerful forces in nature and squeezing it down a skinny wire to your house so you can see in the dark, dry your jeans and toast a bagel.

Well, what's the solution? Sure, we could make those greedy, irresponsible vendors fix their vulnerable gear. And yes, that's a good start, but the utility can only upgrade so fast  (don't forget about that whole "force of nature" thing -- you gotta work carefully and methodically around stuff like that). We could fix the regulatory model. Yeah, that's not a bad idea but remember we're not really set up for that. It would literally take an act of congress to make it happen - which isn't entirely out of the picture but I'm not holding my breath. We could throw money at the problem, but that will bring the carpetbaggers and crooks along for the ride. See my point? There isn't one silver bullet to solve this problem. Like brushing and flossing, like diet and exercise, it's going to take conversation and compromise. This is the hard road, but it's the right road and everyone knows it. They're just too lazy to do it right.

Join me. Grab a spoon, get your seat at the table and let's eat this elephant in the room together. It's not going to taste good. It's not going to be over quick. But we won't get past it until we decide to do it together. I'll bring the Bourbon and sarcasm to help wash it down.

[image credit]

"By Failing to Prepare, You Are Preparing to Fail"

I thought I might take a few moments to provide some backstory given the recent buzz about my post yesterday announcing my move to the independent consulting path, the respective  management transition at EnergySec and the recent post by Dale Peterson of DigitalBond highlighting the struggles the organization went through in December 2012.

First, the astute quote from Benjamin Franklin in title of this post is about preparation. We knew at the onset that the organization would need a heavier than usual executive load to provide the industry clout, clear direction and depth of experience needed to take the organization from its all-volunteer state to a self-sustaining nonprofit. We also knew that the startup mindset and staffing model would need to change once we saw clear evidence that the organization would be able to support itself. This is just smart business.

In December of last year, DOE re-affirmed commitment to the NESCO and extended the agreement but chose to hold their contribution until EnergySec could catch up to its part of the cost-share. At that time, the executives, in discussion with our Board of Directors and the NESCO Advisory Board, made the decision to streamline and enhance the operational capabilities to capture more immediate revenue. This was an easy call because we were already seeing steadily increasing positive traction with the NESCO Tactical Analysis Center and the organizational memberships. We were also running financial models to identify when the organizational changes to senior management would need to take place. The December circumstances required that we make some prompt staffing adjustments, close the cost-share gap, and ultimately re-instate the DOE cost-share contribution. The strategy is working. This approach is aligning the organization with the expected financial targets, but it was invoked slightly sooner than we forecasted.

Sure, partnering with any federal agency for funding is a challenge (ever tried to get VC for a nonprofit?). I've always said that federal grant money is "free like a puppy." We knew the risks going in, and unfortunately a fair share of those risks were realized. The risk landscape also included things like economic recovery/recession, competitive landscape, and politics. But as a result of the experience, we have a well-managed, lean, agile and tightly focused EnergySec operating the NESCO program. This benefits everyone.

I'm genuinely pleased with the direction the organization is going. EnergySec's story is a testament to the challenges inherent in sharing information effectively. It is an example of the complications found in public-private partnerships. It is proof of the strength and determination demonstrated by people dedicated to the security of our industry. We've been doing this longer than anyone else and we're still learning - as we should be.

[image credit]

Next Step

You may have heard about the recent EnergySec change in management. Yes, I'm taking the next step.

But no worries. I founded the organization over 10 years ago, I'm still very much attached to EnergySec and always will be. This transition was actually planned so it's a good thing for everyone. When the EnergySec Board asked me to leave the private consulting world and take the wheel as CEO in 2010 it was really to advance the nonprofit mission to the next level with the inertia created by the funding of the

DOE cooperative agreement and get the NESCO off the ground. My goal was to be the steep-curve leader of the outreach, promotion and community building necessary to bootstrap the nonprofit to a funded and sustainable point. After which, I'd replace myself with a more operational executive and shift to an oversight and governance role. This allows the organization to streamline and focus on delivering quality products and services.

Steve Parker is a perfect choice for CEO. He's been a key player in the organization since inception and he's built most of the operational aspects with his own two hands. I'm still deeply involved in EnergySec as President Emeritus, executive committee member of the NESCO Advisory Board, company liaison to the North American Energy CISO Forum and most importantly through my seat on the Board of Directors. I still wear an EnergySec badge at many of the conferences I attend and I'm a guest instructor for some of their training/classes. I'm their biggest advocate.

In short, I'm not really going anywhere. Just handing the day-to-day EnergySec operations over to a trusted friend and executive while I go back out into the community to do what I do best - be a catalyst, build interesting connections, ask the hard questions and provide the honest answers when no one else will. I'm always drawn to the next wave and I'm seeing some fascinating things on the horizon.  At the moment, I'm doing a lot of independent consulting/advisory around the critical infrastructure security, regulatory and policy areas for utilities as well as software and hardware companies. If you have any interesting projects that might benefit from my participation, please let me know!

While I have your attention, I would like to issue a call to arms. EnergySec and NESCO are doing great things for/with the industry. They are doing the hard work that everyone said couldn't (or shouldn't) be done. I've always told my staff "If it were easy, someone else would have done it already." And they still live by that message today. We knew it would be difficult in ways we couldn't imagine. But it was worth every ounce of effort to make it happen because it was (and will always be) the right thing to do. We can show the rest of the critical infrastructures, the Federal onlookers (regulators, etc), our energy industry peers and even the hype-driven media that we "get it." We are advancing security from the bottom up, as a sector united, dedicated to the mission of safe, reliable power. The industry needs a place where the asset owners can have open, honest and candid discussions about the potentially sensitive subject of security. EnergySec is that place. We must all work together to keep it safe.

So, do your part (if you haven't already) and purchase the organizational membership. Get your executives to join the CISO Forum. Get your people involved. Sign up for the Rapid Notification System. Buy a subscription to the Tactical Analysis Center (TAC). I know there are many organizations out there asking for the roughly same commitment. But I challenge any of them to demonstrate the 10-year history of trust, value and loyalty to the security cause that EnergySec has proven. And as part of the nonprofit mission, we do it at a fraction of the cost. Higher value, higher trust, and lower cost? Sounds like an easy choice to me.

You can always reach me at my EnergySec email address or my personal account if needed.

[image credit]

WARNING: Zombie Policy Ahead

Maybe it's just my dark and twisted sense of humor, but I find it slightly funny when someone hacks a traffic control sign with the message "ZOMBIES AHEAD." It's probably not so funny to the person stuck at the back of the line in the resulting traffic jam though. Less humorous (because of potential panic and such), but along the same lines is the recent emergency broadcast system hack(s) warning of zombie attacks.

What I don't find funny is the seemingly endless night-of-the-living-dead cybersecurity legislation. Apparently, we are doomed to repeat our insane history by doing the same thing over and over again expecting different results. The Congressional graveyard of dead cybersecurity bills is coming back to life. Zombie bills are breaking through the soil and slowly ambling their way around the beltway seeking brains.

I'm sure someone somewhere has a comprehensive list of these dead cybersecurity bills, but from my own rough estimate it's at least a hundred of them in the past few years. What troubles me more is that we already have a rather significant foundation of cybersecurity legislation, regulation or otherwise legally oriented mumbling. I ask a simple question… Do we need new legislation/regulation or can we more effectively enforce (or maybe even refine) what we already have?

My hunch is that we won't make it out of 2013 without seeing at least one new cybersecurity law passing. Another prediction is that whatever we get will focus squarely on critical infrastructure.

If you've got skin in this game, now is the time to grab your virtual pitchfork and torch and storm the legislative castle (figuratively, not for real of course). By this, I mean use the power of your voice to create your future. If we're going to get zombie policy whether we like it or not, we should at least do our best to get something we can live with. That won't happen unless smart, informed people get involved and actually talk to Congressional staff. I know, I know… you're afraid that what you say can and will be used against you. Yep. It probably will. But we can't solve this problem with the same thinking that got us to where we are. The only way out is through. Together.

Here's the counterargument. If you don't engage you'll be labeled as "disinterested," "unresponsive," "apathetic," etc. If you do engage, and all you say is the same thing you've always said, you are no different than they are and you're essentially a zombie too. Something about a pot and a kettle should come to mind. Simply put, if you don't participate - in a meaningful and newly creative way - then you can't complain about what you get.

[image credit]

Not Fast Enough

The problem with SCADA and ICS security isn't going to rapidly change for the better, no matter how bad the situation gets. Let's go ahead and accept the fact that Operational Technology (OT) environments are not able to withstand the degree of constant update enjoyed by traditional and future Information Technology (IT) environments. Complexity, cost, and culture are such formidable barriers at this time that our best short-term approach looks no better than a frantic game of "Whac-a-mole." Sure, long-term we can buy and implement better products - assuming the vendors provide us with better options - but for now, it would probably be best to assume your OT environments are not as secure as you think they are.

So what do you do?

To have any hope of "keeping the lights on," SCADA/ICS security should balance prevention, detection and response. You've heard it before. You've seen this portrayed on too many slide decks. So, has it truly sunk in? Simply put, you need to know that you can operate in a degraded capacity under duress with unknown system integrity. This usually means equipping experienced and engaged people with powerful and effective tools. It usually means changing architectures. It usually means changing culture. All of this means cost. Possibly BIG cost. Will the cost vs. benefit balance reach equilibrium or even tip in favor of benefit? In my opinion, yes. But not fast enough for many counting the coins.

[Photo by Maggie Smith]

Take Your Pick

In the recent TechNewsWorld article by Richard Adhikari "Study: Electric Grid Needs Full-Time Cyberguard," I was quoted as saying "We would welcome a single authority." At the end of that sentence, I also said "whomever that may be: DOE, DHS, etc." Note that I wasn't favoring any agency over another. The discussion was in reference to a recent report released from MIT,"The Future of the Electric Grid."

It is certainly true that the legislative, regulatory and overall policy sands are shifting. The existing bulk power system security regulations (NERC CIP) are changing. Multiple federal agencies are competing for control over the authority for grid security. The Distribution system, with its deep relationship to the consumer side of the grid modernization (smart grid) push, is hotly contested between the State Commissions and the feds.

All of this confusion has a numbing effect the utility executives. They are traditionally risk averse to begin with, and the policy forecast essentially indicates a 70% chance of storms ahead. The unintended consequence is that many organizations will only do the bare minimum required to be compliant with today's regulation. It is difficult to justify dedicating resources to future efforts with the significant possibility that things could change and that money may have been wasted.

With all of this churn, yes, I do think that some focus and harmonization in the policy landscape would be a good thing. Who can do this best? Well, the jury is still out on that one. I think all of the proposed agencies have their pros and cons. Ask me again in a year or so.

14 Seconds

I'm still a nobody (so I don't really qualify for a full 15 seconds) but I've been getting some media hits lately. Everyone told me, and I believed them to a certain extent, that everything you say can be twisted. I've dealt with many people who do just that and I have always felt that the truth will surface no matter what. Truth is like data. It wants to be free.

I sincerely make every effort to be balanced in my statements/positions and always say the good with the bad. My personal belief is that there's always good to be found. At times when I can't see it immediately, I try to drop the ego or emotion and look a little deeper. Invariably, it's there. With this in mind, I decided to dust off my personal blog and use it as the future platform to correct any misstatements, quotes taken out of context or just add the "whole story" where necessary.

So, let me start with the recent set of articles about various SCADA security topics (http://goo.gl/Kty17http://goo.gl/KcvCh and http://goo.gl/txIDp). It is true, those statements are mine, and I did provide them in email interviews. They're actually pretty close to the mark, but I'd like to add a few of my other statements that were omitted, just for context...

"All of the above (and more) lead to a state where many are forced to operate with aging infrastructure extended beyond its lifespan. Note however, that many staff at municipal utilities are actually remarkably dedicated and resourceful people. They have to be, given the circumstances." 
"The threat is somewhat exaggerated, but it is still very real. The vulnerabilities are underestimated."

Please understand that I'm not casting aspersions on the Municipal Utilities of the world. Some are further along the security maturity path than others, but I have worked with many of them and I find them to be amazing people and amazing organizations.

Beyond The Bullets

It’s been a while since I’ve posted. I’ve been busy and various other pseudo-legitimate excuses, but something happened at a meeting recently that caused me enough pause to actually carve out time for a quick blog post.

I was discussing some trends from recent meetings when a participant began venting his frustration with a few of the bullets in one of my past presentations. I made a genuine attempt to explain that I don’t just read the bullets on the screen and go away. Rather, I provide context and backstory to each bullet while presenting. I gave him the context and backstory to the bullets of his concern but that didn’t seem to satisfy his frustration. Which is ok. I know I can’t please everyone. I’m certainly open to constructive criticism (as anyone who knows me from my WECC CIP Audits and Investigations days can attest).The fact that someone was frustrated with me and my content wasn’t the issue that spurred me.

The rub is that people actually think a PowerPoint presentation stands alone by itself. It doesn’t. Bullets are (or should be) used to cue the presenter’s thought process to provide valuable descriptive elements surrounding the bumper-sticker-bulletized blurb on the screen. You might get a fraction of the intended substance from the bullets in the slide deck but, assuming the presenter is worth a dime, then the presentation itself – given by the presenter – is where the real value resides. Whether webcast or in-person, the presenter should make an attempt to go beyond the bullets.

If you read something in a presentation that sets you sideways – or if you read something that really resonates with you – take either emotion with a grain of salt. After all, you’re only getting part of the story. If you can’t make it to the presentation then consider emailing or even calling the presenter to get the complete and intended message.

2010 EnergySec Summit 9/21-9/22, Denver

Sheraton Denver Downtown

For six years running, the EnergySec Summit has proven to deliver the highest value when it comes to Energy/Electric Sector security events. And no, I’m not just saying that because I’m on the Board of Directors [FD]. I say it because I have yet to see another conference where such a diverse body of professionals meet under one roof and actually talk to each other. The variety of attendees ranges from all organizational strata of electric asset owners spanning the operational and business groups, State Regulators, Feds of all agencies, national labs, consulting firms big and small, security product vendors, industrial control systems vendors, research firms, and universities.

But it’s not enough to just get great minds to the table. The magic is when those great minds begin sharing in ways they hadn’t before, thus generating a result that is greater than the sum of its parts. The EnergySec culture draws this out of you. The collaborative spirit is simply pervasive and infectious. There is no singular agenda or axe to grind. The audience isn’t just there to listen and blink at powerpoint slides from the so-called “experts” all day. Instead, all attendees are recognized as experts and they are active participants in the event. You can’t help but leave empowered with new connections, new knowledge and an invigorated desire to do more for your company and your industry. Sure, this happens at many conferences, but there is something unique about the EnergySec vibe that takes it to a different level.

The information sharing grass roots effort known as EnergySec today has matured from its humble beginnings back in 2004 as Energy Security Northwest (E-Sec NW). Those original meetings of adjacent asset owners over lunch and the mailing list were the seeds of something great. The membership started as an all-volunteer group with no real organizational structure other than a loose “board” of moderators who maintained the vision of the EnergySec culture. They fostered it through technological advancements and beyond into non-profit status as a 501(c)(3) company. Today, it boasts 380+ members, 102+ organizations, 50%+ North American Generation, 65.3%+ North American Distribution, etc.

Ok, so back to the Summit... ICF International [FD: my current employer] kicked things off with a networking breakfast on Day One. I followed with my “Energy Sector Crystal Ball” spiel to kick off the event and set the mood. I really tried to keep it FUD-free and just talk about trends I’m seeing within the industry. Carol Hawk of the Department of Energy then walked through some of the projects they are currently working/funding - one of which is the National Electric Sector Cybersecurity Organization (NESCO). More on this again later. Mike Mertz of PNM then presented on security policy. His subtitle was “The most important element for Security and Compliance.” By the time he was done, he had me convinced. Mike always has such a great way of getting to the core of the issue and providing actionable take-aways. Bill Hunteman of DOE was up next to tell the group about the department’s strategic direction for smart grid security, an area in which nearly everyone is interested. Bill was followed by Brian Girardi of NetWitness presenting on network forensics & advanced threat analysis for critical infrastructure - an area where NetWitness is recognized as a practice leader.

Then came lunch, sponsored by NetWitness and more networking. I eavesdropped on several conversations as I usually do and joined in as many as I could. I am always surprised to see how much fusion happens when the walls come down and the silos are broken.

Lunch was followed by an entertaining and insightful presentation from Mike Assante (now with NBISE, formerly NERC Chief Security Officer). I particularly enjoyed the parallels between suicidal squirrels and the threat actors, but this portion of his talk was curiously left out of the presentation he provided to EnergySec for posting (I’ll see if he’d be willing to furnish the un-redacted deck). Suicidal squirrels aside, Mike had some solid and sobering wisdom to share about how regulatory compliance has affected cybersecurity for the industry. Jack Whitsitt, Sr Critical Infrastructure Cyber SME supporting the Transportation SSA followed Mike with a technologist’s admission of inadequacy, the executive’s role in national cybersecurity. Though Jack doesn’t come directly from the power biz, he knows industrial control systems security and his subtle-but-wry humor and acute technical references really fit the audience well in my opinion. Tim Erlin of nCircle was next with a talk on configuration auditing and vulnerability management in the NERC CIP era. Short of the free tool Nessus, nCircle probably has the largest presence in the vulnerability and configuration management space within the electric sector, based on what I’ve seen as an auditor and consultant.

Day One closed with a panel on the Summit’s primary topic, the intersection of security and compliance. The panelists were: Josh Axelrod, WECC CIP Audits and Investigations Team Lead; Dave Norton, Entergy Policy Consultant - Critical Infrastructure Protection; Mike Assante, NBISE President and CEO; Mike Mertz, PNM Resources Sr Project Manager of FERC Compliance and Ben Miller, Constellation Supervisor of Information Security Operations. As you can probably imagine, this was a lively debate. There was no shortage of opinions, from the panelists or the audience. A recurring thread was that the existing NERC CIP standards may be decreasing security in some (but not all) implementations for the sake of compliance. Some great examples of satisfying both security AND compliance were also provided. It was a fantastic way to end the day and really primed the discussions during the reception.

Big thanks to nCircle for providing drinks and hors d'oeuvres. Liquor and food have a funny way of changing the dynamic. After a day immersed in the collaborative culture and high-value content, minds were reeling and ruminating. What was left of the inhibition to release opinions quickly vanished, which made for weighty and direct conversation. I eavesdropped some more, made more new friends and had my fair share of the hotel’s Bourbon.

Day two started with Tim Roxey, NERC Manager - Critical Infrastructure Protection. Tim’s presentation did a fantastic job of capturing the complexity of today’s power grid. The analogy of flocking birds as the intelligent future of power management was very intriguing. The vulnerabilities in such a complex system are not being fully considered - yet. Next came the highlight of the event in my opinion. James Arlen (Push The Stack Consulting) reinforced the information sharing purpose with his presentation “SCAD’oh - An Agnostic’s View: stop the ‘us v. them’ and come together to solve the problem.” James delivered the message with refreshing candor and corrosive humor. No one was safe. We all got a chance to laugh at each other and ourselves. We were all well aware of many opportunities for self-improvement by the time James left the stage. But everyone needed to hear every single word. If there was any ego remaining in the room, they simply weren’t listening.

Up next was Steven Parker, EnergySec Director. Steve used puzzle pieces handed out at the beginning of the event to convey his message of sharing and collaboration. Each attendee put their name on their piece and then handed them back at the end of Day One. On Day Two, the puzzle was available for assembly at the registration table. The message: we are all parts of the cybersecurity puzzle. Only when we work together do we form the complete picture. Steve also included a briefing on the EnergySec state of the union including the announcement of the formation of the NESCO. The event was wrapped up by Sean McBride of Critical Intelligence with a presentation on developments in ICS defense. As usual, Critical Intelligence hit the mark based on their deep involvement in the subject. CI provides the light version of their open-source intelligence reports to EnergySec members.

Day Two ended with a final networking lunch. You probably guessed it but I eavesdropped again, chatted more and exchanged contact details on napkins because I was out of business cards.

Throughout the event I received strongly positive feedback from the attendees on the quality of the content in addition to the networking/collaboration benefit. It may sound like a big hugfest or kumbaya session, and it was to a certain extent but there was also plenty of dissenting opinion and experienced debate to keep things balanced. No doubt that everyone likes to know they're not alone in the difficult struggle to secure the North American power grid, and the level of professional exchange is what makes the EnergySec Kool-Aid taste so good.

Another interesting addition this year was the live tweeting from the event and commentary/discussion from those on the floor and those who were unable to attend. The real-time discussion went beyond the walls of the conference room. Very cool indeed. I finally met a bunch of people from Twitter face to face...

@sintixerr

@myrcurial

@electricfork

@CrucialCarl

@tmdheard

@jholcomb

@pmhesse

...to name just a few.

Lastly, a HUGE thanks to Lisa James, EnergySec Chair and her supporting cast of volunteers, Jeri Freimuth and Ed Croft. I’ve been to very expensive conferences with paid armies of staff and the EnergySec logistics were better.

I’d like to close the post with a brief note about the National Electric Sector Cybersecurity Organization (NESCO) mentioned above. EnergySec was awarded a $5.8M grant over three years to stand up the NESCO. The NESCO will be supported by EPRI and their collaborative as the research element, NESCOR - the “R” is for resource. Some details are in the presentations from Carol Hawk, Bill Hunteman and Steven Parker. You can read more in the official press release. If you are interested in hearing more, contact Lisa James, EnergySec Chair at lisa@energysec.org.

[image credit]

The Cabin Door Is Closed, Please Power Down All Electronic Devices...

I fly a lot. More than many. Sometimes over 50%. As such, I have some experience with the consumer side of the commercial aviation business. I am by no means an expert. Recently, I was asked to speak at an air traffic controller's conference on the subject of cyber security in the Next-Generation Air Transportation System. I was there to provide a perspective from the outside, more of a security technology discussion for what works in the overall CI/KR space. It was a panel, so the slide deck was short - which was good for me, because again, I'm not an expert in aviation. The panel had current and ex-FAA staffers, university professors, aviation consultants from the defense industry/sector and me. Do you remember the old Sesame Street song "one of these things is not like the other?" Throughout the event, I was constantly reaching for my smartphone to Google the acronyms that I'd never heard.

I had a similar newbie feeling when I started working in the electric power sector. Before then, I was just a consumer. I flipped the switch and expected the lights to come on. When the power went out at my house, I was the first to call my local utility and give them a piece of my mind as a paying customer. After all, us geeks can't live without our tech toys for longer than a few seconds. So the first time I actually spent more than five minutes on the Control Center floor, I was.. well.. floored. Even mild spring days in the 'shoulder months' can seem like a delicate balance of order and chaos. The electric system is interconnected, just like the aviation system. What happens in one area will quickly and directly affect other areas, some quite distant. The real-time seat-of-the-pants decisions by system operators is really what keeps the system running - not the technology. Sure, the technology is there, but it is only a tool.

I see a situation, whether it is Smart Grid in the power biz or the Next-Gen Air Transportation System in the aviation sector, where we are inserting a much wider technological distance between the human and the physical/kinetic endpoint. System operators are using ever-increasing layers of technology. Until fairly recently, they looked at some sort of analog or electro-mechanical instrumentation for operational decisions and then they would physically (manually) activate something. Today, we have operators using tools which are in turn, using other embedded tools, which may also be using further embedded tools - and so on. This can be a good thing for many reasons, but it can also be a bad thing. This trend, though perfectly natural - even expected, should be carefully monitored, carefully balanced. Especially when it involves critical infrastructure. We may even need to tip the scale toward sound security engineering instead of focusing solely the profit drivers. At least for a while.

We've ignored our critical infrastructures for so long that we are in desperate need of an overhaul. Nearly every one of the sectors in the National Infrastructure Protection Plan (NIPP) could be called brittle. Some money is starting to flow to these areas for much needed upgrades but the legacy technology and the bleeding edge enhancements need to work together in the same interconnected system. This creates a 'base of sand' problem. Legacy devices are underpinning tomorrow's technology gizmos with incredible distance between the two ends of the spectrum. We need a security engineer to put their stamp on the blueprints BEFORE they get the permit to build. When adding to or modifying an existing structure, the structural engineers factor those old trusses, supports and cracks in the foundation into the new design. I don't want to discount the great work being done here, but I think few would disagree that we have a cart-before-the-horse situation.

  • The most common recommendations I heard at the recent aviation conference were:
  • Test bed for qualifying systems (approaching Certification and Accreditation)
  • Minimizing potential of supply chain attacks
  • Security Training for operators/controllers
  • Situational Awareness (and integrity of decision support data)
  • Information Sharing

Those of you following the power sector for the past few years should see some striking similarities. I'm willing to wager that nearly all CI/KR sectors are facing these same challenges. The only recommendation didn't see was slowing down to get security issues addressed in the design phase. I've been a security professional long enough to expect that, but I can't seem to bring myself to accept it - hence this post/rant.

Like nearly all of my posts, I am writing this as I fly home on a commercial airline. Now if only I can think of a solution to being crammed into a space smaller than my anatomical dimensions. I'm not important or rich enough for First Class seats. But every time I think I've got it bad, I remember my co-worker CJ and his 6'5" span. He's taller than most clearances at drive-thrus and parking garages. Unfortunately TSA frowns on bringing a crowbar to extract him from the seat.

[image credit]

Is Reliability In Your Future?

I'm hearing a new wave of disdain for the NERC Reliability Standards from the industry. This happens from time to time and it isn't just about the CIP Standards. The Order 693 stuff gets its fair share of noise too. The most common thread is how all of this effort doesn't really improve reliability of the power system. I hear it from plant and system operators. I hear it from comm-techs. I hear it from all ranks of management, from the front lines all the way to the executive level (though middle management seems to be the loudest). I even hear it from the IT staff but to a lesser extent.

Granted. The Reliability Standards are a pain. Lots of work, lots of money and lots of time spent to reach the magical state of Compliance. It deserves some of the frustrated noise that it gets, but not all.

One benefit: Accountability.

I know I'll probably take some heat for saying it, but in my opinion, holding people accountable for their actions will improve reliability of the power system. Accountability is a powerful tool for maintaining integrity. Some of the most obvious examples of accountability in action are cameras. They are aimed at cash registers while capturing POS data watching the watchers at daycare centers and schools and even publicly scrutinizing police officer actions via headcams. No, system and plant operators shouldn't be fitted with headcams, but they shouldn't fear accountability either. I can sense a strong authority vibe coming from them and it seems that they perceive these standards to be chipping away at their ability to freely make grid management decisions. The accountability elements built into the standards will only take away your ability to make decisions anonymously. Believe it or not, this could actually help you and your system.

The photo was taken by a friend of mine who says the graffiti isn't his - and I think I believe him. And to quote his response on the message: "no, it isn't, but hopefully it is recoverable." Thanks SHP. Please, no bathroom humor.

CSO706SDT FAIL

Dear friends in the electric power industry: this CIP-010 and CIP-011 draft baffles me. I had a heck of a time trying to audit the first one and this new one leaves me deeply sympathetic for the poor auditors I left behind (sorry guys). You have no idea how challenging it is to call a ball or strike with CIP-002 through CIP-009 as an auditor. Well you might have an idea because you had to implement it - or should have anyway. With that, I hope you see my point that inserting additional flexibility and vagueness will only make your job implementing these requirements even harder. It will also make your auditor's job more difficult. These two facts increase your risk.

So, what happens if you get this one wrong? What happens if FERC remands it? Will it cause a ripple effect that could possibly spell the end of the ERO's oversight of security for the industry? Will Congress decide that our industry can't self-regulate, therefore they need to step in and "save" the grid from the cyber-boogeyman? Sure, these are extreme cases but they are still in the realm of the possible. And if we have an incident, think ESA. Remember what happened to the airline industry. You may not be able to enter a substation unless you've gone through a full body imaging scan and your liquids and gels are all less than 3.4 ounces in a one quart clear baggie.

CSO706SDT, especially after listening to the recent Version 4 Workshop, I implore you to listen to the auditors. They are not the enemy. A few points that bear repeating:

  • Define stuff. If you haven't defined your terms, you haven't written a standard. "Annual" is only one of the many words you need to clarify.
  • Attackers aren't constrained by budget and time. If we are, they have the advantage.
  • Remember Moore's Law. Technology will transform significantly within ten years. Consider more realistic implementation deadlines. In fact, make it simple and give us a single [sane] date.
  • Write the standards in such a manner as to eliminate the need for a Technical Feasibility Exception.
  • Access points matter. Allowing anything is like saying a shoji screen is equivalent to a steel door.
  • Terms like boundary, border, perimeter are all acceptable. Most professionals know that this means "preventive control." Removing the ESP and PSP language may do more damage than good, despite the pre-existing confusion. Require a perimeter, with a DMZ.
  • Low impact systems deserve protection. Packets don't care about arbitrary labels.The way it is currently designed, "stupid" would be a compliant password for low impact systems. Minimize the potential for gaming the system and labeling everything "low."
  • Be thinking, with every requirement you construct, "how would someone evidence this?"

Electric sector, just go secure your systems. It will cost you money. It will take time and resources from other projects. Accept it. Embrace it. The sooner the better. If you start securing your stuff now, you will have less work to do when someone finally hands you a security standard. The situation won't get better in the future. There aren't enough security professionals who can spell R-T-U. The Feds aren't going to let sloppy or weak security standards prevail. The economy isn't going to turn around tomorrow with lavish profits to pay for it all. Somehow, you're going to have to find a way to do it anyway. The time is now.

Furthermore, we owe it to ourselves to step this up. We owe it to ourselves to get it right. We are engineers, operators, security professionals and generally very smart people. We can do this. We've solved harder problems before. The reality, however, is that we will only solve problems we want to solve.

Oh, and Hello World. This is my first official blog post.