Confessions of an OT Network and Systems Engineer

The winner needs to be the OT customer again

My first days within the ICS community began at a Geothermal operating company located off dirt roads in the middle of the California high desert. The challenges at a remote industrial facility surprised me. My first task was "to fix the Internet” - yet there were no known passwords to access servers or firewalls, site power was intermittent, and we weren't even sure if we paid the ISP bill! Days before my start on the job, our two transmission lines had collided in a fierce wind, leading to extended downtime. In hindsight, no power or Internet led us to the tightest cyber-security we would ever achieve.

From those early days the DefCon alarms within my head began ringing on the notion that industrial systems were not ready for the connected world that was emerging before me. Stuxnet was fresh in the minds of my peer community, and incidents like it would only grow more prominent and problematic in the long-lived and distributed world that is ICS and OT. I was committed to being a part of the solution.

The Insatiable Cyber Octopus  

After a year or two into my career in ICS, I had become the insatiable cyber octopus of the operation, maneuvering my tentacles into everything within the company. DCS system upgrades, DNS domain registration, VoIP, firewalls, virtual infrastructure, compliance and RF communication, office 365, energy metering, RTU/RIG replacement, etc. – I was involved with it all. I was managing to increase resiliency and security, despite the increase connectedness. But for how long?

In short, my role was becoming a juggling act that was not scalable for a single person. Moreover, all these connected systems highlighted how much an industrial organization relied on and was connected to IT-like systems. When a control system fails, clever people rise to the occasion and locally operate valves, start/stop pumps, calculate indications, etc. It is true that sections of these facilities can run for a time on brawn and brains. The rest of the story is the human toil this takes, and the loss of operational efficiency. My facility had once operated without centralized IT-like infrastructure, but at that time there were three times the staff.

I also lost a lot of hair and time worrying if my small facilities were Iran's cyber-terrorist playground - a fear I picked up at DefCon. I was beginning to realize just how broad cyber-related risk is. More than just the technology, cyber risk involves people, processes, procedures and so much more.

I began to wonder, were my security efforts warranted? Did I need to annually patch my DCS systems, and re-enable HMI firewalls despite vendor recommendations? I had been operating my OT resiliency efforts mostly blind. The ICS Village welcomed me, and ultimately so did the Operational Technology (OT) team at Forescout.  

From customer to vendor 

In 2019 I took my experiences as a Controls and Network Supervisor in OT to Forescout. As their OT/ICS Systems Engineer, I set out to help other ICS reliability stakeholders work smarter and better manage the challenges that I had faced for nearly a decade in the field.

With the OT deep packet inspection (DPI) that Forecout’s eyeInspect product was able to provide, customers were able to gain a massive amount of visibility and operational context that they did not traditionally have. I found it incredibly rewarding bringing real positive business impact to ICS and OT organizations with innovative technology that I too once needed to improve my organizations’ security risk posture. With this stated, there does reach a point of diminishing returns when it comes to DPI and device data. How much charts, tables and alerts are useful? How many logs does a cybersecurity or operations manager require? The answers were ever-changing and complex, but here’s my take.

Do you work for data, or does data work for you? 

As someone who has spent 5000 hours with ICS detection and visibility tools as both a user and seller, I am certain that they have made energy systems, manufacturing, and critical infrastructure more secure. However, I am also certain that the customer’s needs for cyber resiliency have been largely ignored in a vendor race for features over function. Ask yourself, does your ICS detection and visibility tool aid you in your day-to-day job? Can your vulnerability management teams track the vulnerabilities identified? Are you leveraging the asset inventorying functions of the solution you have deployed? A lot of times these questions are not clearly answerable for a customer investing in a detection and visibility tool for their ICS or OT environment.

ICS detection and visibility tools are increasingly becoming middleware 

ICS detection and visibility tools serve an important middleware purpose that, up until about 5 or 6 years ago, did not exist. Now, however it does, and common functional tasks provided by OT visibility tools are sent to either a SIEM or spreadsheet. Cybersecurity tech is changing and evolving, but not as quick as cybersecurity demands. In short, security and resiliency need to be led from the top of an organization - in terms of dashboards, tools, and titles. As a result, ICS detection and visibility tools are now more of a component, rather than a self-standing solution. Don’t get me wrong, stakeholders will continue to leverage asset inventory, threat intelligence and DPI, but executives should demand more information related to ROI and risk exposure. They already are.

Cyber resiliency technology  

I get excited about tools that automate the collection of impactful data. However, over the years I must admit that despite various cybersecurity tools and vulnerability scanners, I have still devoted thousands of work hours to manual tasks that I had hoped that the correct tooling would have spared me. For me, this is why I see DeNexus as being a truly transformative tool for ICS asset owners today.

The reason I joined DeNexus is linked to the solutions’ ability to tie cybersecurity issues with business outcomes. Data and advanced data analytics to drive sounded business decisions. This allows network engineers, like myself 10 years ago, to link my efforts and actions to loss-event and business impact. Furthermore, is also provides the source of truth that stands to unify the common interest of all ICS detection and visibility tools under a single source of truth and value assessment for the customer. Evolution from data collection to data analytics has already happened in many other areas of activity, and it is finally coming to ICS cybersecurity.

"…despite various cybersecurity tools and vulnerability scanners, I have still devoted thousands of work hours to manual tasks that I had hoped that the correct tooling would have spared me. For me, this is why I see DeNexus as being a truly transformative tool for ICS asset owners today."

When it comes to cyber resilience, cybersecurity and risk tolerance is a conversation that must be answered from a business continuity and even shareholders perspective. In some sense, it is my view that cybersecurity has been deprived of quantification that allows business leaders to engage. These activities should feed conversations at the board level, not incomprehensible reports.

I see DeNexus as the change-agent and potential single source for of truth relative to cybersecurity for the industrial enterprise, risk leaders and underwriter.