Blog > Automation > Your OT Networks Don't Take a Break for the Holidays: Unveiling Lurking Vulnerabilities

11/28/23 | Mark Miozzi, Industrial Automation Sales Manager

Your operational technology (OT) networks don’t take a break for the holidays and the big rivalry football games that go with them. While many were enjoying turkey and Travis Kelce commercials, two industrial manufacturers found themselves grappling with major OT network issues.

In one case, a Windows patch altered all IP addresses on the OT side, and in another, communication with a Programmable Logic Controller (PLC) was lost, resulting in significant downtime for both manufacturers.

It’s no secret that unplanned downtime is bad news. According to an article from Forbes, “Overall, unplanned downtime costs industrial manufacturers as much as $50 billion a year.”

A study by Endeavor Business Intelligence on behalf of Panduit and The Journal From Rockwell Automation and Our PartnerNetwork™ magazine found that 9% of respondents indicated the cost of unplanned downtime is more than $100,000 per hour, and 25% indicated the cost is between $10,000 and $100,000 per hour.

The good news is that these issues are avoidable, and this served as a critical learning opportunity for these two manufacturers.

When the IT team for a composites manufacturer pushed out a Windows patch, it duplicated IP addresses on the OT side. Complicating things further, they lacked network segmentation, a well-defined OT network design, a DMZ, and specific support for OT communication. As the downtime costs ticked higher over the next six hours, we troubleshot the issues and got them back up and running. The manufacturer recognized the benefits of investing in robust OT protection measures compared to the losses of downtime, and we’re helping them implement a remediation plan, design, and providing remote support.

For a consumer packaging manufacturer, a seemingly straightforward addition of sensors to provide analytics turned into a crisis. Their network had flaws prior to the installation, and they were using unmanaged switches and lacked documentation, resulting in chaos. The machine went down, and they lost nearly two days of production time.

The manufacturer reached out to us for help, and through some savvy troubleshooting and technical support from our partners, we got their machine back up and running. They’re now investing in proper segmentation with a managed switch and documentation to implement the initial sensor additions.

Industry statistics emphasize that 83% of vulnerabilities are deeply embedded within the control systems network. That’s why it’s critical that you have an accurate list of vulnerabilities or, at minimum, a complete list of your OT assets.

You need to have a robust and comprehensive OT network infrastructure. The lack of network segmentation, a well-defined OT network design, a DMZ, and dedicated support of OT communications only made matters worse for the composites manufacturer. Make sure to prioritize the development and maintenance of a well-structured OT network to minimize vulnerabilities and enhance overall resilience.

As demonstrated by the consumer packaging incident, any network expansion, no matter how seemingly minor, requires strategic planning. The implementation of sensors and analytics tools should be approached with care, including the use of managed switches, thorough documentation, and a well-thought-out segmentation plan. Proactive planning not only prevents downtime but also ensures that technological advancements seamlessly integrate into your existing industrial infrastructure.

Comprehensive documentation is a critical component in preventing operational disruptions. Ensure that your industrial network includes meticulous documentation, from asset lists to network configurations. This will help you facilitate troubleshooting, accelerate incident response, and ultimately contribute to a more resilient and efficient industrial environment.

The SANS Institute recommends five crucial steps for ensuring OT network reliability, and they've become my guiding principles:

If you need help, contact us today. Our services are guided by industry standards and are manufacturer agnostic.

As you gear up for your next holiday celebration, keep in mind that the key to true enjoyment lies in a well-secured and reliable OT network. Don't let avoidable incidents disrupt your operations and force you to weather the costs of unplanned downtime. Partner with us to fortify your OT systems. Rexel can baseline your assets, network, and vulnerabilities. Through industry-standard remediation, we ensure project success and operational reliability for your industrial operations.

Here's hoping you can truly enjoy your next holiday celebration without any OT network issues.