5 Steps to Industrial Patch Management

An effective patch management program in ICS is necessary for the safe procurement, deployment, testing, and integration of verified patches to keep Industrial Control Systems secure. It ensures the security of ICS against hackers and malicious threat actors. It becomes challenging to address critical security vulnerabilities in OS-based devices within the ICS environment. Therefore, patch management is needed to address these security vulnerabilities and fix security flaws in the ICS network.

The implementation of security patches in ICS environments is a way to mitigate risk and address vulnerabilities. However, patch management in complex ICS environments demands a proactive approach to support the needs of these systems. Hence, it becomes challenging for Industrial Control systems to implement patches effectively. Some of the challenges in ICS patch management are discussed below.

• Asset Visibility & Inventory
  • Lack of inventory/monitoring of end systems, active or passive.
• Patch Acquisition
  • Difficulty in monitoring patch releases for all systems/applications
• Patch Testing & Validation:
  • Lack of testing infrastructure and expertise to review, approve or mitigate patches in a workflow
• Patch Deployment
  • Time to deploy patches on critical assets, due to various different maintenance slots.
• Change Management
  • Additional efforts are required to document changes and update the baseline.
• Lack of a Centralized Dashboard
  • No Central dashboard to get patch posture status across the plant.
• SOC Visibility
  • Limited or no visibility of validated patches in the SOC (Security Operation Center)

This includes building relationships with product suppliers, creating an inventory of up-to-date devices, and evaluating the environment to determine its supportability using active/passive discovery tools.

An accurate inventory assessment is necessary to determine the scope of the devices and the software and patches in use. This will help establish a patch management system and allow the asset owner to identify which assets and devices have the vulnerability.

After getting an accurate inventory of all devices, it is time to collect specific information about each device. This data collection identifies any information that could be used or required to operate the critical infrastructure patch management system.

The following data should be collected: (Additional data is always useful and provide better insights into the existing inventory.

1. Asset Owner – This is the identification of the asset owner, custodian personnel, and resources capable of supporting it.
2. Product Vendor, make, and model number – This information can be used later to contact product suppliers.
3. Asset Versions – This is the version that is associated with any hardware components and their associated firmware. It can also include OS or software versions associated with the hardware.
4. Asset Roles & Classification – This describes each asset function and the classification and grouping of devices that perform similar functions.
5. Network Visualization – This is the network structure and architecture. Remote access systems, such as management and support systems, should be included in this layout.
6. Applications visibility – This information is vital and identifies which software has been installed or not on the assets including its patch status.
7. Product Lifecycle – This indicates the status of each asset support level as per the product manufacture (OEM). It is important to note the date of the expected or known end of support for product suppliers. This data should be reviewed periodically to allow for controlled changes to support plans.
8. Asset Criticality – Each critical process depends on the interdependencies and interaction of computer systems to function.
9. Vulnerability assessment tools Applicability – This indicates whether assessment tools can either be run against the system manually or automatically.
10. Configuration Baseline – Note that any configuration information must be captured before a modification and then reapplied. The latest configuration backup will serve as a baseline for future changes.

Many automatic scanning tools use new plug-ins regularly. Every asset owner should monitor and control the configurations of their tools and test any plug-ins before they are included in an automated scanning tool.

Automated tools might not be capable of collecting all the elements of asset inventory, including patch levels, ownership, and criticality. Therefore, a manual data entry component is often required. Some inventory collection tools can be integrated into automated patch distribution software.

Passive scanning tools are an effective alternative option to discover critical assets without impacting the critical environments.

The monitoring and evaluation process helps to make a clear decision on whether or not to deploy other countermeasures, install the patch, or adopt alternative control.

There are many methods to monitor and identify patches that need to be installed in a critical infrastructure environment, like:

1. The asset owner requests a list of patches available from the manufacturer or supplier of OS platforms to be added to their inventory.
2. A notification is sent by the OS platform manufacturer or product supplier to notify users or post a list of patches that have been released recently or are available.
3. Asset owners can directly evaluate the patches on their critical assets and then compare them against the available or newly released patches.

Determining Patch Applicability (Identification Stage)

Asset owners should assign tasks to determine patch applicability by doing the following:

• Maintaining a list of software and hardware items and managing inventory.
• Notifying and/or documenting patches to start the patch evaluation process
• Investigating and researching to determine if technical advisories or knowledge bases are security-related.
• Verifying that the patch works in the environment by comparing it to the software/hardware inventory.

Impact Analysis:

Assessing the potential risks associated with applying the patch to an environment is important. This involves assessing the potential impact on the production environment and the importance and severity of the vulnerability.

The risk assessment will provide information about the benefits, risks, and challenges that will help you decide whether to install the patch. The priority of deployment and installation should be determined based on severity.

Patch testing is an integral part of the industrial patch management process. The asset owners maintain an identical lab to test the patches and observe the changes and behavior of the asset before approving it for the final deployment.

Before installing security patches, the owner/operator must test and qualify them in a quality assurance environment. This includes live data feeds and interaction between other system components, operators, and operating procedures.

The asset owners sometimes face the challenge of not having a testing environment, they can adopt the methodology to test the patches on secondary systems with a clear roll-out plan to ensure the least impact on the production environment.

This includes the following processes:

1. Patch Testing Process: It includes Patch file authenticity, Review changes, Installation procedure, Qualification & verification, Removal procedure, and Risk mitigation.
2. Determining Patch File Authenticity: it is important to authenticate the patch files before installing or testing them. For this, you can determine the patch source, verify the file size, checksum, and digital signature, and scan the patch for viruses.
3. Installation Procedure: The installation process requires checking the technical notes and platform information, product supplier installation instructions, identification of prerequisites, identifying target devices and testing samples, and installing a patch for creating a new environment.
4. Patch Qualification and Validation: As part of their qualification process, the asset owner might rely upon iterative testing phases performed by different groups.
5. Review Functional and Security Changes from Patches This allows you to verify if the patch has affected the IACS device’s functionality, operability, or reliability. It included:

• • Effects on system reliability
  • Effects on system performance
  • Effect on fault-tolerance capabilities or redundancy
  • If the critical component is operational, will the patch be installed?
  • Ability to roll back the patch in the event of unforeseen effects

In case the above cannot be performed alternative risk mitigation strategies can be adopted to reduce the attack surface.

Risk Mitigation Alternatives: If the patch isn’t installed, but the security vulnerability persists, several options exist to mitigate the security risks: like:

• Reconfiguring a product
• Remove or disable the vulnerable component or feature that requires the patch; remove affected software.
• The patch is required to disable the startup of the vulnerable service.
• Network filtering controls, intrusion detection systems (IPS), rules and signatures, program and executable access control, and security policies that include technical support solutions are all possible.
• Based on cost and business justification, investigate the possibility of replacing the vulnerable device.
• Secure outbound gateways that are only accessible to authorized personnel with strong access controls

The patch deployment and installation process include notification, preparation, scheduling, installation, and verification.

Depending on the complexity and size of the organization as well as the number of devices, physical locations, and available data network communication, there may be extensive preparations before patches can be installed:

• The patch installation personnel may not have been involved in testing.
• They may need to review the instructions and procedures to ensure they are consistent with testing and a reliable installation.

Phased Installation

It can be challenging to schedule patch installation depending on your critical environment. However, there are options:

1. Waiting for the next scheduled outage
2. Queuing for patching activities during the next scheduled outage
3. Patching during unscheduled outages is often seen as an additional risk.
4. First, update standby/secondary devices, fail-over to it, and finally, complete the active/primary devices.
5. Separate maintenance windows to schedule for devices that are offline or unavailable.
6. As per business constraints and the availability of qualified personnel, you can install patches in phases.

Verification of Patch Installation

The final step in the patch deployment and installation process involves verifying that the patch has been installed on all affected devices.

These are the methods to verify the installation of the patch/vulnerability mitigation:

1. Comparison of the software version before and after the update
2. Logs from the configuration change detection system reviewed
3. Examining reports from patch management software to verify that the patches have been installed and that files have been updated
4. Performing vulnerability scans of affected devices.
5. Validating the correct application of compensating controls.

The final step is to report the plant cybersecurity posture state to the higher management and most importantly to the Security Operation Center (SOC) to ensure the non-mitigated vulnerabilities can be monitored and alternative controls such as

• Intrusion Detection System
• Endpoint Controls
• Host-based Intrusion Prevention System
• Honeypots
• Events Monitoring

Can be fine-tuned to detect, protect and mitigate the threat.

This article describes a sequence of activities, tasks, and requirements to help Asset owners effectively manage a patch management system. The information provided provides them with an actionable roadmap and is relevant to the OT/ICS Patch management system’s challenges. If understood and followed correctly, the asset owners can establish effective patch management programs, reduce vulnerabilities, increase effectiveness, and increase overall critical infrastructure reliability.