Best Practices and Recommendations Firmware Update and Verification

IT management, support and cybersecurity teams understand the importance of keeping their business’ OS and applications operational, up to date and free from known vulnerabilities. Businesses often spend significant resources updating and patching their OS however the same processes are often not extended to the firmware that determines the behaviour of system hardware. In many cases, firmware updates are never done or at best, only done so because of a malware incursion or operational failure. Not installing a firmware update may seem fairly low-stakes but these patches are often the difference in the device’s security and working reliably.

The InSight VNOC team believe a disciplined process of both operating system and firmware management is an essential element of good operational and cybersecurity practice and have provided information below that outlines best practice and our recommendations to ensure systems are maintained effectively. These should be considered by clients when developing and implementing their own OS and firmware update strategy.

Other innovations include:

1. Establish a firmware update policy

Every client should aim to create update policy that outlines establishes when OS and firmware should be updated. These should account for factors including the risk of not updating the need to new features and potential cost of downtime. Installing a firmware update can be time-consuming, so it’s important to select a time that is not critical to users in the event of downtime but also in which your staff can see the update through to completion.

The policy should establish buy-in from all IT and technical stakeholders, establish process for emergency updates due to critical security issues and establish criteria to determine if an update should be deployed based on security and feature impact. 

Once developed, this update policy will be shared with our VNOC team to deliver and ensure appropriate change management is followed.

2. Establish OS and firmware update visibility

Visibility into OS and firmware versions can be achieved in a variety of ways. Our VNOC team use a combination of vendor agnostic and vendor specific platforms to identify when OS and firmware updates are available for system components. Daily scans of devices identify current state, vulnerabilities and operational status while also providing live, 24/7 detection of issues that may adversely affect deployed systems.

3. Develop tooling and skills needed for testing, rollout, and rollback

Testing and deployment of OS and firmware is completed based on a series of established process designed to mitigate risk and deliver effective update management across deployed systems. Staff are trained to ensure each specific device or system type is updated effectively and that the processes are repeatable. 

InSight’s VNOC has a phased rollout process that sees initial testing of new OS and firmware occur in our demo environment before it is deployed to a nominated client test system then to wider distribution on acceptance. Each system and hardware type is addressed by a separate process to ensure success. Establishing the rollout program is a critical part of the OS and firmware management. Alongside our VNOC team, each client will identify spaces categorised as “Staging” systems, “General” systems, and “Executive” systems. These designations will determine when updates occur and are referred to as “rings”.

MTRP Patching

All device firmware updates are undertaken in line with the clients update policy and are only deployed following written confirmation by the client’s representative. 

The MTRP update service manages the Windows OS, Microsoft Teams application and the firmware for the connected appliances (NUC, Camera, Microphone, Ingest and control Touch Panel). Patching is handled through the MTRP managed services portal. The VNOC team assigns each client room/system to specific update “rings” which dictate how long after the patch release date, the room inside the rings gets the update applied to it. These ring groups are assigned through consultation with the client. 

Once accepted by our VNOC team, patches are deployed to the client “Staging” ring first. Once deployed, these systems are analysed to ensure successful version upgrade has occurred and that no changes to configuration or operation function. Testing continues for two weeks and once accepted, updates are applied to the “General” ring. Systems are again analysed and tested over a two week period and once accepted the “Executive” rooms are patched.

MTRP Testing and Rollback 

The VNOC team undertake initial OS and Firmware testing in two of our own demo rooms at InSight Systems. These rooms are part of our staging ring and consist of two different hardware configurations. On patch release day and over a two week period, we undertake significant analysis of each system and have an operational checklist that is utilised to determine if there are any functional issues. If faults are detected, we work directly with Microsoft to assess and rectify the problems. Once accepted, we then move to deploy each update to our clients “Staging” ring.

Clients that utilise the MTRP service benefit from being part of a global network of IT/AV professionals that are constantly monitoring Microsoft Teams Rooms for faults, operational and security issues. If an issue is detected, our VNOC team works directly with Microsoft to investigate and resolve any potential impact. This process also helps to improve fault detection and rectification algorithms for the wider global MTRP community. 

If an issue is unable to be remediated immediately and is considered an operational risk to our client and their users, our VNOC team can provide Microsoft supported, streamlined rollback of OS or Firmware.

Non-MTRP Testing and Firmware Update Process

For non-MTRP devices, the firmware update deployment is managed in a similar process to MTRP devices. Utilising vendor specific management platforms, our VNOC team scan devices and are notified when new firmware versions are released for general use. Our team deploy the updates to our demo environments located in the staging ring. On patch release day and over a two-week period, we undertake significant analysis of each system and have an operational checklist that is utilised to determine if there are any functional issues. If faults are detected, we work directly with the specific vendor to assess and rectify the problems or rollback to a previous version if required.

Once accepted by our VNOC team, firmware patches are deployed to the client “Staging” ring first. Following deployment, these systems are analysed to ensure successful version upgrade has occurred and that no changes to configuration or operation function. Testing continues for two weeks and once accepted, updates are applied to the “General” ring. Systems are again analysed and tested over a two-week period and once accepted the “Executive” rooms are patched.

Client-Side Testing and Change management

Our VNOC team recommend that the client designate at least one space at their location to serve as a staging system to test OS and firmware patches. Identifying a staging space on premises enables our VNOC team to test patches within the local network and hardware environment. Testing will be conducted by the VNOC team remotely and verified with the assistance of a designated onsite client contact (IT support tech or office manager). Once User Acceptance Testing (UAT) has been completed, the VNOC team will proceed with the patching as per the sections above.

If the client has change management policies in place, we can assist with creating the documentation required to push this through the approving body.

4. Make OS and firmware support a priority in hardware purchasing decisions

InSight Systems ensure that our systems are designed and deployed using an established list of approved vendors. Too often, the fractured nature of our industry means there can be a wide variance in how vendors support their products post sale. By ensuring we partner with reputable and established manufacturers, we mitigate the risk of products becoming unsupportable and high security risks for clients.

Want to outsource your firmware updates?

As we’ve talked about, our VNOC team helps clients across Australia simplify their device management by taking care of every firmware update that comes available. They can remotely monitor your devices 24/7 to ensure you always have availability. If you would like more information, please contact us on 1300 369 451 or email sales@insightsystems.com.au