Risk assessment for applying patches

by admin on April 6, 2012

What will be the basis for performing a risk assesment while applying patches?

In a regular system implementation or upgrade or enhancement, the risk assessment is based on URS or FS to determine the extent of validation testing required.  To discuss criteria for performing risk assessment I asked SQA Solution folks to establish a risk assessment checklist for applying a patch.

Based on the output of this RA, I would determine the extent of testing and regression testing required.  The patches usually come with technical description / release notes of what they are fixing. If they are patches then they are not necessarily ‘changing’ functionality but ‘fixing’ existing functionality to how it should have been. An example we had was date format could not be changed within in the system, even though the functionality was stated it could in the design. A patch was applied post go live and we had to verify the results did not impact quality critical data (there were a lot of functions using the date in the system). We tested all quality critical modules (by type, not every instance) using the date format and a statistical testing method to verify the data.
The risk assessment determines whether/how these fixes affect critical functions and yes, the extent of verification. Is best for such patches to be tested in a sandbox system to ensure nothing obvious is wrong before updating the live system and verifying results.

The system should also have a procedure for patch management as part of the deliverables, if it doesn’t you might want to consider developing one (there are many templates depending on system type). If it does have an SOP, it should give you the necessary guidance (if it is not good enough then refer to writing one).

The following may be considered as an example for performing a risk assessment applying to patches.

Vulnerability – The vendor has identified flaws in the security design of the system; however, new patches have not been applied to the system

Threat – Unauthorized users (e.g., hackers, disgruntled employees, computer criminals, terrorists)

Threat Action – Obtaining unauthorized access to sensitive system files based on known system vulnerabilities

 

Based upon the purpose of the ‘patch’, and a change impact assessment which carefully reviews the release notes), some patches
– Are automatically applied to Production (e.g. anti-virus definition updates)
– Are manually applied to Production with no testing (e.g. critical security patch for the operating system)
– Are applied first in a QA/Test instance, with limited or no regression testing (e.g. minor application bug fix)
– Are applied first in a QA/Test instance,with more extensive regression testing (e.g. major application fix)
– Are applied in a development environment, configuration changes made, then promoted to QA/test for more extensive testing (e.g. major application upgrade)

For all of these, the User Requirements need not have changed.

Also see Appendix S4 in GAMP 5 (Patch and Update Management).

We generally look at risk / impact analysis of changes to the infrastructure along three dimensions: error – failure – threat. Error is “Oops!” and the realm of things like HACCP. Failure is the realm of “What was that?!?!” and the realm of FMEA, and threat is the realm of the guys In the black hats and Fites-Kratz. etc.

I will leave you to negotiate with whomever as to whether or not you actually have to generate any of these.

Speaking as a software engineer / CISSP, for your situation, I’d recommend an in-service qualification rollout of your patches to a subset of low-criticality systems with careful, well-documented monitoring of functional and operational performance. That is, pick a few non-product-quality systems, apply the patches, exercise them, make sure the numbers add up, and throw a heavy load at them to be sure they don’t buckle under the stress. (NOTE: Do this at 3:00 AM – The voice of experience speaks!)

As you become more confident that the patch is holding water, venture into deeper, rougher waters by rolling the patches onto higher criticality systems, until you have all of the boxes done. Again, document thoroughly. You are, in essence, doing “live fire” regression testing.  A further point is that Risk Assessment should cover the risk of applying the patch versus the risk of NOT applying the patch. A simple example is an OS update where a security flaw has become apparent and there are known exploits in operation. In such circumstances the risk of problems from applying the patch is likely to be low (we can reasonably expect it has been tested by the OS developer) whereas there is high vulnerability of the unpatched system. In such circumstances applying immediately is a defensible position

#SQASolutionShare on FacebookShare on Google+Share on LinkedInTweet about this on TwitterEmail this to someone

Previous post:

Next post: