May 21, 2018 • RBS

Categories: Security News

The other day I happened upon an article titled: “Vulnerability Management: Why the Problem Can’t Be Solved“, which made me curious for all the wrong reasons. As you can imagine, I strongly disagree with the premise that vulnerability management is a problem that cannot be solved. Having worked in the vulnerability intelligence field for more than 15 years, I recognize the challenges to implementing effective vulnerability management, but there are solutions too. For many security practitioners they are, unfortunately, not viewed as the quick and easy fixes they’d prefer. No blinking device is going to solve it. To do it properly it requires hard work. However, if you have access to a reliable vulnerability intelligence solution combined with proper asset management it can certainly reduce cost and resource requirements.

With disclosure of my own bias out of the way, the article referenced above is based on an interview with the Director of Security Strategy at Flexera. It focuses on the various challenges to vulnerability management and how they – based on their latest annual vulnerability report for 2017 – conclude that patch management is a saving grace: 86% of all vulnerabilities supposedly have fixes within the first 24 hours. I won’t spoil much by already saying that this claim is false, though it nicely fits the Flexera narrative, as they provide patch management solutions.

We’ve recently discussed how security vendors make false promises or oversell their products. We even had some fun digging into the F.A.K.E. Security booth at RSA. So we decided to keep the streak going and dig into some of the statistics in Flexera’s annual security report, share some thoughts on why we believe they are wrong, and what that means to the conclusions made about the effectiveness of patch management.

False Claims About Vulnerabilities Patched Within 24 Hours

According to Risk Based Security’s VulnDB Vulnerability Intelligence solution 76.8% of all vulnerabilities disclosed in 2017 have fixes available. That percentage is not a huge difference from the 86% claimed by Flexera… on the surface at least. Remember that Flexera in their report claimed that their percentage was fixes available within the first 24 hours. To check their claims, our percentage of vulnerabilities disclosed in 2017 with fixes available was generated in May 2018, which allowed a window from disclosure to fix of at least 5 months and up to 17 months.

In order to get real numbers, these can be gleaned by referring to VTEM (Vulnerability Timeline and Exposure Metrics) in our VulnDB product, which provide insights into many things including the responsiveness of vendors. When looking at vulnerabilities that have a fix available within 24 hours we found that those only comprised 41.7%. That is much lower than the 86% claimed by Flexera! Not only that, the average time from disclosure to fix for vulnerabilities without a fix immediately available is 37.5 days.

How can there be such a discrepancy? An initial thought and hypothesis could be that Flexera tracks patches better and this, therefore, impacts the metrics. That would be a great explanation, but alas – another spoiler – not the right one. In fact their coverage is not only narrower, but also much shallower than our VulnDB solution. This becomes apparent when reviewing some of their other statistics.

A Problematic Dataset For Generating Statistics

In the interview, Flexera’s Director of Security Strategy shares that their 2017 dataset covers vulnerabilities in about 2,000 products from about 200 vendors. Looking at our VulnDB dataset for 2017 we can see that the 21,661 unique vulnerabilities we documented (compared to the 19,954 “vulnerabilities” documented by Flexera) were across 10,498 products from 3,983 vendors. Flexera’s team seemingly missed vulnerabilities in about 8,500 unique products from about 3,800 vendors. That’s a huge difference and illustrates Flexera’s limited product coverage.

Flexera also shared another statistic: “more than 52 percent of all vulnerabilities counted in 2017 were found outside Microsoft, Adobe, Java, or Google Chrome and Mozilla Firefox browsers.” I think it’s fair to conclude based on this quote that Flexera is suggesting that approximately 48% of all vulnerabilities reported in 2017 are in products from either Microsoft or Adobe or in Google Chrome, Java, or Firefox. Even without digging deep, it is obvious that this is a very flawed statistic.

This is further confirmed by the data from VulnDB: Vulnerabilities in any products from Microsoft, Adobe, Google, Oracle, and Mozilla (yes, on purpose we significantly widened the scope and included all Google, Oracle, and Mozilla products!) accounted for 4,484 unique vulnerabilities i.e. 20.7% of all the reported vulnerabilities in 2017. If a subset of these accounted for almost half the vulnerabilities at Flexera (again, we counted all Google, Oracle, and Mozilla products – not only Chrome, Java, and Firefox), it speaks volumes as to how limited the data used to created the report is, and one is left to assume in their product as well. It also points out that their way of counting vulnerabilities produces inflated results. Remember that Flexera claimed to have covered 19,954 unique vulnerabilities in 2017. If so, there were about 9,578 vulnerabilities in products from those two vendors along with those three products. As we have pointed out, there were clearly not. Unless Flexera erred when calculating the statistics, it more likely suggests that they did not come close to covering 19,954 unique vulnerabilities in 2017.

Naturally, if a dataset is this limited, it has an impact on generated statistics and, more concerning, the conclusions drawn from it. While Flexera’s annual report usually garners some media attention, it has historically been criticized for its flawed statistics. The statistics in the reports are based on a dataset from Flexera’s in-house Secunia Research team, which Flexera acquired in 2015. As some may know, I’m quite familiar with that team as I personally built and managed it for 10 years prior to joining Risk Based Security. This gives me a unique perspective.

The problem is that the Secunia database does not track unique vulnerabilities. The focus is primarily on a per solution basis, which makes sense from a patch management perspective. One Secunia advisory may cover multiple vulnerabilities in the same product, if all have the same (or no) fix. More importantly, one unique vulnerability e.g. in a library may lead to multiple Secunia advisories being issued; one advisory per vendor issuing fixes for their products. If these are all counted as distinct vulnerabilities, it is obvious how this inflation ends up skewing not only their total vulnerability count, but also their statistics. A good write-up that goes into more detail with this counting problem was posted a few years ago using the OpenSSL library as an example. It was determined that they counted CVE-2014-0160, aka Heartbleed (which was a single vulnerability) as many as 210 times.

Regardless of how inflated we believe Flexera’s vulnerability count may be for 2017, it seems fair to conclude based on the numbers above and their limited dataset that it does not cover as many unique vulnerabilities as they claim. Unless Flexera radically changed Secunia’s methodology, their dataset has, to our knowledge, no reliable way of tracking unique vulnerabilities. Based on the numbers cited in the report, we believe that Flexera missed a lot of vulnerabilities in a lot of products, resulting in a skewed dataset that produced questionable statistics.

The Importance of Reliable Statistics

How can we be so sure of our analysis and the VulnDB dataset? We at Risk Based Security do track unique vulnerabilities (even though it can be extremely painful) as part of our VulnDB Vulnerability Intelligence solution. This allows us to generate highly reliable and accurate statistics just from looking at the data. As a result more accurate conclusions can be derived, which is key to a modern Vulnerability Intelligence solution when helping organizations make better risk decisions.

The facts are that 86% of the vulnerabilities that we aggregated in 2017 were not patched within 24 hours. Indeed ¾ of all the vulnerabilities did eventually have fixes available, so we agree that patch management is an important part of proper vulnerability management. However, it is very problematic when security companies and the media promote limited or flawed statistics that mischaracterize the situation and point organizations toward incomplete solutions.

Based on our analysis, patch management – while important – will not efficiently cover as broad a spectre of the reported vulnerabilities as claimed by Flexera. Nor will it do so as quickly. If an organization was solely relying on patch management, they would still potentially be exposed to at least a quarter of all the vulnerabilities reported in 2017 without fixes. In case of the ones with fixes, their infrastructure would not have fixes available in 86% of the cases within 24 hours, but more like 41.7%.

Another point we want to make clear is that we are not arguing that it’s necessary to fix every single vulnerability immediately or at all. It is, however, important to know about as much as possible about the vulnerabilities impacting your IT infrastructure. Otherwise, you cannot make informed decisions and properly prioritize your remediation resources. That is what access to a truly comprehensive and high quality Vulnerability Intelligence solution can provide.

There is no doubt that patch management has been and continues to be an important part of vulnerability management, but vulnerability management is more than just getting alerts whenever your infrastructure needs a patch applied. These days a proper Vulnerability Intelligence solution provides so much more including metrics like Code Maturity that allow assessing the security state of a given product or VTEM metrics that allow understanding how seriously and quickly a given vendor deals with vulnerability reports and addresses vulnerabilities.

Ultimately, accurate decisions require accurate data, and the vulnerability management “problem” can and has been solved! With so many reports being released from security vendors, organizations need to ensure that they understand the data being used and the analysis process – otherwise they are doing themselves a disservice by making any decisions based on it.

Our products
The Platform
Risk Based Intelligence
Learn more
Vulnerability Intelligence
Learn more
Cyber Risk Analytics
Threat Intelligence
Learn more
Risk Management
Learn more