How Reliance on Inadequate Data Can Lead to Faulty Conclusions
March 2, 2021 • RBS
Recently, a company called Redscan released a report that analyzed vulnerabilities in 2020 via the National Vulnerability Database (NVD) data, which is based entirely on MITRE’s CVE. Redscan is a company that performs penetration tests and offers a managed security platform. Based on their web page, they do not run a vulnerability database or aggregate disclosures themselves. Despite that, they made quite a few bold conclusions including a “record breaking number of vulnerabilities reported in 2020”. These are the softballs of vulnerability statistics.
Looking at the actual report in more detail, we wanted to provide some input and offer caveats and qualifications to their conclusions and statistics, as well as contrast their findings to our own, based on a much larger and more comprehensive dataset. For example, in their key findings they say that, according to NVD, there was an average rate of 50 CVEs per day. Compare that to VulnDB, where we saw an average of 69 per day, and one can immediately see a coverage gap between the two.
One of the statistics that they presented stood out to several of us at Risk Based Security, where they say that there were “more high and critical severity vulnerabilities in 2020 than the total number of all vulnerabilities recorded in 2010 (4,639 including low, medium, high, and critical)”. Looking at VulnDB, we see there were at least 9,360 vulnerabilities disclosed that year, almost double what CVE aggregated, rendering their statistic inaccurate. It also reminds us that if one wants to create a shocking statement, one can choose an arbitrary year that best suits your needs. That same comparison to 2015 probably doesn’t work, but if they compared it to 2000, they could have had more “fun” with the even bigger gap.
This comparison between 2020 and 2010 also highlights another problem. Analyzing data may seem like a straightforward endeavor, but you simply can’t do it properly unless you understand how the data was collected and the caveats that come with it. In 2010, CVE had different leadership, higher standards, and different abstraction rules. CVE in 2020 is a very different beast; trying to make a direct comparison of data from those two years is more akin to comparing an alpaca to a llama. They are both camelids but the similarities stop shortly after that.
In a similar vein, on page six of the report they include a section and chart about “Attack Complexity” or “AC” which goes back to 1988. Unfortunately, this chart has no actual value due to a significant change in CVSS. Under CVSSv2, the “AC” metric was formerly “Access Complexity” which “measures the complexity of the attack required to exploit the vulnerability once an attacker has gained access to the target system.” When CVSSv3 was introduced, AC changed to “Attack Complexity” which “describes the conditions beyond the attacker’s control that must exist in order to exploit the vulnerability.” This is a significant difference and means that at some point on this chart, the definition of “AC” changed and a straight comparison cannot be made.
When did that change take place? NVD started scoring CVSSv3 for the first time in 2016 and did not choose to “backfill”, or retroactively score, older vulnerabilities. That means the chart Redscan provided can only be used to compare 1988 – 2015. Everything from 2016 – 2020 would have to be looked at separately as the data is completely different.
RELATED: The Value of Backfilling
With any significant dataset, it is possible to pull out and focus on smaller points of interest. For example, we have written about Electronic Voting Machines which represent a tiny fraction of all vulnerabilities disclosed. However, the context around that was the severity of those vulnerabilities in the face of upcoming elections, and the potential for just one vulnerability to have severe consequences.
The flip side to that is focusing on a tiny subset of vulnerabilities that has no apparent narrative other than “the numbers changed”. Redscan’s report does this by focusing on physical vulnerabilities, saying that they saw a “large spike” in 2020. In reality, there was steady growth from 2018 – 2020. The significant spike occurred from 2017 to 2018 if anything, but we’re still talking about a gap of fewer than 300 physical vulnerabilities in each of those years. Meanwhile, the report also talks about ‘Adjacent’ vulnerabilities, which is considerably more interesting as far as the numbers go, but they chose not to focus on them.
Another issue that can manifest in conducting analysis is focusing on the narrative that accompanies each point of examination, without considering how these comments compare to each other. For example, Redscan says that the “prevalence of low complexity vulnerabilities in recent years means that sophisticated adversaries do not need to ‘burn’ their high complexity zero days on their targets and have the luxury of saving them for future attacks instead.” This is their conclusion after looking at the complexity of vulnerabilities based on NVD’s CVSS scoring. Anyone well-versed in vulnerability intelligence knows this is a really bad metric to use for any real analysis of severity.
Later in the report they examine the privileges required to exploit vulnerabilities in the same year, and come to this conclusion:
“It is also encouraging that the proportion of vulnerabilities requiring high-level privileges has been on the increase since 2016. This trend means that cybercriminals need to work harder to conduct their attacks.”
On one hand, there is a prevalence of low complexity vulnerabilities, and attackers don’t need to work hard. On the other hand, the privileges required to exploit vulnerabilities is on the rise, so attackers have to work harder to exploit them. While both may be technically true, and some attackers may align with one statement or the other, using such blanket statements doesn’t mix well.
We don’t think that Redscan is spreading misinformation or purposefully manipulating the narrative to fit their needs. This dissection just illustrates how nuanced and specific the vulnerability disclosure landscape can be. When it comes to vulnerabilities, there is so much more than what is aggregated in CVE/NVD. In fact, CVE is missing over 80,000 vulnerabilities that you may not know about.
Caveats are important so we always make sure that our clients understand how they can affect their Vulnerability Management programs. For the latest details involving vulnerability trends (with all the disclaimers), check out our 2020 Year End Vulnerability QuickView Report. For organizations wanting a full picture of their risk profile, we invite you to see for yourself the importance of comprehensive, detailed and timely vulnerability intelligence.