5-Star Ratings – Just How Vulnerable Is That Shiny New Application?

Star-based ratings are everywhere you look these days. From hotel and restaurant reviews to doctors and lawyers, practically every service and seller imaginable is subject to some sort of performance score. These rating systems are so familiar in fact that they have become the de facto shorthand for making fast judgements about the quality of a product or service. Despite – or perhaps because of – their prevalence, the basis for how these ratings are developed is often overlooked, which can lead to dubious scoring or ratings of questionable validity.

Consider some of the most common examples of 5-star ratings.  

  • On the more rigorous end of the spectrum, the NHTSA (National Highway Traffic Safety Administration) provides 5-star crash ratings for automobiles.  You’ve probably seen the crash dummies and the slow-motion videos that the NHTSA uses to assess how a given automobile make and model does in certain crash conditions. These ratings are widely respected thanks in large part to the meticulous testing process and are used by organizations and individuals alike to make informed decisions.
  • On the opposite end of the spectrum, you’ll find 5-star ratings on many popular retail buying sites which are typically based on customer reviews.  Are they helpful? Yes, somewhat. Are they based on input from professional analysts who specialize in assessing the quality of that wireless keyboard or fleece jacket you’re eyeing?  Probably not.
  • Then there is the muddy middle ground. Hotels and restaurants have been assessed using 5-star rating systems for a long time. Official ones like those from Forbes, AAA, and Michelin are based on well-defined and established criteria.  They are grounded in data about the hotel and restaurant industries with common measurements used to assess the various establishments being reviewed.  Then there are the crowdsourced scores, largely based on aggregated customer feedback which can be quite subjective.

What does this tell us? That all ratings can bring some value to the table, but it is the more exacting and objective systems that can provide deeper insight into a product or service. When it comes to assessing the risk associated with using a piece of software or selecting a vendor for your next project, it’s all the more important to look to a thorough rating system. One such system is an objective comparison based on data for known vulnerabilities, using criteria such as:

  • How often vulnerabilities get released
  • How exploitable (easy v. hard) the vulnerabilities are
  • How much damage can an exploit cause

This is a helpful way to assess software investments. After all, even the slickest application can quickly lose its luster if it requires constant patching and puts a drain on already tight resources.

Risk Based Security’s VulnDB provides this objective vulnerability intelligence for vendors and their products.  Our expert research team assesses vulnerabilities for risk and exploitability. Our proprietary model calculates 5-star ratings for each software product and then aggregates 5-star ratings for each vendor based on their portfolio of products, using the history for all the vulnerabilities to derive the rating.

What can you use this information for?  

Vulnerability Evaluation: You can evaluate software products for vulnerability quality. For example, here’s ratings data for a screen sharing tool with a 5-star rating of 2.5.  Major factors in that rating are the number of vulnerabilities over the last 11 months and the fact that several vulnerabilities can be exploited remotely.

Product Comparisons: Another use for 5-star ratings is product comparisons. Below is a comparison of three similar products.  The first two are on the lower half of the 5-star rating scale, mostly likely due to the large number of vulnerabilities and relatively short amount of time between when new vulnerabilities are disclosed.  The third product has a very high 5-star rating, and even though the average CVSS scores are similar to the others, vulnerabilities occur on a much less frequent basis (only one every 174 days).

5-star ratings are available for many  products and services. When these ratings are based on objective data, they are useful aids for making informed decisions free of bias. The 5-Star ratings included in VulnDB provide meaningful insight into the performance of products and vendors over time. Whether used on their own or in combination with other objective performance measurements, VulnDB’s 5-Star ratings can provide a powerful tool for selecting your next service provider.