This page is part of a global project to create a better online reviews system. If you want to know more or give your feedback, write at [email protected] and we’ll grab a beer ;)
Following on what was said in “Unclear scale”, even with clear statements for each value, rating systems lack nuance.
As discussed in “Categorization,” all experiences are multi-dimensional. But it’s not just about different categories of review; even within a specific category, a numerical scale can’t capture the full richness of an experience. Reducing an experience to a single dimension (1 to 5) oversimplifies the reality of what it was.
Language, by nature, is nuanced.
Example: “Good and reliable” is not the same as “Excellent” or “Extraordinary,” yet all might be rated as 5 stars.
Nuances can also appear in the details of the experience:
Example: “There was a bit of moisture in the shower” is different from “Some clothes were left on the floor”. Both experiences might get the same rating in the cleaning category on Airbnb, but they resonate differently with individuals based on their priorities.
Fortunately, comments provide more detail about the experience, but comments aren’t aggregated into an “average sentiment.” Potential customers must read the reviews to access this information. 54.7% of shoppers read at least 4 product reviews before purchasing, while around 44% read 3 or fewer . This isn’t much.
A common mistake is to assume that increasing the number of scale values increases nuance. While 10-level or 100-level rating systems offer slightly more nuance, they still miss the semantic richness mentioned above and suffer from the same issues as simpler scales.
We could even argue that 5 stars is not enough to showcase excellence. Bryan Chesky himself has made it clear in the very good episode of Master of Scale with Reid Hoffmann:
CHESKY: So what would a ten-star check-in be? A ten-star check-in would be The Beatles check in, in 1964. I’d get off the plane and there’d be 5,000 high school kids cheering my name, with cards welcoming me to the country. I’d get to the front yard of your house and there’d be a press conference for me, and it would be just a mindfuck experience. So what would an 11-star experience be? I would show up at the airport and you’d be there with Elon Musk and you’re saying, “You’re going to space.” The point of the process is that maybe 9, 10, 11 are not feasible. But if you go through the crazy exercise of “keep going,” there’s some sweet spot between “they showed up and they opened the door” and “I went to space.” That’s the sweet spot. You have to almost design the extreme to come backwards. Suddenly, doesn’t knowing my preferences and having a surfboard in the house seem not crazy but reasonable? It’s actually kind of crazy logistically, but this is the kind of stuff that creates great experience.
“That’s how far numbers can take you”: this summarizes the problem of ratings regarding nuance. When we limit the number of choices, we impoverish the message. Language is complex, and capturing someone’s sentiment often requires a few sentences.
- Remove ratings; words are enough. This simple proposition, also mentioned in “Unclear Scale,” suggests that nuances are inherently captured in language. The challenge remains to aggregate this information meaningfully.
- “Would you say the product was good with X?”. This question, with a limited number of choices (e.g., “Yes, totally,” “Rather yes,” “No idea,” “Rather no,” “Absolutely not”), could capture more precise user sentiment on specific topics. While not as nuanced as a detailed comment, it provides valuable information without requiring potential customers to read all reviews.
Bizrate Insights, 2021