All reviews don’t count the same

This page is part of a global project to create a better online reviews system. If you want to know more or give your feedback, write at [email protected] and we’ll grab a beer ;)

Consider this common scenario: a reviewer gives a rating without a comment. This type of review is less trustworthy than one where the reviewer took the time to describe their experience. Despite this, both ratings impact the average equally. Even short comments don’t offer much help to readers who seek useful information, and still count the same.

I bet you wouldn’t consider this review
I bet you wouldn’t consider this review

A comprehensive meta-analysis by Watson and Wu 1^1 analyzed customers' perceptions of online reviews, gathering many existing papers on the topic. It outlines various parameters that determine a review's credibility. Based on this article, additional literature and my understanding, here are a few criteria that influence the credibility of a review:

  • Age of the Review: Recent reviews carry more weight, especially when there are many. A study showed that 83% of consumers agree or somewhat agree that reviews must be recent and relevant in order to care about them 2^2. Freshness is associated with quality and usefulness 3^3. Trustpilot takes that into account and factor in review age in their TrustScore calculations.
  • Length and Depth: Longer and more comprehensive reviews are deemed more trustworthy. Review depth impacts credibility 4^4 5^5.
  • Precision: The more detailed the review, the more credible it appears. Strong arguments bolster credibility 6^6 7^7.
  • Moderate Opinion: Balanced reviews that highlight both positives and negatives are more trustworthy. Evidence shows that extreme reviews are less helpful for experience goods 4^4 8^8.
  • Context & Information: Reviews that include relevant details are more credible. If specific conditions don't apply to the reader, they may disregard the review.
  • Identity Disclosure: Revealing the reviewer’s identity 9^9, such as their real name and location, as well as their reputation and community participation 1^10^0, enhances perceived helpfulness.
  • Photos: Reviews with photos are more trustworthy as they add depth and context.
  • Integrity: Diplomatic and respectful reviews are seen as more credible. Emotional responses such as entertainment or irritation can affect credibility 1^12^2 1^13^3.
  • Conformity with Other Reviews: Reviews consistent with the majority are perceived as more credible 6^6.
  • Conformity with Expectations: Reviews that align with readers’ prior beliefs about a product or service are viewed as more credible 6^6.

Companies are aware that reviews don’t carry the same weight and adjust the average rating based on other characteristics (e.g., Amazon, Trustpilot). However, these adjustments are often opaque, and the specifics of the algorithm are not disclosed. One certainty is that newer reviews are weighted more heavily.

💡
Exploration:
  • Selection of criteria. Allow potential customers to filter reviews based on their criteria. It’s impractical to read all comments for specific information. A search bar helps, but different words can apply to the same criterion, making it less effective.
    • Amazon's AI-generated summary with filter options is a good example of what can be done. Unfortunately, it’s not available for every product and seems to be in phase test only for now.
    Amazon's AI-generated summary with filter options is a good example of what can be done. Unfortunately, it’s not available for every product and seems to be in phase test only for now.
    Google reviews’ search filter is not enough: different words could relate to the same concept.
    Google reviews’ search filter is not enough: different words could relate to the same concept.
  • Keep user preferences. Store user preferences to show the most relevant reviews first and potentially create a “personal average rating.” This is particularly useful for industries with frequent recurring use, like movies or restaurants. However, this approach raises data privacy concerns that need addressing.
  • Marking reviews as “Helpful”. Websites where readers can upvote reviews use this as a reputation metric. The current challenge is distinguishing whether an upvote indicates the review helped someone make a decision or if it resonated with their own experience. Platforms should clarify this distinction, potentially adding a step in the review process.
    • Amazon's "Helpful" button. It’s not clear whether actual buyers or potential customers marked it as helpful.
    Amazon's "Helpful" button. It’s not clear whether actual buyers or potential customers marked it as helpful.
  • Display similar experiences. Indicate how many reviews share a similar sentiment. This helps users quickly gauge the credibility of a comment without reading through all reviews. AI could capture this information, and reviews with the most similarities should be highlighted first.
    • Simple design proposition for Airbnb.
    Simple design proposition for Airbnb.
  • Weighted average rating. Make the weighted average rating calculation transparent. Platforms like IMDb, Amazon, and Trustpilot use weighted averages but don’t disclose their algorithms, which can affect trust.
  • Peer reviews. Highlight reviews from friends and connections first. Recommendations from people we know are the most influential.
  • “Certified Reviewer” status. Amazon’s certified reviewer status enhances credibility. Detailed and objective reviews from certified reviewers should carry more weight in the average rating. The criteria for certification should focus on review quality rather than the quantity of reviews left.
    • Amazon's reviewer status increases credibility.
    Amazon's reviewer status increases credibility.
  • Include External Context. For example, Airbnb displays information like the length of the stay and how long the user has been on Airbnb.
    • Airbnb displays additional context to reviews.
    Airbnb displays additional context to reviews.

    A useful piece of information for products would be the time elapsed between the purchase and the review, helping readers understand if the satisfaction has lasted over time.

    Original Amazon review. The date of purchase is already present.
    Original Amazon review. The date of purchase is already present.
    Redesign attempt. Changing the date to the time passed since the review, and adding the time passed between the purchase and the review.
    Redesign attempt. Changing the date to the time passed since the review, and adding the time passed between the purchase and the review.

1^1 “The Impact of Online Reviews on the Information Flows and Outcomes of Marketing Systems”, Watson and Gu, 2021.

2^2 2021 State of Reviews, Podium.

3^3 “Analyzing key influences of tourists’ acceptance of online reviews in travel decisions”, Chong et al., 2018.

4^4 “Research Note: What Makes a Helpful Online Review? A Study of Customer Reviews on Amazon.com”, Mudambi and Schuff, 2010.

5^5 “Understanding the importance of interaction between creators and backers in crowdfunding success”, Wang et al., 2018b.

6^6 “Credibility of Electronic Word-of-Mouth: Informational and Normative Determinants of On-line Consumer Recommendations”, Cheung et al., 2009.

7^7 “Beyond the Credibility of Electronic Word of Mouth: Exploring eWOM Adoption on Social Networking Sites from Affective and Curiosity Perspectives”, Fang, 2014.

8^8 “What Makes a Review Voted? An Empirical Investigation of Review Voting in Online Review Systems”, Kuan et al., 2015.

9^9 “Examining the Relationship Between Reviews and Sales: The Role of Reviewer Identity Disclosure in Electronic Markets”, Forman, Ghose, and Wiesenfeld, 2008.

1^10^0 “Did you Tell me the Truth?: The Influence of Online Community on eWOM”, Yang, Mai and Ben-Ur, 2012.

1^12^2 "E-WOM from e-commerce websites and social media: Which will consumers adopt?”, Yan et al., 2016.

1^13^3 “Antecedents of attitudes toward eWOM communication: differences across channels”, Gvili and Levy, 2016.

➡️ Next up: Unrepresentative set of reviewers