Scrape Competitor Reviews

Why this is a goldmine

Your competitors have already spent the time and money to acquire users and learn from them. Their public reviews on sites like G2, Capterra, the App Store, and Trustpilot are a raw, unfiltered feed of their customers' pains and desires. By analyzing this feedback, you get direct insight into what a validated audience in your market actually wants, without spending a dime on discovery yourself.

This process uncovers three key opportunities. First, 'pain-points' you can solve with a new feature or a better product. Second, 'deal-breakers' that cause churn, which you can use for positioning ('Unlike {Competitor}, we offer...'). Third, 'missing features' that you can build to attract a new segment of the market. It's one of the most efficient ways to build a data-driven product roadmap.

Three real examples

Bonus: Plug your competitor's most-praised features into your own customer surveys to see how you stack up on the things their users value most.

The step-by-step playbook

1. 1

   Step 1: Identify your top 3-5 competitors

   List the direct and indirect competitors that your customers mention. Look at 'alternative to' pages and review sites to find the key players in your space. Don't just pick the market leader; include up-and-coming players too.

2. 2

   Step 2: Find the review hotspots

   Locate where their customers leave reviews. This includes software marketplaces like G2, Capterra, and GetApp; app stores like the Apple App Store and Google Play Store; and general review sites like Trustpilot or even Reddit threads.

3. 3

   Step 3: Scrape the reviews

   Use a web scraping tool to extract all the reviews for your chosen competitors from these sites. Aim to get the review text, rating, date, and any other metadata available. You'll want at least a few hundred reviews per competitor to get meaningful data.

4. 4

   Step 4: Clean and consolidate the data

   Your raw data will be messy. Consolidate it into a single spreadsheet or database (like Airtable or Google Sheets). Standardize the columns (e.g., 'rating', 'review_text', 'source', 'date').

5. 5

   Step 5: Tag reviews with AI

   Feed the review text into an AI model to automatically categorize each one. Use tags like 'Bug', 'Feature Request', 'Pricing Complaint', 'Positive Feedback', 'Customer Support Issue'. This is where the magic happens.

6. 6

   Step 6: Analyze the tagged data

   Filter and sort your data to find patterns. What are the most common 'Feature Requests'? What themes emerge from the 1-star reviews? What do people who give 5-star reviews consistently praise? Quantify your findings--e.g., '25% of negative reviews mention poor customer support'.

7. 7

   Step 7: Synthesize and create action items

   Turn your analysis into a concrete list of opportunities. Frame them as user stories or jobs-to-be-done. For example: 'Users need a way to integrate with Slack so they can get notifications without leaving their workflow'. Prioritize these based on frequency and alignment with your strategy.

Template: Internal Insights Memo

You've done the analysis. Now you need to share it with your team to drive action. Use this template to summarize your findings and propose next steps.

Subject: Key Insights from Competitor Review Analysis

Hi Team,

I've completed an analysis of ~{Number} reviews for our top competitors ({Competitor 1}, {Competitor 2}, {Competitor 3}) from sites like G2, Capterra, and the App Store.

The goal was to identify opportunities for our product and marketing roadmap. Here are the key takeaways:

Top 3 Pain Points with Competitors:
1. {Pain Point 1 with data, e.g., '20% of negative reviews mention the slow UI'}
2. {Pain Point 2 with data}
3. {Pain Point 3 with data}

Most Requested Features They Lack:
1. {Feature Request 1, e.g., 'A native mobile app'}
2. {Feature Request 2}
3. {Feature Request 3}

Biggest Marketing Angle Opportunity:
Our competitors' customers consistently complain about {Common Complaint, e.g., hidden fees}. We can leverage this in our messaging by highlighting our {Your Contrasting Feature, e.g., 'transparent, all-inclusive pricing'}.

Proposed Next Steps:
- Product: Let's discuss the feasibility of building an MVP for {Feature Request 1}.
- Marketing: Can we spin up a landing page comparing us against {Competitor 1} on the basis of {Pain Point 1}?

I've attached the full report. Let's schedule a 30-minute meeting to discuss this further.

Best,

{Your Name}

Key Questions to Ask

• What are the most frequent keywords in 1 and 2-star reviews?
• What features do users mention in 5-star reviews that we lack?
• Are there complaints about pricing, and what model are they complaining about (per-seat, usage-based, etc.)?
• What integrations are customers asking for most often?
• What words do customers use to describe the product? (e.g., 'easy', 'clunky', 'powerful', 'confusing')

Focus on the 'why' behind the rating. A 1-star review that says 'It's too expensive' is less useful than one that says 'It's too expensive for a team of my size because the per-seat model punishes growth'.

Recommended tools

Prices are indicative and change - check the vendor before buying.

Do it faster with AI Prompts

Mistakes that kill this play

• ×Focusing on vanity metrics. 'They have 5,000 reviews' is not an insight. '30% of their negative reviews in the last 6 months mention bugs with the new update' is an insight.
• ×Ignoring review recency. A complaint from 3 years ago might have already been fixed. Focus on reviews from the last 6-12 months.
• ×Forgetting qualitative context. Don't just count keywords. Read the reviews to understand the emotion and the 'job-to-be-done' the user is struggling with.
• ×Only reading the bad reviews. Good reviews tell you what the competitor's 'stickiest' features are and what their customers value most. This is crucial for positioning.
• ×Analysis paralysis. The goal isn't a perfect academic paper. The goal is to find 1-2 high-impact opportunities and act on them quickly.
• ×Violating Terms of Service. Be mindful of the scraping policies of the websites you're targeting. Scrape responsibly and ethically.

Your First 5 Days

• Day 1: Finalize your list of 3-5 direct competitors. For each one, find the top 2-3 websites, app stores, or communities where their users post reviews. Create a master list in a spreadsheet.
• Day 2: Set up your scraping tool (e.g., Phantombuster). Start the scraping process for your first competitor. While it runs, manually read through 20-30 reviews yourself to get a qualitative feel for the data.
• Day 3: Your raw data is ready. Clean it up and consolidate it into one master file. Use an AI prompt to start categorizing the first 100 reviews. Refine your categories and prompt as needed.
• Day 4: Batch-process the rest of your reviews with your AI prompt. Once categorized, pivot the data. Create simple charts: 'Review Count by Category', 'Most Common Feature Requests', etc. Identify your top 3-5 insights.
• Day 5: Draft the 'Internal Insights Memo' using the template. Share it with your product and marketing leads. Schedule a meeting to present your findings and propose action items.

More Level 3 hacks