Evidence-Based AI Website Reports: Real Scraping, No Guesswork

Most AI website analysis tools scrape your homepage and hallucinate the rest. You get generic personas. Vague pain points. Messaging that could apply to any B2B SaaS company.

We fixed this with three changes: multi-page scraping, evidence tracking, and low temperature.

Homepage content is marketing fluff. Product pages have the real details.

We now scrape 3-6 pages per site:

• Homepage
• /product or /products
• /platform
• /solutions
• /pricing (if available)
• /about (for company context)

We extract titles, meta descriptions, H1s, H2s, and paragraph content. Filter out cookie banners and GDPR noise.

// apps/web/src/app/api/reports/generate/route.ts
const pagesToTry = [
  cleanUrl,
  `${cleanUrl}/product`,
  `${cleanUrl}/platform`,
  `${cleanUrl}/solutions`,
];

for (const attemptUrl of pagesToTry) {
  const response = await fetch(attemptUrl, {
    headers: { 
      'User-Agent': 'Mozilla/5.0...',
      'Accept': 'text/html',
    },
    signal: controller.signal,
  });
  
  const html = await response.text();
  
  // Extract structured content
  const titleMatch = html.match(/<title[^>]*>([^<]+)<\/title>/i);
  const h1Matches = [...html.matchAll(/<h1[^>]*>([^<]+)<\/h1>/gi)];
  const pMatches = [...html.matchAll(/<p[^>]*>([^<]{30,})<\/p>/gi)];
  
  // Stop after 3 good pages
  if (successfulPages.length >= 3) break;
}

Every AI claim must cite the website content. No invented facts.

We added evidence fields to the prompt:

// Prompt instruction
"For each persona, quote specific phrases from the website 
that prove this is their actual target audience. 

Example:
{
  "persona": "Engineering Managers",
  "evidence": "From /platform page: 'Built for engineering 
  teams of 50-500 developers' and 'Integrate with GitHub, 
  GitLab, Bitbucket'"
}

The AI must ground every insight in scraped content. If it cannot find evidence, it returns "insufficient content" instead of guessing.

Temperature controls randomness. Higher = more creative. Lower = more factual.

We dropped from 0.7 (default) to 0.2 for website analysis.

// apps/web/src/app/api/reports/generate/route.ts
const completion = await openai.chat.completions.create({
  model: 'gpt-4o',
  temperature: 0.2,  // Low temp = factual, deterministic
  messages: [
    {
      role: 'system',
      content: 'Use ONLY the provided website content. 
      Do NOT invent or assume. Quote specific phrases as evidence.'
    },
    {
      role: 'user',
      content: websiteContent
    }
  ]
});

At 0.2, the model sticks to the facts. It repeats exact phrases from the website instead of paraphrasing creatively.

If we cannot scrape enough content (bot protection, login walls, etc.), we do not fake it.

The API returns an error with a clear message:

{
  "error": "insufficient_content",
  "message": "Could not read website content. The site may 
  be blocking automated access. Please try again or 
  contact support."
}

Better to fail honestly than generate garbage.

Before (temperature 0.7, homepage only):

After (temperature 0.2, multi-page, evidence):

Specific. Grounded in real content. Actionable for media planning.

Scraping 3 pages in parallel takes 2-3 seconds. GPT-4o analysis takes 3-5 seconds at temperature 0.2.

Total: 5-8 seconds for an accurate, evidence-backed report.