Why $300K Websites With Perfect Lighthouse Scores Fail in AI Search
High Lighthouse scores measure speed, not whether AI search engines can read your site. Here is why expensive startup websites remain invisible to LLMs.
Many technology startups invest between $100,000 and $350,000 on custom web redesigns, only to discover months later that conversational search tools like ChatGPT and Perplexity fail to reference their products accurately. While engineering and marketing teams often rely on Google Lighthouse to validate site health—celebrating performance scores reaching 97 to 100—these metrics only evaluate client-side speed and basic technical configurations. They do not evaluate whether search engine bots or generative answer engines can properly parse backend architecture, structured data, or current company facts.
The disconnect between speed scores and machine readability
This issue was detailed by technical marketing advisor Roberto Iriondo, who conducted an automated review of his own consulting website. Despite near-flawless Lighthouse marks, an AI-assisted audit surfaced 14 distinct issues ranked by severity. Key files responsible for machine discovery had been neglected: the sitemap had not been refreshed since 2018, contained dead URLs, and included a misspelled domain name. Furthermore, the site's robots.txt file contained leftover rules copied years earlier from a university department, directing crawlers to the wrong external sitemap while blocking non-existent directories.
Other common defects included email contact links obfuscated by Cloudflare protection scripts, rendering them invisible to raw HTML scrapers, and schema markup that listed outdated career history. In many organizations, these oversights stem from post-launch drift. Once an agency delivers a website, nobody actively maintains background configuration files. Product names, pricing tiers, and personnel change over time, but the structured JSON-LD data and robots.txt rules remain static, causing generative engine optimization (GEO) to suffer across platforms such as Google's AI Overviews, Gemini, and Claude.
How modern web frameworks introduce crawler blind spots
The problem spans virtually every major deployment stack, each presenting its own architectural pitfalls:
- Custom JavaScript Frameworks (Next.js, Astro): When marketing sites are coded in the same frameworks as web applications, text that relies strictly on client-side JavaScript execution may be ignored by AI crawlers that only evaluate raw HTML. Additionally, unconfigured public preview URLs generated during deployment cycles risk being indexed by search engines.
- Visual Site Builders (Webflow, Framer): Automated sitemap generators often index test pages, unmapped CMS entries, and internal staging domains (such as
yoursite.webflow.io) unless explicitly suppressed. Structured data typically requires manual custom code integration, which teams regularly bypass during rapid launches. - Content Management Systems (WordPress): Conflicting plugins frequently generate multiple competing sitemaps, while settings intended to discourage search indexing during staging are mistakenly left active after public release.
- Subdomain Fragmentation: Startups frequently separate their documentation onto platforms like Mintlify or GitBook, while hosting blogs on Substack or Ghost. Because every subdomain carries its own isolated sitemap and robots.txt, crawlers may encounter misconfigured barriers on documentation hubs—the exact pages answer engines cite most often.
What it means for developers
For engineers and technical leads, site maintenance cannot conclude once a build is pushed to production. Generative engines construct summaries based on readable data layers; if machine-readable signals conflict with current marketing copy, AI models will reproduce outdated claims.
Developers must treat machine discovery as an ongoing operational discipline:
- Verify Raw HTML Content: Ensure core page text and business information exist directly in the initial HTML response rather than requiring client-side hydration or JavaScript execution.
- Standardize Structured Data: Implement schema.org JSON-LD tags across homepages, documentation, and articles, ensuring properties match active company positioning, executive rosters, and product definitions.
- Audit Subdomains Regularly: Confirm that robots.txt files across all subdomains explicitly point to current sitemaps and do not accidentally block helpful AI crawlers or allow staging mirrors to be indexed.
- Automate Weekly Inspections: Rather than depending on infrequent manual audits, teams can write scripts or prompt AI assistants to inspect sitemaps, robots.txt, and page metadata against a structured checklist. Developers building automated auditing agents can test top AI models cheaply through one API at https://apixoai.online to parse codebases and evaluate markup consistency.
Iriondo notes that while proposed specifications like llms.txt can provide concise site summaries for language models, they do not replace foundational technical hygiene. Fixing critical issues—such as correcting sitemap entries, updating structured metadata, and cleaning robots.txt directives—often takes just a single working session once automated reporting brings them to light.
Source: Your $300K Website Scores 100 on Lighthouse. AI Search Still Can’t Read It. — Towards AI. Written by the Apixo team from that report.
One key for Claude, GPT, GLM, DeepSeek and more. Pay per token with crypto.
Get your API key
