AI crawler HTML inspector: see what a non-rendering fetch gets

How to use AI Crawler HTML Inspector

1. Copy the page source. In your browser, choose View page source from the menu, select all, and copy. The text is close to what a fetch without JavaScript receives.
2. Paste it and set the Page URL. The URL is used only to build citation links. The tool does not request it.
3. Choose a chunk size and press Inspect HTML. 150, 300 or 500 estimated tokens. Load one of the two samples first if you want to see the tool work.
4. Read the Overview, then the Audit. The failing and warning checks come first, ranked by status. Every row on the Audit tab states its reason.
5. Export what you need. Download the Markdown audit, the JSON chunks, or the plain text.

What the inspector measures

The inspector reads the HTML of one page and reports what a fetch without JavaScript receives, and how the tool splits the text it keeps into chunks. It has six views: Overview, Main content, Chunks, Summaries, Audit and Plain text. All six come from one analysis, so the numbers agree from tab to tab.

Use the page source, not the DevTools copy. In your browser, choose View page source from the menu, select all and copy. That text is close to what a fetch without JavaScript receives. The same URL can return different HTML to different clients, so the copy is a sample, not a guarantee. DevTools shows the page after scripts have changed it, so it can include content that a non-rendering reader never saw. Google's own guidance is that "not all bots can run JavaScript," and that Googlebot queues pages for rendering (Google Search Central, JavaScript SEO basics).

The render-gap verdict

The render gap counts four things in the HTML you pasted: visible characters, bytes of inline script (JSON data blocks are counted separately), external script tags, and empty app roots such as #root. Visible characters are text outside script, style, noscript, template and head elements, and outside elements marked hidden. Navigation, footer and cookie wording count as visible text, because a text-only reader sees them too. The verdict uses one threshold: 300 visible characters.

In December 2024 Vercel, with MERJ data, reported that none of the major AI crawlers it measured currently render JavaScript, and that the ChatGPT and Claude crawlers fetched JavaScript files without executing them. The primary data came from monitoring nextjs.org and the Vercel network over a few months. To check the findings across technology stacks, the post also analysed two job-board sites. The same post says AppleBot renders JavaScript through a browser-based crawler, so the statement covers the AI crawlers the post lists, not every bot. Read it as a dated measurement, not a current guarantee (Vercel, "The rise of the AI crawler").

How the main content is picked

A reader mode keeps the article and drops the rest of the page. This tool does the same job with a scoring pass over article, main, section and div containers. The highest score wins, and the blocks inside it become the text that gets chunked.

The rules follow the approach of Mozilla's Readability, which scores candidate blocks by tag and class weights, penalises link-heavy blocks, and drops nodes that match an "unlikely candidate" pattern (Mozilla Readability repository). This tool is its own implementation, not that library, and it will make mistakes on unusual markup. The Main content tab lists every removal rule that matched, with its element count and visible characters, largest first. Script, style and head elements are not listed. If extraction drops the page's first h1, for example from inside a header, the tool adds that heading back as the first block, so the title still appears in the heading paths. If no container scores, the whole body is used and the tab says so.

Chunks, anchors and citation links

Chunks follow the page's structure. Each heading starts a section, and the heading path shows where the chunk sits, for example "Rate limits > Response headers > Retry-After". Paragraphs are packed into chunks of 150, 300 or 500 estimated tokens. A paragraph longer than the limit is split at sentence boundaries first, then at word boundaries. Token counts are estimates. Each vendor's tokenizer gives its own count, so treat the numbers as planning figures.

Every chunk gets a citation candidate. The first choice is an anchor: the nearest element id, on the heading, the paragraph or an ancestor. The link is the page URL with #id added. The HTML standard resolves a fragment to the first element in tree order that has that id (WHATWG HTML Standard, scrolling to a fragment). A duplicated id is therefore reported in the audit.

When a chunk has no id nearby, the tool builds a text-fragment link from the first five words of its first paragraph. A heading-only chunk uses the first five words of its heading instead. A text fragment links to text "without requiring the page author to add an ID". If the text does not match, the whole fragment is ignored and the top of the page is linked. Browser support varies (MDN, text fragments). For a planning comparison of chunk sizes and strategies, see the chunking strategy comparator.

What each audit rule checks

The audit has 23 rules. Each one ends in pass, warn or fail, with a reason that quotes the number or the element it found. Failing and warning rules also carry a suggested fix. A rule that passes shows no fix. The list below groups the rules by the question they answer.

The outbound-link rule has a reason behind it. In the GEO paper, adding citations, quotations and statistics raised the authors' visibility metric on their benchmark, where answers came from gpt-3.5-turbo (Aggarwal et al., arXiv:2311.09735). The paper also reports one experiment on Perplexity.ai, with gains of up to 37% in its introduction. Neither result shows the same effect in live AI search, so the rule checks only that links exist. The FAQPage warning follows Google's changelog, which says the FAQ rich result "will no longer appear in Google Search starting May 7, 2026" (Google Search Central updates).

For how to fix the common failures, the guide on making passages easy for AI answers to cite goes into more detail, with sources for each recommendation.

Limits and what it cannot tell you

• It does not run the pasted page's scripts. It cannot show what a rendering crawler would see after those scripts run. Treat the render-gap verdict as a prompt to check, not as proof.
• It does not know which bot you care about. OpenAI documents GPTBot for training, OAI-SearchBot for search and ChatGPT-User for user-triggered visits. It says that sites opted out of OAI-SearchBot "will not be shown in ChatGPT search answers, though can still appear as navigational links", and that a robots.txt change can take about 24 hours to reach search. Anthropic documents ClaudeBot, Claude-SearchBot and Claude-User. Perplexity says Perplexity-User "generally ignores robots.txt rules" because a user asked for the fetch. The inspector reads no robots.txt file.
• Its Google points rely on Google's own statements. The FAQ and llms.txt notes use Google's documentation changelog. They say nothing about any other AI system.
• It works on one page at a time. It does not crawl a site, read a sitemap or compare pages.
• Its chunking is one design. Other tools can split text differently, and so do the chunking strategies in the strategy comparator.
• A pass is not a prediction. A page that passes every rule can still be ignored, and a page that fails one can still be cited.

For how the crawlers themselves are documented, read OpenAI's crawler overview and Perplexity's crawler page. The background on how a crawler reads a page is in the guide on how AI crawlers read your page.

Privacy: what happens to your HTML

The tool's code parses the pasted HTML with the browser's DOMParser, in this tab. The parsed copy is turned into plain data for the analysis, and no parsed node is attached to the page you are reading. The tool never writes your text into the page with innerHTML. Results are built from text nodes, so markup in a pasted page is shown as text and never runs.

The tool's code makes no request with your HTML, and it writes nothing to localStorage, cookies or IndexedDB, so a reload clears the input. The Page URL is used only to build citation links, and the tool does not request it. The share link holds the chunk size and nothing else. Downloads are created in this tab, and their object URLs are released after a short delay.

This page also loads the site's own promo widgets, which are separate code. The promo banner loads its product list through the site's own server and keeps its display state in your browser. A popup keeps its display flags there too. Neither reads this tool's text box. The two sample pages are fictional and exist only to show the tool working.