Google

How Search Engines and AI Assistants Crawl, Index and Rank Pages

Search engines work in three stages: crawlers fetch pages, the index stores and understands them, and ranking systems pick the best results for each query. AI assistants add a fourth step on top: they retrieve a few relevant pages, often from a search index, and write an answer that may cite them. If a page can’t be crawled, it can’t be indexed, ranked or quoted.

Last updated: October 2026. Prices exclude tax and may change. I may earn a commission if you buy through links on this site.

Understanding this pipeline explains most SEO advice. It also explains why some sites rank on Google but never show up in ChatGPT or Perplexity, and the other way around. Let’s walk through each stage.

What is crawling?

Crawling is discovery. A crawler (also called a bot or spider) requests a URL, downloads the HTML, and pulls out the links it finds so it can visit those next. Googlebot does this for Google, Bingbot for Bing, and AI companies run their own bots.

Crawlers find new pages in a few ways:

  • Following links from pages they already know.
  • Reading XML sitemaps you submit or list in robots.txt.
  • Revisiting known URLs to check for changes.

This is one reason backlinks matter beyond ranking. A link from a page that gets crawled often helps a new page get discovered quickly.

Which AI crawlers should you know about?

AI bots are not one thing. Each company runs several, with different jobs:

Bot Purpose
GPTBot OpenAI training data
OAI-SearchBot ChatGPT search results
ChatGPT-User Fetches a page when a user’s request needs it
ClaudeBot, Claude-SearchBot, Claude-User Anthropic’s training, search and user-triggered fetching
PerplexityBot, Perplexity-User Perplexity’s index and user-triggered fetching
Bingbot Bing’s index, which also feeds ChatGPT search and Copilot
Googlebot Google Search, including AI Overviews

A common misunderstanding: Google-Extended is a robots.txt token, not a separate crawler. It only controls whether your content is used for Gemini training. Blocking it does not remove you from Google Search or AI Overviews.

What blocks crawling by accident?

Plenty of sites are invisible to AI assistants without realizing it. The usual causes are a robots.txt rule that disallows a bot, a CDN or firewall setting that challenges or blocks AI crawlers, content that only appears after JavaScript runs, and pages behind logins or cookie walls. If you want to be cited, check your robots.txt and your CDN’s bot settings first.

What is indexing?

Indexing is understanding and storing. After fetching a page, the search engine renders it, extracts the main text, works out what it’s about, notes its language, links and structured data, and decides whether it’s worth keeping. Not every crawled page gets indexed.

Pages are commonly left out of the index when they:

  • Carry a noindex tag.
  • Duplicate another page, so a canonical version is chosen instead.
  • Have very little unique content.
  • Return errors, redirect loops or soft 404s.

Google Search Console shows which of your pages are indexed and why others aren’t. Bing Webmaster Tools does the same for Bing, which is worth setting up given Bing’s role in ChatGPT search.

How does ranking work?

Ranking is selection. When someone searches, the engine pulls candidate pages from its index and orders them using many signals. Google doesn’t publish a full list, but the broad categories are well known:

  1. Relevance: does the page actually address the query and its intent?
  2. Quality and helpfulness: is the content accurate, original and useful? See why content quality matters.
  3. Authority: do trusted sites link to it? Links are still one of the strongest signals. Types of backlinks explains which kinds count.
  4. Page experience: mobile usability, HTTPS and reasonable load times.
  5. Context: the searcher’s location, language and the freshness the query needs.

Ranking also includes spam detection. Pages or links that look manipulative can be ignored or demoted, which is the main risk any link-building strategy has to manage.

How do AI assistants choose which pages to cite?

Assistants answer in two broad ways. Some answers come purely from what the model learned during training. Others use retrieval: the assistant runs a search, pulls a handful of pages, reads them and writes an answer with citations. ChatGPT search, Perplexity, Claude with web search and Google AI Overviews all lean on retrieval for current questions.

Retrieval means traditional SEO still matters, because the pages an assistant reads usually come from a search index. On top of that, assistants tend to favor pages that:

  • State the answer clearly and early.
  • Use headings that match the question being asked.
  • Contain specific facts such as numbers, names and steps that can be quoted.
  • Look current and trustworthy.

Training data is the slower path. If your brand is mentioned accurately across many independent pages, models trained on that material are more likely to know about you. That is a long game, and nobody can guarantee an AI model will cite you.

What does this mean for a PBN?

Network sites go through exactly the same pipeline. A PBN blog that isn’t crawled passes no value, and one that isn’t indexed is close to worthless. Many PBN owners block backlink crawlers like AhrefsBot, SemrushBot, MJ12bot, DotBot and BLEXBot to make their network harder for competitors to map. That’s a reasonable choice. The mistake is blocking AI crawlers or Googlebot along with them, which throws away the point of the site.

Check that every network site is indexed in Google, has Bingbot access, and allows the AI bots you want reading it. My How to Build a PBN guide covers setup in more detail.

Quick checklist

  • Robots.txt allows Googlebot, Bingbot and the AI crawlers you want.
  • CDN and firewall don’t challenge those bots.
  • Main content is in the HTML, not loaded only by scripts.
  • Sitemap submitted to Google Search Console and Bing Webmaster Tools.
  • Key pages are indexed and canonical.
  • Each page answers its question early, with clear headings.

Frequently Asked Questions

Does blocking Google-Extended remove my site from AI Overviews?

No. Google-Extended only controls whether content is used for Gemini training. It does not affect Google Search or AI Overviews.

Why is my site on Google but not in ChatGPT search?

Common causes are robots.txt or CDN rules blocking OAI-SearchBot or Bingbot, or content that only loads with JavaScript. Bing's index feeds ChatGPT search, so check Bing Webmaster Tools too.

What is the difference between crawling and indexing?

Crawling is a bot fetching a page. Indexing is the search engine understanding and storing it. A page can be crawled but left out of the index.

Should a PBN block AI crawlers?

Usually not. Blocking backlink checkers like AhrefsBot and SemrushBot is a common choice, but blocking Googlebot, Bingbot or AI crawlers removes much of the point of the site.

Leave a Comment

Your email address will not be published. Required fields are marked *