All articles

Showing 1–4 of 4 articles

• Crawling

  What Is a Web Crawler and How Does It Work?

  A web crawler is a program that fetches URLs over HTTP and follows the links it finds. Here is the fetch loop, the rules that stop it, and how to verify one.

  3 min read

• Technical foundations

  HTTP Status Codes That Actually Matter for Search

  The status code is the first thing a crawler reads, and it decides everything after it. Here is what each code means to a search engine, and the traps.

  2 min read

• Indexing

  Crawled, Currently Not Indexed: What It Actually Means

  Google fetched your page and chose not to store it. Here are the five real causes behind that Search Console status and how to tell which one applies.

  2 min read

• Crawling

  Why robots.txt Does Not Remove a Page From Google

  Blocking a URL in robots.txt stops the fetch, not the listing. Here is why blocked pages still appear in results, and what actually removes them from the index.

  2 min read