Hassan
Editor, Search Engine Basics
- 8 years of hands-on SEO and technical search work
- Runs original crawl and log-file experiments on live sites
I started in search in 2018 and have spent most of that time on the technical side: crawl budget, log files, indexing diagnostics, and the gap between what a tool reports and what a search engine actually did.
Search Engine Basics exists because most explanations of how search works are either vendor marketing or a summary of another summary. The site is written the other way round — from search engines’ own documentation, from published papers, and from experiments run on sites I control and can measure.
Corrections are welcome and are published on the page they affect. The editorial policy sets out how sourcing and review work.
Articles written
- What Is a Web Crawler and How Does It Work?
A web crawler is a program that fetches URLs over HTTP and follows the links it finds. Here is the fetch loop, the rules that stop it, and how to verify one.
3 min read
- HTTP Status Codes That Actually Matter for Search
The status code is the first thing a crawler reads, and it decides everything after it. Here is what each code means to a search engine, and the traps.
2 min read
- Crawled, Currently Not Indexed: What It Actually Means
Google fetched your page and chose not to store it. Here are the five real causes behind that Search Console status and how to tell which one applies.
2 min read
- Why robots.txt Does Not Remove a Page From Google
Blocking a URL in robots.txt stops the fetch, not the listing. Here is why blocked pages still appear in results, and what actually removes them from the index.
2 min read