Tools
Search Tools: What Each One Can Actually Prove
Every search tool measures something narrower than people assume. This guide goes through the main ones and states plainly what each can prove, what it can only suggest, and which question you should reach for each tool to answer.
Tools disagree constantly, and most arguments about which is right are really arguments about what each one measures. Sorting them by the evidence they provide makes the disagreements uninteresting.
Google Search Console
The only source of Google’s own view of your site. It can prove: which URLs Google has indexed and why others are excluded, what Google fetched and when, which queries your pages were shown for and how often they were clicked, and whether a specific URL passes or fails a check right now.
It cannot show you a full ranking history, complete query data — long-tail queries are withheld — or anything about competitors.
Bing Webmaster Tools
The same category of evidence for Bing’s index, with more generous query data and a useful crawl diagnostics section. Worth connecting even for sites where Bing traffic is small, because a second independent index is a second opinion on whether your pages are technically accessible.
Server log files
The primary evidence for anything crawl-related. Logs record every request that reached your server: the URL, the timestamp, the user-agent and the status code returned. Nothing else can prove what a crawler actually requested. Verify user-agents by reverse DNS before trusting them.
Site crawlers
Screaming Frog, Sitebulb and similar tools crawl your site the way a search engine might, and report status codes, titles, canonicals, directives and internal link structure. They prove what your site returns to a crawler. They do not prove what a search engine did with it.
Rank trackers
They sample a results page for a query from a location at a time. That is a useful longitudinal signal and a poor absolute measurement, because results are personalised, localised and volatile. Treat direction over weeks as meaningful and a single day’s position as noise.
The rule of thumb
When two tools disagree, prefer the one closer to the source. For “was this
fetched”, logs beat Search Console. For “is this indexed”, Search Console beats
a third-party crawler. For “does this page return the right response”, your own
curl beats everything.
Articles in this guide
The articles for this guide are being written. The guide above covers the whole topic in outline, and the article index lists everything published so far.