TF-IDF
TF-IDF is a scoring formula that weighs a term by how often it appears in a document, divided by how common it is across the whole collection. It gives high scores to terms that are frequent in one document and rare elsewhere, and near-zero scores to words like 'the'.
The name is two ideas multiplied together.
Term frequency counts how often a term appears in this document. A page mentioning “crawler” twelve times is probably more about crawlers than one mentioning it once.
Inverse document frequency divides by how many documents in the collection contain the term. “The” appears everywhere, so it carries almost no information and its weight collapses towards zero. “Inverted index” appears in few documents, so a document containing it is strongly characterised by it.
TF-IDF is the classic baseline for text retrieval, it takes an afternoon to implement, and it produces surprisingly reasonable results on a small corpus. It is worth writing once, because doing so makes the limits obvious: it has no sense of document length, so long pages score higher by accident, and term frequency grows without limit, so repetition is rewarded indefinitely.
BM25 fixes both of those and is where you should go next. No production web search engine uses plain TF-IDF for ranking, but almost all of them use something descended from it as one component among many.