Site popularity meta-search

Overview

We use logs generated by a network of proxy cache servers to build a static database of URLs. For every URL, this database allows us to determine the relative popularity of a corresponding web page in real time. We use this information to re-rank results of web engines (Google, Hotbot, etc.)

Here's how it works: user specifies his query and a set of web search engines he wishes to use. Our engine queries the specified web engines to obtain search results, which get collated into a single result list. After initial pre-processing of the result list (removing duplicates, etc), every page from the list is assigned a numerical rank, that takes into account page popularity, relative position of the page in the original list, how well the page matches the query, and some other factors. The page list gets re-sorted according to these new ranks, and is presented to the user for viewing.