MetaCrawler, HuskySearch, and Grouper

Note: MetaCrawler, HuskySearch, and Grouper were WWW search services operated by the University of Washington.
MetaCrawler operated at the UW from 1995 - 1997; HuskySearch and Grouper operated at the UW from 1997 - 2000.
MetaCrawler was licensed to Go2Net, now InfoSpace, in 1997. HuskySearch and Grouper were retired from service in 2000.

MetaCrawler: 1995 - 1997

MetaCrawler provideded a single, central interface for World Wide Web document searching. Upon receiving a query, MetaCrawler posted the query to multiple search engines in parallel, and performed sophisticated pruning on the responses returned. Experiments indicated that MetaCrawler was able to Prune as much as 75% of the returned responses as irrelevant, outdated, or unavailable. MetaCrawler received upwards of 60,000 queries daily, making it one of the most popular meta-searching services in the world while operated by the University of Washington.

recommended MetaCrawler as the search service of choice!

Go2Net, now InfoSpace, took over exculsive operation of MetaCrawler in 1997 see www.metacrawler.com.

HuskySearch: 1997 - 2000

HuskySearch was based on MetaCrawler and operated as a service that allowed you to search the World Wide Web and in particular the University of Washington Web. HuskySearch, like MetaCrawler, differed from most search services in that it didn't maintain any internal database. Rather, it forwarded your query to multiple search services in parallel and collated the results.

HuskySearch was used in studies concering aspects of Information Retrieval and Artificial Intelligence and how they relate to search on the Internet. HuskySearch explored methods to improve query refinement. Query refinement on most search services consisted of presenting the form you entered with a prompt to modify it. Better search services suggested words to help refine your query. Our research focused on determining which methods worked well.

Grouper: 1997 - 2000

Grouper took the results of HuskySearch and identified phrases that help you find what you were looking for. Unlike most search services that showed results in a linear list ranked by relevance or sorted by URL, Grouper partitioned the results into clusters, or groupings of URLs which contain similar content. By generating high quality clusters with simple descriptions for novice users Grouper provided an effective way of organizing search results into collections for ease of browsing.

Utilizing Parallel Search


Searching the search engines: An illustration of how MetaCrawler, HuskySearch, and Grouper queried multiple search engines in parallel on the World Wide Web.

Additional Information

MetaCrawler and HuskySearch were research projects developed by graduate student Erik Selberg. Grouper was a research project developed by graduate student Oren Zamir. Both students were advised by Oren Etzioni. Erik and Oren received their Ph.D. degrees in 1999. With the research successfully completed and the graduation of these students we were no longer able to maintain these services in a university setting. As a result Go2Net, now InfoSpace, took over exculsive operation of MetaCrawler in 1997 see www.metacrawler.com and HuskySearch and Grouper were shutdown from service in 2000.


Computer Science and Engineering, 2001