BotSeer is the first search engine to provide full text search and analysis of robots.txt files.
For papers on BotSeer please see:
- Yang Sun, I. Councill, C.L. Giles, "BotSeer: An automated information system for analyzing Web robots," 8th International Conference on Web Engineering (ICWE 2008), 2008.
- Yang Sun, Z. Zhuang, I. Councill, C.L. Giles, "Determining Bias to Search Engines from Robots.txt," Proceedings of IEEE/WIC/ACM International
Conference on Web Intelligence (WI 2007), 149-155, 2007.
- Yang Sun, Ziming Zhuang, and C. Lee Giles, "A Large-Scale Study of Robots.txt," 16th International World Wide Web Conference (Poster) (WWW 2007), 1123-1124, 2007.
| Summary of BotSeer |
| Crawler SearchNew! | BotSeer searches 15,257 web crawlers and 2,322 user-agent strings. |
| Robots.txt Search | BotSeer provides search function to the full text index of more than 2 million robots.txt files. |
| Open Source Crawler Search | BotSeer collects open source crawler projects and indexes the documentation files and source codes. |
Feedback
Credits:
Special credits:
The official Robots Exclusion Protocol was proposed by Martijn Koster (http://www.robotstxt.org).
BotSeer would like to give thanks to Andrei Broder, Eric Glover, Oleksandr Vagin
and Uber0n.
|