Aspect Search Web Crawler
Our robot will index our client's site for all publicly available html, pdf, text, xml and office 2007 documents by
navigating internal links discovered during a crawl.
Our robot can be identified by the following agent string:
Mozilla/5.0 (compatible; AspectBot; http://www.aspectsearch.com/features/aspectbot.aspx)
If you are seeing our robot in your web server logs then it means your site has been identified by a client of
ours for indexing.
How do I block access to your robot?
You can use a robots.txt to exclude our robot from certain pages or
folders on your site.
How can I exclude sections of a document from being indexed?
You can tell our robot not to index words or sections within a document.
Use the following tags to surround the section of the document to exclude.
<!--NOINDEX--> Do not index <!--/NOINDEX-->
If you have any more questions regarding our robot, please contact us.
Back to our search features