Index
Build a structured index of publicly accessible websites and resources.
Crawl
INETDEX Bot discovers public pages while identifying itself with a clear User-Agent.
Open Web
Designed around transparent crawling and publicly available web content.