Transparent first-party retrieval

Crawler and bot information.

AgentSourceBot identifies itself as AgentSourceBot/1.0 and links to this page.

Policy

We respect robots.txt and applicable site terms, use bounded first-party retrieval, and do not bypass authentication, CAPTCHAs, paywalls, or technical controls.

  • HTTP and HTTPS only
  • Private-network and metadata targets blocked
  • Bounded response size, time, redirects, pages, and concurrency
  • Content hashing avoids unnecessary extraction
  • Contact support for correction, suppression, or crawl questions