The crawler that builds Sooth's index — a search engine for students with no generative AI, just links back to your site.
Title
SoothBot
Operator
Sooth (sooth.fyi)
Subject
Curated web & academic search indexing for students
Classification
Search — indexing crawler
User-Agent
SoothBot/1
Verification
Cryptographic — HTTP Message Signatures (RFC 9421 / Web Bot Auth). Key directory: /.well-known/http-message-signatures-directory
Contact
SoothBot visits pages to build the search index behind Sooth, a search engine built for students and researchers. Sooth doesn't summarize or rewrite your content with generative AI — every result links straight back to the source page. Indexing your site means students find their way to it, not around it.
SoothBot fetches pages at a deliberately modest, polite rate, and respects standard crawling conventions (see below). It does not fill in forms, click through paywalls, or take any action beyond reading and indexing publicly accessible pages.
SoothBot reads and follows robots.txt directives targeted at its user-agent token, SoothBot.
To block SoothBot entirely:
User-agent: SoothBot Disallow: /
To block specific sections only:
User-agent: SoothBot Disallow: /private/ Disallow: /account/
To allow SoothBot (default, if no rule matches):
User-agent: SoothBot Allow: /
Changes to robots.txt are picked up on SoothBot's next scheduled visit to your site — no need to contact us for routine allow/block changes.
User-Agent strings can be spoofed by other traffic, so we don't recommend relying on it alone for anything but robots.txt targeting. Every SoothBot request is cryptographically signed per RFC 9421 (HTTP Message Signatures), with our public key published at /.well-known/http-message-signatures-directory. If you're behind Cloudflare, this is checked automatically as part of their Verified Bots program — you can independently confirm SoothBot's status in Cloudflare's public Bots Directory.
Questions, abuse reports, or crawl-rate requests: contact@sooth.fyi
