ShadowxBot
ShadowxBot is ShadowX’s website classification crawler. It helps organizations identify websites that offer interactive AI services and apply appropriate enterprise security policies.
What does ShadowxBot do?
ShadowxBot analyzes publicly accessible web pages to determine whether a website offers capabilities such as:
AI chat and large language model conversations.
Interactive AI assistants.
Prompt-based text, code, or other content generation.
AI applications that accept user instructions or uploaded content.
Its purpose is to distinguish websites that provide AI functionality from websites that only publish AI-related news, articles, or educational material.
Why does ShadowxBot visit my website?
ShadowxBot may visit your website as part of ShadowX’s AI service discovery and website categorization process.
It reviews a limited selection of relevant public pages, such as your homepage, product descriptions, and feature pages. This helps determine whether your website provides AI functionality relevant to enterprise access and data protection policies.
A visit from ShadowxBot does not indicate that your website is malicious or has a security vulnerability.
What information does it process?
ShadowxBot processes publicly accessible page content, including page titles, descriptions, visible text, and relevant links. Content may be converted into Markdown or another text format for AI-assisted classification.
Classification records may include the website URL, detected category, supporting excerpts, and the time of assessment.
ShadowxBot is intended for website classification, not for collecting private user conversations, account data, or content behind authentication. Its crawling purpose is not to build datasets for training general-purpose AI models.
How does ShadowxBot identify itself?
ShadowxBot uses the following crawler identifier:
ShadowxBot
Its intended HTTP User-Agent format is:
ShadowxBot/1.0 (+https://shadowx.ai/bot)
This documentation page identifies the crawler’s purpose and operator. The User-Agent alone should not be treated as proof that a request originated from ShadowX.
Crawling behavior
ShadowxBot is designed to:
Respect applicable robots.txt directives.
Limit crawling to pages relevant to website classification.
Apply per-site request limits and avoid unnecessary repeat requests.
Back off when a website signals overload or rate limiting.
Respect website access restrictions.
ShadowxBot does not attempt to bypass authentication, paywalls, CAPTCHAs, or other access controls. It does not submit prompts to your AI service, create user accounts, or perform vulnerability scans as part of website classification.
How can I block ShadowxBot?
To disallow crawling, add the following to your website’s robots.txt file:
User-agent: ShadowxBot
Disallow: /
Website owners may also apply their own access controls.
Contact and classification corrections
For questions about ShadowxBot, crawling concerns, or requests to review a website classification, contact:
What to include
Please include your website URL and, where relevant, the request timestamp and request details.
Operator
Operated by ShadowxAI OÜ, under the ShadowX brand.
