# CityZeen® — public discovery and private-surface crawl policy # # Search and user-requested retrieval are allowed on approved public pages. # Model-training crawlers remain blocked. Authentication and authorization — # never robots.txt — protect private product and client data. # AI search and user-request agents. Repeat private exclusions here because a # crawler can prefer this more-specific group over the wildcard group below. User-agent: OAI-SearchBot User-agent: ChatGPT-User User-agent: Claude-SearchBot User-agent: Claude-User User-agent: PerplexityBot User-agent: Perplexity-User Allow: / Disallow: /desk Disallow: /desk/ Disallow: /vault Disallow: /vault/ Disallow: /account Disallow: /account/ Disallow: /api/ Disallow: /passport- Disallow: /recovery- Disallow: /admin-backup/ Disallow: /internal/ Disallow: /.env Disallow: /backup/ # Training-oriented and bulk-corpus crawlers. User-agent: GPTBot User-agent: ClaudeBot User-agent: Claude-Web User-agent: anthropic-ai User-agent: CCBot User-agent: Google-Extended User-agent: Bytespider User-agent: Amazonbot User-agent: Applebot-Extended User-agent: Diffbot User-agent: Omgilibot User-agent: Omgili User-agent: cohere-ai User-agent: FacebookBot User-agent: meta-externalagent User-agent: ImagesiftBot Disallow: / # Commercial SEO/recon crawlers not needed for public discovery. User-agent: DataForSeoBot User-agent: MJ12bot User-agent: DotBot User-agent: SemrushBot User-agent: AhrefsBot User-agent: Scrapy Disallow: / # Conventional search engines: public marketing pages are indexable. User-agent: * Allow: / Disallow: /desk Disallow: /desk/ Disallow: /vault Disallow: /vault/ Disallow: /account Disallow: /account/ Disallow: /api/ Disallow: /passport- Disallow: /recovery- # Honeypot paths remain disallowed. See api/trap.js. Disallow: /admin-backup/ Disallow: /internal/ Disallow: /.env Disallow: /backup/ Sitemap: https://www.cityzeen.co/sitemap.xml