# 搜索引擎与 AI 爬虫均可抓取全部公开页面(营销页、工具/模型落地页、博客) User-agent: * Allow: / Disallow: /admin Disallow: /api # 放行博客公开 API:Googlebot 渲染页面时遵守 robots.txt,前端 SPA 需 fetch /api/blog/* 才能渲染正文, # 被Disallow阻止会导致渲染出 "Article not found" → Google 判软 404(Google 最长前缀匹配,Allow 覆盖 Disallow) Allow: /api/blog Disallow: /user # 主流 AI 爬虫显式放行(内容用于 AI 搜索与引用,如 ChatGPT、Perplexity、Claude) User-agent: GPTBot Allow: / User-agent: OAI-SearchBot Allow: / User-agent: ChatGPT-User Allow: / User-agent: ClaudeBot Allow: / User-agent: Claude-User Allow: / User-agent: Claude-SearchBot Allow: / User-agent: PerplexityBot Allow: / User-agent: Perplexity-User Allow: / User-agent: Google-Extended Allow: / User-agent: Applebot-Extended Allow: / User-agent: CCBot Allow: / User-agent: meta-externalagent Allow: / User-agent: Amazonbot Allow: / User-agent: Bytespider Allow: / User-agent: YouBot Allow: / User-agent: cohere-ai Allow: / User-agent: Diffbot Allow: / User-agent: ImagesiftBot Allow: / Sitemap: https://zaotu.vip/sitemap.xml