# Skill: Publish RFC 9309 Compliant robots.txt for AI Crawlers

> **Standard:** RFC 9309
> **Category:** DISCOVERABILITY
> **Classification:** SPECIFICATION_AND_GUIDE
> **Author:** TinyCTO Architecture Guild (https://tinycto.tv)

## Overview
Architect and deploy an RFC 9309 compliant /robots.txt with explicit crawler permissions for AI agents (ClaudeBot, GPTBot, PerplexityBot) and sitemap declarations.

## Learning Objectives
1. Define explicit User-agent rules for general search engines and dedicated AI training/search bots.
2. Advertise canonical sitemap locations without circular redirects.
3. Ensure HTTP 200 plain text responses with proper caching headers.

## Prerequisites
- Understanding of web crawler User-agents
- Access to root web server / reverse proxy config

## Implementation & Execution Guide

### Step 1: Declare Dedicated Bot Blocks
Specify rules for AI search agents and general crawlers separately.

```http
User-agent: *
Allow: /
Disallow: /admin/

User-agent: GPTBot
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: PerplexityBot
Allow: /

Sitemap: https://tinycto.tv/sitemap.xml
```

### Step 2: Validate Content-Type
Ensure the web server delivers 'text/plain; charset=utf-8' without HTML wrapper boilerplate.

## Architectural Takeaway
Clear robots.txt directives allow autonomous agents to distinguish between permitted indexable content and internal operational dashboards.