# Skill: Configure AI Bot Crawler Rules & Disclosures

> **Standard:** AI Bot Governance
> **Category:** BOT_ACCESS_CONTROL
> **Classification:** SPECIFICATION_AND_GUIDE
> **Author:** TinyCTO Architecture Guild (https://tinycto.tv)

## Overview
Establish transparent crawling and training guidelines for OpenAI, Anthropic, Google, and Perplexity autonomous agents.

## Learning Objectives
1. Permit AI search agents to index public documentation and failure catalogs.
2. Optionally manage training permissions with granular User-agent rules.
3. Prevent indiscriminate scraping of non-public admin portals.

## Prerequisites
- robots.txt editing access

## Implementation & Execution Guide

### Step 1: Configure Dedicated AI Bot Allow Rules
Ensure AI agents are not blanket blocked by aggressive generic WAFs.

## Architectural Takeaway
Explicit AI bot rules ensure your technical domain is cited accurately during user queries on AI search engines.