Monitor AI crawler policy signals via HTTP, Data Tables, email and Slack

## What it does Compare robots.txt, llms.txt, TDM reservation records and homepage X-Robots-Tag signals for training, search and user-triggered tokens. Send one HTML email for baselines or policy changes. ## Who it is for Website owners and teams tracking published AI crawler preferences on a small list of sites. ## How it works 1. Load per-site snapshots from a Data Table. 2. Fetch public files with a descriptive User-Agent, pacing, timeout and transient retries. 3. Match product-token groups and longest path rules. Compare bot statuses and all four signals; classify changes by severity. 4. Email a severity digest, optionally escalate to Slack, then save successful snapshots. Fetch errors preserve old state. Malformed TDM content is a warning, never a legal verdict. ## Setup 1. Create the five-column Data Table in SETUP.md. 2. Edit Config for sites, bots, table ID and email addresses. 3. Select SMTP credentials and run twice manually before publishing the daily schedule. ## Requirements n8n 2.8.4, Data Tables, SMTP and outbound public HTTP access. Keep executions sequential. robots.txt is a voluntary convention; this template reports what sites publish and does not assess legal compliance. ## How to customize Edit Config for origins, bot tokens, llms.txt checks, baseline email and policyCheckHour (0 to 23 in the schedule timezone). Config controls evaluatedPaths and botCategories. High means training access reopened or search root blocked; other rule and TDM/header changes are medium; llms-only changes are low. Optional Slack is disabled; enable it with credentials for severity escalation. These are published signals, not legal advice. A fuller version could audit meta robots tags across many pages.