Generate llms.txt Files from Screaming Frog Data with OpenAI

## Text Classifier 🚫 This node is deactivated by default in the template. You can enable it if you want to add a more "intelligent" 🤓 filter to refine the URLs included in the llms.txt file, helping LLMs discover and prioritize valuable content. ### How It Works: This node has two outputs: - useful_content → Pages that are likely to contain high-quality content, making them suitable for inclusion in a file that aids content discovery for an LLM. - other_content → Pages that should not be included (e.g., pagination or low-value content). You can modify the description in the node to fine-tune the classification according to your needs. ### Input Fields: - url → {{ $json.url }} - title → {{ $json.title }} - description → {{ $json.description }} - word_count → {{ $json.word_count }} ### Why use an LLM? A language model (LLM) can analyze the URL, title, and description to identify pages that most likely contain meaningful and relevant content. This allows it to prioritize valuable pages and structure the data for better content discovery and training purposes. ### For large websites If you have a very large website, consider using a Loop Over Items node to make the workflow more robust and ensure all pages are processed. Also, using a Loop Over Items node make it easier to handle: - Timeouts - API quotas - Other scalability issues ### Tokens usage Finally, keep in mind that more pages mean more tokens and more billed LLM API calls.