Bibliothek / Web Scraping & Recherche
Web Scraping & Recherche
Gescrapten HTML-Code bereinigen und für die Verarbeitung durch OpenAI in Markdown umwandeln
Unterworkflow, der von anderen Workflows aufgerufen wird. Er entfernt Skripte, Medien und Kommentare aus abgerufenen Seiten, ersetzt optional URLs und Bilder und wandelt den Rest für OpenAI in Markdown um.
Was sie bewirkt
## Post-processing of the HTML page: 1. Keep only content 2. Remove inline tag entirely, as well as: NOSCRIPT, IFRAME, OBJECT, EMBED, VIDEO, AUDIO, SVG, and HTML comments. 3. In case query parameter method=simplified, replace all page URLs (a href) and IMG (src) with NOURL / NOIMG - this may save up to 20% of the page length 4. Convert the remaining HTML to Markdown. This step further reduces the length of the page: long HTML tags and styles are eliminated, but the markdown syntax keeps some page structure. This gives much better results compared to just a blank text. 5. Finally, check the page length. If it's too long, send an "ERROR: PAGE CONTENT TOO LONG" instead of the actual page. Of course, you could split the page content in chunks, but sometimes long pages just don't have a needed content, so it makes little sense to burn tokens on them.
Die Schritte
- When chat message received
- AI Agent
- OpenAI Chat Model1
- When Executed by Another Workflow
- QUERY_PARAMS
- CONFIG
- HTTP Request
- Is error?
- Stringify error message
- Exctract HTML Body
- Remove extra tags
- Simplify?
- Simplify output
- Convert to Markdown
Insgesamt 16 Schritte.
Ähnliche Automatisierungen
Texterstellung & Content-GenerierungBaserow-Markdown in HTML umwandeln18 SchritteTexterstellung & Content-GenerierungAirtable-Markdown per Webhook in HTML umwandeln18 SchritteDokumente, PDF & OCRBaserow-Markdown-Felder in HTML umwandeln8 SchritteDokumente, PDF & OCRAirtable-Markdown-Felder in HTML konvertieren8 Schritte