Library / Web Scraping & Research
Web Scraping & Research
Scrape Any Web Page with Selenium Behind a Webhook, Including Login Pages
Webhook-triggered workflow that uses a self-hosted Selenium container to collect data from any web page, with or without login, and return it.
What it does
## N8N Ultimate Scraper - Workflow This workflow's objective is to collect data from any website page, whether it requires login or not. For example, you can collect the number of stars of the n8n-ultimate-scraper project on GitHub. ## Requirements Selenium Container: Selenium is an open-source automation framework for web applications, enabling browser control and interaction through scripts in various programming languages. You can deploy the Docker Compose file from the associated GitHub project to set up your Selenium container and configuration: {{ $env.WEBHOOK_URL }} Residential Proxy Server: To scrape data at scale without being blocked, I personally recommend GeoNode. They offer affordable, high-quality residential proxies: {{ $env.WEBHOOK_URL }} OpenAI API Key: For using GPT-4. ## Optional Session Cookies Collection: To use login functionality with the n8n Ultimate Scraper, you need to collect session cookies from the target website. You can do this using the extension created for this application in the GitHub project: {{ $env.WEBHOOK_URL }} Follow the installation procedure to use it. ## How to use Deploy the project with all the requiremnts and request your webhook. Example of request: curl -X POST {{ $env.WEBHOOK_URL }} \ -H "Content-Type: application/json" \ -d '{ "subject": "Hugging Face", "Url": "github.com", "Target data": [ { "DataName": "Followers", "description": "The number of followers of the GitHub page" }, { "DataName": "Total Stars", "description": "The total numbers of stars on the different repos" } ], "cookies": [] }' You can also scrape link like this : curl -X POST {{ $env.WEBHOOK_URL }} \ -H "Content-Type: application/json" \ -d '{ "Target Url": "{{ $env.WEBHOOK_URL }}", "Target data": [ { "DataName": "Followers", "description": "The number of followers of the GitHub page" }, { "DataName": "Total Stars", "description": "The total numbers of stars on the different repo" } ] }' Note The maximum nimber of Target data is 5.
The steps
- Extract First Url Match
- OpenAI Chat Model
- Clean Webdriver
- Delete Session
- Delete Session2
- If Block1
- Delete Session3
- Limit
- Delete Session1
- Delete Session4
- Success with cookie
- Respond to Webhook2
- Code
- Delete Session5
56 steps in total.
Similar automations
Web Scraping & ResearchScrape Any Website Page with Selenium and OpenAI, Including Logins56 stepsWeb Scraping & ResearchScrape Any Website with Selenium and OpenAI56 stepsWeb Scraping & ResearchSelenium Web Scraper with OpenAI for Any Website56 stepsWeb Scraping & ResearchScrape Any Website with Selenium and OpenAI362 steps