Skip to content
NEWMarketplace and the trig8n Agent: sell what you build, and get plans and changes that are tested before you see them. See what is new

Library / Web Scraping & Research

Web Scraping & Research

Pull Website Data Without an n8n Node

When you run it, paginated web pages are requested and article titles are extracted with the HTML Extract node, showing how to handle missing integrations.

Import it free Sign in to open it

What it does

## 3. Handle Pagination ### Sometimes you need to make the same request multiple times to get all the data you need (pagination). ### The pagination process goes as follow: ### 1. Loop through the pages of the input source (HTTP Request node named "Get my Starts") ### 2. Increment the page at the end of each loop (done with the set node named "Increment Page") ### 3. Stop looping when there are no pages left (checked at the If node named "Are we Finished?")

The steps

  1. On clicking 'execute'
  2. Set
  3. Item Lists - Create Items from Body
  4. HTML Extract - Extract Article Title
  5. Item Lists - Fetch Body
  6. If - Are we finished?
  7. Set - Increment Page
  8. HTTP Request - Get Mock Albums
  9. HTTP Request - Get Wikipedia Page
  10. HTTP Request - Get my Stars

Similar automations

Code, Git & Developer ToolsRestore n8n Workflows from Merged GitHub Pull Requests13 stepsCode, Git & Developer ToolsBack Up n8n Workflows to GitHub via Pull Requests22 stepsCode, Git & Developer ToolsReview GitHub Pull Requests with OpenAI8 stepsCode, Git & Developer ToolsReview GitHub Pull Requests with OpenAI, Log to Sheets20 steps