# Deep web scraping with AI-powered RAG: n8n workflow template

> Started by another system calling it, and works across Pinecone. It puts OpenAI behind a retrieval step, so answers are drawn from documents you supply...

Source: https://millionrounds.com/workflow/deep-web-scraping-with-ai-powered-rag (Millionrounds; every figure on the page links to its filing or article)

[Automation library](https://millionrounds.com/workflows) · [AI assistants](https://millionrounds.com/workflows?category=AI+assistants#library)

# Deep web scraping with AI-powered RAG

Started by another system calling it, and works across Pinecone. It puts OpenAI behind a retrieval step, so answers are drawn from documents you supply rather than from the model alone. 17 steps, 1 branch, 1 loop or waits: a working afternoon to understand properly.

Medium Calls a model: OpenAI Retrieval

[Download free](https://millionrounds.com/login?next=%2Fapp%2Fworkflow%2Fdeep-web-scraping-with-ai-powered-rag)

**17 steps** Medium: a working afternoon

**1** Branch: places the flow splits

**1** Loop or waits

**1** Service it connects to

## The flow

Starts it 4 Model 3 Logic 8 Service 2

The steps in the order they run, from what starts it.

- 1 **Configure Website & Limits** Set *Logic*
- 2 **Configure Your Question** Set *Logic*
- 3 **Trigger - Start Scraping** Webhook *Starts it*
- 4 **Trigger - Ask Question** Webhook *Starts it*
- 5 **Fetch Homepage** Http Request *Service*
- 6 **Search Similar Content** Vector Store Pinecone *Model*
- 7 **Extract All Page URLs** Code *Logic*
- 8 **Assemble AI Context** Code *Logic*
- 9 **Batch Processing Controller** Split In Batches *Logic*
- 10 **Generate Answer with GPT-4** Open Ai *Model*
- 11 **Fetch Page Content** Http Request *Service*
- 12 **Return AI Answer** Respond To Webhook *Starts it*
- 13 **Parse & Clean HTML** Code *Logic*
- 14 **Create Semantic Chunks** Code *Logic*
- 15 **Store in Pinecone Vector DB** Vector Store Pinecone *Model*
- 16 **Check Completion Status** If *Logic*
- 17 **Return Success Response** Respond To Webhook *Starts it*

## How it's built

- **Starts on:** Another system calling it
- **Steps:** 17, with 1 branch, 1 loop or waits
- **Size:** Medium: a working afternoon
- **Connects to:** Pinecone
- **Model:** OpenAI
- **Retrieval:** Answers are drawn from documents you supply, not from the model alone

## Questions

What does it need to run?

An n8n account, plus your own Pinecone account, and a key for OpenAI. You connect your own accounts after importing it; none are included.

What starts it?

It starts on another system calling it.

Does it use AI?

Yes. It calls OpenAI. Its answers draw on documents you give it, not on the model alone. Model calls are billed by the provider, not by n8n.

How long does it take to set up?

Plan an afternoon. 17 steps is enough that you should read each one before running it.

How do I import it?

Copy the workflow, open a new workflow in n8n and paste. The steps appear on the canvas, ready for your own accounts.

**Before you run it**

Nothing here has been tested by us. It is somebody's working automation, read and described from its own steps. Treat the connections it wants as a list of accounts it will touch, and read the 17 steps before you point it at anything that matters.

## Run it in n8n

- **Create your n8n account** Free to start. n8n is where the workflow runs. [Create an n8n account ↗](https://millionrounds.com/n8n)
- **Download the workflow** Free with a Millionrounds account. [Download free](https://millionrounds.com/login?next=%2Fapp%2Fworkflow%2Fdeep-web-scraping-with-ai-powered-rag)
- **Paste it into n8n** Open a new workflow in n8n and paste. Then connect your own accounts.

Stored with the author's traces removed: canvas notes, instance identifiers, saved credential names and any live document they had left behind.

## Connects to

Each opens the other workflows that use it.

[Pinecone](https://millionrounds.com/workflows?service=Pinecone#library)

## Workflows like it

Also ai assistants, with the same services where possible, closest in size first.

- [**Route Chatwoot customer chats to AI and live agents with Groq** 17 steps · calls a model · Gmail, Pinecone](https://millionrounds.com/workflow/route-chatwoot-customer-chats-to-ai-and-live-agents-with-gro)
- [**Build a RAG knowledge assistant with Google Drive and OpenAI** 16 steps · calls a model · Google Drive, Pinecone](https://millionrounds.com/workflow/build-a-rag-knowledge-assistant-with-google-drive-and-openai)
- [**Personalized email automation with Google Docs and Pinecone** 18 steps · calls a model · Gmail, Google Docs](https://millionrounds.com/workflow/personalized-email-automation-with-google-docs-and-pinecone)
- [**WhatsApp group chat with OpenAI and Google Drive** 16 steps · calls a model · Google Drive, Pinecone](https://millionrounds.com/workflow/whatsapp-group-chat-with-openai-and-google-drive)
- [**AI-powered Upwork cover letter generator – Pinecone, Groq** 15 steps · calls a model · Pinecone](https://millionrounds.com/workflow/ai-powered-upwork-cover-letter-generator-pinecone-groq)
- [**Build a chatbot trained on your internal company data in 20mins** 15 steps · calls a model · Google Drive, Pinecone](https://millionrounds.com/workflow/build-a-chatbot-trained-on-your-internal-company-data-in-20m)