Automation library · AI assistants

Deep web scraping with AI-powered RAG

Started by another system calling it, and works across Pinecone. It puts OpenAI behind a retrieval step, so answers are drawn from documents you supply rather than from the model alone. 17 steps, 1 branch, 1 loop or waits: a working afternoon to understand properly.

MediumCalls a model: OpenAIRetrieval

17 stepsMedium: a working afternoon
1Branch: places the flow splits
1Loop or waits
1Service it connects to

The flow

Starts it 4Model 3Logic 8Service 2

The steps in the order they run, from what starts it.

  1. 1Configure Website & LimitsSetLogic
  2. 2Configure Your QuestionSetLogic
  3. 3Trigger - Start ScrapingWebhookStarts it
  4. 4Trigger - Ask QuestionWebhookStarts it
  5. 5Fetch HomepageHttp RequestService
  6. 6Search Similar ContentVector Store PineconeModel
  7. 7Extract All Page URLsCodeLogic
  8. 8Assemble AI ContextCodeLogic
  9. 9Batch Processing ControllerSplit In BatchesLogic
  10. 10Generate Answer with GPT-4Open AiModel
  11. 11Fetch Page ContentHttp RequestService
  12. 12Return AI AnswerRespond To WebhookStarts it
  13. 13Parse & Clean HTMLCodeLogic
  14. 14Create Semantic ChunksCodeLogic
  15. 15Store in Pinecone Vector DBVector Store PineconeModel
  16. 16Check Completion StatusIfLogic
  17. 17Return Success ResponseRespond To WebhookStarts it

How it's built

Starts on
Another system calling it
Steps
17, with 1 branch, 1 loop or waits
Size
Medium: a working afternoon
Connects to
Pinecone
Model
OpenAI
Retrieval
Answers are drawn from documents you supply, not from the model alone

Questions

What does it need to run?

An n8n account, plus your own Pinecone account, and a key for OpenAI. You connect your own accounts after importing it; none are included.

What starts it?

It starts on another system calling it.

Does it use AI?

Yes. It calls OpenAI. Its answers draw on documents you give it, not on the model alone. Model calls are billed by the provider, not by n8n.

How long does it take to set up?

Plan an afternoon. 17 steps is enough that you should read each one before running it.

How do I import it?

Copy the workflow, open a new workflow in n8n and paste. The steps appear on the canvas, ready for your own accounts.

Before you run it

Nothing here has been tested by us. It is somebody's working automation, read and described from its own steps. Treat the connections it wants as a list of accounts it will touch, and read the 17 steps before you point it at anything that matters.