Automation library · Data collection

From sitemap crawling to vector storage: Creating an efficient

Started by an event in a connected app, and works across Postgres, Supabase and Supabase vectors. It puts OpenAI behind a retrieval step, so answers are drawn from documents you supply rather than from the model alone. 35 steps, 4 branches, 4 loops or waits: large, and worth reading before you trust it with anything live.

LargeCalls a model: OpenAIRetrieval

Download free
35 stepsLarge: read it before trusting it with anything live
4Branches: places the flow splits
4Loops or waits
3Services it connects to

The flow

Starts it 1Model 4Logic 19Service 11

The steps in the order they run, from what starts it.

  1. 1When clicking ‘Test workflow’Manual triggerStarts it
  2. 2Character Text SplitterText Splitter Character Text SplitterModel
  3. 3Embeddings OpenAIEmbeddings Open AiModel
  4. 4CREATE TABLE scrape_queue in SupabasePostgresService
  5. 5CREATE TABLE scrape_queue in Supabase1PostgresService
  6. 6HTTP RequestHttp RequestService
  7. 7Default Data LoaderDocument Default Data LoaderModel
  8. 8Supabase Vector Store_documentsVector Store SupabaseModel
  9. 9XMLXmlLogic
  10. 10Update a row in scrape_queue TableSupabaseService
  11. 11Split OutSplit OutLogic
  12. 12Loop Over ItemsSplit In BatchesLogic
  13. 13Loop Over Items1Split In BatchesLogic
  14. 14Get a row - scrape_queue TableSupabaseService
  15. 15Split Out1Split OutLogic
  16. 16Format the URLSetLogic
  17. 17If2IfLogic
  18. 18Check if the URL is in the Supabase TableSupabaseService
  19. 19Crawl4ai Web Page ScrapeHttp RequestService
  20. 20Format the Output from the Supabase nodeCodeLogic
  21. 21WaitWaitLogic
  22. 22Wait1WaitLogic
  23. 23If "shouldInsert" is trueIfLogic
  24. 24Crawl4AI_Task StatusHttp RequestService
  25. 25URL in a new rowSupabaseService
  26. 26IfIfLogic
  27. 27Update a row in scrape_queue Table1SupabaseService
  28. 28Remove redundant data from the scrapingCodeLogic
  29. 29Edit FieldsSetLogic
  30. 30Quality Filter NodeCodeLogic
  31. 31Task_id CounterCodeLogic
  32. 32Content Type DetectionCodeLogic
  33. 33If1IfLogic
  34. 34Better Metadata ExtractionCodeLogic
  35. 35Update a row in scrape_queue Table2SupabaseService

How it's built

Starts on
An event in a connected app
Steps
35, with 4 branches, 4 loops or waits
Size
Large: read it before trusting it with anything live
Connects to
Postgres, Supabase, Supabase vectors
Model
OpenAI
Retrieval
Answers are drawn from documents you supply, not from the model alone

Questions

What does it need to run?

An n8n account, plus your own Postgres, Supabase and Supabase vectors accounts, and a key for OpenAI. You connect your own accounts after importing it; none are included.

What starts it?

It starts on an event in a connected app.

Does it use AI?

Yes. It calls OpenAI. Its answers draw on documents you give it, not on the model alone. Model calls are billed by the provider, not by n8n.

How long does it take to set up?

Take your time. At 35 steps it is large: read it before trusting it with anything live.

How do I import it?

Copy the workflow, open a new workflow in n8n and paste. The steps appear on the canvas, ready for your own accounts.

Before you run it

Nothing here has been tested by us. It is somebody's working automation, read and described from its own steps. Treat the connections it wants as a list of accounts it will touch, and read the 35 steps before you point it at anything that matters.