# From sitemap crawling to vector storage: Creating an efficient: n8n workflow

> Started by an event in a connected app, and works across Postgres, Supabase and Supabase vectors.

Source: https://millionrounds.com/workflow/from-sitemap-crawling-to-vector-storage-creating-an-efficien (Millionrounds; every figure on the page links to its filing or article)

[Automation library](https://millionrounds.com/workflows) · [Data collection](https://millionrounds.com/workflows?category=Data+collection#library)

# From sitemap crawling to vector storage: Creating an efficient

Started by an event in a connected app, and works across Postgres, Supabase and Supabase vectors. It puts OpenAI behind a retrieval step, so answers are drawn from documents you supply rather than from the model alone. 35 steps, 4 branches, 4 loops or waits: large, and worth reading before you trust it with anything live.

Large Calls a model: OpenAI Retrieval

[Download free](https://millionrounds.com/login?next=%2Fapp%2Fworkflow%2Ffrom-sitemap-crawling-to-vector-storage-creating-an-efficien)

**35 steps** Large: read it before trusting it with anything live

**4** Branches: places the flow splits

**4** Loops or waits

**3** Services it connects to

## The flow

Starts it 1 Model 4 Logic 19 Service 11

The steps in the order they run, from what starts it.

- 1 **When clicking ‘Test workflow’** Manual trigger *Starts it*
- 2 **Character Text Splitter** Text Splitter Character Text Splitter *Model*
- 3 **Embeddings OpenAI** Embeddings Open Ai *Model*
- 4 **CREATE TABLE scrape_queue in Supabase** Postgres *Service*
- 5 **CREATE TABLE scrape_queue in Supabase1** Postgres *Service*
- 6 **HTTP Request** Http Request *Service*
- 7 **Default Data Loader** Document Default Data Loader *Model*
- 8 **Supabase Vector Store_documents** Vector Store Supabase *Model*
- 9 **XML** Xml *Logic*
- 10 **Update a row in scrape_queue Table** Supabase *Service*
- 11 **Split Out** Split Out *Logic*
- 12 **Loop Over Items** Split In Batches *Logic*
- 13 **Loop Over Items1** Split In Batches *Logic*
- 14 **Get a row - scrape_queue Table** Supabase *Service*
- 15 **Split Out1** Split Out *Logic*
- 16 **Format the URL** Set *Logic*
- 17 **If2** If *Logic*
- 18 **Check if the URL is in the Supabase Table** Supabase *Service*
- 19 **Crawl4ai Web Page Scrape** Http Request *Service*
- 20 **Format the Output from the Supabase node** Code *Logic*
- 21 **Wait** Wait *Logic*
- 22 **Wait1** Wait *Logic*
- 23 **If "shouldInsert" is true** If *Logic*
- 24 **Crawl4AI_Task Status** Http Request *Service*
- 25 **URL in a new row** Supabase *Service*
- 26 **If** If *Logic*
- 27 **Update a row in scrape_queue Table1** Supabase *Service*
- 28 **Remove redundant data from the scraping** Code *Logic*
- 29 **Edit Fields** Set *Logic*
- 30 **Quality Filter Node** Code *Logic*
- 31 **Task_id Counter** Code *Logic*
- 32 **Content Type Detection** Code *Logic*
- 33 **If1** If *Logic*
- 34 **Better Metadata Extraction** Code *Logic*
- 35 **Update a row in scrape_queue Table2** Supabase *Service*

## How it's built

- **Starts on:** An event in a connected app
- **Steps:** 35, with 4 branches, 4 loops or waits
- **Size:** Large: read it before trusting it with anything live
- **Connects to:** Postgres, Supabase, Supabase vectors
- **Model:** OpenAI
- **Retrieval:** Answers are drawn from documents you supply, not from the model alone

## Questions

What does it need to run?

An n8n account, plus your own Postgres, Supabase and Supabase vectors accounts, and a key for OpenAI. You connect your own accounts after importing it; none are included.

What starts it?

It starts on an event in a connected app.

Does it use AI?

Yes. It calls OpenAI. Its answers draw on documents you give it, not on the model alone. Model calls are billed by the provider, not by n8n.

How long does it take to set up?

Take your time. At 35 steps it is large: read it before trusting it with anything live.

How do I import it?

Copy the workflow, open a new workflow in n8n and paste. The steps appear on the canvas, ready for your own accounts.

**Before you run it**

Nothing here has been tested by us. It is somebody's working automation, read and described from its own steps. Treat the connections it wants as a list of accounts it will touch, and read the 35 steps before you point it at anything that matters.

## Run it in n8n

- **Create your n8n account** Free to start. n8n is where the workflow runs. [Create an n8n account ↗](https://millionrounds.com/n8n)
- **Download the workflow** Free with a Millionrounds account. [Download free](https://millionrounds.com/login?next=%2Fapp%2Fworkflow%2Ffrom-sitemap-crawling-to-vector-storage-creating-an-efficien)
- **Paste it into n8n** Open a new workflow in n8n and paste. Then connect your own accounts.

Stored with the author's traces removed: canvas notes, instance identifiers, saved credential names and any live document they had left behind.

## Connects to

Each opens the other workflows that use it.

[Postgres](https://millionrounds.com/workflows?service=Postgres#library)[Supabase](https://millionrounds.com/workflows?service=Supabase#library)[Supabase vectors](https://millionrounds.com/workflows?service=Supabase+vectors#library)

## Workflows like it

Also data collection, with the same services where possible, closest in size first.

- [**Find and score event and speaker opportunities with Gumloop** 35 steps · Gmail, Postgres](https://millionrounds.com/workflow/find-and-score-event-and-speaker-opportunities-with-gumloop)
- [**Create a self-updating RAG chatbot with Google Drive** 34 steps · calls a model · Google Drive, Postgres](https://millionrounds.com/workflow/create-a-self-updating-rag-chatbot-with-google-drive)
- [**Websites into a conversational knowledge base with OpenAI RAG** 34 steps · calls a model · Supabase, Supabase vectors](https://millionrounds.com/workflow/websites-into-a-conversational-knowledge-base-with-openai-ra)
- [**AI chatbot call center: Taxi service with Grok and Postgres** 33 steps · calls a model · Postgres, Redis](https://millionrounds.com/workflow/ai-chatbot-call-center-taxi-service-with-grok-and-postgres)
- [**Filter URLs with Gemini and Groq** 37 steps · calls a model · Postgres](https://millionrounds.com/workflow/filter-urls-with-gemini-and-groq)
- [**Build an omnichannel OTP verification flow with Postgres** 33 steps · Email, Postgres](https://millionrounds.com/workflow/build-an-omnichannel-otp-verification-flow-with-postgres)