# Autonomous AI crawler with OpenAI and Supabase: n8n workflow

> Started by an event in a connected app, and works across Supabase. It hands the judgement calls to OpenAI.

Source: https://millionrounds.com/workflow/autonomous-ai-crawler-with-openai-and-supabase (Millionrounds; every figure on the page links to its filing or article)

[Automation library](https://millionrounds.com/workflows) · [Data collection](https://millionrounds.com/workflows?category=Data+collection#library)

# Autonomous AI crawler with OpenAI and Supabase

Started by an event in a connected app, and works across Supabase. It hands the judgement calls to OpenAI. 28 steps: large, and worth reading before you trust it with anything live.

Large Calls a model: OpenAI

[Download free](https://millionrounds.com/login?next=%2Fapp%2Fworkflow%2Fautonomous-ai-crawler-with-openai-and-supabase)

**28 steps** Large: read it before trusting it with anything live

**0** Branches: places the flow splits

**0** Loops or waits

**1** Service it connects to

## The flow

Starts it 1 Model 5 Logic 18 Service 4

The steps in the order they run, from what starts it.

- 1 **Text** Tool Workflow *Model*
- 2 **URLs** Tool Workflow *Model*
- 3 **OpenAI Chat Model** Lm Chat Open Ai *Model*
- 4 **JSON Parser** Output Parser Structured *Model*
- 5 **Execute workflow** Manual trigger *Starts it*
- 6 **Set domain (text)** Set *Logic*
- 7 **Set domain (URL)** Set *Logic*
- 8 **Crawl website** Agent *Model*
- 9 **Get companies** Supabase *Service*
- 10 **Add protocool to domain (text)** Set *Logic*
- 11 **Add protocool to domain (URL)** Set *Logic*
- 12 **Set social media array** Set *Logic*
- 13 **Select company name and website** Set *Logic*
- 14 **Get website (text)** Http Request *Service*
- 15 **Get website (URL)** Http Request *Service*
- 16 **Merge all data** Merge *Logic*
- 17 **Map company name and website** Set *Logic*
- 18 **Convert HTML to Markdown** Markdown *Logic*
- 19 **Retrieve URLs** Html *Logic*
- 20 **Insert new row** Supabase *Service*
- 21 **Set response (text)** Set *Logic*
- 22 **Split out URLs** Split Out *Logic*
- 23 **Filter out empty hrefs** Filter *Logic*
- 24 **Remove duplicated** Remove Duplicates *Logic*
- 25 **Set domain to path** Set *Logic*
- 26 **Filter out invalid URLs** Filter *Logic*
- 27 **Aggregate URLs** Aggregate *Logic*
- 28 **Set response (URL)** Set *Logic*

## How it's built

- **Starts on:** An event in a connected app
- **Steps:** 28
- **Size:** Large: read it before trusting it with anything live
- **Connects to:** Supabase
- **Model:** OpenAI

## Questions

What does it need to run?

An n8n account, plus your own Supabase account, and a key for OpenAI. You connect your own accounts after importing it; none are included.

What starts it?

It starts on an event in a connected app.

Does it use AI?

Yes. It calls OpenAI. Model calls are billed by the provider, not by n8n.

How long does it take to set up?

Take your time. At 28 steps it is large: read it before trusting it with anything live.

How do I import it?

Copy the workflow, open a new workflow in n8n and paste. The steps appear on the canvas, ready for your own accounts.

**Before you run it**

Nothing here has been tested by us. It is somebody's working automation, read and described from its own steps. Treat the connections it wants as a list of accounts it will touch, and read the 28 steps before you point it at anything that matters.

## Run it in n8n

- **Create your n8n account** Free to start. n8n is where the workflow runs. [Create an n8n account ↗](https://millionrounds.com/n8n)
- **Download the workflow** Free with a Millionrounds account. [Download free](https://millionrounds.com/login?next=%2Fapp%2Fworkflow%2Fautonomous-ai-crawler-with-openai-and-supabase)
- **Paste it into n8n** Open a new workflow in n8n and paste. Then connect your own accounts.

Stored with the author's traces removed: canvas notes, instance identifiers, saved credential names and any live document they had left behind.

## Connects to

Each opens the other workflows that use it.

[Supabase](https://millionrounds.com/workflows?service=Supabase#library)

## Workflows like it

Also data collection, with the same services where possible, closest in size first.

- [**Automated email responses with OpenAI and Supabase** 27 steps · calls a model · Outlook, Postgres](https://millionrounds.com/workflow/automated-email-responses-with-openai-and-supabase)
- [**Automate viral video generation with Shotstack, OpenAI, and S3** 30 steps · calls a model · S3, Supabase](https://millionrounds.com/workflow/automacao-gerar-videos-virais-shotstack-with-openai-and-s3)
- [**Sync a Telegram RAG chatbot with Google Drive and OpenAI** 25 steps · calls a model · Google Drive, Supabase](https://millionrounds.com/workflow/sync-a-telegram-rag-chatbot-with-google-drive-and-openai)
- [**Classify and log AI use cases with Anthropic and Supabase** 25 steps · Email, Supabase](https://millionrounds.com/workflow/classify-and-log-ai-use-cases-with-anthropic-and-supabase)
- [**Match product manager jobs with Supabase and Claude** 25 steps · Supabase](https://millionrounds.com/workflow/match-product-manager-jobs-with-supabase-and-claude)
- [**Google Drive to Pinecone vector storage workflow with OpenAI** 32 steps · calls a model · Google Drive, Supabase](https://millionrounds.com/workflow/google-drive-to-pinecone-vector-storage-workflow-with-openai)