Automation library · Data collection

Filter URLs with Gemini and Groq

Started by a clock, and works across Postgres. It hands the judgement calls to Gemini, Groq and one other. 37 steps, 4 branches: large, and worth reading before you trust it with anything live.

LargeCalls a model: Gemini, Groq

Download free
37 stepsLarge: read it before trusting it with anything live
4Branches: places the flow splits
0Loops or waits
1Service it connects to

The flow

Starts it 2Model 5Logic 16Service 14

The steps in the order they run, from what starts it.

  1. 1Schedule TriggerSchedule triggerStarts it
  2. 2Mistral Cloud Chat ModelLm Chat Mistral CloudModel
  3. 3Groq Chat ModelLm Chat GroqModel
  4. 4Google Gemini Chat ModelLm Chat Google GeminiModel
  5. 5Delete robots.txt TablePostgresService
  6. 6StartExecute Workflow triggerStarts it
  7. 7Check all robots.txt TablePostgresService
  8. 8Check all forbidden urls TablePostgresService
  9. 9Delete forbidden urls TablePostgresService
  10. 10Manual Upsert robots.txt TablePostgresService
  11. 11Manual Upsert forbidden urls TablePostgresService
  12. 12Manual Create forbidden urls tablePostgresService
  13. 13Manually Create robots.txt TablePostgresService
  14. 14Start (Tests)SetLogic
  15. 15Model SelectorModel SelectorModel
  16. 16Get Base URLCodeLogic
  17. 17Information ExtractorInformation ExtractorModel
  18. 18Create forbidden urls tablePostgresService
  19. 19If Link Allowed 2IfLogic
  20. 20Assume False OutputSetLogic
  21. 21Check forbidden urls tablePostgresService
  22. 22Prepare Output for Link AllowedSetLogic
  23. 23Prepare Output for Link DisallowedSetLogic
  24. 24If forbidden url DetectedIfLogic
  25. 25Prepare OutputSetLogic
  26. 26Return DisallowSetLogic
  27. 27Create robots.txt TablePostgresService
  28. 28Upsert robots.txt TablePostgresService
  29. 29Check robots.txt TablePostgresService
  30. 30OutputSetLogic
  31. 31If robots.txt Found and UpdatedIfLogic
  32. 32Prepare Robots.txt Check 2SetLogic
  33. 33Prepare Robots.txt CheckSetLogic
  34. 34Check Robots.txtCodeLogic
  35. 35Get Robots.txtHttp RequestService
  36. 36If Link AllowedIfLogic
  37. 37Assume DisallowSetLogic

How it's built

Starts on
A clock
Steps
37, with 4 branches
Size
Large: read it before trusting it with anything live
Connects to
Postgres
Model
Gemini, Groq, Mistral

Questions

What does it need to run?

An n8n account, plus your own Postgres account, and a key for Gemini and Groq. You connect your own accounts after importing it; none are included.

What starts it?

It starts on a clock.

Does it use AI?

Yes. It calls Gemini, Groq and Mistral. Model calls are billed by the provider, not by n8n.

How long does it take to set up?

Take your time. At 37 steps it is large: read it before trusting it with anything live.

How do I import it?

Copy the workflow, open a new workflow in n8n and paste. The steps appear on the canvas, ready for your own accounts.

Before you run it

Nothing here has been tested by us. It is somebody's working automation, read and described from its own steps. Treat the connections it wants as a list of accounts it will touch, and read the 37 steps before you point it at anything that matters.