Automation library · Research and reporting

Evaluate AI prompts with OpenRouter LLM-as-judge and Notion

Started by someone submitting a form, and works across Notion. It hands the judgement calls to OpenRouter. 17 steps, 1 loop or waits: a working afternoon to understand properly.

MediumCalls a model: OpenRouter

17 stepsMedium: a working afternoon
0Branches: places the flow splits
1Loop or waits
1Service it connects to

The flow

Starts it 1Model 8Logic 6Service 2

The steps in the order they run, from what starts it.

  1. 1When Form SubmittedForm triggerStarts it
  2. 2OpenAI Dataset GeneratorLm Chat Open RouterModel
  3. 3OpenAI Answer AgentLm Chat Open RouterModel
  4. 4OpenAI Grader AgentLm Chat Open RouterModel
  5. 5OpenAI Synthesis AgentLm Chat Open RouterModel
  6. 6Set Notion IDsSetLogic
  7. 7Dataset GeneratorAgentModel
  8. 8Answer AgentAgentModel
  9. 9Grader AgentAgentModel
  10. 10Synthesis AgentAgentModel
  11. 11Flatten Dataset CodeCodeLogic
  12. 12Extract Grader ResultsCodeLogic
  13. 13Upload Report to NotionNotionService
  14. 14Batch Process ItemsSplit In BatchesLogic
  15. 15Collate Evaluation ResultsCodeLogic
  16. 16Upload Item to NotionNotionService
  17. 17Prepare Evaluation ItemCodeLogic

How it's built

Starts on
Someone submitting a form
Steps
17, 1 loop or waits
Size
Medium: a working afternoon
Connects to
Notion
Model
OpenRouter

Questions

What does it need to run?

An n8n account, plus your own Notion account, and a key for OpenRouter. You connect your own accounts after importing it; none are included.

What starts it?

It starts on someone submitting a form.

Does it use AI?

Yes. It calls OpenRouter. Model calls are billed by the provider, not by n8n.

How long does it take to set up?

Plan an afternoon. 17 steps is enough that you should read each one before running it.

How do I import it?

Copy the workflow, open a new workflow in n8n and paste. The steps appear on the canvas, ready for your own accounts.

Before you run it

Nothing here has been tested by us. It is somebody's working automation, read and described from its own steps. Treat the connections it wants as a list of accounts it will touch, and read the 17 steps before you point it at anything that matters.