Automation library · AI assistants

Evaluate AI agent response correctness with OpenAI and RAGAS

Started by someone opening a chat with it. It calls services over plain HTTP rather than through connectors. It hands the judgement calls to OpenAI. 24 steps: large, and worth reading before you trust it with anything live.

LargeCalls a model: OpenAI

24 stepsLarge: read it before trusting it with anything live
0Branches: places the flow splits
0Loops or waits
0Services by connector: it calls them over plain HTTP

The flow

Starts it 2Model 5Logic 12Service 5

The steps in the order they run, from what starts it.

  1. 1Examples1Output Parser StructuredModel
  2. 2OpenAI Chat ModelLm Chat Open AiModel
  3. 3OpenAI Chat Model1Lm Chat Open AiModel
  4. 4When fetching a dataset rowEvaluation triggerStarts it
  5. 5When chat message receivedChat triggerStarts it
  6. 6Correctness ClassifierChain LlmModel
  7. 7AI AgentAgentModel
  8. 8Remap InputSetLogic
  9. 9Calculate F1 ScoreCodeLogic
  10. 10EvaluationEvaluationService
  11. 11MergeMergeLogic
  12. 12Set Input FieldsSetLogic
  13. 13No Operation, do nothingNo OpLogic
  14. 14Correctness ScoreCodeLogic
  15. 15Get Embeddings1Http RequestService
  16. 16GroundTruth to ItemsSplit OutLogic
  17. 17Update OutputsEvaluationService
  18. 18Remap EmbeddingsSetLogic
  19. 19Get EmbeddingsHttp RequestService
  20. 20Update MetricsEvaluationService
  21. 21Create Embeddings ResultMergeLogic
  22. 22Remap Embeddings1SetLogic
  23. 23Calculate Similarity ScoreCodeLogic
  24. 24AggregateAggregateLogic

How it's built

Starts on
Someone opening a chat with it
Steps
24
Size
Large: read it before trusting it with anything live
Connects to
Nothing by connector; it calls services over plain HTTP
Model
OpenAI

Questions

What does it need to run?

An n8n account, and a key for OpenAI. You connect your own accounts after importing it; none are included.

What starts it?

It starts on someone opening a chat with it.

Does it use AI?

Yes. It calls OpenAI. Model calls are billed by the provider, not by n8n.

How long does it take to set up?

Take your time. At 24 steps it is large: read it before trusting it with anything live.

How do I import it?

Copy the workflow, open a new workflow in n8n and paste. The steps appear on the canvas, ready for your own accounts.

Before you run it

Nothing here has been tested by us. It is somebody's working automation, read and described from its own steps. Treat the connections it wants as a list of accounts it will touch, and read the 24 steps before you point it at anything that matters.