Automation library · AI assistants

Evaluations metric: answer similarity with OpenAI

Started by someone opening a chat with it. It calls services over plain HTTP rather than through connectors. It hands the judgement calls to OpenAI. 18 steps: a working afternoon to understand properly.

MediumCalls a model: OpenAI

Download free
18 stepsMedium: a working afternoon
0Branches: places the flow splits
0Loops or waits
0Services by connector: it calls them over plain HTTP

The flow

Starts it 2Model 2Logic 9Service 5

The steps in the order they run, from what starts it.

  1. 1OpenAI Chat Model1Lm Chat Open AiModel
  2. 2When fetching a dataset rowEvaluation triggerStarts it
  3. 3When chat message receivedChat triggerStarts it
  4. 4AI AgentAgentModel
  5. 5Remap InputSetLogic
  6. 6EvaluationEvaluationService
  7. 7Set Input FieldsSetLogic
  8. 8No Operation, do nothingNo OpLogic
  9. 9Get Embeddings1Http RequestService
  10. 10GroundTruth to ItemsSplit OutLogic
  11. 11Remap EmbeddingsSetLogic
  12. 12Get EmbeddingsHttp RequestService
  13. 13Create Embeddings ResultMergeLogic
  14. 14Remap Embeddings1SetLogic
  15. 15Calculate Similarity ScoreCodeLogic
  16. 16AggregateAggregateLogic
  17. 17Update OutputEvaluationService
  18. 18Update MetricsEvaluationService

How it's built

Starts on
Someone opening a chat with it
Steps
18
Size
Medium: a working afternoon
Connects to
Nothing by connector; it calls services over plain HTTP
Model
OpenAI

Questions

What does it need to run?

An n8n account, and a key for OpenAI. You connect your own accounts after importing it; none are included.

What starts it?

It starts on someone opening a chat with it.

Does it use AI?

Yes. It calls OpenAI. Model calls are billed by the provider, not by n8n.

How long does it take to set up?

Plan an afternoon. 18 steps is enough that you should read each one before running it.

How do I import it?

Copy the workflow, open a new workflow in n8n and paste. The steps appear on the canvas, ready for your own accounts.

Before you run it

Nothing here has been tested by us. It is somebody's working automation, read and described from its own steps. Treat the connections it wants as a list of accounts it will touch, and read the 18 steps before you point it at anything that matters.