Automation library · Documents and files

Evaluation metric example: RAG document relevance with OpenAI

Started by someone opening a chat with it, and works across Google Sheets and in-memory vectors. It puts OpenAI behind a retrieval step, so answers are drawn from documents you supply rather than from the model alone. 19 steps: a working afternoon to understand properly.

MediumCalls a model: OpenAIRetrieval

19 stepsMedium: a working afternoon
0Branches: places the flow splits
0Loops or waits
2Services it connects to

The flow

Starts it 3Model 9Logic 4Service 3

The steps in the order they run, from what starts it.

  1. 1When fetching a dataset rowEvaluation triggerStarts it
  2. 2When chat message receivedChat triggerStarts it
  3. 3Embeddings OpenAIEmbeddings Open AiModel
  4. 4Recursive Character Text SplitterText Splitter Recursive Character Text SplitterModel
  5. 5When clicking ‘Execute workflow’Manual triggerStarts it
  6. 6OpenAI Chat ModelLm Chat Open AiModel
  7. 7Embeddings OpenAI1Embeddings Open AiModel
  8. 8Match chat formatSetLogic
  9. 9AI AgentAgentModel
  10. 10Simple Vector StoreVector Store In MemoryModel
  11. 11Default Data LoaderDocument Default Data LoaderModel
  12. 12Get datasetGoogle SheetsService
  13. 13Simple Vector Store1Vector Store In MemoryModel
  14. 14Evaluating?EvaluationService
  15. 15Remove DuplicatesRemove DuplicatesLogic
  16. 16Extract documentsSetLogic
  17. 17Return chat responseNo OpLogic
  18. 18Calculate doc relevance metricOpen AiModel
  19. 19Set metricsEvaluationService

How it's built

Starts on
Someone opening a chat with it
Steps
19
Size
Medium: a working afternoon
Connects to
Google Sheets, in-memory vectors
Model
OpenAI
Retrieval
Answers are drawn from documents you supply, not from the model alone

Questions

What does it need to run?

An n8n account, plus your own Google Sheets and in-memory vectors accounts, and a key for OpenAI. You connect your own accounts after importing it; none are included.

What starts it?

It starts on someone opening a chat with it.

Does it use AI?

Yes. It calls OpenAI. Its answers draw on documents you give it, not on the model alone. Model calls are billed by the provider, not by n8n.

How long does it take to set up?

Plan an afternoon. 19 steps is enough that you should read each one before running it.

How do I import it?

Copy the workflow, open a new workflow in n8n and paste. The steps appear on the canvas, ready for your own accounts.

Before you run it

Nothing here has been tested by us. It is somebody's working automation, read and described from its own steps. Treat the connections it wants as a list of accounts it will touch, and read the 19 steps before you point it at anything that matters.