The flow
Starts it 1Logic 18Service 11
The steps in the order they run, from what starts it.
- 1Index Dataset from HuggingFaceManual triggerStarts it
- 2Check Collection Exists1QdrantService
- 3Get Dataset SplitsHttp RequestService
- 4Check Collection ExistsQdrantService
- 5If1IfLogic
- 6Split Them All OutSplit OutLogic
- 7IfIfLogic
- 8Create Collection1QdrantService
- 9Get Dataset Rows (Pagination)Http RequestService
- 10Create CollectionQdrantService
- 11Divide Per RowSplit OutLogic
- 12Restructure for DeduplicatingSetLogic
- 13Deduplicate TextsSummarizeService
- 14Restructure for BatchingSetLogic
- 15LimitLimitLogic
- 16MergeMergeLogic
- 17Calculate #words in Each TextSetLogic
- 18Loop Over BatchesSplit In BatchesLogic
- 19Loop Over Batches1Split In BatchesLogic
- 20Sum them UpSummarizeService
- 21Aggregate a BatchAggregateLogic
- 22Edit FieldsSetLogic
- 23Merge1MergeLogic
- 24Aggregate a Batch to EmbedAggregateLogic
- 25Get the Average Text LengthSetLogic
- 26Upsert PointsQdrantService
- 27Aggregate a Batch to UpsertAggregateLogic
- 28Get OpenAI embeddingsHttp RequestService
- 29Upsert Points1QdrantService
- 30Split OutSplit OutLogic