embeddings

[fadil4u]

Semantic Deduplication

clusters = dt.reduce(df, action="dedup", embedding_model="text-embedding-3-small", llm=llm)

Pass merge data to Datatune primitves

mapped = dt.map(
    prompt=mapping_prompt,
    output_fields=["passenger_trend_comment"],
    input_fields=["year","month","passengers"],
    clusters=clusters,
)(llm, df)

Reduces the amount of semantically similar rows sent to the LLM thereby reducing tokens and therefore cost

Dedup

Semantic deduplicator does the following things

1. Embeds rows and saves embeddings to disk. Embeds partition by partition
2. Cluster embeddings using FAISS HNSW index (approximate nearest neighbor)
3. An additional LLM evaluation step on each cluster
4. __call__ returns clusters

[{'canonical_id': 5, 'duplicate_ids': [1, 2, 3, 4, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19]}]

when passed to datatune primitves , only the canonical row is sent to the LLM and it's output is transmitted to its duplicate rows.