The more data companies gather on their customers, products, and code, the more duplicated information accumulates, confusing the teams who depend on it. You don't go through millions of records manually. So deduplication is now a machine learning problem. This article looks at the current state of duplicate detection in ML systems where the old techniques still work, and what embeddings and large language models have brought to the table. How Do ML Systems Detect Duplicates?It is possible for a person to look at two pictures or two strings and know if they are the same. Teaching a machine to do the same raises difficult questions. What is 'like'? There are degrees of resemblance. Is “Jon Smith, 12 Oak St.” the same customer as “Jonathan Smith, 12 Oak Street?”