DEV Community
Follow
How to Find Duplicate Rows in SQL (and Decide What Counts as One)
This article explains how to identify and handle duplicate rows in a database table using SQL queries and a clear three-stage process. Before querying, it's crucial to define what constitutes a duplicate for a specific table. The first step involves a quick test: comparing the total row count with the count of distinct key values to determine if duplicates exist. If they do, the next stage is to group rows by the defined duplicate criteria and use the HAVING clause to identify groups with more than one row. To view all the actual duplicate rows, a subquery using the IN operator can retrieve all rows matching the identified duplicate keys. The article then introduces the ROW_NUMBER window function to assign a unique number to each row within a partition, allowing for the designation of a keeper and extras. It emphasizes marking duplicates for potential later deletion rather than immediate deletion to preserve data and allow for review. Finally, it provides a DELETE query pattern for cases where removing exact duplicate rows is necessary, stressing the importance of running a SELECT statement first to preview the rows that would be deleted. The core idea is that grouping by specific columns defines what a duplicate means, making them visible and manageable.