SQLite3 – remove duplicates by multiple columns

duplicationrowsqlite

I'm looking for the most efficient way to remove duplicates from a table where id's are unique but there are equal rows when you check other columns.

Example:

id   name   age   x
1   peter   25    II
2   peter   25    II

The table has tens of thousands of rows.

Best Answer

Supposing you have no foreign keys referencing that table, you could do something like

create table some_table as
select min(id), name, age, x
from t
group by name, age, x

then you can drop the old table, rename the new table so that it has the same name as the old one before, and create indexes and other things you need on that table.

(Verification) Rows that will be deleted

;with cte as
(
    select
        *,
        row_num = 
            row_number() over (partition by road order by ID)
    from dbo.YourTable
)
select *
from cte
where row_num > 1;

(Verification) Rows that will not be deleted

;with cte as
(
    select
        *,
        row_num = 
            row_number() over (partition by road order by ID)
    from dbo.YourTable
)
select *
from cte
where row_num = 1;

Data Delete

Note: the below query, when modified to fit your environment will actually delete data. Therefore, you should ensure that you really want to delete the data, and also have a backup of the data/database prior to performing any data modification.

;with cte as
(
    select
        *,
        row_num = 
            row_number() over (partition by road order by ID)
    from dbo.YourTable
)
delete from cte
where row_num > 1;

Postgresql – How to remove duplicates and keep the one with the most additional information

Your question is a bit light on definitions. Assuming:

You define the "least amount of information" with how many of the relevant columns are NULL.
Primary key is adr_id.
Duplicates are marked with a column dupe_id to indicate groups of duplicates.

Since it's also vague what to do exactly, I create a separate table with the dupe ranking:

CRATE TABLE adr_dupe_rank AS
SELECT adr_id, dupe_id
     , rank() OVER (PARTITION BY dupe_id
                    ORDER BY (nr_of_beds  IS NULL)::int
                           + (nr_of_baths IS NULL)::int
                        -- + (...)::int  -- more terms
                   ) AS rnk
FROM   address;

false translate to 0, true translates to 1. So rows with the fewest empty columns are ranked first. Master rows end up with rnk = 1. Dupes get higher rnk numbers.

The window function rank() assigns 1 to all rows sharing the lowest score per dupe_id. Add enough columns or other terms to break ties and get one winner per dupe_id. Or deal with multiple winners separately.

You can then do as you please. To just delete dupes:

DELETE FROM address a
USING  adr_dupe_rank d
WHERE  a.adr_id = d.adr_id
AND    d.rnk > 1;

Alternatively you can use the above query as derived table in the DELETE directly.

Best Answer

Related Solutions

Sql-server – How to delete unique rows from a table with duplicate keys

(Verification) Rows that will be deleted

(Verification) Rows that will not be deleted

Data Delete

Postgresql – How to remove duplicates and keep the one with the most additional information

Related Question