Sql-server – SQL Server 2012 Import Wizard appending new data only

sql serversql-server-2012ssisssis-2012

I am using SQL Server 2012's Import wizard to import data from DatabaseA to DatabaseB.

  • Source: DatabaseA table A
  • Destination: DatabaseB table B

I have written a source query that does the job fine, but I have only 2 options

  1. Delete destination data – and pull in everything
  2. Append data – but this creates duplicates (i.e. pulls in everything all over again)

So I wanted to try modify my source query to be something like below

select col1, col2, col3 from A where col1 not in (select col1 from 
DatabaseB.dbo.B)

In other words I want to test that the data I am pulling from DatabaseA's table A does not already exist Database B's table B where col1 is a unique value.

I would like to be able to do this over and over again, only ever pulling across the delta data

Whenever I try this I get an error stating

Deferred prepare could not be completed.
Statement(s) could not be prepared.
Incorrect syntax near the keyword 'in'. (Microsoft SQL Server Native Client
11.0)

Some extra things worth noting (might give some clues)

  1. DatabaseA and DatabaseB are on separate servers
  2. DatabaseA.A.col1 is an identity column
  3. DatabaseB.B is a table I created to match the columns of A, it has no
    identity column, or primary keys etc as it is only a staging table for a migration script I will run after

Best Answer

The Import/Export wizard builds out an SSIS package behind the scenes to perform EL (Extract and Load). It really doesn't have a Transform capability since it's a streamlined tool.

What you're trying to do is conditionally load data based on existence. Which is fine, that's a common pattern.

You're attempting to solve it by a non-working query.

select col1, col2, col3 from A where col1 not in (select col1 from

DatabaseB.dbo.B)

This assumes that DatabaseB is on the same instance as A. If you established a Linked Server between A and B, then your query would work - whether it performs at acceptable levels is a different matter.

select col1, col2, col3 from A where col1 not in (select col1 from

Server2.DatabaseB.dbo.B)

A more SSIS native way of doing this would be to add a Lookup between the Source and Destination that matches based on col1 but that cannot be done using the Import Export wizard. You'd have to edit the package using Visual Studio/SQL Server Data Tools (SSDT).

If you absolutely cannot edit the package for reasons, then you have to build 2 packages. The first will export all the Col1 IDs from Destination back to Source. You can then modify the source query to reference the local cache of ids instead of trying to cross servers.

Good all purpose read -> Stairway to Integration Services I think step 3 is an incremental load which is what you're looking for

Related Question