Sql-server – Copy data from multiple databases to a central database for reporting and analysis

replicationreportingsql serverssis

I have more than 8 databases and all are in a SQL Server environment. I need to extract data from specific tables (more than 50 tables) of each database and dump into a central database for reporting and analysis purposes (a one way transfer).

Table structures are the same in all databases. Note that all my databases reside on the same server, except the central database, which is on a separate server. I have evaluated a couple of options. My current thinking and related questions are below:

  1. SSIS

    With SSIS I can do data transfer from the source to the destination databases; however, there is no change tracking. I have to track the changes myself and then extract and push the data manually. Apart from this I will end up creating a number of control flows and related data flows as the data sources are different which would be hard to manage in future.

  2. Replication

    Replication has change tracking, so the data transfer would be quick enough. However, I have to create those many publishers against each database. I am afraid server performance would be impacted due to this. I am also not sure which replication type to use, or if any performance issues could be handled with a better configuration. I am okay with a nightly update and data transfer need not be real time.

How should I proceed from here?

Best Answer

I have more than 8 databases and all are in a SQL Server environment. I need to extract data from specific tables (more than 50 tables) of each database and dump into a central database for reporting and analysis purposes (a one way transfer).

Table structures are the same in all databases.

REPLICATION : If the objects (tables) you are replicating are not undergoing frequent schema changes then you can use many-to-one replication using T-Rep (this is a very thorough answer that I wrote with scripts and end result), provided you can slightly modify your table schema. This will give you a near-to-realtime data transfer to your central server.

SSIS : If you dont need a near-to-realtime data transfer, then you can look into SSIS - Incremental load using Lookup transform as described in Updating Rows in Incremental Loads – Level 4 of the Stairway to Integration Services. ALso, refer to this excellent series : Stairway to Integration Services for more details. Alternatively, you can test out Change Data Capture (SSIS) (as @srutzky suggested).