You don't have sufficient disk space to process everything you're trying to do concurrently. Data flow operations that spill to disk and temp files that get created are just some of the many reasons you might run out of space.
You should be able to run Perfmon during one of these runs to see how disk space and activity play out.
It's next to impossible for us to recommend anything without intimate knowledge of your system. The fact you're running several processes concurrently - where the actual speed and state of each process can vary widely from run to run - makes it impossible to know when or why the processes are encountering the "perfect storm" of starving each other of disk space.
You will have to understand the resource usage of each one of your processes independently then you'll be able to know if you should even try to run them concurrently.
I wouldn't want to have 200 data flows in a single package. The time it'd take just to open up and validate would make you old before your time.
EzAPI is fun but if you're new to .NET and SSIS, oh hell no, you don't want that. I think you'll spend far more time learning about the SSIS object model and possibly dealing with COM than actually getting work done.
Since I'm lazy, I'll plug BIML as a free option you didn't list. From an answer on SO https://stackoverflow.com/questions/13809491/generating-several-similar-ssis-packages-file-data-source-to-db/13809604#13809604
- Biml is an interesting beast. Varigence will be happy to sell you a license to Mist but it's not needed. All you would need is BIDSHelper and then browse through BimlScript and look for a recipe that approximates your needs. Once you have that, click the context sensitive menu button in BIDSHelper and whoosh, it generates packages.
I think it might be an approach for you as well. You define your BIML that describes how your packages should behave and then generate them. In the scenario you describe where you make a change and have to fix N packages, nope, you fix your definition of the problem and regenerate packages.
Or if you've gained sufficient familiarity with the framework then use something like EzAPI to go and fix all the broken stuff. Heck, since you've tagged this as 2005, you could also give PacMan a try if you're in need of making mass modifications to existing packages.
SSIS Design considerations
Generally speaking, I try to make my packages focus on solving a single task (load sales data). If that requires 2 data flows, so be it. What I hate inheriting is a package from the import export wizard with many un-related data flows in a single package. Decompose them into something that solves a very specific problem. It makes future enhancements less risky as the surface area is reduced. An additional benefit is that I can be working on loading DimProducts while my minion is dealing with loading SnowflakeFromHell package.
Then use master package(s) to orchestrate the child work flows. I know you're on 2005 but SQL Server 2012's release of SSIS is the cat's pajamas. I love the project deployment model and the tight integration it allows between packages.
TSQL vs SSIS (my story)
As for the pure TSQL approach, in a previous job, they used a 73 step job for replicating all of their Informix data into SQL Server. It generally took about 9 hours but could stretch to 12 or so. After they bought a new SAN, it went down to about 7+ hours. Same logical process, rewritten in SSIS was a consistent sub 2 hours. Easily the biggest factor in driving down that time was the "free" parallelization we got using SSIS. The Agent job ran all of those tasks in serial. The master package basically divided the tables into processing units (5 parallel sets of serialized tasks of "run replicate table 1", table 2, etc) where I tried to divide the buckets into quasi equal sized units of work. This allowed the 60 or so lookup reference tables to get populated quickly and then the processing slowed down as it got into the "real" tables.
Other pluses for me using SSIS is that I get "free" configuration, logging and access to the .NET libraries for square data I need to bash into a round hole. I think it can be easier to maintain (pass off maintenance) an SSIS package than a pure TSQL approach by virtue of the graphical nature of the beast.
As always, your mileage may vary.
Best Answer
In addition to adding the new column to the variable User::vSQLBase, you need to open the Data Flow for the staging table being loaded. Edit the source, and click on Columns, verify you see the new column, click OK. Then edit the Destination, and click on Mappings - and make sure that the new column from the source is pointing to the right column in the staging table. That will do the new column mapping the consultant was talking about. You'll need to make sure the mapping is updated for whatever moves data from staging to the destination table too. Then save and deploy again.
An incremental load only populates new or changed data since the last successful run. Do you care about the new column being populated in the destination only going forward? Or is it important for that data to exist in records previously imported as well? For an accounting system, I'd guess you'd want the new column populated for all the records in the table. In that case, I agree with your consultant that a full load is necessary.