You've come across a fairly well-known problem. When SSIS is trying to import the data, it provides locale-aware conversions and the format YYYYMMDD doesn't convert nicely. You can use the "import as string and convert to actual datetime" approach you've outlined in your solution but that's slower and consumes more resources than using the native approach of telling SSIS to quit being so damn smart.
Right click on your flat file source and select the Show Advanced Editor. In the Input and Output Properties, expand Output Columns, find your column (BirthDate) and change the FastParse
property from False to True
.
With only that change, the package will execute successfully
Also, an excellent answer from SO on the same issue
Two options come to mind, both are features of the can't-say-enough-good-things-about-this-free-tool BIDS Helper.
Create Fixed Width Columns
The first is the Create Fixed Width Columns feature. It's as simple as Create a Flat File Manager pointing at the correct file and define 1 column.
Once that's done, then you right click back on the Connection Manager and use the context sensitive feature of "Create Fixed Width Columns..."
In the next window, you paste the tab delimited list (Excel) into the editor
Oh yeah, it's that's simple. If you need to do some tweaking for column types and such, then you go in and change it as normal with edit.
Biml
I like the Business Intelligence Markup Language for my SSIS development. There's lots of benefit to it but at a most basic level, use it for things like godawful flat files, especially when they provide no header rows (I'm looking at you, you mainframes).
It has often been my experience when dealing with legacy systems, that they have their file definitions in COBOL copybooks. Those devs will send me Excel files with the column layout like
DATABASE FIELD NAME START LOC LENGTH
SEND.DT 1 STRING(08)
SEND.TIME 9 STRING(08)
DT 17 STRING(08)
TERM 25 STRING(04)
%ZONE 29 STRING(01)
It ain't fancy, but Excel formulas made short work of it. So I'd create 4 new columns based on the above.
Sanitized Name (removed invalid characters for column names)
=SUBSTITUTE(SUBSTITUTE(B3, ".", "_"), "%", "PCT_")
Length (extracted the numbers, note this still has a leading 0 but it doesn't hurt)
=MID(E3,LEN("STRING(")+1,LEN(E3)-LEN("STRING(")-1)
XML (this builds out the actual XML for a flat file definition)
=CONCATENATE("<Column Name=""",F3,""" Length=""",G3,""" DataType=""AnsiString"" ColumnType=""FixedWidth"" CodePage=""1252"" />")
DDL (Might as well build the target table while I'm at it)
=CONCATENATE(", ", F3, " varchar(", G3, ")")
The first column of the DDL gets the lead comma dropped and then wrapped with CREATE TABLE dbo.FOO()
Ultimately, that XML results in something that gets slapped into a FlatFileFormat
tag and then it's a simple matter of creating the Connections collection and finally the package itself. Looks a lot but it's actually pretty simple, once you have a pattern.
<Biml xmlns="http://schemas.varigence.com/biml.xsd">
<FileFormats>
<FlatFileFormat
Name="FFF Pickup"
CodePage="1252"
RowDelimiter="CRLF"
IsUnicode="false"
FlatFileType="RaggedRight">
<Columns>
<Column Name="SEND_DT" Length="08" DataType="AnsiString" ColumnType="FixedWidth" CodePage="1252" />
<Column Name="SEND_TIME" Length="08" DataType="AnsiString" ColumnType="FixedWidth" CodePage="1252" />
<Column Name="DT" Length="08" DataType="AnsiString" ColumnType="FixedWidth" CodePage="1252" />
<Column Name="TERM" Length="04" DataType="AnsiString" ColumnType="FixedWidth" CodePage="1252" />
<!--
ad nauseum
-->
<Column Name="RPRTD_PU_PCS" Length="5" DataType="AnsiString" ColumnType="FixedWidth" CodePage="1252" />
</Columns>
</FlatFileFormat>
</FileFormats>
<Connections>
<FlatFileConnection
Name="FF Pickup"
FileFormat="FFF Pickup"
FilePath="C:\ssisdata\Operations\Input\Pickup Report Pickups.txt"
CreateInProject="false"
/>
</Connections>
<Packages>
<Package Name="PickupLoad" ConstraintMode="Linear" ProtectionLevel="DontSaveSensitive">
<Tasks>
<Dataflow Name="DFT Load Pickups" >
<Transformations>
<FlatFileSource
Name="OLE_SRC Pickup"
ConnectionName="FF Pickup"
RetainNulls="true">
</FlatFileSource>
</Transformations>
</Dataflow>
</Tasks>
</Package>
</Packages>
Oh and a similar question from SO with a disturbingly similar answer. Wish I had looked at Skinner's comments before I rewrote this answer.
Best Answer
Casting a string of
000000000000017943
to decimal(18,2) is going to result in a value of17943
The source doesn't have a decimal place, so why would the cast know you need a decimal there?Also, given your source data, your substring formula is off. It's resulting in a source string of
R00000000000001794
That said, I take one of 2 approaches. Either I use a string operation to insert the decimal place or I use math to divide by 100. If I'm already dealing with strings, I usually stick with that approach as I hate to keep casting values about.
Off to lunch, but here is the Biml I used to generate the package. Grab bidshelper and use that to generate the same SSIS package as me.