How to Output More Than 256 Characters to a File in SQL Server

csvsql serversqlcmd

I am using the following command to write the output from a stored procedure to a file for further processing:

DECLARE @shellCommand VARCHAR(2000)
SET @shellCommand = 'SQLCMD -S ' + @@SERVERNAME + ' -d ' + @dbName + ' -U ' + @UserName + ' -P ' + @Password + ' -Q "' + @sqlCommand + '" -s "," -h -1 -k 1 -W -o "' + @outputfileName + '"'

@sqlCommand is set correctly, to "EXEC StoredProcedureName parameterValue" and when run independently apparently produces the correct data.

The problem I'm having is that one of the fields contains text greater than 256 characters long and is getting truncated when fed through the SQLCMD command.

On reading the documentation I see that the default column width is 256 characters. So I'm looking at using the -y or -Y option to allow more than 256 characters to be output. However, these options aren't compatible with the -W option which I was using to remove trailing spaces from the output. I started by using -y 0 (despite the performance warning – once I get the output working I can look at the performance). However, that produces some strange results.

The file consists of a header row followed by the data rows and should look something like this:

First,ABC,Number,String,ReallyLongString,...
A,0123,14.99,"Short string","Longish string",...
B,0456,23.99,"Normal string","Really, really long string that's causing problems",...

With the -W option the file format is correct but the really long string is truncated. The reason we found it was because the trailing " was missing and the file wasn't being read properly.

With -y 0 the really long string is getting output but large numbers of spaces are being added to apparently random columns. We're getting something like this:

First,ABC ,Number    [...]      ,String,ReallyLongString,...
A    ,0123,14.99     [...]      ,"Short string","Longish string",...
B    ,0456,23.99     [...]      ,"Normal string","Really, really long string that's causing problems",...

(There are many, many more spaces between "Number" and the next column as represented by the "[…]", I'm just showing a few).

There are more numeric values that have to be formatted in a similar way and it does appear that it's these that are causing the extra spaces after the value and before the next comma. We can live with a few extra spaces, but not as many as this as the resultant file is too large to be read by the target program.

The data is generated by a stored procedure that looks like this:

SELECT
    'First' AS First,
    'ABC' AS ABC,
    'Number' AS Number,
    'String' AS String,
    'ReallyLongString' AS ReallyLongString,
    ...

UNION ALL

SELECT
    'A' as First,
    Column1 as ABC,
    REPLACE(FORMAT(Column2, 'N2', 'en-GB'), ',', '') as Number, -- to get the format correct
    '"' + Column3 + '"' as String,
    '"' + Column4 + '"' as ReallyLongString,
    ....
FROM Table
WHERE <condition>

I'm thinking that the problem is the output from the stored procedure. I only added the -W option to the SQLCMD because of the extra trailing spaces being output, but looking at the stored procedure I can't work out where they are coming from. I changed the number formatting to include RTRIM:

RTRIM(REPLACE(FORMAT(Column2, 'N2', 'en-GB'), ',', ''))

but that appeared to make no difference.

Is there something I can do to the stored procedure or is there a combination of options to SQLCMD that will produce the desired output? Or am I going to have to find some other way of producing this file?

Best Answer

While trying out the various suggestions in the comments I have stumbled across a solution.

I replaced:

'"' + Column4 + '"' as ReallyLongString,

with:

'"' + REPLICATE('O', 4000) + '"' AS ReallyLongString,

and all of the text was output.

However, using a variable:

DECLARE @longText NVARCHAR(MAX)
SET @longText = REPLICATE('O', 4000);

'"' + @longText + '"' AS ReallyLongString,

didn't.

What did work was declaring the variable with a specific length:

DECLARE @longText NVARCHAR(4000)

So, as the text in this case is never going to be longer than 2000 characters (the device reading the file can't cope with text that long) I convert all the text to 2000 character long strings:

SELECT
    'A' as First,
    Column1 as ABC,
    REPLACE(FORMAT(Column2, 'N2', 'en-GB'), ',', '') as Number, -- to get the format correct
    '"' + CAST(Column3 as NVARCHAR(2000)) + '"' as String,
    '"' + CAST(Column4 as NVARCHAR(2000)) + '"' as ReallyLongString,
    ....
FROM Table
WHERE <condition>

This is the how. I still don't really know the why.

Related Question