I am new to using BCP but I am trying to extract data generated via a stored proc. This is a continuation of my original question which was a stored procedure with temp tables. This procedure has grown a bit but the core is the same, generate temp tables, run dynamic SQL to populate the tables and then return data.
I realized quickly that bcp does not like temp tables via the error:
SQLState = S0002, NativeError = 208
Error = [Microsoft][SQL Server Native Client 10.0][SQL Server]Invalid Object name '#SurveyData
So I changed my procedure to remove the #temp tables and use actual tables. But now I get an error:
SQLState = 37000, NativeError = 156
Error = [Microsoft][SQL Server Native Client 10.0][SQL Server]Incorrect syntax near the keyword 'as'
I believe the issue is with the dynamic SQL. There is an as
in the statement.
SET @sql = 'INSERT INTO cl.Results ' +
' SELECT s.SurveyId
, s.InstanceId
, CASE WHEN columnName = ''' + @ColumnName + ''' THEN REPLACE(columnName, ''Q_'', '''') ELSE '''' END
, Cast(s.' + @ColumnName + ' as varchar(1000))
FROM cl.SurveyData t
INNER JOIN dbo.' + @TableName + ' s' +
' ON REPLACE(t.tableName, ''Library_'', '''') = s.SurveyID ' +
' WHERE t.columnName = ''' + @ColumnName + ''''
exec(@sql)
The code that I am using in BCP to execute is:
bcp "exec dbo.getresults" queryout "c:\temp\mytext.txt" -S <myserver> -T -c -t^|
I am open to any suggestions about how to get this data extracted, even if I have to rewrite the stored procedure.
Can I not use BCP to execute a stored proc with dynamic SQL? If not, how else should I consider extracting this data? SSIS does not seem to like either the temp tables or dynamic SQL either.
Edit: using print(@sql)
INSERT INTO cl.Results
SELECT s.SurveyId
, s.InstanceId
, CASE WHEN columnName = 'Q_1' THEN REPLACE(columnName, 'Q_', '') ELSE '' END
, Cast(s.Q_1 as varchar(1000))
FROM cl.SurveyData t
INNER JOIN dbo.Library_1 s
ON REPLACE(t.tableName, 'Library_', '') = s.SurveyID WHERE t.columnName = 'Q_1'
INSERT INTO cl.Results
SELECT s.SurveyId
, s.InstanceId
, CASE WHEN columnName = 'Q_2' THEN REPLACE(columnName, 'Q_', '') ELSE '' END
, Cast(s.Q_2 as varchar(1000))
FROM cl.SurveyData t
INNER JOIN dbo.Library_2 s
ON REPLACE(t.tableName, 'Library_', '') = s.SurveyID WHERE t.columnName = 'Q_2'
INSERT INTO cl.Results
SELECT s.SurveyId
, s.InstanceId
, CASE WHEN columnName = 'Q_3' THEN REPLACE(columnName, 'Q_', '') ELSE '' END
, Cast(s.Q_3 as varchar(1000))
FROM cl.SurveyData t
INNER JOIN dbo.Library_2 s
ON REPLACE(t.tableName, 'Library_', '') = s.SurveyID
WHERE t.columnName = 'Q_3'
INSERT INTO cl.Results
SELECT s.SurveyId
, s.InstanceId
, CASE WHEN columnName = 'Q_4' THEN REPLACE(columnName, 'Q_', '') ELSE '' END
, Cast(s.Q_4 as varchar(1000))
FROM cl.SurveyData t
INNER JOIN dbo.Library_2 s
ON REPLACE(t.tableName, 'Library_', '') = s.SurveyID
WHERE t.columnName = 'Q_4'
Edit: Here is the full script that I am running without any temp tables:
truncate table cl.Results
truncate table cl.SurveyData
-- insert the survey table structures for use
insert into cl.SurveyData (tableName, columnName, columnId, rownum)
select tables1.name, cols1.name, column_id, ROW_NUMBER() over(order by tables1.name, column_id)
from sys.all_columns cols1
inner join
(
SELECT *
FROM sys.all_objects
WHERE type = 'U'
AND upper(name) like 'LIBRARY%'
) Tables1
ON cols1.object_id = tables1.object_id
WHERE cols1.name Like 'Q_%'
ORDER BY tables1.name, column_id;
declare @sql varchar(max) = '';
declare @RowCount int = 1;
declare @TotalRecords int = (SELECT COUNT(*) FROM cl.SurveyData);
Declare @TableName varchar(50) = '';
Declare @ColumnName varchar(50) = '';
WHILE @RowCount <= @TotalRecords
BEGIN
SELECT @TableName = tableName, @ColumnName = columnName
FROM cl.SurveyData
WHERE @RowCount = rownum
SET @sql = 'INSERT INTO cl.Results ' +
' SELECT s.SurveyId
, s.InstanceId
, CASE WHEN columnName = ''' + @ColumnName + ''' THEN REPLACE(columnName, ''Q_'', '''') ELSE '''' END
, Cast(s.' + @ColumnName + ' as varchar(1000))
FROM cl.SurveyData t
INNER JOIN dbo.' + @TableName + ' s' +
' ON REPLACE(t.tableName, ''Library_'', '''') = s.SurveyID ' +
' WHERE t.columnName = ''' + @ColumnName + ''''
exec(@sql)
SET @RowCount = @RowCount + 1
END
SELECT r.SurveyId,
r.InstanceId,
CASE
when chat.DataName = 'BRKR_ACCT_ID' and chat.DataValue is not null then chat.DataValue
when email.PropName = 'ACCT_ID' and email.PropValue is not null then email.PropValue
END Account_ID,
CASE
when chat.DataName = 'BRKR_CUST_ID' and chat.DataValue is not null then chat.DataValue
when email.PropName = 'CUST_ID' and email.PropValue is not null then email.PropValue
END Cust_ID,
case
when chat.LoginName is null
then email.LoginName
else chat.LoginName
end SchwabId,
case
when chat.CustEmail is null
then email.ReplyTo
else chat.CustEmail
end CustomerEmail,
sp.DateSent,
CONVERT(varchar, DATEADD(ms, DATEDIFF(SECOND, sp.datesent, sp.datecompleted) * 1000, 0), 108) ResponseTime,
r.QuestionNumber,
r.Response
FROM cl.Results r
INNER JOIN dbo.SurveyParam sp
ON r.InstanceId = sp.InstanceID
AND r.SurveyId = sp.SurveyID
-- chat surveys uses Param4
LEFT JOIN
(
SELECT si.SessionID,
si.CustEmail,
u.LoginName,
ltrim(rtrim(sie.DataValue)) DataValue,
ltrim(rtrim(sie.DataName)) DataName
FROM dbo.SessionInfo si
LEFT JOIN dbo.Users u
ON si.LastAgent = u.id
LEFT JOIN dbo.SessionInfoExternals sie
ON si.SessionID = sie.SessionID
WHERE sie.DataName IN ('BRKR_ACCT_ID', 'BRKR_CUST_ID')
) chat
ON sp.Param4 = chat.SessionID
-- email surveys uses Param3
LEFT JOIN
(
SELECT mm.MsgID,
mr.ReplyTo,
u.LoginName,
ltrim(rtrim(mie.PropValue)) PropValue,
ltrim(rtrim(mie.PropName)) PropName
FROM dbo.MailReply mr
INNER JOIN dbo.MailMessage mm
ON mr.ReplyToID = mm.MsgID
LEFT JOIN dbo.Users u
on mr.AgentID = u.ID
LEFT JOIN dbo.MsgInfoExternals mie
ON mm.IncidentID = mie.Instance
WHERE mie.PropName IN ('CUST_ID', 'ACCT_ID')
) email
ON sp.Param3 = email.MsgID
WHERE sp.DateCompleted is not null
Best Answer
I believe that BCP does a two phase process. The first is done with a format only where the query isn't actually executed but it is parsed so that BCP can figure out the columns that will be returned and in the second phase it actually runs the query. You are failing on the first phase as the temp table doesn't exist so it can't be used for the format only option.
Basically the use of the temp table isn't going to work.