SQL Server 2012 Import Wizard appending new data only
I am using SQL Server 2012's Import wizard to import data from DatabaseA to DatabaseB.
- Source: DatabaseA table A
- Destination: DatabaseB table B
I have written a source query that does the job fine, but I have only 2 options
- Delete destination data - and pull in everything
- Append data - but this creates duplicates (i.e. pulls in everything all over again)
So I wanted to try modify my source query to be something like below
select col1, col2, col3 from A where col1 not in (select col1 from
DatabaseB.dbo.B)
In other words I want to test that the data I am pulling from DatabaseA's table A does not already exist Database B's table B where col1 is a unique value.
I would like to be able to do this over and over again, only ever pulling across the delta data
Whenever I try this I get an error stating
Deferred prepare could not be completed.
Statement(s) could not be prepared.
Incorrect syntax near the keyword 'in'. (Microsoft SQL Server Native Client
11.0)
Some extra things worth noting (might give some clues)
- DatabaseA and DatabaseB are on separate servers
- DatabaseA.A.col1 is an identity column
- DatabaseB.B is a table I created to match the columns of A, it has no
identity column, or primary keys etc as it is only a staging table for a migration script I will run after
sql-server sql-server-2012 ssis ssis-2012
edited May 19 '18 at 17:31
MDCCL
6,75731745
asked May 19 '18 at 12:38
CrudlerCrudler
145129
bumped to the homepage by Community♦ 4 mins ago
This question has answers that may be good or bad; the system has marked it active so that they can be reviewed.
add a comment |
I am using SQL Server 2012's Import wizard to import data from DatabaseA to DatabaseB.
- Source: DatabaseA table A
- Destination: DatabaseB table B
I have written a source query that does the job fine, but I have only 2 options
- Delete destination data - and pull in everything
- Append data - but this creates duplicates (i.e. pulls in everything all over again)
So I wanted to try modify my source query to be something like below
select col1, col2, col3 from A where col1 not in (select col1 from
DatabaseB.dbo.B)
In other words I want to test that the data I am pulling from DatabaseA's table A does not already exist Database B's table B where col1 is a unique value.
I would like to be able to do this over and over again, only ever pulling across the delta data
Whenever I try this I get an error stating
Deferred prepare could not be completed.
Statement(s) could not be prepared.
Incorrect syntax near the keyword 'in'. (Microsoft SQL Server Native Client
11.0)
Some extra things worth noting (might give some clues)
- DatabaseA and DatabaseB are on separate servers
- DatabaseA.A.col1 is an identity column
- DatabaseB.B is a table I created to match the columns of A, it has no
identity column, or primary keys etc as it is only a staging table for a migration script I will run after
sql-server sql-server-2012 ssis ssis-2012
edited May 19 '18 at 17:31
MDCCL
6,75731745
asked May 19 '18 at 12:38
CrudlerCrudler
145129
bumped to the homepage by Community♦ 4 mins ago
This question has answers that may be good or bad; the system has marked it active so that they can be reviewed.
Do you have a linked server defined to DatabaseB from A?
– billinkc
May 21 '18 at 14:08
No, it is not a linked server
– Crudler
May 21 '18 at 14:33
add a comment |
I am using SQL Server 2012's Import wizard to import data from DatabaseA to DatabaseB.
- Source: DatabaseA table A
- Destination: DatabaseB table B
I have written a source query that does the job fine, but I have only 2 options
- Delete destination data - and pull in everything
- Append data - but this creates duplicates (i.e. pulls in everything all over again)
So I wanted to try modify my source query to be something like below
select col1, col2, col3 from A where col1 not in (select col1 from
DatabaseB.dbo.B)
In other words I want to test that the data I am pulling from DatabaseA's table A does not already exist Database B's table B where col1 is a unique value.
I would like to be able to do this over and over again, only ever pulling across the delta data
Whenever I try this I get an error stating
Deferred prepare could not be completed.
Statement(s) could not be prepared.
Incorrect syntax near the keyword 'in'. (Microsoft SQL Server Native Client
11.0)
Some extra things worth noting (might give some clues)
- DatabaseA and DatabaseB are on separate servers
- DatabaseA.A.col1 is an identity column
- DatabaseB.B is a table I created to match the columns of A, it has no
identity column, or primary keys etc as it is only a staging table for a migration script I will run after
sql-server sql-server-2012 ssis ssis-2012
edited May 19 '18 at 17:31
MDCCL
6,75731745
asked May 19 '18 at 12:38
CrudlerCrudler
145129
I am using SQL Server 2012's Import wizard to import data from DatabaseA to DatabaseB.
- Source: DatabaseA table A
- Destination: DatabaseB table B
I have written a source query that does the job fine, but I have only 2 options
- Delete destination data - and pull in everything
- Append data - but this creates duplicates (i.e. pulls in everything all over again)
So I wanted to try modify my source query to be something like below
select col1, col2, col3 from A where col1 not in (select col1 from
DatabaseB.dbo.B)
In other words I want to test that the data I am pulling from DatabaseA's table A does not already exist Database B's table B where col1 is a unique value.
I would like to be able to do this over and over again, only ever pulling across the delta data
Whenever I try this I get an error stating
Deferred prepare could not be completed.
Statement(s) could not be prepared.
Incorrect syntax near the keyword 'in'. (Microsoft SQL Server Native Client
11.0)
Some extra things worth noting (might give some clues)
- DatabaseA and DatabaseB are on separate servers
- DatabaseA.A.col1 is an identity column
- DatabaseB.B is a table I created to match the columns of A, it has no
identity column, or primary keys etc as it is only a staging table for a migration script I will run after
sql-server sql-server-2012 ssis ssis-2012
sql-server sql-server-2012 ssis ssis-2012
edited May 19 '18 at 17:31
MDCCL
6,75731745
asked May 19 '18 at 12:38
CrudlerCrudler
145129
edited May 19 '18 at 17:31
MDCCL
6,75731745
asked May 19 '18 at 12:38
CrudlerCrudler
145129
edited May 19 '18 at 17:31
MDCCL
6,75731745
edited May 19 '18 at 17:31
MDCCL
6,75731745
edited May 19 '18 at 17:31
MDCCL
6,75731745
6,75731745
asked May 19 '18 at 12:38
CrudlerCrudler
145129
asked May 19 '18 at 12:38
CrudlerCrudler
145129
asked May 19 '18 at 12:38
CrudlerCrudler
145129
145129
bumped to the homepage by Community♦ 4 mins ago
This question has answers that may be good or bad; the system has marked it active so that they can be reviewed.
bumped to the homepage by Community♦ 4 mins ago
This question has answers that may be good or bad; the system has marked it active so that they can be reviewed.
Do you have a linked server defined to DatabaseB from A?
– billinkc
May 21 '18 at 14:08
No, it is not a linked server
– Crudler
May 21 '18 at 14:33
add a comment |
Do you have a linked server defined to DatabaseB from A?
– billinkc
May 21 '18 at 14:08
No, it is not a linked server
– Crudler
May 21 '18 at 14:33
Do you have a linked server defined to DatabaseB from A?
– billinkc
May 21 '18 at 14:08
Do you have a linked server defined to DatabaseB from A?
– billinkc
May 21 '18 at 14:08
No, it is not a linked server
– Crudler
May 21 '18 at 14:33
No, it is not a linked server
– Crudler
May 21 '18 at 14:33
add a comment |
1 Answer
1
active
oldest
votes
The Import/Export wizard builds out an SSIS package behind the scenes to perform EL (Extract and Load). It really doesn't have a Transform capability since it's a streamlined tool.
What you're trying to do is conditionally load data based on existence. Which is fine, that's a common pattern.
You're attempting to solve it by a non-working query.
select col1, col2, col3 from A where col1 not in (select col1 from
DatabaseB.dbo.B)
This assumes that DatabaseB is on the same instance as A. If you established a Linked Server between A and B, then your query would work - whether it performs at acceptable levels is a different matter.
select col1, col2, col3 from A where col1 not in (select col1 from
Server2.DatabaseB.dbo.B)
A more SSIS native way of doing this would be to add a Lookup between the Source and Destination that matches based on col1 but that cannot be done using the Import Export wizard. You'd have to edit the package using Visual Studio/SQL Server Data Tools (SSDT).
If you absolutely cannot edit the package for reasons, then you have to build 2 packages. The first will export all the Col1 IDs from Destination back to Source. You can then modify the source query to reference the local cache of ids instead of trying to cross servers.
Good all purpose read -> Stairway to Integration Services I think step 3 is an incremental load which is what you're looking for
answered May 21 '18 at 15:43
billinkcbillinkc
13.1k33775
add a comment |
Your Answer
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "182"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: false,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: null,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fdba.stackexchange.com%2fquestions%2f207276%2fsql-server-2012-import-wizard-appending-new-data-only%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
The Import/Export wizard builds out an SSIS package behind the scenes to perform EL (Extract and Load). It really doesn't have a Transform capability since it's a streamlined tool.
What you're trying to do is conditionally load data based on existence. Which is fine, that's a common pattern.
You're attempting to solve it by a non-working query.
select col1, col2, col3 from A where col1 not in (select col1 from
DatabaseB.dbo.B)
This assumes that DatabaseB is on the same instance as A. If you established a Linked Server between A and B, then your query would work - whether it performs at acceptable levels is a different matter.
select col1, col2, col3 from A where col1 not in (select col1 from
Server2.DatabaseB.dbo.B)
A more SSIS native way of doing this would be to add a Lookup between the Source and Destination that matches based on col1 but that cannot be done using the Import Export wizard. You'd have to edit the package using Visual Studio/SQL Server Data Tools (SSDT).
If you absolutely cannot edit the package for reasons, then you have to build 2 packages. The first will export all the Col1 IDs from Destination back to Source. You can then modify the source query to reference the local cache of ids instead of trying to cross servers.
Good all purpose read -> Stairway to Integration Services I think step 3 is an incremental load which is what you're looking for
answered May 21 '18 at 15:43
billinkcbillinkc
13.1k33775
add a comment |
The Import/Export wizard builds out an SSIS package behind the scenes to perform EL (Extract and Load). It really doesn't have a Transform capability since it's a streamlined tool.
What you're trying to do is conditionally load data based on existence. Which is fine, that's a common pattern.
You're attempting to solve it by a non-working query.
select col1, col2, col3 from A where col1 not in (select col1 from
DatabaseB.dbo.B)
This assumes that DatabaseB is on the same instance as A. If you established a Linked Server between A and B, then your query would work - whether it performs at acceptable levels is a different matter.
select col1, col2, col3 from A where col1 not in (select col1 from
Server2.DatabaseB.dbo.B)
A more SSIS native way of doing this would be to add a Lookup between the Source and Destination that matches based on col1 but that cannot be done using the Import Export wizard. You'd have to edit the package using Visual Studio/SQL Server Data Tools (SSDT).
If you absolutely cannot edit the package for reasons, then you have to build 2 packages. The first will export all the Col1 IDs from Destination back to Source. You can then modify the source query to reference the local cache of ids instead of trying to cross servers.
Good all purpose read -> Stairway to Integration Services I think step 3 is an incremental load which is what you're looking for
answered May 21 '18 at 15:43
billinkcbillinkc
13.1k33775
add a comment |
The Import/Export wizard builds out an SSIS package behind the scenes to perform EL (Extract and Load). It really doesn't have a Transform capability since it's a streamlined tool.
What you're trying to do is conditionally load data based on existence. Which is fine, that's a common pattern.
You're attempting to solve it by a non-working query.
select col1, col2, col3 from A where col1 not in (select col1 from
DatabaseB.dbo.B)
This assumes that DatabaseB is on the same instance as A. If you established a Linked Server between A and B, then your query would work - whether it performs at acceptable levels is a different matter.
select col1, col2, col3 from A where col1 not in (select col1 from
Server2.DatabaseB.dbo.B)
A more SSIS native way of doing this would be to add a Lookup between the Source and Destination that matches based on col1 but that cannot be done using the Import Export wizard. You'd have to edit the package using Visual Studio/SQL Server Data Tools (SSDT).
If you absolutely cannot edit the package for reasons, then you have to build 2 packages. The first will export all the Col1 IDs from Destination back to Source. You can then modify the source query to reference the local cache of ids instead of trying to cross servers.
Good all purpose read -> Stairway to Integration Services I think step 3 is an incremental load which is what you're looking for
answered May 21 '18 at 15:43
billinkcbillinkc
13.1k33775
The Import/Export wizard builds out an SSIS package behind the scenes to perform EL (Extract and Load). It really doesn't have a Transform capability since it's a streamlined tool.
What you're trying to do is conditionally load data based on existence. Which is fine, that's a common pattern.
You're attempting to solve it by a non-working query.
select col1, col2, col3 from A where col1 not in (select col1 from
DatabaseB.dbo.B)
This assumes that DatabaseB is on the same instance as A. If you established a Linked Server between A and B, then your query would work - whether it performs at acceptable levels is a different matter.
select col1, col2, col3 from A where col1 not in (select col1 from
Server2.DatabaseB.dbo.B)
A more SSIS native way of doing this would be to add a Lookup between the Source and Destination that matches based on col1 but that cannot be done using the Import Export wizard. You'd have to edit the package using Visual Studio/SQL Server Data Tools (SSDT).
If you absolutely cannot edit the package for reasons, then you have to build 2 packages. The first will export all the Col1 IDs from Destination back to Source. You can then modify the source query to reference the local cache of ids instead of trying to cross servers.
Good all purpose read -> Stairway to Integration Services I think step 3 is an incremental load which is what you're looking for
answered May 21 '18 at 15:43
billinkcbillinkc
13.1k33775
answered May 21 '18 at 15:43
billinkcbillinkc
13.1k33775
answered May 21 '18 at 15:43
billinkcbillinkc
13.1k33775
answered May 21 '18 at 15:43
billinkcbillinkc
13.1k33775
13.1k33775
add a comment |
add a comment |
Thanks for contributing an answer to Database Administrators Stack Exchange!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fdba.stackexchange.com%2fquestions%2f207276%2fsql-server-2012-import-wizard-appending-new-data-only%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Do you have a linked server defined to DatabaseB from A?
– billinkc
May 21 '18 at 14:08
No, it is not a linked server
– Crudler
May 21 '18 at 14:33