I am currently designing a SSIS package to integrate data into a data warehouse fact table. This fact table has about 70 columns among which 17 are foreign keys for dimension tables.
To insert data in that table, I have to make several transformations and lookups. Given the fact that the lookups I have to make are a little complicated, I have about 70 tasks in my Data Flow.
I know it's a lot, but I can't find a way to make it simpler. It seems I really need all these tasks.
Now, the problem is that every new action I try to make on the package takes a lot of time. At design time, everything is very slow. My processor is eavily loaded each time I change a single setting in one of the tasks, and executing the package in debug mode takes for ages. If I take a look at the size of my package file on disk, it's more than 3MB.
Hence my question : Are there any limitations in terms of number of columns or number of tasks that can be processed within a Data Flow ?
If not, then do you have any idea why it's so slow ?
Thanks in advance for any answer.
Two things. One, the XML on your package has to be extremely large and cumbersome for the engine to work with. I would imagine this is one source of your slowness. If you can break your package up into smaller packages, that would be much better and likely easier to support.
Two, you might have some success in working offline via the "Work Offline" switch under the SSIS menu in BIDS. That is, perhaps some of the slowness is in validating your data flows against the connections.
No comments:
Post a Comment