ETL Logging vs Audit Dimension

View previous topic View next topic Go down

ETL Logging vs Audit Dimension

Post  arowshan on Tue Mar 06, 2012 4:41 pm

I have been reading and playing around with the sample SSIS packages in the Microsoft Data Warehouse Toolkit and was hoping to get some other opinions on this:

The Audit Dimension that they propose in the book contains the package logging as well which means they are adding logging information such as package start and end time in the Audit Dimension both for Extract and Load pacakges. I was thinking ETL execution and error logging should be in the back room not part of the model.

arowshan

Posts : 23
Join date : 2011-10-18
Location : Vancouver, Canada

View user profile

Back to top Go down

Re: ETL Logging vs Audit Dimension

Post  ngalemmo on Tue Mar 06, 2012 5:15 pm

Well, it has to be somewhere. It really all depends on wither there are requirements for users to see it. You can go either way with it.
avatar
ngalemmo

Posts : 3000
Join date : 2009-05-15
Location : Los Angeles

View user profile http://aginity.com

Back to top Go down

Re: ETL Logging vs Audit Dimension

Post  bciampa on Sat Mar 10, 2012 5:23 pm

Arowshan,

I have not read the book that you reference but I suppose that package logging (if I understand how you are using the term) may have a different purpose from error logging. In the environment in which I work, we have a table in our presentation area that logs the last successful refresh time of an ETL job. When reports are written from that star, the report writer can display that date and time so that users will know that they are looking at salary data (for example) that is current as of that date and time. If an ETL job "bombed" at some point the fact that the ETL job ended in an error (but not the reason) is also logged. Logic is written into the report to alert the user that the report is unavailable and we are working on the problem. The reason for the error, however, is not made available to the user. That is stored in the back room for the ETL developers to see.

Thanks,
Brian

bciampa

Posts : 8
Join date : 2012-02-24

View user profile http://valuabledata.blogspot.com

Back to top Go down

Re: ETL Logging vs Audit Dimension

Post  Mike Honey on Tue Mar 13, 2012 12:06 am

Hi arowshan,

I look at the Kimball Audit Dimension as primarily performing an Audit function, not Logging. It offers a much more convenient way to review your overall package execution history than trying to query Logging data.

IMO there's very little overlap between the two. Both are useful.

Good luck!
Mike

avatar
Mike Honey

Posts : 185
Join date : 2010-08-04
Location : Melbourne, Australia

View user profile http://www.mangasolutions.com

Back to top Go down

Re: ETL Logging vs Audit Dimension

Post  Sponsored content


Sponsored content


Back to top Go down

View previous topic View next topic Back to top

- Similar topics

 
Permissions in this forum:
You cannot reply to topics in this forum