Datawarehouse Load Best Practices

View previous topic View next topic Go down

Datawarehouse Load Best Practices

Post  mr_neal on Wed Aug 22, 2012 3:38 pm

In other Datawarehousing projects that I have worked on, I have always presumed that we load all data that we receive from source systems as long as the data model and supports it. However, on my current project, they want to restrict what data we load based on reporting requirements. So in my case, we ask certain users what project codes they want to see on monthly reports and only load those project codes and related data.

My thought is to load all the data and then limit the users selection to only their project codes in the report. This way, if the user mistakenly gave us the wrong codes, or changed their minds, the effort would quite less then trying to go back and reload all the data. We do not have a storage issue, so that is not the reasoning. Are there any "Best Practices" that you can refer me to in order to make the decision? I welcome your feedback as well. thanks


mr_neal

Posts : 17
Join date : 2012-01-26

View user profile

Back to top Go down

Re: Datawarehouse Load Best Practices

Post  LAndrews on Wed Aug 22, 2012 4:24 pm

Your instincts are correct - load all the data, then gracefully filter in the BI/Reporting layer.

Ideally there can be an attribute associated with the project code that can be used as a filter, allowing for a changing project code list (because reporting requirements ALWAYS change) with no ETL or BI changes.

LAndrews

Posts : 132
Join date : 2010-05-13
Location : British Columbia, Canada

View user profile

Back to top Go down

Re: Datawarehouse Load Best Practices

Post  BoxesAndLines on Thu Aug 23, 2012 5:51 pm

Some concepts are so elementary that to consider this a best practice is silly. I mean I interview folks all the time and ask about ETL best practices and I would never expect to hear this. As such, I don't believe you will find any papers or blogs that identify what your current project is doing is bad practice.
avatar
BoxesAndLines

Posts : 1212
Join date : 2009-02-03
Location : USA

View user profile

Back to top Go down

Re: Datawarehouse Load Best Practices

Post  Sponsored content


Sponsored content


Back to top Go down

View previous topic View next topic Back to top

- Similar topics

 
Permissions in this forum:
You cannot reply to topics in this forum