Confused over Hybrid Architecture - the NDS store

View previous topic View next topic Go down

Confused over Hybrid Architecture - the NDS store

Post  meb97me on Wed Mar 27, 2013 11:59 am

I'm currently charged with implementing a new data warehouse.

I've been trying to understand how a hybrid model would work with regards to storing\handling historical data.

My understanding so far is that the flow of data would be

Source (OLTP systems) > Staging > Normalised Data Store (NDS) > Dimensional Data Store (DDS) > Data Marts

As I understand it the data within the NDS should be able to rebuild the DDS complete with historical changes. The definition I have read is
"The NDS is the master data store, meaning the NDS contains the complete data sets, including all historical transactional data, and all historical versions of master data"

So therefore in reality if we had a customer (for example) entity in the source OLTP system and as part of the daily load we found one or more of the attributes had changed we would need to add a new row into the NDS for that customer. Presumably the NDS needs to have validfrom validto fields to capture this.

So the reality of this would be that if we have a fairly volatile dimension\entity we're going to have a very large NDS as well as a very large DDS very quickly, as obviously these multiple versions in the NDS translate into multiple SCD records.

Is my understanding and the reality of this correct?

meb97me

Posts : 34
Join date : 2010-07-28

View user profile

Back to top Go down

Re: Confused over Hybrid Architecture - the NDS store

Post  BoxesAndLines on Wed Mar 27, 2013 1:56 pm

Source > Staging > DDS

That's all you need.
avatar
BoxesAndLines

Posts : 1212
Join date : 2009-02-03
Location : USA

View user profile

Back to top Go down

Re: Confused over Hybrid Architecture - the NDS store

Post  ngalemmo on Wed Mar 27, 2013 6:14 pm

In my opinion, the 'hybrid' DW is a political compromise for nay sayers of the Kimball approach.

It is basically the Inmon architecture. The difference is, Kimball Group is saying the analytic environment should be a full blown atomic level dimensional implementation, whereas Inmon is fairly vague on what should and shouldn't be published. Inmon basically refers to targeted data marts, i.e. department specific aggregates or extracts or whatever.

So, essentially you are implementing two data warehouses that store the same information. One in 3NF to satisfy the entrenched traditional modelers, and a dimensional DW for users. The idea is you put everything out there for users to access without having to continually 'go back to the well', and start a new IT project, when analytic requirements change.

Frankly, its a FUD (fear, uncertainty, doubt) thing. For example, a common argument is you cannot enforce 'business rules' in a dimensional model. While that is not entirely true, I often wonder why you would want to? What rules are there that the operational systems should not have been enforcing from the beginning? Are they not based on 3NF models? The DW should be an accurate depiction of what is happening in the operational systems, and if rules are being violated, fix it at the source, or change the rules.

avatar
ngalemmo

Posts : 3000
Join date : 2009-05-15
Location : Los Angeles

View user profile http://aginity.com

Back to top Go down

Re: Confused over Hybrid Architecture - the NDS store

Post  Jeff Smith on Tue Apr 01, 2014 5:33 pm

So we have a couple of vendors to help us fix a big issue. One of the vendors has used the "hybrid" term. Basically he wants a staging area in 3NF to support the STAR (or STARFLAKE). The "hybrid" term is really BS because it's really Kimball. It's Kimball because they aren;t building a huge staging area/DW in 3NF before building the Marts. A Mart is conceptually created and then supporting data structures are created in the staging area.

Kimball doesn't say the staging area can't be in 3NF.

The argument for a Staging Area in 3NF is that it cushions the Stars from changes to source systems. Granted, that's more work up front, but less work in the long run. If a source is added or changed, work is primarily done in the staging area. Or so goes the argument.

To me, a change has to be accommodated somewhere. But I kind of like working with 3NF as opposed to flat files or whatever is thrown at me. If the company wants to pay for a staging in 3NF, who am I to fight it. But I'm not sure it's necessary.

Jeff Smith

Posts : 471
Join date : 2009-02-03

View user profile

Back to top Go down

Re: Confused over Hybrid Architecture - the NDS store

Post  Sponsored content


Sponsored content


Back to top Go down

View previous topic View next topic Back to top

- Similar topics

 
Permissions in this forum:
You cannot reply to topics in this forum