design for data load failures

View previous topic View next topic Go down

design for data load failures

Post  Ishwar on Tue May 31, 2011 10:00 am

how to design for data load failures? I have seen a system where if (days) data load has issues, ETL stops all days data to go into the destination tables. Any reading material for the same will help. I am more concerned about database model. I have following questions,

1) With which data I need to check the transformed data?

2) database model for example

3) Stop few data or the entire data load?


Ishwar

Posts : 6
Join date : 2011-05-31

View user profile

Back to top Go down

Re: design for data load failures

Post  ngalemmo on Tue May 31, 2011 11:46 am

This is something particular to an installation. It all depends on the business and what the user expects.

With that said, I have never implemented a process that, when doing database to database load processes, would ever reject data. The only situations would be when loading from flat files, and it was determined the file was corrupt in some manner. In which case the entire file is rejected.
avatar
ngalemmo

Posts : 3000
Join date : 2009-05-15
Location : Los Angeles

View user profile http://aginity.com

Back to top Go down

Re: design for data load failures

Post  Ishwar on Tue May 31, 2011 2:56 pm

In my current work (Retail data warehouse development), after studying underlying data sources I have picked up tables and columns to work on. Data validation will decide whether I have picked up the correct fields and tables from data sources.

What if some of setting in underlying data source changes, (for example addition of new functionality or addition of new promotion which I didn't consider while designing data warehouse)?
At which step this data should be checked?

What if user identifies issue in data? I need to check and reload the data for the data source?

Any thoughts!!!!

Ishwar

Posts : 6
Join date : 2011-05-31

View user profile

Back to top Go down

Re: design for data load failures

Post  ngalemmo on Tue May 31, 2011 4:18 pm

If there is a material change in the structure of the source system (new tables, columns etc...) then effecting such changes is a matter of changing the DW model and load processes. That is part of the system lifecycle and would hardly be considered a 'load error'.

If simply adding a new promotion (or any new values to an existing structure) causes problems, then you need to reconsider how you are loading things in the first place. Any load process should gracefully handle new data values... such as adding a new row to the promotion dimension if a new promotion code is encountered.
avatar
ngalemmo

Posts : 3000
Join date : 2009-05-15
Location : Los Angeles

View user profile http://aginity.com

Back to top Go down

Re: design for data load failures

Post  Ishwar on Tue May 31, 2011 4:28 pm

I agree, but does that mean after implementing data warehouse there will not be any data validation?

Ishwar

Posts : 6
Join date : 2011-05-31

View user profile

Back to top Go down

Re: design for data load failures

Post  Sponsored content


Sponsored content


Back to top Go down

View previous topic View next topic Back to top

- Similar topics

 
Permissions in this forum:
You cannot reply to topics in this forum