Best Practice for ETL Incremental File loading

View previous topic View next topic Go down

Best Practice for ETL Incremental File loading

Post  chime101 on Mon Apr 14, 2014 11:09 am

Hi - Newbie to the forum, so please excuse any indiscretions …..

I am looking for any advice for a best practice guide for loading incremental files to STAGE Data Warehouse layer.

Essentially looking for the mechanism -

e.g.
Source files arrive on server -  what filename - use of a sequence ?
How do we know that files have arrived fully ?
What if files don't arrive.
What happens after a file is loaded ?
How can we rollback - what if a file is missed completely?


Many thanks
chime101


Last edited by chime101 on Mon Apr 14, 2014 11:18 am; edited 1 time in total (Reason for editing : spelling :-()

chime101

Posts : 1
Join date : 2014-04-14

View user profile

Back to top Go down

Re: Best Practice for ETL Incremental File loading

Post  nick_white on Wed Apr 23, 2014 2:51 am

The specific answers to your questions are going to be completely dependent on your environment and your requirements e.g. most of what you have asked could be achieved through scripting and/or an ETL tool - but without knowing what environment you have these questions couldn't be answered.
Also, questions such as "what do we do if a file doesn't arrive" depend on your business and the answer could be anything ranging from "do nothing, it doesn't matter" to "stop everything, alert everyone as our business is about to collapse" - there is no generic answer.

Also, effectively what you are asking is how to manage an ETL process. To cover this would take many chapters in a book on ETL (Kimball covers it in a number of his books) so you are unlikely to get an answer in a forum like this - you are more likely to get an answer if you ask specific questions that can be answered in a couple of paragraphs.

nick_white

Posts : 366
Join date : 2014-01-06
Location : London

View user profile

Back to top Go down

View previous topic View next topic Back to top

- Similar topics

 
Permissions in this forum:
You cannot reply to topics in this forum