How to test deployment of ETL jobs (dev-test-production)?

View previous topic View next topic Go down

How to test deployment of ETL jobs (dev-test-production)?

Post  larus on Wed Aug 17, 2011 2:23 am

We have been developing a Data Warehouse and bunch ETL jobs for loading data from source systems into the DW. Now we are on the egde of deploying these ETL jobs into production environment. ETL jobs should be heavily tested and I would like to know if you guys could give some hints and guidence how to do this testing? What should be tested (data validation, job function, database performance, indexing)? What kind of reports should be written etc.?

larus

Posts : 5
Join date : 2011-03-01

View user profile

Back to top Go down

Re: How to test deployment of ETL jobs (dev-test-production)?

Post  BoxesAndLines on Thu Aug 18, 2011 9:45 am

Did you use any documentation to build the ETL logic? Source to target mappings, design documents, etc? These documents are the source of the rules to test your ETL.
avatar
BoxesAndLines

Posts : 1212
Join date : 2009-02-03
Location : USA

View user profile

Back to top Go down

Re: How to test deployment of ETL jobs (dev-test-production)?

Post  larus on Thu Aug 18, 2011 2:18 pm

Well, yes of course, but I think that there is much more than just testing ETL jobs (logic, function). I would be delighted if someone could give some advice what kind of testing should be done at minimum.

larus

Posts : 5
Join date : 2011-03-01

View user profile

Back to top Go down

Re: How to test deployment of ETL jobs (dev-test-production)?

Post  BrentGreenwood on Mon Jan 02, 2012 12:42 pm

Here are some additional ideas for testing. The list is really endless. Just depends on how much budget/time you are able to spend on setting up the tests and framework.

- Failure / Restart testing (at each logical break point - by landing zone - raw, cleansed, transformed, integrated, staged, final load)
- Unit Testing (isolated logic and function tests - ideally, these would be created in a way that they can be automated and used for regression on later builds)
- Performance testing (hardware)
- Load testing (ensure load process will scale to handle peak loads within processing window)
- Security testing (ensure users don't have access to the kitchen)
- Integration testing (over multiple units and/or entire process - test functions as well as data integrity from source to target)

A good resource in this space is Ralph Hughes' book Agile Date Warehousing. He also teaches classes specifically in Agile DW Testing that are quite informative, and go into great detail on a testing framework he built for a massive DW at a telecom client recently.

Hope that helps.
Brent Greenwood

BrentGreenwood

Posts : 6
Join date : 2011-12-25
Location : San Diego, CA

View user profile http://brentgreenwood.blogspot.com

Back to top Go down

Re: How to test deployment of ETL jobs (dev-test-production)?

Post  burgerboy123 on Thu Jan 22, 2015 1:24 am

So... is it the dropping and reloading that is causing me the problem? or should i generate a surrogate key a different way ensuring it gets the same key after any necessary rebuild?



________
adil

burgerboy123

Posts : 1
Join date : 2015-01-22

View user profile

Back to top Go down

Re: How to test deployment of ETL jobs (dev-test-production)?

Post  Sponsored content


Sponsored content


Back to top Go down

View previous topic View next topic Back to top

- Similar topics

 
Permissions in this forum:
You cannot reply to topics in this forum