Kimball Forum
Would you like to react to this message? Create an account in a few clicks or log in to continue.

ETL Job Scheduler and Monitor - Need Guidance with Architecture

2 posters

Go down

ETL Job Scheduler and Monitor - Need Guidance with Architecture Empty ETL Job Scheduler and Monitor - Need Guidance with Architecture

Post  surfinwind Wed Oct 09, 2013 12:14 pm

Hi everyone,

I need to design an application to allow scheduling and monitoring ETL jobs that load an EDW. Buying a 3rd party scheduling tool is not an option. I work for a shop that mainly uses open source software, UNIX and shell scripting. The only purchased product is Netezza TwinFin appliance.

My goal is to create an architecture such that the ETL jobs could be defined, scheduled and controlled in the database tables. Unix shell script would then read the tables, execute and log metadata as defined. In a nutshell, can someone recommend any resources to study specific to this subject?

I am not a good UNIX hands-on, but would like to find some examples or patterns to clone. I might be asking for too much, but please chime in if you can give me any guidance.

Also, if anyone could recommend strong cases for buying 3rd party Scheduling tool, I could use that to sell to my manager/business too...

Thank you in advance!

Mike

surfinwind

Posts : 6
Join date : 2009-02-03

Back to top Go down

ETL Job Scheduler and Monitor - Need Guidance with Architecture Empty Re: ETL Job Scheduler and Monitor - Need Guidance with Architecture

Post  ngalemmo Mon Oct 28, 2013 3:34 pm

Job scheduling is such a basic, fundamental part of operating a production system, I find it hard to believe that building a system from scratch to do that is even being considered. There are a lot of complexities that have been worked out over many, many years. It is well worth investigating an established tool. I am sure there are open-source options as well.
ngalemmo
ngalemmo

Posts : 3000
Join date : 2009-05-15
Location : Los Angeles

http://aginity.com

Back to top Go down

Back to top

- Similar topics

 
Permissions in this forum:
You cannot reply to topics in this forum