ETL Job Scheduler and Monitor - Need Guidance with Architecture

View previous topic View next topic Go down

ETL Job Scheduler and Monitor - Need Guidance with Architecture

Post  surfinwind on Wed Oct 09, 2013 12:14 pm

Hi everyone,

I need to design an application to allow scheduling and monitoring ETL jobs that load an EDW. Buying a 3rd party scheduling tool is not an option. I work for a shop that mainly uses open source software, UNIX and shell scripting. The only purchased product is Netezza TwinFin appliance.

My goal is to create an architecture such that the ETL jobs could be defined, scheduled and controlled in the database tables. Unix shell script would then read the tables, execute and log metadata as defined. In a nutshell, can someone recommend any resources to study specific to this subject?

I am not a good UNIX hands-on, but would like to find some examples or patterns to clone. I might be asking for too much, but please chime in if you can give me any guidance.

Also, if anyone could recommend strong cases for buying 3rd party Scheduling tool, I could use that to sell to my manager/business too...

Thank you in advance!

Mike

surfinwind

Posts : 6
Join date : 2009-02-03

View user profile

Back to top Go down

Re: ETL Job Scheduler and Monitor - Need Guidance with Architecture

Post  ngalemmo on Mon Oct 28, 2013 3:34 pm

Job scheduling is such a basic, fundamental part of operating a production system, I find it hard to believe that building a system from scratch to do that is even being considered. There are a lot of complexities that have been worked out over many, many years. It is well worth investigating an established tool. I am sure there are open-source options as well.
avatar
ngalemmo

Posts : 3000
Join date : 2009-05-15
Location : Los Angeles

View user profile http://aginity.com

Back to top Go down

View previous topic View next topic Back to top

- Similar topics

 
Permissions in this forum:
You cannot reply to topics in this forum