ETL Job Scheduler and Monitor - Need Guidance with Architecture
2 posters
Page 1 of 1
ETL Job Scheduler and Monitor - Need Guidance with Architecture
Hi everyone,
I need to design an application to allow scheduling and monitoring ETL jobs that load an EDW. Buying a 3rd party scheduling tool is not an option. I work for a shop that mainly uses open source software, UNIX and shell scripting. The only purchased product is Netezza TwinFin appliance.
My goal is to create an architecture such that the ETL jobs could be defined, scheduled and controlled in the database tables. Unix shell script would then read the tables, execute and log metadata as defined. In a nutshell, can someone recommend any resources to study specific to this subject?
I am not a good UNIX hands-on, but would like to find some examples or patterns to clone. I might be asking for too much, but please chime in if you can give me any guidance.
Also, if anyone could recommend strong cases for buying 3rd party Scheduling tool, I could use that to sell to my manager/business too...
Thank you in advance!
Mike
I need to design an application to allow scheduling and monitoring ETL jobs that load an EDW. Buying a 3rd party scheduling tool is not an option. I work for a shop that mainly uses open source software, UNIX and shell scripting. The only purchased product is Netezza TwinFin appliance.
My goal is to create an architecture such that the ETL jobs could be defined, scheduled and controlled in the database tables. Unix shell script would then read the tables, execute and log metadata as defined. In a nutshell, can someone recommend any resources to study specific to this subject?
I am not a good UNIX hands-on, but would like to find some examples or patterns to clone. I might be asking for too much, but please chime in if you can give me any guidance.
Also, if anyone could recommend strong cases for buying 3rd party Scheduling tool, I could use that to sell to my manager/business too...
Thank you in advance!
Mike
surfinwind- Posts : 6
Join date : 2009-02-03
Re: ETL Job Scheduler and Monitor - Need Guidance with Architecture
Job scheduling is such a basic, fundamental part of operating a production system, I find it hard to believe that building a system from scratch to do that is even being considered. There are a lot of complexities that have been worked out over many, many years. It is well worth investigating an established tool. I am sure there are open-source options as well.
Similar topics
» SLA: Architecture
» ETL Architecture
» DWH architecture problem
» Extraction layer on SOA/ESB architecture
» Hybrid Architecture
» ETL Architecture
» DWH architecture problem
» Extraction layer on SOA/ESB architecture
» Hybrid Architecture
Page 1 of 1
Permissions in this forum:
You cannot reply to topics in this forum