ETL Execution & Scheduling

View previous topic View next topic Go down

ETL Execution & Scheduling

Post  ETLDev on Wed Feb 03, 2016 9:12 am

I had a question about the ETL design. Given that ETL would involve tools/scripts/programs written in different programming languages and platforms, what platforms/tools are used to manage the execution and scheduling of these? How do we make sure that these tools/scripts/programs run at the same time and pull data into the data warehouse?

Thanks,
Amit

ETLDev

Posts : 1
Join date : 2016-02-03

View user profile

Back to top Go down

Re: ETL Execution & Scheduling

Post  nick_white on Wed Feb 03, 2016 10:41 am

I would use a dedicated ETL tool (Informatica, SSIS, Talend, Datastage, ODI, etc.) otherwise, as you say, you have to hand code not only the actual data transformations but also the control framework to make sure everything runs with the correct precedents, errors are handled appropriately, etc.
Except for the simplest of processes I think you would have to be a bit of a masochist these days (or very confident in you abilities) to hand code your ETL system

nick_white

Posts : 366
Join date : 2014-01-06
Location : London

View user profile

Back to top Go down

Re: ETL Execution & Scheduling

Post  ngalemmo on Wed Feb 03, 2016 2:18 pm

Some etl tools have built in scheduling and execution capabilities, but generally, they are pretty limited in functionality. Most sites use an enterprise scheduling tool and execute etl processes using scripts calling the etl tool's CLI (command line interface). The CLI will usually return a status code that the scheduling tool would use to check progress of the load.
avatar
ngalemmo

Posts : 3000
Join date : 2009-05-15
Location : Los Angeles

View user profile http://aginity.com

Back to top Go down

Re: ETL Execution & Scheduling

Post  Sponsored content


Sponsored content


Back to top Go down

View previous topic View next topic Back to top

- Similar topics

 
Permissions in this forum:
You cannot reply to topics in this forum