Overnight Load Failure Frequency - Benchmark?

View previous topic View next topic Go down

Overnight Load Failure Frequency - Benchmark?

Post  djschiff on Fri Oct 23, 2015 2:44 pm

I'm wondering if there are any benchmarks as to the percentage of time that the overnight load process for a data warehouse is successful (on-time, untouched by developers or support staff) vs late or encountering a problem requiring some sort of intervention.  Obviously there's no "one size fits all" answer; there are many factors including the number and types of source systems, data volume, data type, system/data maturity, etc.  But are there rough benchmarks, such as problems once a month is "normal" vs. "above/below avg," etc.

djschiff

Posts : 2
Join date : 2010-01-18

View user profile

Back to top Go down

Re: Overnight Load Failure Frequency - Benchmark?

Post  ngalemmo on Fri Oct 23, 2015 3:46 pm

If you are dealing with a well defined source with decent data management, such as an ERP system, load failures (those not caused by external issues such as hardware or network failure) should be near zero. Even uncontrolled sources should have sufficient processes in place to separate bad data from good to prevent a full failure.

In other words, if you regularly encounter load issues, your process is broken.
avatar
ngalemmo

Posts : 3000
Join date : 2009-05-15
Location : Los Angeles

View user profile http://aginity.com

Back to top Go down

Re: Overnight Load Failure Frequency - Benchmark?

Post  djschiff on Sat Oct 24, 2015 9:27 am

Well "external issues," I guess that's the rub, isn't it? A network issue here, a database problem or server glitch there. Suddenly there are three mornings that appear only as "data warehouse load issues" to the users, but there's little the data warehouse staff can do.

djschiff

Posts : 2
Join date : 2010-01-18

View user profile

Back to top Go down

Re: Overnight Load Failure Frequency - Benchmark?

Post  ngalemmo on Sat Oct 24, 2015 2:54 pm

That's the nature of the beast. There are a lot of moving parts and all IT has to do their bit to keep things running. The best you can do is grin an bear it, and keep documentation.
avatar
ngalemmo

Posts : 3000
Join date : 2009-05-15
Location : Los Angeles

View user profile http://aginity.com

Back to top Go down

Re: Overnight Load Failure Frequency - Benchmark?

Post  nick_white on Mon Oct 26, 2015 3:11 pm

If another system has caused the problem then don't report them as "data warehouse load issues"

nick_white

Posts : 366
Join date : 2014-01-06
Location : London

View user profile

Back to top Go down

Re: Overnight Load Failure Frequency - Benchmark?

Post  ron.dunn on Tue Oct 27, 2015 3:17 am

Nick, you must never have been a DBA (Default Blame Acceptor), otherwise you'd know that any system problem is automatically the fault of the biggest database in the chain, until incontrovertibly proven otherwise by the over-worked, under-valued database team

ron.dunn

Posts : 55
Join date : 2015-01-06
Location : Australia

View user profile http://ajilius.com

Back to top Go down

Re: Overnight Load Failure Frequency - Benchmark?

Post  Sponsored content


Sponsored content


Back to top Go down

View previous topic View next topic Back to top

- Similar topics

 
Permissions in this forum:
You cannot reply to topics in this forum