Any experience with or feedback on Kalido for Data Warehousing ?

View previous topic View next topic Go down

Any experience with or feedback on Kalido for Data Warehousing ?

Post  BIenthusiast on Thu Nov 17, 2011 4:40 pm


At my workplace we are looking at kalido as a BI tool for Data warehousing. Their claim to deliver a DW within 90 days as well as ability to change with minimun effort is intriguing. I would appreciate any feedback based on your experience with this product.

Thanks

BIenthusiast

Posts : 1
Join date : 2011-11-17

View user profile

Back to top Go down

Kalido DIW From Experience

Post  Peter Lamb on Fri Nov 18, 2011 8:30 pm

I have been working with Kalido DIW in a number of environments for almost 10 years, through multiple versions, upgrades, new builds, extensions...

I have seen their claim of 90 days, and it is possible - of course that has to be qualified. Some implementations are far more complex that other. A lot depends on complexity and clarity of requirements - just like any other BI build. The product does support iterative build and very effective prototyping. One thing to keep in mind is that it has to be used correctly. As an example, I have had projects 'go live' before any ETL is built because the team has been able to load the DW in a temporary way using the Kalido loaders. Some BI capabilities can be delivered with Kalido Query Defs. That is not what they mean by 90 days, but it gives an idea. Used properly, this same capability allows you to complete the model and load data while completing the detail specs for ETL. At the same time your BI reporting can start. You effectively parallelize a lot of build. On top of which you have loaded data and been able to provide it to the business for confirmation before ETL is built reducing rework a huge amount. This applies more on second and subsequent phases that the first phase as you can reuse.

But.. if you use a typical big waterfall approach you will get far less value. Approach Kalido as if it is ERwin or a DDL generator and you will certainly not see much value.

Hope this helps.

Peter Lamb

Posts : 7
Join date : 2011-11-18
Location : Toronto Canada

View user profile

Back to top Go down

Re: Any experience with or feedback on Kalido for Data Warehousing ?

Post  DilMustafa on Sat Nov 19, 2011 2:17 pm

Thanks Peter for the insight. My few concerns with this technology are as follows,

( I am not an expert in Kalido... This is all based on reading I have done..... So please correct me if I am wrong)

1. Kalido does assume that you have quality data?
2. Kalido does assume that the business processes are mature enough to be modeled?

Also not sure if Kalido can or atleast so far I have not seen any thing regarding Kalido's abilities on handling and transforming content data(emails,PDF,docs,bitmaps and stuff).

Your comments on this post are much appreciated.

Thanks,

Dil

DilMustafa

Posts : 49
Join date : 2009-02-03
Location : Calgary, Canada

View user profile

Back to top Go down

Re: Any experience with or feedback on Kalido for Data Warehousing ?

Post  Peter Lamb on Sat Nov 19, 2011 3:13 pm

1) The Kalido model is a 'business model' and business integrity rules are enforced as defined. In that sense you cannot load invalid data (data that breaks constraints) into the DW. Kalido manages suspense on all this data allowing invalid data to be identified and then re-processed. This is actually one of the values as it means you get these capabilities (as well as delta detection, surrogate key generation, slowly changing dimension handling) out of the box.
2) Yes and no ... The 'correct' approach with Kalido is iterative build. The intention is that because of the iterative build approach and the fact that you can get data loaded easily in a prototype process and expose it to the business you can surface issues - whether data quality or business understanding - can be surfaced.

Your last point - handling various document types is true, but these are not typical for a DW. Kalido also has an MDM product and I have heard people say it would be improved if it could store these types of objects. It cannot, although it can hold links to such objects.

Peter Lamb

Posts : 7
Join date : 2011-11-18
Location : Toronto Canada

View user profile

Back to top Go down

Re: Any experience with or feedback on Kalido for Data Warehousing ?

Post  DilMustafa on Sun Nov 20, 2011 4:03 am

Thanks once again Peter. You are a great help to us on this. Points you made in this thread are going to help us in evaluating this technology for our future architecture and beyond. One little clarification Kalido does generate DW and Staging Models from the Business Information Model. This part makes sense to me then there are landing tables which are created based on the source systems. Where do we do the mapping between Landing tables and the staging tables and how is the lineage tracked. Please, disregard if I am asking for too much at this point.

Thanks once again.

Dil

DilMustafa

Posts : 49
Join date : 2009-02-03
Location : Calgary, Canada

View user profile

Back to top Go down

Re: Any experience with or feedback on Kalido for Data Warehousing ?

Post  Peter Lamb on Sun Nov 20, 2011 5:54 am

The Kalido loaders can pull from ODBC or flat files although normally I use ODBC. The process depends on source. I have used Kalido to reach directly into a source such as Manugistics, planning cubes and even Oracle ERP. Sources often restrict that and it is necessary to use an ETL to 'land' information. An example is using ETL to pull from an SAP BW DSO directly to a DW landing table. I have an enhancement request out for Kalido to pull directly from a DSO as it seems like an obvious enhancement.

Consider the data at the point where Kalido can reach it with ODBC as the 'source'. The latest version - just released last week, provides a loading strategy they call the 'Unified Load Controller. With it you define a query against the source(s) - this can be a complex query. The results of the query are 'landed' by Kalido into Kalido managed landing tables. There is then a mapping to to one more target objects in the warehouse. The targets may be a mix of dimensional and fact tables as required. This may result in multiple loads be executed and you have control over how that is defined. Kalido actually creates staging tables for each of the targets based on its definition and manages suspense handling, error reporting etc.

Prior versions work in a similar way although there is normally a single target (although it is possible to load multiple levels in a hierarchy). Because there is only one target the process is simpler and no Kalido managed landing table is used. Also, the older version tended to rely more heavily on the Kalido application server for processing, while the new loader pushes much more to the database and is significantly faster. There was a 'faster' way to load in the older versions also, again allowed pushing to the database, but it was not as flexible in terms of transformations and could not reach out through ODBC so relied more on ETL.

I guess the way to think of it is:
1) You define the model and rules based on requirements
2) You load the data in a prototyping way to surface the data (including issues and gaps in requirements) for the business and BI
3) The model drives the fact table definition and the Kalido managed staging tables
4) Your source is queered using the Kalido loader and this effectively defined the Kalido managed landing table
5) There is a mapping which may include transformations that maps the landing to the model
6) Kalido manages / logs / audits the load process

It is important to understand that ETL is reduced but rarely done away with. There are normally requirements for complex derivations that cannot be handled in the Kalido mappings. The Kalido mappings do not provide Informatica like capabilities. You can do a fair bit for transaction loading, but when it comes to complex KPI calculations you need another tool. Not that ETL tool always provide the entire solution either. I have had to resort to stored procedures in Oracle or Teradata for requirements such as allocations and KPIs that are managed as formulas in master data that are in the control of the business.

Seeing is believing with any product. Kalido normally do a POV with potential customers, demonstrating a limited build of the customers real requirements. If you are interested, that is the route I would recommend. They can also put you in touch with reference customers either in your region or business area.

I certainly do not mind providing my view on the product. It is a view that will differ a little from Kalido sales of course as I do not work for them!

Peter Lamb

Posts : 7
Join date : 2011-11-18
Location : Toronto Canada

View user profile

Back to top Go down

Re: Any experience with or feedback on Kalido for Data Warehousing ?

Post  DilMustafa on Sun Nov 20, 2011 6:27 am

Awesome. So the mapping happens between landing tables and facts and dimensions etc.... Staging tables are there to support facts and dimensions but are not mapped directly against landing tables... Please correct me if I am wrong.

DilMustafa

Posts : 49
Join date : 2009-02-03
Location : Calgary, Canada

View user profile

Back to top Go down

Re: Any experience with or feedback on Kalido for Data Warehousing ?

Post  Peter Lamb on Sun Nov 20, 2011 9:08 am

You are right.

In the Unified Load Controller scenario Data flows from Source to Landing and through Staging into the real warehouse objects. Mapping / transformations are between Landing and the model which reflect the real target objects. Because you can use SQL on the pull from source you could also complete transformations / filters at that level to some degree.

It can get more complex also... You can create partial stream. For example you could stop at landing, but pull multiple data sets. Then create a stream that uses those multiple landed data sets together.

Kalido also supports the definitions of calculated and aggregated measures as well summary data sets where it manages the summaries. Using these capabilities it is also possible to make derivation on data already loaded and preset it to BI.

There is a lot to the product...

Peter Lamb

Posts : 7
Join date : 2011-11-18
Location : Toronto Canada

View user profile

Back to top Go down

Re: Any experience with or feedback on Kalido for Data Warehousing ?

Post  DilMustafa on Mon Nov 21, 2011 1:49 am

Thank you very much Peter. I owe you a beer for this:) May be we can meet on beer (on me ofcourse)when I am in Toronto or you visit Calgary next time. Thanks once again for the help.

DilMustafa

Posts : 49
Join date : 2009-02-03
Location : Calgary, Canada

View user profile

Back to top Go down

Re: Any experience with or feedback on Kalido for Data Warehousing ?

Post  Peter Lamb on Mon Nov 21, 2011 2:23 am

Hope it turns out useful. Ping me if there is anything else.

Peter Lamb

Posts : 7
Join date : 2011-11-18
Location : Toronto Canada

View user profile

Back to top Go down

without any personal hands on experience

Post  Ad Stam on Mon Nov 21, 2011 11:19 am

As stated I have no personal hands on experience with Kalido, (Peter Lamb has much more personal exeprience). However when I was responsible for a BI/DWH team in a consulting firm we introduced Kalido to one of our major clients that was back in 2000. That company was looking for a way to consolidate some 20 BI/DWH implementations and wanted to build an enterprise data warehouse. In first instance we advised for Informatica, but then we came across Kalido (it was just available for the outside Shell market), we asked the client whether or not they had money for two tools, they had not and they decided to start with Kalido (Informatica was brought on board some 3,4 years later). It is really a tool to work incremental (step by step, we used steps of 6-8 weeks), evolutionary (chaning requierements even during an increment) and prototyping (almost changing the model with the user at the developers desk). We did at some point in time 5-6 different projects next to eacht other. Working so, we experienced that we had to make use of an overall architect to safeguard a unity. And keep development,ongoing maintenance and day to day operation in one hand or at least as close as possible together. It was a big success for some 7-8 years. Then it was decided at corporate level (we worked for one of the divisions) hat they would use the data warehouse of the ERP vendor........... Well I have no figures buy I think that conversion costed more money than the initial building. Besides this client we were involved in a number of other projects as well. When you think about using Kalido indeed do a PoV and bring 1 or 2 experienced Kalido people on board. Well and there might be a set back; I do'nt know how it is in Canada, but overhere (Europe) Kalido experience is rather scarce. All and all I dare to say, you should have reasons (and as not really knowing the newer versions I do'nt know what those should be) not to use Kalido. In those days we try to do a PoV with a customer with 1.5 customer records with some 5-600 attributes and their Kalido was too slow, but they claimed they worked the product over to correct that

Ad Stam

Posts : 1
Join date : 2011-11-21

View user profile

Back to top Go down

Re: Any experience with or feedback on Kalido for Data Warehousing ?

Post  Prepper on Mon Nov 21, 2011 12:37 pm

I have some hands-on experience working with the Kalido applications and I can honestly tell you that their claim of 90 days is absolutely believable. My company already uses their MDM product in our data warehouse and we were looking to speed up EDW development and was introduced to their DIW solution. Our management wanted them to do a "bake off" against our current, in-house development methodology and I was assigned to the POC.

I worked with a small team of folks (both in-house and from Kalido) and in 14-days we were able to deliver a production-ready-lite warehouse for a major data component of our EDW. I say "lite" because for the constraints of the POC we limited the data to just 2 of our 30+ markets but given the full 90 days we would certainly have had a fully-realized, production-level EDW complete with a semantic layer and actual reports based on the data -- from all data sources.

Our situation is a little unique in that we have 2 separate and distinct billing-systems in-house and we had to bring completely different datasets together from scratch. The harmonization is one of the reasons that we looked at Kalido in the first place and even in the short time we worked on the POC, it was evident that their product made the task much more manageable.

Having worked with the Kalido folks on several projects (and products) over the last 18 months or so I can honestly say that you would be hard-pressed to find a better, faster solution for your EDW needs.

Sorry I couldn't be more specific to the nature of our business, but I hope I was able to help. Feel free to ask any other questions and I'll do my best to answer them.

Prepper

Posts : 1
Join date : 2011-11-21

View user profile

Back to top Go down

Re: Any experience with or feedback on Kalido for Data Warehousing ?

Post  Sponsored content


Sponsored content


Back to top Go down

View previous topic View next topic Back to top

- Similar topics

 
Permissions in this forum:
You cannot reply to topics in this forum