Home | Topics | TIBCO

TIBCO Spotfire Application Data Services - Innovation and Validation – Usefulness

Printer-friendly versionPDF version

hand upA new product from TIBCO Spotfire, came to my attention in the summer of '09, for me its one of those really important innovations which don't come around too often. Such innovations can leave one thinking 'am I the only person in the world (apart from the developers) who can see the usefulness of this product?' After the Basel II Pillar 2 debacle I don't mind being the only person in the world banging on about the importance of stuff but in this instance the usefulness of this product seems to me such a no-brainer that will not be the case.

Obviously first of all I am going to describe what the product is and where you can find details of it but secondly I am going to explain why I think it is so useful. To do the second bit I have to make a wee bit of a detour into a change in my own thinking of late which I will summarize here but I hope to elaborate upon that topic in a later blog this year.

The product which I find so useful and I think you should too, is TIBCO Spotfire Application Data Services, you can fund an introductory web page here Lets ease ourselves gently into this amazing product which has significant application in Risk Management. TIBCO Spotfire Application Data Services enables organizations to access, transform, and cache enterprise data sources through Spotfire Information Services. TIBCO Spotfire Application Data Services (SADS for short) is an add-on to Spotfire Information Services that enables data retrieval from: 

SAP Netweaver BI (formerly SAP BW),

SAP ERP (formerly SAP R/3),


Siebel eBusiness Applications,

Oracle E-Business Suite,

and Web Services

spotfire SADS

The key functionality applicable in Risk Management is that  SADS offloads queries against mission critical transaction systems to a "just in time" caching layer. SADS can introspect data models from packaged solutions or in house built API-based applications and deliver mission critical data as standards-based SQL to Spotfire Information Services. SADS functionality accelerates data modeling with a pre-built set of the most common data views for standard packages or BI environments (SAP BW BI, ORACLE), it then transforms the data into the form required by Spotfire. Extracting data from packaged applications is not a trivial task. Specialized consultants are often needed to decipher these systems' proprietary interfaces and data structures. Combining packaged application data with other data sources is even more challenging. SADS enables all that for you as an aspect of your overall Data Integration toolset.

To get to the end point, the rendering, SADS is utilizing Spotfire Information Services which is a meta-data management layer facilitating ransparent integration with the TIBCO Spotfire Analytic Development Kits. Spotfire Information Services make data easily accessible from any JDBC, ODBC, or OLE-DB accessible data source including Microsoft® Excel®, delimited flat text files, and a wide range of databases such as Oracle®, IBM® DB2®, Microsoft SQL Server, Microsoft AccessTM, MySQL®, SAS/SHARE®, PostgreSQL®, Sybase® or Informix®.

The key downstream application (TIBCO Spotfire Analytic Developers Kit) which I am interested in is the TIBCO S+ Server, although the functional advantages of the visualization technologies of TIBCO Spotfire are not lost on me but I do not want this blog to have your head spinning around like The Master did in the recent Dr. Who finale, it wouldn't do to create clones of me, would it? (Pushing the analogy too far!) Links to all the product details referenced here are provided below. Upstream the most relevant applications for this product in my world are SAP NW BI and ORACLE of course, although SYBASE may be of interest.

Innovation and Validation - Usefulness

This type of packaged integration is where the world is going now, you can see it in IBM Infosphere or the strategic direction of travel of SAP BO DS DI our technology world is moving from ETL to ETI finally so many years after Kimball predicted this would happen, essentially deployment of these types of technology facilitates data integration at the meta-data level, which used to be so difficult but is now enabled by development environments which analysts can actually use to orchestrate this integration. In SADS Semantics and data types are derived directly from the packaged applications, allowing users (integration developers) to easily find the data required to meet their business needs. 

Here is my little detour; I won't take you too far along this crooked mile. I come from a hybrid background, I am an economist really who began building large databases for essentially econometric applications in investment banking and asset management many years ago.  Hence I got sucked into the Basel II vortex half way through the last decade. My best work was done with SAP and Insightful (now TIBCO Spotfire) in addressing the requirements of Basel II Pillar 2 (if I say so myself). Through the 2nd half of the last decade I banged my head against a wall trying to convince the banks to build Model Development Environments (MDE) so that they could address the issues of;-

Risk Management



Compliance and Regulatory reporting

Supervisory Analytics

All that stuff! Recently I have come to understand that the banking industry cannot practically implement MDE all over the place. There is just not the quantity of quants on the supply side of the equation to manage and implement full blown MDE to address the requirement. Thus sensibly the banking industry requires to purchase, implement and integrate packaged solutions to each of the sub requirements of the general list I have given above (e.g. Liquidity Risk or VaR). In my view, unless a bank wants to hand over all of its information processing to a single vendor then there is no choice but to consider integrating 'best of breed' applications now to meet this onslaught of transparency and governance requirements coming down the pipe. One data set which will never be outsourced or sent to the cloud is the risk management and transparency numbers and in my view. Handing that data set to one single vendor is placing the bank as hostage to fortune.

But the need to have an MDE at the front and centre of your banking transparency and governance is well understood; indeed the UK and Swiss authorities have been specific in their insistence about that. The necessity of the MDE is to allow the bank to demonstrate that it can validate the models which 'best of breed' applications are utilizing to produce the transparency and supervisory numbers which are now required. Validation is not a censorious process as some pontificatory balloons would like to have you believe. It's just a process whereby you run a quantitative model (in this case in S+) which you have built, which you own, against the same data set which your actual production engine is using. The production engine being a closed-source code object which is part of the package you have bought to solve a specific requirement. If you can build a quantitative model which produces the same or tolerably similar results to that which your production engine produces then you can validate the veracity of the numbers your systems are producing. More than that, you can demonstrate to the supervisor that you understand your own numbers. Further you have a platform which allows you to engage with your package vendor in a process to innovate the underlying modeling algorithm appropriate to you and of course finally since you are using toolsets which integrate the data sets for your MDE and your 'best of breed' package applications you can conduct stress testing or data mining on the data sets which your risk packages inevitably create as a by-product of their normal day to day working operation.

SADS screenshotThis is where TIBCO Spotfire Application Data Services is a really useful data integration toolset for risk management. The bank is going to make a great investment in a Liquidity Risk solution a high performance VaR engine, not only in the license fees but also in the IP exchange between vendor and bank, not least of which is the data integration process. Implementation of such a best of breed package then creates an asset for the bank in the underlying data which the package collects and importantly historicizes to support its (new) normal operations. SADS then gives you the possibility to farm that asset, turn it into an early warning system, innovate it, test it and to kill that supervisory bird at the same time.

It's a no brainer! I have focused upon the Risk Management domain and thus the applicability of S+ but S+ itself is also (arguably with more users) relevant in the domain of customer analytics and the whole marketing thing (which I do not understand) and the applicability of the visualization techniques where Spotfire is a world leader are relevant too since quick communication of tactical and strategic concepts is being innovated and developed by the visualization technologies right now, picking up on advanced mathematical ideas of the 1970s.

Its unlikely that as a Financial Institution you do not have some kind of MDE right now, its also highly likely that as a CTO or CRO (CXO) you are considering how to meet the new transparency and governance requirements. In an earlier related blog I waxed lyrical about the challenges of this challenge. In general you will find that the best of breed applications supporting the new requirements will be creating highly valuable historic data sets in ORACLE or DB2 or SAP platforms, SADS gives you a window on that asset, allows you to make use of it, puts it under your control.


 Spotfire Information Services

 TIBCO S+ Server



 Liquidity Risk - The Definitive Methodological Paper - Drehman



 Analyzing Mortgage Backed Securities with TIBCO Spotfire





Short URL
Asymptotix on Twitter

Are the key legislative pillars such as Basel II & III, UCITS IV and Solvency II forcing you to re-examine how you identify, measure and manage risk and capital?

Asymptotix work closely with our partners to help clients develop a more proactive, systematic and integrated approach to governance and risk management to deliver proper value.

Asymptotix can offer the support you need to deliver on time. Read more...

Is the goal of your website to sell services or products, educate, or collect data?

A positive customer experience is vital to conversion, no matter what your conversion goals may be. Our designers and developers will create a positive experience to maximize your conversions and deliver the optimal return on your investment. We strive to find the perfect balance between the web site’s design and functionality.

Asymptotix implements interactive solutions for European companies. From corporate websites to social communities, our clients will tell you an investment in building a scalable online experience will deliver long-term tangible benefits.

Based in Luxembourg we can help you all over Europe. Our multi-lingual team can work with projects and speak your language! Read more...