In today’s competitive environment, critical & timely business decisions significantly impact business outcomes such as improving customer relationships, increasing revenues, optimizing cost & resources, improving performance, maximizing operational efficiencies and even saving lives. The ability to make business decisions intuitively & pertinently is heavily dependent upon availability & accessibility of business information & data. Every business event, such as a customer purchasing a product, yields business data. Such data, resulting from business applications, processes, transactions, operations, business-partnerships, competition etc. inherently contains valuable knowledge & business insights about customers, products, policies, systems, operations, competitors etc., that helps in making business decisions. Typical steps in deriving decisions involve collecting required data, analyzing data by applying intelligence-mining techniques & business rules, extracting interesting insights & new intelligence, understanding context & applicability of such information and finally arriving at decisions in terms of what business actions can be taken.
The value proposition of a business decision is measured in terms of its effectiveness in generating expected benefits while accomplishing one or more business goals & outcomes. There are many factors that affect the effectiveness or the value of a business decision. One of the key factors is the decision-action-latency which is defined as the total time taken, after business event(s) occurred, to collect required data, analyze data, extract new insights & intelligence, understand the applicability of such new information and finally arrive at actionable decisions. According to Dr. Richard Hackathorn, an eminent BI analyst & creator of Time-Value curves, the value of data required to make an actionable business decision degrades as time lapses by after pertinent business events have occurred. This is shown in the following Time-Value curve:
Click on the image to enlarge
The decision-action-latency in turn is cumulative of 1] ‘data-latency’ defined as time taken to collect and store the data, 2] ‘analysis-latency’ defined as time taken to analyze the data & extract new insights & new intelligence and 3] ‘decision-latency’ defined as time taken to understand the context & applicability of such new insights & intelligence and to arrive at decisions in terms of what business actions can be taken.
It has to be mentioned here that business decisions can be strategic or tactical in nature. In case of strategic decisions, the value or effectiveness is potentially realized even though the underlying data used to make decisions can be very old accumulated over longer periods of time. Essentially the slope of the Time-Value curve would be small per se with very gradual decrease in value over time. Typically, strategic decisions are based on mining large data comprising of historical observations collected from several business events over a period of time. A retail store making a decision about when to run beer sales is an example of a strategic decision. For example, a retail store after inferring from store sales data that men who purchase diapers over the weekend also tend to buy beer at the same time can make a strategic decision to capitalize on this information to put the beer near the diapers and run beer sales over the weekends.
In case of tactical decisions, the value or effectiveness of business decisions is very short-lived because underlying data/information is highly volatile and inherently contains time-sensitive intelligence reflecting upon the momentary business performance. Essentially, the slope of the Time-Value curve would be very high with the curve being extremely steep. Typically, a tactical decision pertains to a single business event or transaction and hence is based on data collected from a single event that gets compared to or correlated with associated/related data collected from most-recent related business events. Credit card fraud detection is an example of a tactical decision. For example, a credit card company after inferring that a credit card, being used to purchase an item in Chicago, was used thirty minutes earlier in a place somewhere across the globe, can make an immediate tactical decision to capitalize on this information to mark the transaction as fraud and place a hold on the card.
So how do companies ensure that value proposition of business decisions is retained & realized, regardless of strategic or tactical decisions?
Traditional Data Warehouses:
As IT evolved over the years, companies automated their operations to collect data for analysis & reporting. In early days, each business application would capture such data in its own 'Reporting' database. As more operational automation got implemented, such 'islands of information' became siloed and proliferated. Soon companies realized the analytical value if the data from all such siloed islands is collectively mined together & correlated. However, collecting & correlating data from all such siloed systems was a challenge due to the incompatibilities between systems and lack of easy ways for these systems to interact & interoperate. A need for an infrastructure to exchange, store and analyze the data so that a unified view of insights & intelligence across the enterprise can be created, was recognized and thus Traditional data warehouse evolved to fill this need. Traditional data warehouse organized information from different sources so that data can be effectively analyzed to generate interesting & meaningful reports. Such reports would provide key insights to make business decisions which would then lead to course-corrective actions. To help smoothen the decision making process, a broad range of tools were developed over the years such as Extract, transform, and load (ETL) utilities for moving data from the various data sources to common data warehouse, Data-mining & Analytical engines to perform complex analysis and Reporting tools & Digital Dashboards to provide management & business decision makers with easy-to-comprehend analysis results.
In spite of such attempts to automate the decision making process, the process task to analyze the data & extract new insights & intelligence, the task to understand the context & applicability of such new insights & intelligence and the final task to arrive at decisions in terms of what course-corrective business actions can be taken remained manual to large extent. As shown in the following diagram, the Time-Value curve in case of leveraging traditional data-warehouses had long time-latencies. As such, traditional data warehouses were predominantly leveraged for making strategic business decisions in supporting strategic goals such as reducing costs, increasing sales, improving profits, maximizing operational performance and fine-tuning operational efficiencies by mining & analyzing massive amounts of data collected from across the enterprise over a long period of time. However, traditional data warehousing has little tactical value since the data in it is generally quite stale and can be weeks or months old. There were attempts to incorporate new technologies to minimize time-latencies however such attempts could only be successful in reducing data-latency by further automating data capture processes while both analysis task and decision task remained mostly manual.
Click on the image to enlarge
Active or Real-Time Data Warehouses:
The need for a solution that satisfies both the strategic and the tactical goals of an enterprise resulted in the emergence of Active Data Warehouses or Real-Time Data Warehouses. Sometimes these are also referred to as Real-Time or Right-Time Business Intelligence (RTBI) systems. Active Data warehouses not only support the strategic functions of data warehousing for deriving intelligence and knowledge from past enterprise activity, but also provide real-time tactical support to drive enterprise actions that react immediately to events as they occur. The new breed of data warehouses are designed to reduce all three latencies as much as possible by revamping utility tools. The traditional ETL process involved downtime of the data warehouse while the loads were performed. As such they are characterized as being offline ETL facilities. However Active data warehouses needed online ETL facility that not only preserved historical strategic data but also provided current tactical data. The online ETL’s job is to create and maintain a synchronized copy of source data in active data warehouse while constantly keeping it up-to-date. Besides, Active data warehouses needed improved data-mining & analytical engines with ability to incorporate business rules and with flexibility to run analytical models that can consume & adapt to more recent data blended with historical data. In effect, Active data warehouses markedly reduced overall decision-action-latency and thereby tremendously increased the value-proposition of business decisions in meeting business goals as compared to that of traditional data warehouses. In addition, they offered flexibility in making tactical decisions as and when needed by an enterprise. The following diagram shows the Time-Value curve in case of leveraging Active/Real-Time Data Warehouses.
Click on the image to enlarge
Real-Time-Intelligence-Based Decision Systems:
Even though Active data warehouses reduced overall decision-action-latency & thereby increased the value-proposition of business decisions, they were still predominantly used in a traditional sense with a strategic intent albeit leveraging most recent/current data. They were never considered as systems that can act as pure providers of 'Real-Time-Intelligence-Based' decision services. Such Real-Time-Intelligence-Based decision systems would churn & process varying business operational & transactional data on a real-time basis, sense transitory business insights, predict business fore-sights and use such reasoning to make real-time decisions that can then effect immediate actions through business transactions & operations. Such decision system would agglomerate capabilities such as Machine Learning, Data Mining, Rules Processing, Complex Event Processing, Predictive Analytics, Operations Research type of Optimizations, Artificial Intelligence & other Intelligence-Generating Algorithmic techniques and would provide flexibility to mix & match such capabilities for more complex decision orchestrations. The breadth of decision frameworks is necessary because different business objectives require different analytical approaches. For example, a rules engine works great when recognizing a customer for a milestone. Likewise, event processing is well suited for identifying potential customer dis-service scenarios. Finally, optimization techniques work well when making decisions about which promotions to place in front of the customer.
Real-Time-Intelligence-Based decision system would process live-data from business events as they occur, combine the event data with other valuable data or other events data, extract intelligence from such data and derive a decision as to what action should be taken. Sometimes, knowledge of the event is sufficient information to derive an insight and take action. More often, additional data is needed to correlate & improve intelligence. One another key feature of such 'Real-Time-Intelligence-Based' decision systems would be to instantaneously learn, adapt and adjust decision models & business rules as soon as new data is fed-back from business events. Such spontaneous processing of business events data combined with instantaneous adaptation of decision models based on data fed-back, effectively eliminates 'data-latency', 'analysis-latency' and any latency incurred otherwise in re-engineering the decision models from ground-up. As such, the maximum tactical & transient value associated with business event data is fully preserved & exploited while effecting an immediate business action based on real-time business decision. The value proposition of such as system is depicted using a similar Time-Value curve where the latencies are in micro to milli seconds and any perceived loss in business value is almost nil.
Click on the image to enlarge
Entrigna’s proprietary product RTES falls under ‘Real-Time-Intelligence-Based’ Decision System. For more information, refer to the blog titled ‘From Real-Time Insights To Actionable Decisions - The Road Entrigna Paves’.
Hope you found this blog informative.