Archive for Big Data & Real Time Decisions

Big Data in Agriculture

Over the next few months, I'll be writing blogs on some non-traditional industries that use big data. I'm looking forward to sharing updates and information on how big data can be used in all industries, not just the ones we typically associate with technology.

Farming is something most of us take for granted. We go to the grocery store and pick out our food without giving much thought to where our food came from or what went into growing it. We think of small quaint farms where farmers plant seeds, ride small tractors and then harvest their crops. However, many farms in the United States rely heavily on technology and are turning to big data to help them become more efficient, cost-effective, and less environmentally impactful.

Today’s tractors not only use sensors to collect information that help with preventative maintenance, but tractors also have multiple computer screens and sensors on them that collect information from everything from nitrogen and pH levels in the soil to how far apart the seeds are. Farmers tend to use this information while planting, however, many farmers do not use the information they’ve collected after the fact.

Farmers are also using “precision farming” to help make farming more efficient. This technique can mean many things, but ultimately it means using information about the soil and crops in a specific area to maximize the output of the crop and minimize the production cost for a crop. Farmers can use this information for everything from identifying the best places to plant certain crops to how many plants per acre they can plant.

In the future, we can expect to see more farmers adopting precision farming and other big data techniques. The big data market for agriculture is expected to grow from a $2.4B industry in 2014 to a $5.04B industry in 2020 (Research and Markets Global Precision Agriculture Market 2014-2020) and with the population projected to grow to 9 billion people by 2050, farmers will need to increase outputs significantly to keep up with demand. We’re already seeing some very interesting ways that precision farming and big data solutions us can be implemented at larger facilities. For example, Gallo Winery recently implemented a system that takes satellite imagery of their vineyards and determines which plants are getting too little or too much water. The images are processed, analyzed and then the sprinkler that is connected to an individual plant is automatically adjusted to give it either more or less water.  Water consumption at Gallo Winery has been reduced by 25% since the system was implemented, the health and production of the plants has increased and the costs associated with workers manually watering individual plants has decreased.

The real power of big data will be when farmers start sharing their data with companies. In the past, farmers have been very hesitant to share the data they collect to corporations. Many farmers view the information from their fields as propriety and are worried that the information generated from their farms will be shared with commodity traders or other farmers. They are also worried that seed and equipment companies will use the information to sell farmers higher prices goods. However, seed and equipment companies need information from individual farms in order to improve their software and products so farmers can keep achieving the best results possible. In the next few years, I believe seed and equipment companies will start focusing on how to earn the trust of farmer and proactively show farmers how sharing this information will lead to substantial ROIs for the farmers. Also, as time progresses farmers will become more comfortable with big data and the technologies and realize that the payoff of higher yields and ultimately lower costs will persuade farmers to share their data.

Trends in Big Data and the IoT in 2016

As we enter the new year, it’s always an exciting time to reflect upon the previous year and ask “What new things will happen next year?” Over the past year, it’s been really cool to see how executives at companies are realizing the value of using big data instead of just collecting it.  Because of this trend, 2016 should bring about disruptive changes in the big data and internet of things markets.

Some of the top trends in 2016 that I see happening are

Customer satisfaction levels will be influenced by an automatic personalized experience

As consumers become more tech savvy and more millennials have discretionary income, more consumers will continue to adopt and use mobile apps such as Target’s Cartwheel or PriceGrabber while they’re shopping. These consumers are looking for a personalized experience that will give them some benefit, whether it’s a lower price or targeted advertising or coupons based on past behaviors, when shopping. Consumers have many options to choose from when shopping both online and in-person and will ultimately pick the store that gives them the most value and the best shopping experience.

Additionally, with the increase of internet shopping and the multitude of stores available to consumers, companies will start relying more on what an individual is clicking on and posting on-line about products and her shopping experience. In the past, companies have had challenges making sense of this information in a timely manner and then reacting. However, companies are starting to discover solutions that can help them not only react in real-time to a customer’s shopping experience but also personalize the customer’s shopping experience based on past behaviors or trends. These proactive actions should lead to higher level of customer satisfaction for the customer.

Using ROI in big data

Executives are pushing for the adoption of big data solutions however, many executives want to see a measureable ROI and meaningful use cases before they make a large investment in a solution. In 2016, solution providers will start partnering with their users to determine the ROI of using a solution. Many times these measurements can be straightforward, such as calculating how much revenue is saved when using data sensors to predict when parts will wear out.  However, calculating the ROI on other solutions that combine structured and unstructured data will be more challenging to determine.

Data in the Internet of Things will start to be used instead of just collected

Sensors on many devices will help companies predict when parts need to be serviced and can also predict anomalies in the overall system. However, many companies have yet to realize the full potential of this data. In 2016, more companies who collect this type of information will no longer just store it but start to use this information to help prevent down time and achieve better customer service. Also, with the increased adoption of personal healthcare devices, such as Fitbits and smart watches, more consumers are going to start tracking their own healthcare.  Companies that provide solutions that monitor and make recommendations on a consumer’s heart rate, blood pressure or fitness activity will grow.

The need for simplified Big Data

Currently, many of the traditional big data solutions that make real-time decisions require users to be very tech-savvy and require substantial coding. However, in 2016, we will probably see more companies purchasing tools that can be easily used by non-technical users. This is because there is currently a shortage of data scientists and the average salary of an entry level data scientist is quite high compared to that of an entry level analyst. Many companies just can’t afford to have data scientists on staff.  Also, customer facing groups want to be able to see results in real-time and not wait for the IT or data science group to get them the information they need. Solutions will still need to be set up by data scientists and software engineers, however, once the solution is set up, non-technical groups such as marketing and customer service will be the ones accessing the data and writing simple queries to find the information that they need in real-time.

2016 will definitely be an exciting time for big data! The Entrigna team is looking forward to working with companies in the next year to discover how we can help them make and achieve their big data goals! For more information on Entrigna please e-mail

Value Proposition of Business Decisions - A Systemic Perspective

In today’s competitive environment, critical & timely business decisions significantly impact business outcomes such as improving customer relationships, increasing revenues, optimizing cost & resources, improving performance, maximizing operational efficiencies and even saving lives. The ability to make business decisions intuitively & pertinently is heavily dependent upon availability & accessibility of business information & data. Every business event, such as a customer purchasing a product, yields business data. Such data, resulting from business applications, processes, transactions, operations, business-partnerships, competition etc. inherently contains valuable knowledge & business insights about customers, products, policies, systems, operations, competitors etc., that helps in making business decisions. Typical steps in deriving decisions involve collecting required data, analyzing data by applying intelligence-mining techniques & business rules, extracting interesting insights & new intelligence, understanding context & applicability of such information and finally arriving at decisions in terms of what business actions can be taken.

The value proposition of a business decision is measured in terms of its effectiveness in generating expected benefits while accomplishing one or more business goals & outcomes. There are many factors that affect the effectiveness or the value of a business decision. One of the key factors is the decision-action-latency which is defined as the total time taken, after business event(s) occurred, to collect required data, analyze data, extract new insights & intelligence, understand the applicability of such new information and finally arrive at actionable decisions. According to Dr. Richard Hackathorn, an eminent BI analyst & creator of Time-Value curves, the value of data required to make an actionable business decision degrades as time lapses by after pertinent business events have occurred. This is shown in the following Time-Value curve:

Click on the image to enlarge

The decision-action-latency in turn is cumulative of 1] ‘data-latency’ defined as time taken to collect and store the data, 2] ‘analysis-latency’ defined as time taken to analyze the data & extract new insights & new intelligence and 3] ‘decision-latency’ defined as time taken to understand the context & applicability of such new insights & intelligence and to arrive at decisions in terms of what business actions can be taken.

It has to be mentioned here that business decisions can be strategic or tactical in nature. In case of strategic decisions, the value or effectiveness is potentially realized even though the underlying data used to make decisions can be very old accumulated over longer periods of time. Essentially the slope of the Time-Value curve would be small per se with very gradual decrease in value over time. Typically, strategic decisions are based on mining large data comprising of historical observations collected from several business events over a period of time. A retail store making a decision about when to run beer sales is an example of a strategic decision. For example, a retail store after inferring from store sales data that men who purchase diapers over the weekend also tend to buy beer at the same time can make a strategic decision to capitalize on this information to put the beer near the diapers and run beer sales over the weekends.

In case of tactical decisions, the value or effectiveness of business decisions is very short-lived because underlying data/information is highly volatile and inherently contains time-sensitive intelligence reflecting upon the momentary business performance. Essentially, the slope of the Time-Value curve would be very high with the curve being extremely steep. Typically, a tactical decision pertains to a single business event or transaction and hence is based on data collected from a single event that gets compared to or correlated with associated/related data collected from most-recent related business events. Credit card fraud detection is an example of a tactical decision. For example, a credit card company after inferring that a credit card, being used to purchase an item in Chicago, was used thirty minutes earlier in a place somewhere across the globe, can make an immediate tactical decision to capitalize on this information to mark the transaction as fraud and place a hold on the card.

So how do companies ensure that value proposition of business decisions is retained & realized, regardless of strategic or tactical decisions?

Traditional Data Warehouses:

As IT evolved over the years, companies automated their operations to collect data for analysis & reporting. In early days, each business application would capture such data in its own 'Reporting' database. As more operational automation got implemented, such 'islands of information' became siloed and proliferated. Soon companies realized the analytical value if the data from all such siloed islands is collectively mined together & correlated. However, collecting & correlating data from all such siloed systems was a challenge due to the incompatibilities between systems and lack of easy ways for these systems to interact & interoperate. A need for an infrastructure to exchange, store and analyze the data so that a unified view of insights & intelligence across the enterprise can be created, was recognized and thus Traditional data warehouse evolved to fill this need. Traditional data warehouse organized information from different sources so that data can be effectively analyzed to generate interesting & meaningful reports. Such reports would provide key insights to make business decisions which would then lead to course-corrective actions. To help smoothen the decision making process, a broad range of tools were developed over the years such as Extract, transform, and load (ETL) utilities for moving data from the various data sources to common data warehouse, Data-mining & Analytical engines to perform complex analysis and Reporting tools & Digital Dashboards to provide management & business decision makers with easy-to-comprehend analysis results.

In spite of such attempts to automate the decision making process, the process task to analyze the data & extract new insights & intelligence, the task to understand the context & applicability of such new insights & intelligence and the final task to arrive at decisions in terms of what course-corrective business actions can be taken remained manual to large extent. As shown in the following diagram, the Time-Value curve in case of leveraging traditional data-warehouses had long time-latencies. As such, traditional data warehouses were predominantly leveraged for making strategic business decisions in supporting strategic goals such as reducing costs, increasing sales, improving profits, maximizing operational performance and fine-tuning operational efficiencies by mining & analyzing massive amounts of data collected from across the enterprise over a long period of time. However, traditional data warehousing has little tactical value since the data in it is generally quite stale and can be weeks or months old. There were attempts to incorporate new technologies to minimize time-latencies however such attempts could only be successful in reducing data-latency by further automating data capture processes while both analysis task and decision task remained mostly manual.

Click on the image to enlarge

Active or Real-Time Data Warehouses:

The need for a solution that satisfies both the strategic and the tactical goals of an enterprise resulted in the emergence of Active Data Warehouses or Real-Time Data Warehouses. Sometimes these are also referred to as Real-Time or Right-Time Business Intelligence (RTBI) systems. Active Data warehouses not only support the strategic functions of data warehousing for deriving intelligence and knowledge from past enterprise activity, but also provide real-time tactical support to drive enterprise actions that react immediately to events as they occur. The new breed of data warehouses are designed to reduce all three latencies as much as possible by revamping utility tools. The traditional ETL process involved downtime of the data warehouse while the loads were performed. As such they are characterized as being offline ETL facilities. However Active data warehouses needed online ETL facility that not only preserved historical strategic data but also provided current tactical data. The online ETL’s job is to create and maintain a synchronized copy of source data in active data warehouse while constantly keeping it up-to-date. Besides, Active data warehouses needed improved data-mining & analytical engines with ability to incorporate business rules and with flexibility to run analytical models that can consume & adapt to more recent data blended with historical data. In effect, Active data warehouses markedly reduced overall decision-action-latency and thereby tremendously increased the value-proposition of business decisions in meeting business goals as compared to that of traditional data warehouses. In addition, they offered flexibility in making tactical decisions as and when needed by an enterprise. The following diagram shows the Time-Value curve in case of leveraging Active/Real-Time Data Warehouses.

Click on the image to enlarge

Real-Time-Intelligence-Based Decision Systems:

Even though Active data warehouses reduced overall decision-action-latency & thereby increased the value-proposition of business decisions, they were still predominantly used in a traditional sense with a strategic intent albeit leveraging most recent/current data. They were never considered as systems that can act as pure providers of 'Real-Time-Intelligence-Based' decision services. Such Real-Time-Intelligence-Based decision systems would churn & process varying business operational & transactional data on a real-time basis, sense transitory business insights, predict business fore-sights and use such reasoning to make real-time decisions that can then effect immediate actions through business transactions & operations. Such decision system would agglomerate capabilities such as Machine Learning, Data Mining, Rules Processing, Complex Event Processing, Predictive Analytics, Operations Research type of Optimizations, Artificial Intelligence & other Intelligence-Generating Algorithmic techniques and would provide flexibility to mix & match such capabilities for more complex decision orchestrations. The breadth of decision frameworks is necessary because different business objectives require different analytical approaches. For example, a rules engine works great when recognizing a customer for a milestone. Likewise, event processing is well suited for identifying potential customer dis-service scenarios. Finally, optimization techniques work well when making decisions about which promotions to place in front of the customer.

Real-Time-Intelligence-Based decision system would process live-data from business events as they occur, combine the event data with other valuable data or other events data, extract intelligence from such data and derive a decision as to what action should be taken. Sometimes, knowledge of the event is sufficient information to derive an insight and take action. More often, additional data is needed to correlate & improve intelligence. One another key feature of such 'Real-Time-Intelligence-Based' decision systems would be to instantaneously learn, adapt and adjust decision models & business rules as soon as new data is fed-back from business events. Such spontaneous processing of business events data combined with instantaneous adaptation of decision models based on data fed-back, effectively eliminates 'data-latency', 'analysis-latency' and any latency incurred otherwise in re-engineering the decision models from ground-up. As such, the maximum tactical & transient value associated with business event data is fully preserved & exploited while effecting an immediate business action based on real-time business decision. The value proposition of such as system is depicted using a similar Time-Value curve where the latencies are in micro to milli seconds and any perceived loss in business value is almost nil.

Click on the image to enlarge

Entrigna’s proprietary product RTES falls under ‘Real-Time-Intelligence-Based’ Decision System. For more information, refer to the blog titled ‘From Real-Time Insights To Actionable Decisions - The Road Entrigna Paves’.

Hope you found this blog informative.

The Relationship Between CRM and Real Time Marketing Offers

An exciting marketing capability is gaining momentum that engages customers "in the moment" based on real time events through marketing offer optimization.  This goes well beyond the traditional approaches of personalization by leveraging optimization and predictive analytic techniques rather than static rules.  As a customer interacts with a merchant (e.g., enters a store, visits a website), the merchant intelligently engages with the customer with real time offers and communication to significant improve conversion of a sales opportunity.

When marketers and technology professionals are exploring how to implement to real time offers, there is often confusion on what role CRM plays.  CRM is not sufficient to deliver real time offers.  Rather, CRM is an important partner to real time offers with each process helping the other.

The ability to execute real time offers requires a combination of complex capabilities including data virtualization, eventing, decision frameworks and decision integration into business operations (e.g., website, mobile app, trigger email).  The process is highlighted in the figure below and more details can be found at How to Execute Real Time Decisions.

Entrigna Blog - Real Time Decisions Process

Role of CRM in Real Time Offers

As you can see from the process flow above, CRM plays two important roles with real time offers.

1) CRM provides an important source of information into the virtualized data store used for real time offers.  Data elements like customer profile, transaction history and contact history coupled with the "event" data can be powerful inputs into the decision frameworks (e.g., optimization, rules engine, complex event processing).

2) Real time customer engagement creates new data elements.  Each customer "event" with the associated "decision" and ultimately the customer response to the "decision" (e.g., did he or she review, discard or convert the offer) provide tremendous insight into the customer about what his or her interests are and what is motivating.

This data should be stored in CRM to enhance any other business process that consumes CRM data.  For example, the target list generation for email marketing campaigns can be greatly informed by the real time offer interaction history.  Of course as outlined in 1), this data should be passed along back into the virtualized data store to help improve the next real time offer decision.  This essentially creates a virtual cycle of improvement with marketing effectiveness.

Final Thoughts

CRM does not deliver real time offers nor is real time offers a replacement for CRM.  You can have one without the other but having both work together seamlessly can create tremendous value and synergy.


The Importance of Loyalty Programs to Retail Omnichannel

Retailers are playing catch up to consumers with omnichannel.  Many shoppers are already behaving in a seamless manner across channels either through webrooming (research online and buy in store), showrooming (research in store and buy online) or using their mobile device to do both while in store.  In other words, consumers are already embracing omnichannel.

Loyalty Programs Role in Omnichannel

For retailers, data limitations and technology complexities have slowed their ability to execute omnichannel.  The single biggest challenge to implementation is building a customer centric view that associates all activities of a shopper (e.g., transactions, web searches, store visits, mobile app usage).

The data simply doesn't exist for most retailers to build this customer centric view and the challenge remains how to "mark" the shopper as he or she interacts with them.  This goes beyond a customer just leaving a footprint.  It requires identifying the shopper while he or she is interacting with the retailer in real time in order to achieve optimal impact of omnichannel.

Although there is hype about potential new technology innovations, for the foreseeable future, the best answer to build this customer centric view is through a loyalty program.  The loyalty program provides the unique identifier to link customer activity to a shopper.

Design of Loyalty Programs for Omnichannel

In order to best deliver omnichannel, a new way of thinking about loyalty program design is required.  Traditionally, loyalty programs reward shoppers for transactions - e.g., "buy 5 and get 1 free".  The reward motivates shoppers to purchase more than he or she would have done otherwise.  This is the traditional "value of loyalty" business case for loyalty programs.

With omnichannel, loyalty program design must reward shoppers for identifying themselves while interacting with the retailer (e.g., "checking in" when entering a store, saving online research to your cart).   Essentially, the retailer puts the burden of data capture on the shopper but if the loyalty rewards are designed appropriately, he or she will do so willingly.

The business case for the omnichannel loyalty program design is built upon optimizing customer engagement in real time.  Because the retailer knows who the shopper is and his or her prior interactions, the retailer will improve closing sales leads (e.g., webrooming, showrooming), more effectively cross / upsell and entice new transactions.

It's worth noting that traditional loyalty program design can work synergistically with an omnichannel focused program.  In fact, if a traditional loyalty program exists, it accelerates implementation of the omnichannel component because core loyalty infrastructure already exists.

However, in either case, new capabilities, especially technology related, are needed to enable an omnichannel focused loyalty program (e.g., rewarding geo-location).  As an alternative to building proprietary technology, companies like Bckstgr provide these capabilities in a turn key fashion for retailers.

Once the loyalty program in place, the retailer has the platform to effectively execute omnichannel.  This coupled with real time customer engagement capabilities provides the retailer the ability to achieve the full benefits associated with omnichannel - that is, improve closing sales leads, more effectively cross / upsell and entice new transactions.

Three Customer Buying Behaviors Omnichannel Should Solve

It seems like omnichannel is discussed everywhere in retail these days.  However, even with all this talk, there isn't significant content about the specific customer buying behaviors that omnichannel should solve.

So, as a way to get the discussion going on, we'll share three customer buying behaviors that omnichannel should focus on and by doing so, should generate incremental revenue for the merchant.

Definition of Omnichannel

Before we share the buying behaviors, let's ground ourselves with a definition.  Although there are several floating around, we define omnichannel as:

Omnichannel is a customer focused approach that works alongside the customer as he or she interacts with the merchant.  With omnichannel, when a customer engages in a particular channel, his or her prior interactions in other channels are known and that knowledge is used to optimize the interaction in the current channel.  Omnichannel should also leverage knowledge of one other huge "channel" - what's going on in the outside world.  This includes information like location, weather, traffic and national / local events.

Three Customer Buying Behaviors Omnichannel Should Solve

1. Buy online, pick up in store

Ok, this is definitely not new.  Many merchants, like The Container Store, have implemented this customer solution as it delivers real value to the shopper.  The shopper gains the efficiency of the website purchase process and same day fulfillment of an in store visit while minimizing the time spent at the store.  Implementation requires in store operational changes; namely, establish a process to receive these orders from online channel, have staff available to package order, and create a customer service line for customers to pick the orders up.  In addition, the merchant must work through supply chain and inventory management requirements in order to ensure items bought online are available at the local store.

2. Right offer, right time, right place

As the shopper fluidly moves back and forth and across channels, he or she seeks information during the buying process.  This includes marketing offers, as a way to identify saving opportunities, but it must be relevant and timely.

Ideally, a merchant would want a 360 degree view of all the shopper activity and be able to uniquely identify the shopper at each and every channel interaction.  This would enable the merchant to best select which offer to present and ensure consistency across channels.

However, the 360 degree view will not be available in all cases and for every merchant.  Fortunately, it's not needed to optimize marketing offers.  Classification algorithms coupled with real time decision capabilities can take whatever data is known at the channel interaction (e.g., customer transaction history, mobile identifier, IP address, profile data) and assign the customer to a micro segment.  Optimization algorithms will decide which offers are best for each micro segment. Again, knowing exactly who the customer isn't necessary because segment specific offers will perform far better than a "one size fits all" approach.

 3. Research one in channel, buy in another

More frequently, shoppers research items online and then go to a store to make the actual purchase - i.e., webrooming.   However, this behavior also works in the opposite direction where a shopper may touch, feel and try on a product in a store and then go home to shop online for the best deal - i.e., showrooming.

In the online to offline scenario, the merchant would ideally want to identify the customer as he or she walks into the store and greet the shopper with a message - e.g., "the product you've researched online is sold in aisle 10, here's 5% off coupon and you should consider purchasing this additional item with it".

To execute tactics like this, the merchant must possess exact knowledge and tracking of the customer across channels.  The Loyalty ID of a merchant's loyalty program is the best way to enable this.  Of course, the merchant must motivate the shopper through the loyalty program to "mark" themselves along his or her buying process.  In addition, real time decision capabilities are necessary in order to know when a shopper is interacting with a channel and intelligently decide the optimal action to take based on prior interactions.


What do you think?  Are there other customer buying behaviors omnichannel should solve?

What Retailers Can Learn From Airlines About Omnichannel

The financial troubles of airlines are well documented. So what can retailers learn from a historically struggling industry about how to deliver omnichannel, an innovation that has promise to drive incremental revenue and deliver a fantastic customer experience?  Well, actually quite a lot.

Airline Omnichannel Requirements

Airlines have been executing omnichannel long before the omnichannel buzz word existed.  Fulfillment of an airline's product (getting a person from point A to point B) is arguably the most complex, high touch consumer experiences for relatively frequent purchases.  After a ticket is purchased, a traveler could have 10 or more customer service interactions before the final destination is reached on issues like

  • Seat assignments
  • Upgrades
  • Special requests (e.g., meals, wheelchair)
  • Changing travel plans
  • Flight delays and cancellations
  • Buying ancillary products
  • Flight check in
  • Boarding plane
  • In flight experience
  • Lost baggage and other baggage claims.

It gets even more complicated when considering the number of channels which must be in synch with one another, including:

  • Distributors (i.e., travel agencies like American Express and Expedia)
  • Website
  • Mobile app
  • Call center
  • Email and text for proactive communications
  • Airport kiosks
  • Airport customer service reps
  • Flight attendants.

Each of these channels perform some combination of selling air travel, selling ancillary products and handling a myriad of customer service issues.

Further, a customer often interacts with multiple channels for a specific need.  For example, if a flight is delayed, the traveler may visit the website, check the mobile app, contact the call center, receive an email, look at the airport kiosk and speak with a gate agent all for the same issue.

Putting aside everyone's travel horror stories (and we all have them), airlines do a lot of things right in this challenging environment.

Airline Lessons for Retailer Omnichannel

There are many lessons a retailer can learn from airlines about omnichannel.  We'll share our four biggest insights.

1. Effective use of unique identifiers

Like other retail sectors, airlines have a unique transaction ID for purchased tickets called a passenger name record or PNR.  The PNR ensures linkage of customer service interactions across the channels and through the travel experience. In addition, loyalty frequent flier numbers connect transactions over time to customers.  The loyalty ID has been used effectively for personalization and customer life cycle management.  Leveraging these identifiers, especially loyalty IDs, can be an effective enabler to deliver omnichannel for retailers.  For example, loyalty IDs can be used to link customer research of products in digital channels with a purchase transaction in a brick & mortar store.

2. "Intelligent" middleware investments to speed cross channel deployment

Most companies have legacy technology systems that are difficult to enhance.   Airlines are no different and some have learned the hard way of not investing in "intelligent" middleware that can act as channel "brains".  Without channel "brains", new cross-channel features become a different project in each channel.  This drives significant costs and elongates timelines.  Since investment dollars are constrained, often functionality is implemented in one channel but not another.  This leads to missed revenue opportunities and poor customer experiences.  Instead with "intelligent" middleware, new functionality can be launched much more like a single deployment, enabled all at once.

3. Leveraging events for insights to trigger action

In several contexts, airlines have leveraged the occurrence of an event to take action in real time.  Events are essentially customer digital footprints that can be captured at every touch point and is usually initiated by a customer action; examples include,

  • Customer visits website
  • Customer uses mobile app
  • Customer visits store
  • Customer purchases product
  • Customer logs a complaint
  • Retailer geo-locates customer.

By applying predictive analytics and correlation techniques on events, intelligence can be extracted in real time that can be used for instantaneous decision making and subsequent actions.  For example, airlines might take several immediate actions if a flight delay causes a missed connecting flight - rebook the traveler on next available flight, ensure the bags of traveler are transferred accordingly, offer a food and hotel compensation voucher to the traveler.

4. Data management and data virtualization

Data management and access are critical to the execution of omnichannel.  Airlines have many legacy systems with each acting as a source of truth for different elements of essential data fields.  It is a natural tendency to undergo a large scale enterprise data management project to bring all of these sources of information together into one system (e.g., CRM or MDM ).  However, going down this path will significantly delay omnichannel delivery.  Instead, airlines that have enabled virtualized data stores fed from the different systems have improved time to market for critical omnichannel functionality like real time decisions and offer optimization.

Omnichannel is relatively new for retail and the sector is still trying define the opportunity.  But retail has the opportunity to learn from the lessons of airlines, an industry that has been executing omnichannel for decades.

Tell us what you think?  We'd like to hear.

The Value of Big Data - Defining It Once and For All

Big Data is a huge buzz word.  Like so many "next big things", there is often misunderstanding about what it is and it's value.  This leaves many companies and industries with an open and lingering question about what they should be doing with Big Data.

However, unlike other "next big things", Big Data is not overhyped in its possibilities.  It truly opens up a tremendous frontier of business intelligence.  Unfortunately, given the misunderstandings, it is not always clear how to take advantage of it.

What is Big Data

Before answering the value of Big Data, it's worth a quick summary of what it is.  Big Data is generally thought of as the "3 Vs"; i.e., data that has

  • Volume (terabytes & beyond)
  • Velocity (streaming real time)
  • Variation (structured & unstructured).

Twitter is a great example.  It's data is very large, generated real time and unstructured (admittedly hashtags and handles provide some structure but no where near a traditional relational database).

It's also important to understand that not all "3 Vs" need to be present.  The concept of Big Data is relevant if Big Data processing technologies are needed to unlock the value of a company's data or take advantage of external data.

At a high level, Big Data processing technologies offer two key capabilities.  First, it is a novel way to store data that is especially well suited for any of the "3 Vs" with the focus of providing fast access in terms of queries and updates.  Second, it provides Map / Reduce functionality that identifies relationships between unstructured data elements and helps in building ‘keys’ or ‘indices’ that are needed for fast access and cross-data relationships.

Big Data's "4th V" - Value

It is critical to emphasize that processing Big Data is not an objective unto itself and value is not created by just implementing Big Data processing and storage.  Big Data must be directly used to help improve core objectives such as

  • Optimize offers (marketing offers, cross/up sell, product configuration)
  • Improve customer service (omnichannel, surprise & delight, recovery)
  • Predict and prevent customer churn
  • Improve inventory management / product forecasting.

Big Data can be leveraged to accomplish this and create value if used an input into any of the following business intelligence processes within your organization:

  1. Collection and maintenance of enterprise data assets
  2. Batch analytics and static decisions
  3. Real time analytics and decisions

Each process is described in more detail below as how Big Data can create value.  We've shared a Marketing related example for each but obviously there are many other operational processes that leverage business intelligence and can benefit from Big Data.

Enterprise Data Assets

By processing Big Data, intelligence can be extracted to build data assets over time which plug into existing enterprise data management solutions like customer relationship management (CRM).  These new data elements help improve performance for any process that leverages CRM data (e.g., marketing campaign management).  It is analogous to the objective of building marketing databases that contain customer profile and email address.  However, with Big Data, the profile information that is built contains detailed digital activity and social activity, sentiment and influence.  By knowing these attributes, marketing campaign target lists, channel selection and offer/content can all be improved.

Batch Analytics and Static Decisions

Batch analytics and static decisions are traditionally how companies make decisions.  Historical data is compiled and analysis is done to inform decisions like marketing mix / planning decisions.  However, with Big Data, a company can now bring in granular data about digital interactions that could be terabytes in size.  This granular, cross channel data enables much more sophisticated and accurate marketing optimization models leading to more effective marketing campaigns and resource allocation.

Real Time Analytics and Decisions

Real time decisions is the ability to intelligently engage customers and improve outcomes based on real time events. It requires processing events as they occur, combining the event with other valuable data, gaining intelligence from the data and deciding on an action to improve the customer interaction.  All of this done in real time.  Big Data opens up new data sources like granular data on digital interactions and external data like weather to feed algorithms that optimize which marketing offer to present on a customer by customer basis.

In the end, Big Data is similar to a lot of innovations.  It is a new and innovative way to improve solutions for existing core objectives like marketing offer optimization.  Perhaps, Big Data's value is not mysterious after all.

What do you think?  We'd love to hear your thoughts.

How to Deliver Omnichannel Real Time Decisions

Omnichannel execution has become imperative for retailers as consumers increasingly combine their shopping activity across online, brick & mortar and mobile channels.

With omnichannel, when a customer engages in any channel, the retailer is aware of their prior interactions in other channels and uses that knowledge to optimize the interaction in the current channel.

A well executed omnichannel strategy requires real time decision capabilities - the ability to process events as they occur, combine the event with other valuable data, gain intelligence from the data and decide on an action to improve the customer interaction.  All of this done in real time.

Real Time Decision Process


Entrigna Blog - Omnichannel Strategy - Real Time Decisions Process

Real time decisions process

A Real Time Decision Process can be viewed as five distinct steps.


Execution of omnichannel real time decisions is triggered by an event.  An event can be any number of things but is usually initiated by a customer action; examples include,

  • Customer visits website
  • Customer uses mobile app
  • Customer visits store
  • Customer calls contact center
  • Retailer geo-locates customer.

The event presents an opportunity to engage and must be captured in order to initiate the real time decision process.

Virtualized Data

Sometimes, knowledge of the event is sufficient information to take action.  More often, additional data must be leveraged to improve intelligence.  Many different types of information is potentially needed including

  • Customer profile
  • Transaction/sales history
  • Channel interaction history
  • Social activity history
  • External data like weather, traffic, national/local events.

Data virtualization is the technical process that integrates these disparate data sources in real time into a usable format.


Intelligence must be derived based on the event and virtualized data to determine the optimal action.  Predictive analytic capabilities are necessary and a wide range of decision frameworks must be available, including

  • Rules engine
  • Complex event processing
  • Classification / clustering
  • Optimization
  • Machine learning
  • Artificial intelligence.

The breadth of decision frameworks is necessary because different business objectives require different analytical approaches.  For example, a rules engine works great when recognizing a customer for a milestone.  Likewise, event processing is well suited for identifying potential customer dis-service scenarios.  Finally, optimization techniques work well when making decisions about which promotions to place in front of the customer.


Once determined, the decision must be integrated with a customer facing channel or business process in order to impact the outcome in real time.  The types of actions should be related to achieving core objectives such as

  • Offer optimization
  • Winning/completing the sale
  • Customer lifecycle milestones
  • Customer service (cross channel coordination, surprise & delight, recovery).

Feedback takes two forms.  First, the outcome of the action/decision is fed back to the algorithms used in the Intelligence step.  This can be done in real time for online learning algorithms or stored and leveraged in a batch mode for off line learning algorithms.

Second, data and insight from the Real Time Decision Process is fed into enterprise data management processes like customer relationship management (CRM) and customer data management (CDM).

It's important to note that real time decisions are related but separate to enterprise data management processes in which both rely on each other as inputs to one another.  As a consequence, implementation of real time decisions does not require a multi-year, multi-million dollar enterprise data management project to be complete.

Execution of real time decisions is a complex set of capabilities that include predictive analytics, data virtualization and real time decision software.  However, when done well, real time decisions delivers the full promise of omnichannel benefits.

What do you think?  We'd love to hear your thoughts.

Importance of Real Time Decisions in Omnichannel Marketing

With the growing trend of consumers combining their shopping activity across online, brick & mortar and mobile channels, significant discussion has arisen in the retail industry about the need for omnichannel marketing.

What is Omnichannel

While perhaps a single definition doesn't yet exist, one way of thinking about omnichannel marketing is that it is centered around the customer.  Historically, multi-channel strategies have tried to ensure brand consistency across channels and optimize performance in each channel based on respective strengths.  Multi-channel is more of an inward focused approach.

With omnichannel, when a customer engages in a particular channel, their prior interactions in other channels is known and that knowledge is used to optimize the interaction in the current channel.  Omnichannel is a more of a customer focused approach that works alongside the customer as they interact with the retailer.

Taking this one step further, omnichannel should also leverage knowledge of one other huge "channel" - what's going on in the outside world.  This includes information like location, weather, traffic and national / local events which can have as much impact on optimizing the customer interaction as anything else.

Entrigna Blog - Omnichannel Strategy & Marketing

High level concept of omnichannel

Omnichannel Execution

In order to create maximum value, omnichannel strategies must be directly applied to improving performance of core business objectives.  Otherwise, retailers run the risk of implementing a very complex initiative without an end goal in sight. For the marketer, a well executed omnichannel strategy can improve

  • Offer optimization
  • Conversions / transactions of sales leads
  • Customer lifecycle management - recognition of events & milestones
  • Customer service - cross channel coordination, surprise & delight tactics, recovery.

Omnichannel implementation requires many capabilities including enterprise data management solutions, predictive analytics and real time decision software.  It may also require changes to organizational design and operational processes.

Role of Real Time Decisions in Omnichannel

Real time decisions is critical for a well executed omnichannel strategy.  Omnichannel leverages knowledge of prior interactions across channels to optimize the interaction in the current channel.  In most cases, the opportunity for the retailer to interact with the customer in the current channel is "now" or real time.

Execution of omnichannel and real time decisions is triggered by an event.  An event can be any number of things but is usually initiated by a customer action; examples include

  • Customer visits website
  • Customer visits brick & mortar store
  • Customer engages in a social channel
  • Retailer determines geo-location of customer.

A retailer must be able to capture this event and leverage additional data like prior interactions across channels in order to derive intelligence.  A wide range of tools from sophisticated predictive analytics like machine learning algorithms to more straight forward business rules engines should be leveraged to derive intelligence.  From this intelligence, a decision is determined for the optimal action.  The decision must be integrated into a customer facing channel or business process in order to impact the outcome in real time.

The entire process, outlined above, is called real time decisions.   As you can see, it is central to enabling the knowledge and coordination of actions across channels.  In the next post, we'll talk more about the execution of omnichannel real time decisions.

In the meantime, what do you think?  We'd love to hear your thoughts.