Understanding innovations in data science

Yali Sassoon at Snowplow explains why business leaders need to understand evolving data innovations including behavioural data and data creation

 

Data is an invaluable commodity that is at the core of every business decision being made today. In fact, recent research estimates that the behaviour analytics market will exceed USD 2.9 billion by 2028, from USD 414.5 million in 2021.

 

Organisations that embrace innovations in data can unleash their full potential to create new value and transform the customer experience.

 

Two major innovations in data science that are quickly gaining traction are behavioural data and data creation, which, when combined, can deliver significant leaps forward in artificial intelligence (AI), Machine learning (ML) and business intelligence use cases, as well as solving many of today’s privacy challenges.

 

Outside of data science circles, many business leaders won’t be familiar with these terms, let alone the potential for them to improve their organisations radically. So what exactly are behavioural data and data creation? What are the differences between them? Why should the C-Suite be aware of them? And how can they be utilised?

 

What is behavioural data?

Behavioural data describes, in granular detail what people do second- by- second and minute- by- minute. It describes the decisions they make and the context in which those decisions are made.

 

For this reason, it is very powerful and allows organisations to understand what drives a person’s decision making and creates the opportunity to predict and influence future decisions. Although highly valuable, many companies face the challenge of gaining deep insights from behavioural data itself.

 

For example, many organisations face challenges deploying advanced behavioural data use cases, as much of the time of their data teams is taken up by ad-hoc requests. Also, they often find the data they’re working with is unreliable and requires significant preparation.

 

Another often encountered challenge is remaining compliant with privacy regulations. This arises because data collected across different jurisdictions, under different circumstances, together with a lack of ownership of the data, makes it difficult to ensure compliance.

 

However, when behavioural data is used in combination with data creation, it’s a different story, as these challenges can be overcome.

 

Data creation – the gold standard for organisational data

Data creation is the process of deliberately creating high-quality behavioural data in real-time, rather than extracting "exhaust" - data that backs different applications via APIs and database extracts. The main difference is that when working with "exhaust" data (i.e. extracted data), organisations are repurposing data that was designed for one purpose (to power an application, such as a CRM system) for another purpose (to power AI or ML), which means that a lot of work is required to repurpose this data for analytics or AI. Even when that work is completed, it may not be of sufficient quality to deliver good results.

 

These data sets have different field types, granularity, quality, and completeness and need significant wrangling before they can be used. While this data is plentiful, it’s limited in that each source arrives in a structure that requires validation to become potentially useful.

 

Not only is this technically challenging, but it also introduces a high degree of variability to the data over time. Data teams often find it time-consuming, unproductive, and difficult to work with.

 

In contrast, when organisations deliberately create data for AI or analytics, they start with the requirement they need. For example if they want to detect fraudulent credit card applications, they figure out what features they think are indicative of fraud (e.g., users copying and pasting multiple social security numbers) and then go about deliberately creating the data that describes that behaviour (e.g., a dataset describing how people fill out their social security number, possibly keystroke by keystroke)

 

The result is a dataset with the required granularity and ideal structure that makes it easy for a data scientist to perform the analysis or develop the required feature with minimal data preparation.

 

In addition, data creation is a far more compliant approach to handling organisational data. This is because organisations can create metadata with the data that describes exactly how the data can and cannot be used, especially for downstream processing. (In the above example the company would update its website terms and conditions to indicate that it is collecting data on how people fill in the form for the sake of fraud prevention.) When data is repurposed for AI, it can be unclear what purposes that data can and cannot be put to.

 

A growing number of leading organisations are embracing data creation as an alternative to current approaches for dealing with data. This shift results in more intelligible, flexible, compliant, real-time, and trusted data.

 

Behavioural data - the high-performance fuel for AI

Behavioural data is the high-performance fuel for AI. For example, in retail environments, it provides a more explanatory and predictive alternative to demographic and transactional data.

 

While transactional and demographic data are useful, behavioural data delivers much more powerful insights. It can show not only what a customer buys but also how they buy, other decisions they make and how they make those decisions - including all the context for those decisions.

 

Behavioural data is extremely explanatory and predictive and delivers a deep understanding of customers that allows retailers to drive personalised digital experiences. As a result, it can also be used to power real-time product recommendations, dynamic pricing, demand prediction, multi-channel attribution and supply chain monitoring.

 

Behavioural data, in combination with data creation, has applications across every sector. Looking beyond its retail applications, other examples of its applications can be seen in the healthcare and media sectors. It can offer providers complete data ownership in healthcare, enabling end-to-end auditability for HIPAA, GDPR, and CCPA compliance.

 

It can also build an understanding and predict patient engagement, as well as drivers of churn or retention. For media organisations, it can provide real-time streams of behavioural data to the data cloud and for in-session engagement. It can also identify subscription intent signals for in-session personalised paywalls, optimise ad placement and protect subscription income.

 

Purpose-built for privacy

Data privacy is a growing challenge for all organisations. At present, organisations are grappling with balancing personalisation efforts against data privacy rules. At the same time, consumers are increasingly exercising their right to privacy, especially as regulations tighten.

 

Many organisations still rely on third-party digital analytics solutions, yet there is an urgent need for clarity around which data can be used, and when. Once understood, privacy requirements need to be balanced against using data effectively to deliver the best possible customer experience.

 

The most effective way to address data privacy is to adopt Data Creation, which, as mentioned earlier, allows organisations to create metadata with the data that governs its use. This helps ensure that the data is handled correctly in any downstream processing.

 

Paradoxically, data extraction means you have to have a separate process to audit how to use the data, and there is no easy way to then incorporate that information into the processing of that data.

 

A new data revolution

There’s no doubt that behavioural data that incorporates data creation provides rich, contextual, and granular detail and insights. Organisations using this data to fuel advanced analytics, AI models, and applications are getting further out in front—thanks to powerful personalisation, product innovation, marketing optimisation, and other advances.

 

Now is the time for business leaders to ensure their organisations are benefiting from this new data revolution.

 

---

 

Yali Sassoon is Co-Founder and Chief Strategy Officer at Snowplow

 

Main image courtesy of iStockPhoto.com