Mind the data gap

Heiko Claussen at AspenTech asks whether industrial AI is the key to solving the data scientist shortage

 

Organisations across the UK and the US are funnelling record levels of investment into artificial intelligence as they look to improve operational performance, uncover fresh revenue avenues, and remain competitive. 

 

Market data underscores this trend, according to new findings from Ernst & Young LLP (EY US), 95% of senior executives from the US say their organisations are already investing in AI, and those investments are poised to increase significantly. While today 16 percent of companies are investing US$10 million or more into the technology, that number is to nearly double, to 30 percent, by next year. In the UK, meanwhile, a recent poll by Deloitte revealed that about 75% of business leaders expect their organisations to boost AI spending in the next fiscal year.

 

Yet, as investment surges, a major obstacle has arisen: the shortage of skilled AI professionals. The Pluralsight AI skills report for 2024 found that 95% of executives believe their AI initiatives will fail without staff proficient in AI tools.

 

When you consider that more than half (53%) of organisations polled for Software One’s Cloud Skills report believe that they currently have inadequate skills to make use of AI, that’s a significant problem, to say nothing of the fact that half of all organisations are struggling to find employees that are AI-skilled.

 

Asset-intensive sectors, including energy, chemicals, and manufacturing, are feeling the pinch especially hard. To make effective use of AI, companies need teams that understand algorithms, safety regulations, industrial equipment, and specific production processes, and not only offer generic coding skills. 

 

Encouragingly, over 7,600 students in the UK have enrolled in AI and data science courses but businesses need a more immediate solution that both helps existing workers and attracts tomorrow’s talent.

 

Finding a way forward

Industrial AI offers one possible way to address the current shortfall. It combines domain expertise like first principles and engineering knowledge with data, resulting in robust, interpretable and data efficient solutions. Data cleaning, real-time monitoring and analytics can often be automated, freeing time and allowing efficient scaling of custom fit solutions.

 

Where historically, experts could only afford to model and optimise large equipment, Industrial AI allows efficient scaling and adaptation to a much broader set of use cases thus driving significant optimisation of plant performance.

 

Also, these tools incorporate sector-specific knowledge, enabling more precise insights that operations staff can interpret and act on confidently.

 

In many workplaces, small data science teams are overwhelmed by the volume of raw information. Industrial AI platforms can streamline key steps, such as data quality checks and anomaly detection, and generate targeted alerts whenever unusual patterns emerge. One data scientist can then supervise multiple workflows rather than coding each one, and plant operators can address issues quickly without requiring deep algorithmic knowledge.

 

Experienced engineers and operators also benefit when industrial AI models deliver easily accessible recommendations. A large chemicals producer, for instance, could automatically screen thousands of temperature, pressure, and flow signals, detect early maintenance issues, and provide a range of recommended actions.

 

That proactive approach helps avoid costly downtime, it could, for example, make it easier for utilities to manage fluctuating renewable sources, proactively spot grid issues before they lead to outages, and maintain smooth operation of vital infrastructure or, a manufacturer could rely on adaptive learning models to update demand forecasts, freeing data scientists to review changes and to keep the system accurate.

 

The backbone of a strategic approach

Companies benefit on several fronts. They lower expenses related to unplanned downtime or resource mismanagement and improve production quality. They also provide a more dynamic environment for experienced data specialists, who can shift their attention toward advanced model design or specialised optimisation projects. 

 

That is, an Industrial AI based toolchain allows efficient creation of models by even non-experts that fulfil critical constraints like mass balances. Experienced data specialists can focus their time on the difficult edge cases and market differentiators like modelling proprietary processes.

 

The creation of these custom models is streamlined through the utilisation of an OT data fabric that overcomes data silos and provides contextualised data to the expert. After the custom model is build, it can easily be integrated into the overall plant representation and be jointly optimised to maximise the enterprise efficiency. 

 

The use of an OT data fabric allows operators, maintenance personnel and data specialists to work from the same dashboards and metrics. Because the industrial AI engine embeds domain knowledge into its predictions, there is less confusion around where recommendations come from. Everyone gains confidence in the system, and the technology becomes a practical part of the daily workflow. This is key for adoption of the system and therefore leveraging its operational efficiency improvements.

 

These changes can also help companies attract talent. Using AI effectively across the enterprise can help attract new hires, and potential recruits notice which employers have integrated well-designed digital tools that make a difference in real-world settings. Even individuals without extensive coding backgrounds can find meaningful roles maintaining, interpreting or extending these AI systems in collaboration with their more specialised colleagues.

 

At the same time, businesses that have struggled to hire full-fledged data scientists can make better use of existing teams by automating repetitive tasks. 

 

Looking to the future 

While industrial AI addresses part of the data scientist shortage, organisations must still look for ways to expand their pool of qualified personnel. These efforts can include forming partnerships with universities to provide real-world cases or sponsoring programs that combine data science with business knowledge. Building internal training paths or supporting certification courses can also help current employees shift into new AI-oriented positions. 

 

There is no single fix for the lack of data scientists, and competition for top candidates remains fierce. However, industrial AI can help businesses cope more efficiently with their immediate needs. It reduces the day-to-day strain on limited data teams, distributes analytical functions more widely among operations staff and allows experts to devote time to crucial priorities. 

 

Over the long run, that helps foster an organisational culture where data-driven decision-making feels natural. The short-term benefit is that pressing concerns about talent are addressed in a practical way. Industries that must juggle safety constraints, complex equipment and fluid market conditions stand to gain the most from this approach. 

 

By rolling out AI applications that require less manual upkeep, companies can bridge the expertise gap and unlock new sources of productivity. As the future workforce gradually fills data science roles, today’s teams can maintain momentum.

 

The result is a more balanced workload, better operational insight and a working environment that encourages AI-focused innovation while making the best use of expert data scientists already on board.

 

---

 

Heiko Claussen is Co-Chief Technology Officer at AspenTech

 

Main image courtesy of iStockPhoto.com and gorodenkoff