'Double busy' data scientists fear work is going to waste

Data professionals believe their work is far more important since the pandemic struck and are even able to get more done, but cite over 20 barriers which are standing in the way of effective outcomes, including a perceived lack of top-level support and funding, and the eternal issue of dirty data.

The driving force behind the rise of the data scientist has been the acceleration of digital transformation, according to a new study by analytics company SAS, with more than half (53%) saying their work is of greater importance, 38% saying it is equally important and just 9% saying it is less important.

Meanwhile, 45% say they get more work done, 28% about the same, and 26% say they get less work done.

A typical data science project involves a variety of activities, almost always beginning with preparing data. One notable finding in the survey is that data professionals are spending much more time (58%) gathering, exploring, managing and cleaning data, than they would like to (42%). And that is in contrast with an average of 11% of their time spent creating computer models, compared with their ideal of 24%.

The report states: “The fact is that, regardless of your level in the organisation, data management will probably take a large share of your time, even with the development of low code/no code tools and AI and machine learning algorithms being written for it.

“The likely reason is that the data you have and how you decide what’s relevant is probably specific to your industry and organisation. As is the case for how you approach your model-building, knowing which data is relevant and why has a lot to do with the issues you are trying to solve.”

When it comes to end result, nearly three-quarters (73%) of respondents were satisfied with the outcomes from analytical projects, yet less than half (46%) were satisfied with their company’s use of analytics and model deployment.

This suggests there is a problem with how analytical insights are used by organisations to inform decision-making.

This is backed by the finding that 42% of those surveyed said data science results were not used by business decision-makers, while even more (46%) believe they are being hampered by company politics and a lack of management/financial support for their team.

Dirty data, cited by 42% of respondents, is also a major challenge, while other issues include explaining data science to others (35%), a lack of data science talent in the organisation (34%), the need to co-ordinate with IT (33%), a lack of clear direction (32%), an inability to integrate findings into the organisation’s decision-making process (32%), difficult access to data (29%) and difficulties in deployment/scoring (26%).

Interestingly, only 10% saw privacy issues as a problem, suggesting the vast majority have no issue with current data protection legislation.

The survey also highlighted some specific skills gaps. Less than a third of the respondents reported having advanced or expert proficiency in program-heavy skills, such as cloud management and database administration. This is an issue given that use of cloud services is up significantly, with 94% saying they experienced the same or greater use of cloud since Covid-19 struck.

SAS UK & Ireland head of data science Iain Brown said: “There have clearly been more demands placed on data scientists as the pandemic has accelerated digital transformation projects that many organisations were planning anyway.

“A major source of frustration is finding a way for organisations to implement the insights from analytics projects and use them in their decision-making, which means giving data scientists a seat at the boardroom table might be a way forward.

“Linked to this, we found concerns around support for data science teams and a lack of talent, which has been an issue for some time with demand outstripping supply. Organisations must realise that investing in a team of data scientists with complementary skills could reap huge value for the business, so the cost of hiring needs to consider the return on that investment as we move to significantly more digital and AI-driven business processes.”