Skip to main content

The hunt for data scientists and evolution in analytics

Searching for the most effective ways to deal with data has long been a preoccupation of the business world. Storage, security, performance, and multiple other concerns have all come under scrutiny. However, it’s the fruits of analysis that really make business leaders sit up and listen. Good data analytics yields insight, which informs strategy, which impacts the bottom line. In other words, there’s a direct link between effective data analytics and business success.

Unsurprisingly, as data volumes explode so does the corporate appetite for skilled professionals who can handle, cleanse, prepare, and analyse data. Enter the data scientist. These specialists have long been considered the sole gatekeepers of the vast oceans of valuable insight represented by data. But should this be the case?

Where are all the data scientists?

The global shortage of data scientists has been heavily reported over the last couple of years, as companies of every size look for these experienced data superstars. According to the Harvard Business Review, ‘data scientist’ is the sexiest job of the 21st Century (opens in new tab), despite the term being coined less than ten years ago. These technical experts are now held in the highest regard in the business community, and little wonder, given the huge potential that lies in the pools of data that they curate and interrogate.

Twitter recently announced it will build its first data science team outside of the US, and a couple of months ago both Sainsbury’s and Channel 4 made well-publicised calls for data scientists to join their companies (opens in new tab). Amongst a laundry list of requirements, the supermarket chain was specifically hunting for candidates with expertise in Python, NoSQL, and R, as well as knowledge of Microsoft Power BI, AWS, and Spark, to name but a few. Meanwhile, Channel 4 wanted junior applicants with a BSc in a quantitative discipline, such as statistics, mathematics, or econometrics, to assist with data cleansing and preparation for analysis.

These extensive technical requirements – all important for expert and in-depth data handling – belie a barrier that can hold companies back from harnessing their data effectively. Many businesses are rushing to deal with the exponential growth of their data without considering what is truly needed.

While the full-time maintenance and broad management of data is certainly a job for the technical experts, it’s no longer possible to leave data analysis to these few specialists. Data should be spread throughout an organisation, putting information into the hands of line-of-business users, rather than keeping it locked in technical siloes.

There simply aren’t enough data scientists to solve the world’s problems. Data holds the answers to innumerable questions, but we need a way to access these answers. We will never have enough data scientists for this, so it’s up to other analytical thinkers and business innovators to forge a path beyond basic spreadsheets into meaningful data interrogation. We must equip business users, not technical specialists, with the ability to analyse data.

Equipping more people with the right tools and training

The fundamental aim of effective data analytics in a business context is to glean insights that help the company prosper. In this fast, competitive environment, speed is critical. The ability to react to business opportunities swiftly is often the difference between success and failure.

Companies should therefore strive to remove the multiple layers between line-of-business employees and the data they need to make effective decisions. Those people who spot an opportunity should be able to quickly and effectively decide the appropriate business reaction, based on solid data. This means examining the data themselves to draw their conclusions, rather than going through an intermediary data scientist. These employees will know the key business questions better than anyone else, so they should be given the ability to interrogate data, as and when they need to.

Fundamentally, a company’s key decision makers need to be armed with data-led insights as quickly as possible, and this means providing easy-to-use analysis tools and best practice training for more people. There are options out there which provide self-service capabilities as well as a drag-and-drop interface, and yield answers in minutes rather than days. These tools equip business users with the ability to dissect and analyse data, in more depth than ever before, all without the need for a background in programming. The ultimate sweet spot is to find technology that unites the intuitive usability that business users want, with the powerful capability that data science professionals need.

Once these tools are in place, there’s another critical step that companies must take. They must strive to foster a culture of data-driven decision making through every department and level of the business.

A culture of data

Using data to inform decision making is vital, but it must permeate the entire organisation. This ethos needs to be encouraged everywhere, even if senior management must lead the charge. Best practice examples should be shared with all, alongside findings and everyday processes. New employees should be introduced to data analysis tools as part of their induction. Creative problem-solving and ingenuity, based in data, should be a daily occurrence. There should be an atmosphere of collaborative working which allows all employees to learn from each other.

It goes without saying that there will already be data-driven employees within any organisation. By supporting their development and progress, they’ll be encouraged to grow their analytical capabilities and ‘spread the word’ on a day-to-day basis.

There will always be a place for data scientists, and their technical expertise will remain invaluable. But this shouldn’t mean that companies create capability siloes. Business users can, and should, become data analysts.

For those companies looking to hire more data scientists, there’s a fundamental question they must ask themselves. ‘Do we need more people to curate and manage data, or do we need people who can extract business insights from the data?’ If the latter rings true, then self-service analytics tools could be the game-changer they really need.

Stuart Wilson, EMEA VP, Alteryx, Inc. (opens in new tab)

Stuart Wilson is CEO of network security firm Endace, the leading provider of network recording products which closely monitor computer network histories and prevent cyberattacks through immediate identification of breaches.