Big data makes research more difficult say data scientists

Given the amount of hype surrounding it you could be forgiven for thinking that big data is the answer to most business and scientific problems.

But a new survey by database specialists Paradigm4 reveals that almost three-quarters of data scientists believe that big data has made their research harder. The reason for this is not, as you might expect, the volume of data but the variety.

The results, from a survey of 111 US data scientists, show that 40 per cent say they struggle managing new types and sources of data and 36 per cent say that getting answers from big data takes too long.

Read more: Open source is the best way to innovate big data

Hadoop comes in for criticism too with 76 per cent saying it's too slow, takes too much effort to program or has other limitations. So much so that 35 per cent of scientists who have tried using Hadoop or Spark say they have now stopped using it.

Problems cited include 39 per cent saying it takes too much effort to program and 37 per cent saying it's too slow for interactive ad-hoc queries. 30 per cent also say it's too slow for performing real-time analytics.

It seems that despite all this, however, big data still has a key role to play. 91 per cent of those surveyed say they’re using or plan to use complex analytics on their big data within two years.

If you want further insight into the trials of being a data scientist, you can get hold of the full survey results on the Paradigm4 website or there's a summary of the findings in handy infographic form below.

Image Credit: aslysun / Shutterstock

Topics