Why are BI engineers so frustrated? (It's not money)

(Image credit: Image Credit: Sergey Nivens / Shutterstock)

Unemployment is at an all-time low – which means there are plenty of jobs for talented, smart people. But even though opportunities abound, changing jobs is always going to be stressful – so if people are doing it, there must be a reason. And if they are doing it in droves in the same industry, there must be a real good reason.

It could be due to frustration or other issues, according to a 2017 LinkedIn study. According to the report, over 13 per cent of software industry workers changed jobs that year – and nearly half (45 per cent) cited the lack of opportunities to advance as the main reason. Other top reasons included discontent with leadership (41 per cent), discontent with the work environment (36 per cent), and desire for a more challenging job (36 per cent). Salary came in as only the fifth reason (34 per cent) why professionals decided to change jobs.

If frustration and lack of opportunity is the top reason for employee turnover, it stands to reason that workers saddled with more frustrating work are more liable to change jobs. Among software industry jobs, there are few that are more frustrating then Business Intelligence.

That's especially notable because BI staff salaries, while not in the highest decile of IT pay, are on the higher side for IT. The average BI analyst in the U.S. earns $67,263 a year, comparable to that of an Information Technology Specialist ($68,478), ahead of Network Administrators ($67,051), and not too far behind IT Analysts ($71,334). So, if pay’s not the problem, then what?

The term “back-office work,” is an appropriate description of BI. Staff needs to remediate and reconcile data, ensuring it is consistent throughout the data containers, “clean up” data and metadata, identify gaps and redundancies in data that degrade its quality, transform data taken from different sources into a single framework so it can easily be utilised by staff on front-end applications, and more.

As it happens, much of this work needs to be done manually – and given the explosion of data and the plethora of data sources, there is more information to work through and remediate than ever. It can sometimes feel like using a bucket to empty the ocean; no matter how hard you work, you'll never reach your goal.

One of the biggest issues in BI is metadata - the label that is used to classify data in storage areas and databases. Metadata makes it possible to search for data by category, type, relationships, etc. Without metadata, the information in databases would basically be inaccessible.

What can companies do

In order to make data usable and searchable, metadata needs to be accurate, and organisations may have standards and specifications for metadata. But this tends to go by the wayside; departments that need to get their work done develop their own lingo and labels, and over the years, as organisations build up large caches of data and implement new databases, those differences grow – essentially rendering the data stored by a department useless to anyone in the organisation other than itself.

For example, an organisation might record information about a customer's location with a label called “location,” “address,” “city and state,” etc. Whatever search system that is implemented needs to take into account these issues. This is a chronic - and central - problem for many organisations, and one that could seriously hamper their ability to find data at all.

Of course, BI will come in and save the day – hence the frustration. Subject your BI staff to a few cycles of this metadata confusion and remediation, and you begin to understand the staff turnover numbers. Organisations that want to retain their BI staff need to take another approach that will make the work far less frustrating for staff.

There are several things companies can do: Education and policy enforcement would be a good place to start. Establishing a set policy defining metadata terms and data storage protocols – data catalogues, data dictionaries, glossaries – and ensuring that all departments follow them can ensure that all new data entering the system is usable and searchable by everyone, and thus eliminate some of the more frustrating aspects of BI work.

For data already in the system, an automated remediation system that inspects data and metadata could be useful. With such systems, BI teams can explore databases and ensure that metadata is uniform across all systems, and also matches the catalogues, dictionaries, and glossaries that have been implemented.           

With the frustrating back-end work behind them, BI teams can turn their attention to  development and analysis. For example, all that mislabelled data that is now properly labelled could contain valuable insights about the company, markets, products, and customers.           

As the data experts, BI teams are in the perfect position to explore that information and come up with those insights. That's the kind of creative work these team members got into IT to do in the first place – and if they get an opportunity to do that kind of creative work, BI analysts will be much happier with their jobs, and much less likely to search for a new one.

Amnon Drori, CEO and Co-Founder, Octopai