Skip to main content

How to manage the process of data warehouse development

data
(Image credit: Shutterstock / Gorodenkoff)

Developing a data warehouse for the first time is not easy for any business. This initiative’s complexity makes it quite challenging, especially for an in-house team that has never handled such a project before.

In this article, we take a look at the steps involved in managing data warehouse development and implementation.

1. Discern if external help could be relevant 

Before you look into building a future proof data warehouse, you need to analyze your in-house team’s capability. Unlike common IT projects, building a robust Data solution is a different ball game.

Most often, the right approach is to bring a consultant who will have a chat with your IT staff and decide if you need a data warehouse and what architecture might fit your business structure. Ideally, your staff understands the company operations, and while the incoming consultant might be an expert at building high-end systems. A combination of these two teams will reduce the number of possible flaws and cut down on costs and time.

2. Form high-level strategies and tactics 

Like we said before, a DWH project is a complex project with a lot of moving wheels. This is not a departmental system with only one decision-maker. Instead, it affects all the branches of the company which is why you need to strategize on a high-level.

First, start with your company’s needs. What exactly are the core pain points? In which direction is your company headed? Do you need an insight-driven engine? Is there a need for advanced analytics? Is there a probability of re-platforming with an aim to include cloud technologies? These are merely a few questions you need to deliberate on before the project takes off.

The best way to develop a sound plan and tactics is by involving all the key decision-makers from CEO, COO to Chief IT officer.  Whatever you do, don’t Sonic-dash through the entire development process. Instead, start gradually from one department to another before spreading your wings for the whole enterprise. This way, it’s easy to analyze the strengths and weaknesses of each process and adjust accordingly.

3. Share your project with stakeholders 

Once you have a sound plan and strategies on how you wish your data warehouse project to be carried out, it’s time to look for a stakeholder that will transform your paperwork idea into reality. These people don’t necessarily need to be decision-makers. It could be end users like engineers and data scientists or business analysts that run day to day profit forecasting.

The idea behind including stakeholders is to ensure all the key individuals within the corporation not only recommend a DWH but understand all the details of what is to happen. This does away with all the knowledge gap and power struggles that could throttle the project’s success.

4. Carry out a high-level assessment of both current and future targets 

An in-depth assessment of a DWH project is essential as a reassessment. This assessment answers the key questions like what, why, and how everything needs to be done. The good news is that you can restructure an assessment at any point of this project, especially where direction and value are not clear.

At this stage, your assessment should describe the beginning and the end of every step, review the technical and corporate environment within which the initiative will take place for smooth integration, and whether the proposed DWH concepts will work with your existing tech stack.

During an advanced assessment, you also need to come up with metrics that will tell or measure the effectiveness of the initiative. Don’t just build a system and assume it works. You must have a way to track its performance. 

Understand that assessment is not a one-time task. Company expectations and customer demands change. This means you will need to assess and reassess DWH projects up until your last day of work.

5. Selecting DWH concepts and tools 

There is an explosion of technologies, tools, and concepts you can adopt while building a data warehouse. The sheer number of those technologies makes it quite challenging to decide what would be apt for your business processes. So before making a choice, you must narrow down to a few potential alternatives. Of course, your IT team will help you to figure out what’s suitable for your company.

If you can’t reach unanimity on the appropriate technologies to use, the best middle ground is to settle for hybrid solutions. While making the selection, bear in mind your business needs, the costs, and the scalability of the technology. Be on the lookout for any chances of incompatibilities.

6. Aim for an MVP 

As of this moment, there are many instances of failed data warehouse development projects. Why? Because those in charge went in full swing without even establishing small pilot projects.

The only way to know if a DWH is going to work is by kicking off a Minimal Viable Product. You will need to put it in action to get real-time results of how it performs. This strategy is a budget-friendly way to figure out if you are on the right or wrong track.

7. Create and deploy a clear roadmap 

With an MP in place, you now have an idea of what works. So, at this point, you would be expected to chart a broad roadmap highlighting all the DWH delivery points. Usually, this states if and when incremental implementation will be adopted, when to increase the use of more tools and architecture, and the project will be ready for the first business processes. 

As you can see, this is another point where the help of experienced consultants would be invaluable. You can also arrange for your in-house team to receive more training, so they have an idea of what to do when the consultant’s contract ends.

8. Deploy, monitor, and optimize 

Once your data warehouse engine is up and running, you need to have it regularly monitored, re-assessed, and optimized for smooth performance. This is the only to establish if the initiative needs additional cloud infrastructure and modernization. 

Conclusion 

Managing the process of data warehouse development is hectic. In particular, it can be frustrating and devastating if your pilot projects don’t go through as planned.

However, with a combination of the right team, sufficient resources, and well-captured business goals, you can develop a powerful DWH that complies with all the data warehouse best practices (or at least most of them). If the journey is well planned and everything falls in place, you should see some promising results in a few weeks.

We hope the information in this article gives you a glimpse of what needs to be done while managing the process of building a data warehouse. In case you need help, feel free to reach out to established data scientists.

Alexey Utkin, Systems Architect and Team Leader, DataArt

Alexey Utkin joined DataArt as Systems Architect and Team Leader in 2004, and has been in charge of leading major finance enterprise accounts since. With over 14 years in the IT industry, eight of them in the financial services sector, Alexey brings a wealth of industry expertise to DataArt and has become a core member of its Finance Practice. With a dedicated focus on solution, technology, regulation and process consulting, he now leads DataArt’s most seasoned industry practice from its London’s office. A sought-after speaker and media contributor, Alexey regularly attends key industry events and is often quoted in the press. He holds a MS in Applied Mathematics from St. Petersburg State University (Russia), with a specialization in Computer Science.