This article refers to the Sports Development teams and documents their journey spanning technologies. There are several development teams within Hillside Technology, each facing diverse challenges.
What do you do when the tools of your trade are no longer best fit for your changing demands?
It’s a question that we grapple with a great deal and one that most development teams must face. It’s a tactical decision to either continue to work with legacy technology and accumulate the inevitable technical debt or strike out into the unknown in search of a better way.
For us consistent evolution and innovation has been the proven path to success.
For example, in 2012 we could see that our relational database management systems were struggling to scale in order to meet user demand. There were certain data sets we were struggling to process through our legacy systems as the database solution was not able to match demand.
In that instance, it was an older technology that provided a solution to our massive parallel processing problems. Erlang was developed in the 1990s by Ericsson for telephone switches and we found that our problems of reliability and scale were very similar to those encountered by the engineers of their exchanges.
With Erlang we could greatly reduce the size and complexity of our code base and build relatively simple, highly scalable systems at pace. This has seen us transition away from Java and as a result we have reduced the complexity of our code, while enjoying unprecedented scale, yet employing fewer physical machines within our estate.
Today in our Sports Development teams we are experiencing something similar. As we continue to innovate, we’ve found that we’re reaching the limits of our current Windows based .NET systems. This scenario is highlighted where we need to consistently achieve sub-second latency results.
The .NET framework is an excellent set of tools for many jobs. But for Sports, which sits at the front line of the business and is subject to the fastest rate of change, languages like C# are too verbose and prescriptive to achieve our goals.
The bet365.com site provides a sports betting product with a global reach that offers real time betting opportunities to over 21 million customers in 19 different languages.
At its busiest periods, the site can have concurrency peaks into the high hundreds of thousands of customers, who often want to wager on a small number of high-profile sporting events.
Consequently, we face a set of challenges that have driven the diversification of our software choices and the evolution of our development approach.
For the popular in-play offering, the customer interactions must be in real time. Think of a cricket, tennis or darts match. Data must be accurate for each bowl, rally, or throw. In many cases we will also display a live video stream of the event alongside betting opportunities.
Scalability and concurrency
During major sporting events the customer base will be in the hundreds of thousands. On an average Saturday there can be over 300 sporting events, each with hundreds of betting opportunities being traded in real time.
Across our global offering, we attempt to localise our product, tailoring the site by language and ordering of the sporting events to ensure they are the most relevant to that market.
Speed of development
Often our deadlines are set by the sporting calendar. Think the Grand National horse race or the Rugby World Cup. These deadlines are set in stone. They are not a moving target.
There is certainly the ongoing challenge of scale and concurrency as the company continues to grow. However, we must also innovate and introduce new services that ensure we meet the demands of the customer base, which are becoming more and more sophisticated.
Bet Builder is one such service. Launched last year, it enables customers to create their own unique bets in real-time. The system was the first of its kind in the industry and many felt that it may not be possible to implement. This is especially apparent during the real time activity of the in-play calculations.
More challenging still, services like Bet Builder don’t place incremental load on the system, they create exponential load.
As our systems become more interconnected and sophisticated, so ongoing maintenance and further development also increases in complexity. The more we can simplify our codebase, the better we can respond to challenges that occur in the live environment.
Speed of delivery, performance and maintainability are cornerstones of our success but are becoming harder to achieve with our current toolkit. However, unlike where we had used Erlang previously, the solution to our problems was not found in the past but in something rooted very much in the present day.
When investigating the challenges we faced we found they were very similar to those experienced by Google. Therefore, it should have come as no surprise that their own language has turned out to be the right solution for the job at hand.
Golang is a statically typed, compiled programming language designed at Google, whose syntax is very similar to C like languages such as C#. It’s similarity to .NET languages and streamlined syntax makes it easy to pick up and enables us to build applications at pace.
It’s excellent at transforming data, is faster out of the box than .NET, is more forgiving and performs well at scale on modern Multi-CPU architectures. Fundamentally, in Golang you write less code to perform the same function that you would in .NET. It’s easier to write, debug, merge and maintain.
This is essential. Our systems are made up of a multiplicity of different interconnected technologies and systems that must all communicate together and work in harmony. Any change to the system infrastructure can cause issues and it’s not unusual that new services behave very differently when live, than they did in testing.
Therefore, anything we create must be easy to maintain. The more challenging it is to develop, the more people you need to run it or bring in to troubleshoot should something go wrong.
Golang simplicity of syntax and forgiving nature removes a great deal of this complexity. What’s more it is also an excellent “gluing” technology that ensures our Linux systems can connect to our Windows systems. It has been our experience never to underestimate the cost of cross linking operating systems.
Within Sports development we had also considered .NET Core due to familiarity of syntax to our existing developer community, but our testing showed for our betbuilder product that runtime performance was not as high as equivalent tasks written in Golang, and at the time, the knowledge gap between .NET and Golang was not sufficient enough to offset this decision.
Perhaps more importantly, Golang allows us simple transition away from the Windows platform onto the Linux eco system, this then is allowing us to move into the space of other opportunities such as container-based delivery, which is more mature in the Golang space, thus furthering our ability to cope with the challenge of instantaneous demand at high volumes.
We are now moving to a space beyond Windows. Linux is becoming our natural home as we find ourselves using a suite of technologies, led by Golang, that are more at home on the open-source platform.
Alan Reed, Head of Sports Development, Hillside Technology
Image Credit: everything possible / Shutterstock