The Age of Compliance: How Bitemporality makes every data second count

Your Starter for 10: What do Arthur C. Clarke, Douglas Adams and HG Wells have in common? Answer: They all created and – ultimately profited from - time travel. 

As today’s organisations grapple with ever increasing amounts of data, often trapped in individual wormholes – or unconnected, disparate data silos – of their own, a spot of tech time travel could be just the ticket to enable businesses get faster, more accurate insights into their historical and concurrent data.  

How an organisation’s data is stored, analysed and managed is key to staying competitive. With a galaxy of regulatory requirements now affecting industries across the board, a database that makes it easy to rewind and explore what you knew at any point in time is crucial to mitigate risk and maintain compliance.

With an increasing need to have strong data governance strategies in place, how a database supports - or rather handles - time could be the difference between identifying new business opportunities and revenue streams or the painful consequences of non-compliance including hefty fines, the closure of a business unit and irreparable brand damage.    

Bitemporal: Adding a new dimension to data 

The vast majority of information in today’s applications is temporal, in other words tracking valid time and recording changes to data ‘when it actually occurred’. The next iteration is bitemporal, which handles data along two different timelines by capturing both valid time and system time, which is when something was recorded to the database. Organisations can store and query data along these two timelines with timestamps for both valid time — when a fact occurred in the real world or ’what you knew’, and also system time — when that fact was recorded to the database  or ‘when you knew it’.  

Think of bitemporal as a kind of tech time travel. By being able to identify what a user knew at any point in time, this time machine function ensures businesses - and regulators have a full and accurate picture of data.    

Savvy global financial firms and leading investment banks are embracing bitemporality to better manage their data and maintain and demonstrate compliance with, for example, Dodd-Frank data analysis processes and legislation.     

In the financial trading business, trades can be amended at a later date. After trades are made, they are later reconciled with counterparties and updates often occur before the trade is closed. Future analysis of trading positions could be compromised—and organisations could face major fines by regulators - if there is no time-based record of how and when updates were made to trades. Bitemporal design provides an accurate picture of the entire lifecycle of a trade, including when any changes were made to counter-party names, transaction IDs or price corrections. 

There are use cases for bitemporal across many sectors. For example, ABN AMRO, the international bank based in Amsterdam, The Netherlands, is currently using the MarkLogic database platform to bring vast amounts of unstructured and structured trade data into one central operational data store. MarkLogic’s bitemporal capability allows ABN AMRO to minimise risk through tech time travel— time-stamping and rewinding trades in the data store. With a consistent, transparent record of every order and trade event, ABN AMRO can more easily comply with existing and forthcoming financial regulations. The required data is all in one place, ensuring data lineage and provenance through an append-only model, and providing a uniform view of orders and trades.

Bitemporality is also critical for the legal profession. For legal firms it’s vital to understand what laws are applicable to specific legal documents at a given point in time. For Legal Process Outsourcing (LPO) firms providing services such as processing legal documents, bitemporal functionality gives the essential ability to understand the validity of the documents on a specific date and time.   

LPO companies need to verify the date that the document was issued, the applicable laws at that time and which laws came into effect at a later date. In some cases, laws can be retroactive, which changes the effect of the document in the past. Sometimes the laws don’t take effect until a future date, which requires understanding the implications so they can address potential consequences or issues that come from those legal changes. More time travel.

Similarly, in the insurance industry, bitemporal gives a clear determination of coverage over the course of history, so in the case of health insurance coverage for example, even if retroactive changes and updates are made to the policy, the data is never overwritten.   

In healthcare, bitemporal capabilities can give a more accurate picture of a patient’s medical history and can help connect the dots between an illness and pre-existing conditions. Pharmaceutical and biotechnology companies can also use bitemporal to enhance their decision-making in research or reporting.  With the new Identification of Medicinal Products (IDMP) standards set to create a few IT headaches, bitemporal can also help ensure that pharmaceutical firms demonstrate and maintain compliance.        

Likewise, government intelligence and law enforcement agencies can use databases with bitemporal support to paint the full picture of a situation even when disparate facts have been gathered ad hoc or out of context. As bitemporal databases never erase data and make it possible to see exactly how data was updated based on what was known at the time, bitemporal capabilities can assist when old investigations are reexamined by enabling law enforcement officers to go back and ask why a particular line of enquiry was pursued.    

A 360-Degree Data View At Any Point In Time 

Whether you’re a multi-million dollar global enterprise, or an enterprising entrepreneur, being able to aggregate data from disparate sources to get a single, 360-degree view of customers or transactions is essential. When the data and requests from regulators change over time, using an operational data hub – or virtual filing cabinet - built on a flexible, enterprise-grade NoSQL database can pay dividends. 

Many companies have been able to fully leverage bitemporal capabilities through the use of such databases. They fit the bill because they can handle any data type, and also remove the need to resort to many costly hours of complex data integration challenges and wrangling - extraction, transformation and loading (ETL) - a major weakness of relational databases.  With enterprise-hardened features, these databases offer high levels of performance, flexibility, data resiliency and enterprise-grade security as well as the ability to fully exploit bitemporal capabilities.  

Bitemporal design also keeps storage in check and improves data security because it avoids the need to set up additional databases for historical data. The more copies of an organisation’s data that are made, the more vulnerabilities are introduced.  With bitemporal, as the data is all in one place, this creates a more streamlined, simple approach to maintaining historical data which helps improve an organisation’s overall data governance. 

Back to the data future 

With due respect to Douglas Adams’ theory that 42 was the answer to life, the universe and everything, in today’s digital age it is clear that an organisation’s insight into its past and present data is vital to its future. Bitemporal enables enterprises to tech time travel with ease and gives the organisation vital time-based insights. Understanding precisely what was known and when supports an organisation in its quest for good data governance and security whilst keeping the regulators happy. 

David Northmore, VP of EMEA, MarkLogic 

Image Credit: Alexskopje / Shutterstock