Skip to main content

GDPR and the art of consent management

(Image credit: Image source: Shutterstock/Wright Studio)

There is a consent management headache brewing for organisations. The new rights afforded to individuals under the EU General Data Protection Regulation (GDPR) (opens in new tab), which comes into effect on 25th May 2018, mean that individuals will have the right to request that companies delete any information relating to them, just by withdrawing their consent.

We saw how complex the undertaking was for Google when individuals in the EU won the right to be forgotten, or more accurately, the ‘right to delist’ back in 2014. Of course pre-Internet, when customers’ details were simply recorded in a Rolodex, this task would have been straightforward. Today it’s a data minefield because, in most organisations, details on individuals (or “data subjects" in privacy speak) are scattered across multiple, siloed systems – e-commerce applications, marketing databases, CRM systems, billing systems, etc.

The data silo problem 

Imagine the following scenario:  an online retailer suffers a data breach and many of its databases are compromised. This triggers a response from a large group of disgruntled customers or employees who decide to join forces and issue data portability requests, requiring a full export of all their personal data within the timeline mandated by the GDPR of 2 months. With the volumes of data stored across multiple IT systems it would be like looking for a needle in a haystack. And while one-off requests can probably be handled by going to the business and IT owners and manually collating data, this approach will clearly not scale to tens or hundreds of simultaneous requests.

Even what looks like a simple one-off ‘right to be forgotten’ request from a customer to remove them from all databases isn’t actually that simple. The marketing team can relatively easily remove the customer from their database, but data on this customer is likely to be residing in other systems too. And if one of these systems is changed internally, that customer’s data may suddenly - and unwittingly - make its way back onto the marketing database. This would undoubtedly be another cause for disgruntled customers to initiate litigation.

A moving target

This consent management headache is compounded because there are other moving parts to consider too. First, the GDPR is still a moving target from a regulatory point of view - and will probably continue to be even after the deadline. And second, some of this personal data has probably been shared with and stored by third parties – and could be lying in a data lake or lurking on a shared drive somewhere. The likelihood is that no one really knows.

Lack of clarity requires flexibility

Faced with this lack of clarity, what can organisations do? The GDPR requires firms to demonstrate that they have taken all reasonable actions with regard to compliance. Yet worryingly, according to a new survey from Veritas, only two per cent of "GDPR-ready" organisations are compliant today.

Some large businesses have engaged consultants, at considerable expense, to undertake a manual data mapping exercise. This can be a useful first step but isn’t a sustainable approach because the output tends to be static documentation such as PDFs. As a result, responding to changes in consent or data export requests requires a time-consuming manual examination of this documentation every time – making it difficult and expensive to adapt to changing regulations.

The upshot is that most firms today either have no insight because they haven’t done any data mapping or have static pdfs with all the data mapped out, which will probably be out-of-date by next week. To solve this dilemma, many organisations are opting for an automated approach.

Enter stage left the Consent Management Hub. Think of it as the equivalent of a supercharged digital Rolodex - or a digital filing cabinet. Essentially this is a secure, scalable metadata store that can be used to actively manage the output of any data mapping exercises, and as a way to automate the operational aspects of consent management. 

Customer contracts, website terms and conditions (T&Cs), and all personal data can be added to the hub and searched and queried, making it easy to see where consent has been obtained from individuals. Once data protection officers see where the consent ‘holes’ are, they can work on obtaining additional consent and updating the metadata store accordingly. 

Honing in on an enterpise NoSQL hub

From an IT architecture point of view, an enterprise-grade NoSQL database platform is the best option on which to build a Consent Management Hub, because it can handle the data diversity problem. Relational databases simply can’t cope with vast tracts of unstructured data and any schema changes are expensive to rework and can take anywhere from 4-18 months. Data lakes aren’t a viable option either, because they are open and so not secure by design. When it comes to data management, a secure platform is critical. A lesson many organisations have unfortunately learned the hard way. 

A Consent Management Hub also offers far greater data agility in that they can handle unstructured (e.g jpgs, PDFs, Word documents, etc.) as well as structured data. And they can handle multiple queries of the data, for example, find all of the system owners of a particular set or type of data, find the number of customer consents per region, as well as show consent trend data (e.g. whether consent is going up or down over time).

Enterprise-grade NoSQL databases can also accommodate changes to the GDPR and other regulations with the benefit that modifying and adding to the schema can be done in a matter of days or weeks, rather than months.

·         Automate the data mapping exercise: ingest and index any file that contains even a hint of personal data from the siloed systems as it comes into the hub, and apply some rules (e.g. look for email address, name, address and policy number).
·         Conduct advanced data matching analysis:  It is critical to understand how personal data from each system corresponds to each data subject. Without this step, it isn’t possible to know which data is directly relevant to each request, or consent documentation.
·         Once this up-front analysis is complete, retain the ‘citizen entity’ in the hub (e.g. a person’s name, or some unique identifier). This contains the pointers back to the data in the original silos, meaning that the rest of the personal data can be removed from the hub.
·         From an operational perspective, the hub can also be used to store “consent entities” (all consent-related documentation). These entities contain information that points back to pertinent records in databases and applications (e.g. a marketing database), and includes metadata detailing the system owner, etc. This ensures that any customer data needing to be deleted is handled by the specific system owner rather than at the consent management hub level.
·         These entities help to identify exactly what data relates to which person and are useful for data portability or data access requests. They can also help solve the problem of ‘How do I remember who I’ve been told to forget?’
·         Ensure support for semantic querying, which makes it far easier to correlate personal data and consent documentation see whether you have consent to use the data. If you don’t have consent, then you have to ask the individual for consent or delete the data to ensure compliance.
·         Use a database that supports ‘digital time travel’, or bitemporality, so that it is possible to digitally roll back time to see what, for example, the website T&Cs looked like a year ago, when today’s disgruntled customer originally signed up to receive marketing emails.
·         Choose a database that has advanced security such as redaction built-in to support the GDPR, for example to ensure GDPR compliance, a call centre handler might see the name and phone number of the caller on her screen, but no other personal information. Further, the database should support encryption at rest to prevent it being subject to data breach notification requirements.
·         Build a set of smart user interfaces to the Consent Management Hub that customer service representatives - or even customers themselves via self-serve portals – can use to easily manage consent.
·         Use the hub to store all versions of legal contracts with third-party firms who handle any personal data on your behalf (data processors) to ensure compliance.


With the GDPR deadline almost upon us, the risk associated with not doing building a Consent Management Hub surely outweighs the costs. Plus, once you have mapped this data for regulatory purposes, you have a ready-made launchpad for new, valuable business services based on the deep insight you will have gained by being able to flexibly integrate all types of customer data.

Christy Haragan, Principal Sales Engineer, MarkLogic (opens in new tab)
Image source: Shutterstock/Wright Studio

Christy Haragan is the Global GDPR Lead at MarkLogic.