Data migration from legacy systems to the ECM


Setting a data migration scheme is necessary as there are numerous challenges with legacy systems. Before migrating take into consideration all the risks and costs, perhaps archiving data and starting fresh with new ECM might be a feasible solution.

Why migration to new system?

Why migrate to a new system, is the current system obsolete, will the new system resolve our issues and improve efficiency? These are some of the questions asked by records information and archives managers as well as information technology officers.

Many organisations have acquired ECM solutions from TRIM (RM, CM) ZyFind, Documentum, OpenText, to name few, to manage their information with collaboration, work product management, back-end process automation and retention disposal management. Often, driven by budgetary constraints companies tend to delay with necessary system upgrades that may lead to a system breakdown and unresponsiveness, poor performance, inability to answer user demands, lack of technical expertise in dealing with legacy systems that leads to total obsolescence of those systems. An old legacy system might no longer be suitable for update or change as the Software Company may have discontinued support to their obsolete systems or perhaps seized to exist leaving you in a vacuum situation. Of course, often it might be possible to continue to use the legacy system, but lack of technical and domain experts means that your organisation will be paying exceptionally high maintenance costs to keep it alive. Some of the main challenges of legacy systems are incapability to carry on with growing demand for performance and capacity. Adding to this, restricted update features and patches creates an unstable environment prone to security threats and data losses. Simply, legacy systems are hard to maintain but what are the options? Desirably, migrating to new ECM systems where all legacy data is proficiently mapped and migrated is the desirable way to go. But, the process is not as easy as it may sound.

Project management best practices and international standards

ECM migrations or upgrades are complicated and expensive projects. For organisations intending to perform an ECM migration, effective project management is vital. Defining a project management scheme in advance is an appropriate manner to maintain an ECM migration. Appointing project team, project manager and a project board and defining the key roles, and following accredited and recognised project management methodology (PRINCE2, PMI) would help streamline and control project work.[1] Read best practices and international standards to make sure your ECM migration runs effectively and proficiently. There are many ISO standards that touch on the proper data migration methods, records management, information security and systems, to name few: ISO/IEC 29121:2013 [2]; ISO/CD 30300 [3]; ISO 23081-1:2017 [4]; ISO 16175-3:2010 (ICA) [5].

What next?

The approach to this task depends on the nature of legacy systems, and specifically how existing data has been stored and managed there. It would be necessary to map the structure of the old system to that of the new system so that where possible, the automatic transfer of data can be managed by programmatic means. If old systems have been subject to extensive customisation, this might be difficult, and some level of manual intervention may be required. It would also be necessary to appraise legacy data to assess whether it deserves to be migrated, as it may anyway have exceeded its useful retention period.

To achieve this, start by creating an inventory of all such legacy systems, detailing the volumes, file types, and then work closely with business owners to determine and document which data (documents, metadata) need to be migrated into the new ECM, identifying anticipated challenges. For example, depending on the age of the legacy systems, obsolete file types might present an obstacle, making it additionally necessary to convert files into file types operable with the new system.

Additionally, it is possible that the ECM might have a different folder structure, for example, one built around functional business classifications (FAT) than that found in the legacy system, making it difficult to merely transfer folder contents directly from one system to the other. For these reasons, it is likely that human involvement will be needed to ensure that the task is performed correctly, especially if sensitive documents are involved. Guaranteeing senior management is adequately informed of this need to assure that adequate resources are allocated.

Collaborate with ICT

Work with ICT colleagues to make sure that required data is extracted from each legacy system (leaving such systems intact for the time being) and then uploaded into the new system. Following an agreed period of quality control, the legacy systems could be retired. This would have to be part of an overall implementation strategy, to ensure business continuity and that users are adequately instructed in the use of the new system.

In coordination with superiors take into consideration, depending on the complexity, appointing a project board, project manager and delivery team to assure proper and smooth migration. Planning and managing migration process must be done carefully to ensure that information is available and usable during the process. Making sure that no digital information is erased until it is successfully migrated to the new system and that it has been checked for errors or migration failures are paramount.

The most significant impact on physical records collection would involve migration of the electronic systems and databases that may have been used to manage these records. Unique IDs generated by earlier systems could be retained in the new system to avoid complications and the inconvenience of creating new labels etc.

Data migration process

Given the fact that you have selected and customised a new ECM system, which is a project on its own, follow these steps:

> Create a project plan, defining how long the migration will take, estimate costs, outline roles, duties and responsibilities following and of recognised project management methodologies. Determine the stakeholders, records managers, IT infrastructure team, and developers.

> Evaluate legacy system and describe requirements and possibilities. Determine the total size of repository, the number of records to be migrated, what should not be migrated, what is not possible to migrate and what solutions are available.

> Data mapping - Define each data source location and target location in the new system. Metadata set mapping - content types, profiles, categories, permission.

> Arrange for proper infrastructure and design the new (target) system environment for data migration.

> Prepare and test migration.

> Extract data.

> Define rules for data cleaning and develop scripts in cooperation with developers.

> Run scripts and load data into the new system.

> Validate data and check potential issues.

> New system goes live.

Data Quality

Data quality attributes must be taken into consideration when assessing data migration: authenticity, reliability, integrity, usability.

The data quality model specified in the standard “ISO/IEC 25012:2008 defines a general data quality model for data retained in a structured format within a computer system. It can be used to establish data quality requirements, define data quality measures, or plan and perform data quality evaluations. It could be used, for example, to define and evaluate data quality requirements in data production, acquisition and integration processes, to identify data quality assurance criteria, also useful for re-engineering, assessment and improvement of data, to evaluate the compliance of data with legislation and/or requirements.

ISO/IEC 25012:2008 categorises quality attributes into fifteen characteristics considered by two points of view: inherent and system dependent. Data quality characteristics will be of varying importance and priority to different stakeholders.

ISO/IEC 25012:2008 is intended to be used in conjunction with the other parts of the SQuaRE series of International Standards, and with ISO/IEC 9126-1 until superseded by ISO/IEC 25010.” [6]

The Data Quality model outlined in the standard ISO/IEC 25012 is composed from 15 attributes, and presented in the following image:

Points to take into consideration:

·        The file formats used in the system

·        Size of information to be transferred, - space allocation

·        Encryption, if some records are encrypted, type of encryption keys, etc. 

·        Process quality management

·        Change management

·        Governance

·        Implementation

Think outside the box

Organisations are required to properly manage their information based on national regulations, business' statutory documents and regulations, objectives, expected results, mission and vision. System data must be retained to fulfil stakeholders demand to complete their work, internal and external auditors, financial authorities, tax auditors. In short, information held legacy systems are essential to the business. As mentioned in many cases the older systems are coming to the end of life, and you do not want to lose any of that information but on another hand, you might be limited in your resources and data migration is out of the question. As a relatively cheaper and easier option, you might consider archiving that information, with the vision that the enterprise legacy repository is going to be in full compliance with legal requirements such as audit proof storage, proper access controls, accurate data mapping and strong retrievable solution. This would be a challenging process too, requiring a lot of planning, analysis and effort.

The Essential requirement is not technology but skilled people

Depending on the available resources and expertise you might decide on data migration to the new ECM or on properly archiving the legacy system data. Either way is complex and requires a collaborative effort among several groups and individuals. The essential requirement for success in this field is not the technology but skilled people. The collaboration requires collaborative leadership, that starts, according to David Chrislip and Carl Larson, in Collaborative Leadership, from the basis that "...if you bring the appropriate people together in constructive ways with good information, they will create authentic visions and strategies for addressing the shared concerns of the organisation or community."

Ertan Čako, Information Management Assistant, International Criminal Court
Image Credit: Flickr / janneke staaks