When starting on any project, the proper preparations must be made. On data migration projects, this initial preparation is called data assessment. The goal of data assessment is to figure out what legacy data sources need to be brought forward and what is in those legacy data sources. The initial data assessment is a critical piece of the overall project and is also more difficult than it initially sounds.
Identifying data sources might sound trivial, but in many organizations there might be several known systems that drive a lot of the current business and many supplemental data sources. The main data sources are going to be the current ERP systems that are being used and are easy to identify. The supplemental ones are sometimes crucial to the business and can be a challenge to discover. These data sources could be additional ERPs, Access databases, Excel spreadsheets, handwritten lists, etc., and are not initially known at the onset of the project. The people using them might be breaking corporate policy by maintaining them and might be afraid to divulge their existence. To get around some of these difficulties in finding these data sources, communication needs to go out to the business users that it is important to share all sources with the team and that there will not be any repercussions for divulging any rogue data sources.
As data sources are identified, it is important to figure out what is contained within each data source. The second major aspect of the initial data assessment is to figure out where each major data area resides and what the actual data values are in each column. This initial data assessment will uncover where the item, customer, sales order, employee, etc., data is stored, the tables that contain the relevant data, the values of the data contained with the tables, the data relationships within and across tables and data sources, and begin to uncover various data issues. This initial assessment is used as the starting point for the following four major activities in the data migration process. Data assessment will continue throughout the course of the project as additional business requirements and supplemental data sources are discovered.
Once information starts to flow out of the data assessment effort, several of the subordinate portions of the data migration can begin. The first one of these is data cleansing. Data in legacy systems is never better and often worse than anticipated. The data assessment activities will lead to the starting point of identifying areas where data will need to be cleaned. The data cleansing effort will dive deeper into these areas and flesh out the details of the records that need to be cleaned up. This effort will stretch and evolve over the course of the entire data migration project as you discover new requirements and develop a better understanding of the data and what’s needed in the target systems. Typical data cleansing activities are de-duplicating data, correcting invalid codes, correcting invalid dates, inactivating no-longer-used records, populating missing values, correcting invalid addresses, closing old orders, etc.
As the initial data assessment effort starts to finish up, the data mapping process can begin. Data mapping is looking at the data that is needed for the target system and figuring out the logical rules to fit the legacy data into that format. Data mapping initially might sound like a simple task, but it is difficult and often painful. Data migration projects deal with different systems built in different eras, with different technologies, with different business requirements. This means data critical for the operation of the legacy systems might not even be needed in the new target system. There may also be features in the target system that are not in the legacy systems. The business might also want current business processes to work a completely different way in the target systems. The differences between the systems can result in a scenario where a square peg needs to fit into a round hole. Lastly, all of these complexities multiply when there are multiple legacy systems that are combining into a single target system.
At some point during the initial data assessment or the data mapping portions of the project, there will be data that is required for the target system to work that does not exist in the legacy systems. The process of incorporating this data is called data enrichment. Some typical data enrichment activities might be to attach product codes onto the items, add VAT numbers to customers, add counties to addresses, etc. Some of this data will need to come from a third-party source, some can be generated through mapping rules, and other values might need to be created by the business.
After the initial cut of the data mapping phase is completed, the data transformation process can begin. Data transformation is the actual journey from point A to point B. In order to execute the data transformation, the data migration team will take the mapping specifications, build programs that will transform the square legacy data to fit the round hole of the target system, and report any errors that are encountered along the way.
Once the data is loaded into the new system, the next step is to reconcile the data to make sure that everything in the target system matches to what was expected to end up there. There are different levels of data reconciliation, from just tracking record counts to running fully automated parallel processing that compares every field. Different businesses and conversions will require different levels of reconciliation. There are also different audiences that will require different reconciliation metrics to be captured. In every major data migration, an official audit will need to take place. Project management, business users, and master data management groups might also require different sets of reconciliation metrics. It is important to meet with all the stakeholders well ahead of when the reconciliation is scheduled to allow the team to make sure all of the proper metrics are set up to be captured before, during, and after the migration.
The last major aspect of a data migration project is system retirement. Once the data migration is complete and the new system is operational, the legacy systems or portions of the legacy systems are no longer the systems of record. At this point, some functionality of these systems can be turned off, changed to operate in an inquiry-only mode, or decommissioned entirely. After the system retirement plan has been executed, the data migration journey is complete.