logo for information-management-architect.com
leftimage for information-management-architect.com

Data Conversion Solutions

Need to design information management data conversion solutions and want a checklist and practical timesaving suggestions

Data conversion is a key component of an information managementData Conversion Solutions projects.

Typical data conversion solutions involve the creation of a data warehouse and the loading of history data, which the business owner hopes to use for historical trend analysis. 

Sounds easy

If we can load the data warehouse with current data, why can’t we just use the same process to load history data?

Nothing is really simple these days. Even with home computers, we sometimes have challenges converting data. Remember the days when we had “floppy” disks and then we got a new computer that only used CD’s.

A lot of people decided to move all the data off “floppies” and store it on CD’s for back-up purposes. That’s when the fun started—The old files did not open properly, or the right version of software to read them was trashed long ago ….You are probably aware of some of the issues.

Now think of some of the things that might have happened to a computer system during the past three or four years.

The system may have been modified. Early versions of back-up data may have different data structures than later versions and documentation explaining these changes might be hard to locate.

Data quality improvements might have been introduced, and there could be different business rules that need to be considered to ensure quality data in the warehouse.

A company might have been acquired during the history period and there might even be two or three sets of the same related data e.g. billing data, and each set of data might have different ways of using reference data.

What are some data conversion design considerations?

The following items should be considered:
Should data be excluded from conversion based on age?
Are there any audit reporting requirements for data conversion?
Do we need to maintain any linkages with the source system?
Interface specifications, although data conversion is usually a “one-time” effort, we still need to specify the interfaces and this might involve re-storing several versions of history data to get an idea of what’s involved—We might find that we have several interfaces for the same history file or table;
Data mapping needs to address any changing source data structures. It’s possible that data cleansing and data transformation might be required just to get the data into a usable format; and
Data quality and data profiling is a big concern. Data profiles on history data will need to ensure that possible changes in data quality are considered.
Consider history data conversion as a sub-project within the requirements analysis phase and “time-box” the work effort—It is very easy to spend a lot of time on requirements analysis and this effort needs to be carefully managed; and
Consider establishing a small design/development team specifically dedicated to this task.

Data conversion solutions and history data requirements must be identified early in the project to ensure clear understanding of the anticipated development work effort.  Anticipate technical challenges with history data and plan for them.

footer for Information management page