Data
Conversion Solutions
Need to design information management data
conversion solutions and want a checklist and practical timesaving
suggestions
Data conversion is a key component of an information management
projects.
Typical data conversion solutions involve the creation of a data
warehouse and
the
loading of history data, which the business owner hopes to use for
historical trend analysis.
Sounds easy
If we can load the data warehouse with current data, why can’t we just
use the same process to load history data?
Nothing
is really simple these days. Even with home computers, we sometimes
have challenges converting data. Remember the days when we had “floppy”
disks and then we got a new computer that only used CD’s.
A lot of people decided to move all the data off “floppies” and store
it on CD’s for back-up purposes. That’s when the fun started—The
old files did not open properly, or the right version of
software to read them was trashed long ago ….You are probably aware of some
of the issues.
Now think of some of the things that might have happened to a computer
system during the past three or four years.
The
system may have been modified. Early versions of back-up data may have
different data structures than later versions and documentation
explaining these changes might be hard to locate.
Data quality
improvements might have been introduced, and there could be
different business rules that need to be considered to ensure quality
data in the warehouse.
A company might have been acquired during
the history period and there might even be two or three sets of the
same related data e.g. billing data, and each set of data might have
different ways of using reference data.
What are some data
conversion design considerations?
The following items should be considered:
 Should data be excluded from
conversion based
on age?
Are there any audit reporting
requirements for
data conversion?
Do we need to maintain any linkages
with the
source system?
Interface
specifications, although data conversion is usually a “one-time”
effort, we still need to specify the interfaces and this might involve
re-storing several versions of history data to get an idea of what’s
involved—We might find that we have several interfaces for the same
history file or table;
Data mapping needs to address
any changing source data structures. It’s possible that data cleansing
and data transformation might be required just to get the data into a
usable format; and
Data quality and data profiling is
a big concern. Data profiles on history data will need to ensure that
possible changes in data quality are considered.
Consider
history data conversion as a sub-project within the requirements
analysis phase and “time-box” the work effort—It is very easy to spend
a lot of time on requirements analysis and this effort needs to be
carefully managed; and
Consider establishing a small
design/development team specifically dedicated to this task.
Summary…
Data conversion solutions and
history
data requirements must be identified early in the project to ensure
clear understanding of the anticipated development work
effort.
Anticipate technical challenges with history data and plan for them.
|
