Secret To Successful Data Migration: Put Data Mapping First!
Let’s admit the key to a successful organization lies not only in gathering more and more data but also in ensuring how organized is their data management service. We are living in a dynamic world which is fast changing and companies going through mergers and acquisitions is nothing news, which is also leading to the need of data migration from existing or legacy systems. A discussion on data migration will see the focus immediately shifting to mapping legacy systems. Understanding the role of mapping legacy systems is a critical factor for a successful data migration project.
The critical role of data mapping in data migration
Understanding the successful data migration begins with understanding what approaches, methodologies, technologies and tools will take center stage in the project life cycle. But more often, organizations mainly focus on acquiring technologies. However, ground reality speaks something else, without strong business focus and a proven approach there is a high risk that the data migration may fail. In fact, according to Gartner, 83 % data migrations fail outright or exceed their allotted budgets.
OK, 83 % data migrations fail but when will you come to know about it
ETL Job reports errors during execution. After the migrating data and during unit testing if it is noticed that the job reports are bumping into the errors then it means something went wrong with the data migration. Tracing of error back to the origin is the need of the hour.
Analysis of data migration projects over the years has shown that they meet with mixed results.While mission-critical to the success of the business initiatives they are meant to facilitate and support lack of planning structure and attention to risks causes many data migration efforts fail.
— Gartner, “Risks and Challenges in Data Migrations and Conversions,” February 2009, ID Number: G00165710
Most of the time the errors has got to do with the data mapping issues. Just to make things even more clear often the errors are traced back to data type issues, transformations and buffering.
When it comes to data migration project, ignorance is not bliss
The fact that over a half of projects involving data migration fail only goes on to say that a majority of organizations embarking on data migration initiatives are not aware of the fundamentals that make up the project. Strangely none of the organizations that embarked on initiative involving data movement do not expect project to fail.
Are organizations ignoring basics of successful data migration
Organizations that have lots of data and data movement will be the worst affected. As Bloor Research rightly puts “Approximately 60 percent of data migration projects have overruns on time and / or budget, which affect business continuity and disrupt operations”. Sadly many of the data movement projects fail because we do not recognize the fact that we need to apply proven methodologies, best industry practices and automate legacy processes – all essentially boils down to underestimating what it takes to make a project successful. For strange reasons, the focus is always on getting the project completed on time, but the efforts needed to make successful data migration always takes a back seat.
Manual tasks are the leading cause of enterprise quality issues
Business users can derive valuable business insights only when decision support systems or information systems are supplied with accurate data as raw material. When data is migrated to a new application, there is every chance that it may contain certain inaccuracies, typos and wrongful entries. Data inaccuracies can be traced back to manual mapping process which presents significant hurdles to an IT organization. Pre-ETL source to target mappings using spreadsheets involves frequent manual intervention, which makes the process expensive, time consuming, resource-intensive and error-prone.
What does it take to make a project a success depends on how the team defines the plan and approach besides best practices, tools, technologies and processes they will leverage to manage the project.
So many things come into focus while this date movement is on, like:
- What elements define the best migration methodology
- Are there any aspects of that process that could be automated
- As it turns out, many organizations typically rely on manual STM’s that consumes significant amount of time and resources when existing data is transferred to another application.., etc..!
Pre-ETL data mapping doesn’t get the attention it deserve
Pre-ETL space is usually a part of large implementations, and majority of the time the focus is on ETL, QA profiling and requirements phase rather than on mission critical and error-prone mapping phase. Currently Excel spreadsheets or other such home grown solutions are extensively used in data design phase but it is not free from errors and far from being a perfect solution. So using these excel spreadsheets to feed expensive enterprise systems is not only recommended but can be dangerous.
The data mappings specification challenges:
If you are planning to invest heavily in modelling tools, ETL tools and other parts of architecture and if you want to leverage spreadsheets for your design phase, there are a number of essential questions to ask yourself beforehand.
Your data design is among those phases- one we call essential for any large implementation project that should be free from errors.
These questions are very important in the sense that they will not only help prevent hiccups in execution but also device a data design strategy that can accelerate data integration.
Creating manual mappings using excel spreadsheets is one of the difficult task. Some of the fairly common challenges that are faced and making the job of data designing a little cumbersome are:
Mappings specifications built using excel spreadsheets cannot be easily managed Data mappings cannot be versioned and auditability of what and who has changed mappings remains a distant possibility.
There is hardly any automation and reuse options are limited in this approach.
A recent Gartner study indicates that IT Process Automation (ITPA) will be poised as a key component in the future of business success.
If this describes your organizations, we would like you to introduce Mapping Manager that can help you get rid of the legacy excel based mapping approach and put an automated process in place. You will notice that the solution can save you 70% in design phase, in ETL it saves 30 % and in testing phase it saves you nearly 10-20 %.
We encourage you to try Mapping Manager, we’ve got a free 30 day trial.
The best way to discover the capabilities of Mapping Manager is by seeing a live capabilities demonstration. We invite you to visit our webinar schedule and see the capabilities for yourself.