Our new software has been implemented and is now running as a production system. Of course, we still have some development and testing instances but real business is being taken care of. We have been slowly increasing the amount of work being done and the activity has been increasing also as people have become more comfortable with the interface.

Up to this point, I had been working mostly with the back end of the application and planning a second db which will tie into this one and support other external activities for our office. I say this because I had not paid much attention to the person data that was migrated during the vendor’s last visit. I started noticing that a few people had more than one personal record – each user should only exist once – but then I started seeing 3 or 4 times the same name. This peaked my curiosity and I needed to find the exact number of duplicate data.

Dealing with data duplicates can be a bit tricky, especially when you start dealing with last names like Jones, Adams or Smith and names like John, Steve, and David. However, we have the luxury of having a large student population with foreign names so that made my list of conflicting records a little bit smaller. So I opened up Enterprise Manager and started working my way around the data.

To make a long story short, we had about 600 records (out of ~ 19,000) which were duplicated at least once and had to be addressed. Out of those 600, approximately 150 were cleaned up automatically with some TSQL magic. The rest of them had to be checked out manually because some were legitimate duplicates (don’t ask!).

The point to this whole thing is that next time that you need to migrate data from one system to another, don’t rush into things and put some due diligence in the process. I took the user’s word for granted and I let the migration go as planned by the vendor. I am starting to wonder how much value this migrated data will have in the future? I think we could’ve been just as well off starting from scratch as we did with the other parts of the system. Lesson learned