Converting to a New Genealogy Program - RootsMagic and GEDCOM Changes

This is a continuation of my series on converting to RootsMagic version 5 (RM5) genealogy software. Earlier articles can be found at:

1. Software Conversion - Moving to RootsMagic

2. Before Converting to a New Genealogy Program - RootsMagic Conversion Notes

As part of my conversion to RM5 I decided to modify the GEDCOM file directly prior to import to get the cleanest possible import of my data. This should only be attempted by those who understand the GEDCOM spec and how to edit files in a pure text mode (not using a word processor that inserts invisible formatting characters). See the "Better GEDCOM Wiki" for more information on GEDCOM.

I've decided to describe the changes I made generically instead of including the actual code. First, the descriptions will be more readable. Second, I don't want to give enough information to allow novices to cause problems they then take to RootsMagic tech support. Anyone who understands GEDCOM will probably understand exactly what I did based on the descriptions below. If you try this and have problems, it’s your problem, not something the vendor should have to solve for you. Also, if I include all the details from the notes I made as I implemented the changes, this blog post would be many pages longer.

If I had implemented more of the custom features in my old genealogy program I would have needed to do a lot more manual work to get a clean import. Each person will have to look at their own usage and determine what additional GEDCOM changes might be needed. If you can modify a GEDCOM file directly you probably know enough about the customizations you used to figure that out.

This is the list of things I changed in the GEDCOM file before I got a fairly clean import:

1. Moved place details from the place field to place details. The output GEDCOM file includes all place fields at one level. RM5 has a place details subfield for things like "Baylor Hospital" so the master place contains city, county, state, and country if you use it.
   
2. Moved long text notes from the occupation details field. There seems to be a character limit on how much RM5 will import. Or something in my data was causing an issue on the events where I had a long set of sentences. RM5 uses the occupation detail field to hold the occupation name.
   
3. Moved long text notes from the military details field. There seems to be a character limit on how much RM5 will import. Or something in my data was causing an issue on the events where I had a long set of sentences.
   
4. Renamed my custom event tags to a standard one used by RM5, where a standard one will work. I kept custom tags I still use or that contain data I need to transfer inside of RM5 (my Research tags which may become ToDo items, Research Logs, and Research Notes in RM5).
   
5. Changed the path to exhibits (media in RM5 lingo). As part of the conversion I decided to move my document images and photographs to a higher level folder in my document area. It shortens the full path name if I want to use links to these images from other programs.
   
6. Removed references to customizations that don't transfer through GEDCOM such as [WO], [LINDEX], and a few others.
   
7. Removed special characters my old program used for privacy that are not understood by RootsMagic. RootsMagic knows how to handle my sensitivity brackets {} so I can leave those. It does not know how to handle hyphens used as exclusion markers.
   
8. Removed pseudo-people such as the "census" people I added years ago and no longer use.
   
9. Modified the sources so I could get a clean, consistent first footnote or endnote. This will be usable until I convert the sources to formatted source types within RM5. RM5 imports sources as free-form sources. My old program exported them with the data mixed with the field names where I had customized the sources to match Evidence!¹ and later Evidence Explained.²
   
10. Modified repository references in source citations. I also took this opportunity to consistently refer to repositories with the same name. National Archives, NA-Washington, and NA all became NARA. I chose to use the abbreviation NARA (and TSLAC for Texas State Library and Archives Commission, dw. for dwelling, fam. for family, etc.) in the free-form sources as there will be a lot more "subsequent" footnotes than "first" footnotes. If I can remember to change the first abbreviation usage in a report to use the full name and show the abbreviation in parentheses, there will be fewer changes to make in output reports until I convert my sources within RM5. RM5 has a nice way of handling first and subsequent references and the abbreviations within.
    
11. Used a Perl script (a simple programming language) to remove the reference number for each person so I did not have to do this manually. Years ago I used a modified Henry number, as defined in William Dollarhide's Managing a Genealogical Project.³ Once genealogy software was capable of printing an indented descendant list I really didn't need these numbers any longer, but they were still in my old database.
    
12. Fixed a few spelling errors.
    

After importing the GEDCOM file I reviewed the import errors, fixed them, and then imported to a clean database. Repeat until no import errors are reported. I then spent some time reviewing the data and reports in RM5. I exported a GEDCOM file from RM5 and was pleasantly surprised. Not only does it export the basic data, it exports many of the customizations. If you color code people in your RM5 database, even the color is exported. This means imports to another RootsMagic database will be cleaner. Other programs likely won't know how to import all of the information that RM5 exports. But maybe someday all genealogy programs will do this.

RM5 also exports all three types of printed sources—first notes, subsequent notes, and bibliography format sources. Because my sources were imported as free-form sources, all three formats contain identical text right now. I considered taking the time to edit the GEDCOM file output by RM5 to make my citations clean in all three forms. But I decided to wait until I convert from free-form to formatted sources instead of spending a lot of time on the free-form sources.

There was one major issue I had with the way my data printed after my last GEDCOM import. RootsMagic has a different philosophy about how event notes should display in reports. The event data such as date, place, and details print as specified in a sentence template. Then a citation reference is printed. Then any "notes" associated with that event are printed.

Basic writing guidelines say a citation reference should follow all of the information that comes from the cited source. But I now have a lot of information that is printed after the citation reference for the source of the information. At this time there doesn't seem to be a really good workaround for this. This may not be a big deal for those who aren't printing many reports or don't need the report to follow a particular set of style guidelines. But it is a big deal to me. Not big enough to keep me from using RM5 for now.

As the best workaround I could think of after asking experienced users, I made one last set of changes to the GEDCOM file output from RM5. After making this change I then imported it into a clean RM5 database. The change I made was to add the phrase "Additional information: " to every note attached to an event. This won't be pretty when I print a report. But it gives me a unique phrase to search for that marks the places where the text will need to be cleaned up in each report. If (hopefully when) the RootsMagic program is changed to print citation references at the end of a note, I can use the search and replace option inside of RootsMagic to remove this phrase from my notes. For now I can search and replace in my word processor so I know where I need to make changes before sharing a report.

So is my database clean and perfect now? Not by a long shot. But it wasn't perfect in my old program either. Data entry standards change over the decades. Not all of my sources of a similar type were entered the same way in my old program. At least now they are consistent. And I have a usable database I can add to and get output reports from—even in a word processor format in my 64-bit operating system.

One thing I did notice when I imported the RootsMagic GEDCOM back into RM5, DNA events don't seem to be imported and DNA test result values don't seem to be exported via GEDCOM at this time. I had entered DNA results for one person so I could test some things as I played with RM5. A DNA event was included in the GEDCOM file created by RootsMagic. The marker values were not exported. Even the included DNA event was not imported. The lesson learned here is be sure you know what is not included in a GEDCOM export or import—don't assume everything is included even if a GEDCOM file includes a lot data more than some other vendors include.

Good luck if you decide to convert to a different genealogy database. It's not an activity for sissies. If you want more information you can contact me using the e-mail address on my website or add a comment below.

---

1. Elizabeth Shown Mills, Evidence! Citing History Sources from Artifacts to Cyberspace (Baltimore, Maryland: Genealogical Publishing Company, 2007).
2. Elizabeth Shown Mills, Evidence Explained: Citing History Sources from Artifacts to Cyberspace (Baltimore, Maryland: Genealogical Publishing Company, 2007). Evidence Explained expands on and updates concepts presented by Ms. Mills in Evidence! Citation & Analysis for the Family Historian. There is a newer edition of Evidence Explained than the one I have. See evidenceexplained.com for current version and ordering information.
3. William Dollarhide, Managing a Genealogical Project: A Complete Manual for the Management and Organization of Genealogical Materials (Baltimore: Genealogical Publishing Co., rev. ed. 1998).

© 2012, Debbie Parker Wayne, CG, All Rights Reserved

Before Converting to a New Genealogy Program - RootsMagic Conversion Notes

This is a continuation of my series on converting to RootsMagic version 5 (RM5) genealogy software. Earlier articles can be found at:
1. Software Conversion - Moving to RootsMagic

I assume you have your own reasons for wanting to convert to RM5 or a different genealogy program than the one you use now. I gave mine in part one of this series where I discussed my old program, The Master Genealogist (TMG). As part of my decision and testing process I watched all of the free RootsMagic webinars. I played with data I entered directly and data I transferred into RM5. At some point I decided RM4, and later RM5, would meet most of my needs—enough to make me go through the agony of a software conversion process.

As part of my decision-making process, I read (almost) every blog post, mail list message, forum message, and FAQ I could find on converting in general and converting specifically from my old program. Search terms included GEDCOM, import, and similar words, as well as the name of my old program and RootsMagic. In addition to general searches using Google, I did searches on RootsMagic lists and forums.

According to the RootsMagic Features list, RM5 does direct imports from specific versions of some software programs. TMG isn't on the list; the move requires transfer of my data via GEDCOM. This almost always means some data does not transfer properly and requires a lot of cleanup work after the transfer. This is not the fault of any particular software company. Those custom features we all demand were mostly conceived after the GEDCOM spec was written. Each vendor has to interpret how to include data that isn't part of the GEDCOM spec. Each vendor has to decide how to interpret that data when it is imported. All vendors don't agree on the how. See the "Better GEDCOM Wiki, Shortcomings of GEDCOM" for more information.

A typical GEDCOM transfer process would be:

1. Create a GEDCOM file from TMG. Look at all the export options offered and try to choose what looks like it will provide the cleanest import to RM5.
   
   
2. Import the GEDCOM file to RM5.
   
   
3. Review the data on the screen and in printed reports and use RM5 to change everything that didn't import cleanly.
   

If you aren't familiar with GEDCOM specs and editing plain text files, you are better off making changes inside of the RootsMagic program. Because I am comfortable changing raw GEDCOM files, I decided to make some changes and redo my import. (Details on this will be in a future post.)

For those who understand the GEDCOM specs and prefer making the changes in the GEDCOM file then reimporting it, the process might be:

1. Create a GEDCOM file from TMG. Look at all the export options offered and try to choose what looks like it will provide the cleanest import to RM5.
   
   
2. Import the GEDCOM file to RM5.
   
   
3. Review the data on the screen and in printed reports to determine what changes will be easier to make in the GEDCOM file and which can be easily changed after the final import. Check placement of imported data in the person events, notes, and sources—every database field into which your imported data was placed. Print narrative reports with footnotes/endnotes and charts. Look at placement of citation reference numbers, wording on footnotes/endnotes, etc.
   
   
4. If you see things that don't look right on screen or in reports, search to see if there is a way you can modify the GEDCOM file and reimport the data to correct the problem. Whether changes will be needed or not will depend on how much customization you did in TMG. If all you have in your database are basic facts with dates and places and minimal notes, your data may import properly without requiring any changes.
   
   
5. Add a sample family to RM5 with data and source citations properly entered. Create an output GEDCOM file from RM5. (I discovered RM5 GEDCOM output is much more complete than what I got from TMG using what I thought were the best options. I am not sure how many other programs will correctly import all of this RM5 data, but at least transfers to other RM databases would be fairly complete.)
   
   
6. Review the GEDCOM file created by RootsMagic to determine how to modify the GEDCOM file created by TMG so imported data goes into the correct fields in RootsMagic.
   
   
7. Repeat the process from step 2 (import to RM5) until you get an import you can live with in step 3, then stop. If you try to go for a perfect import you could be stuck in this cycle forever. You have to be realistic about how much must be corrected at the import stage and what you can do later as time allows.
   

© 2012, Debbie Parker Wayne, CG, All Rights Reserved

Software Conversion - Moving to RootsMagic

Disclaimer: This post contains opinions based on my understanding of events and my experiences. Others may have different interpretations of these events.

Don’t tell anyone, but I’ve been a secret heretic for almost a year now. At least, according to my genealogy software buddies. I’m switching to RootsMagic as my main genealogy software.

My software of choice changed in the beginning years of my research, but I’ve used one program for over sixteen years now. For a few months I used PAF which was limited in source citation implementation and other ways. I tried FTM and quickly decided it was only good for pretty charts as far as I was concerned. I found Everyone’s Family Tree (EFT) by Bill Dollarhide and happily used it for many years. By 1995 EFT wasn’t being updated and I had been reading about The Master Genealogist (TMG) and its advanced features. As a techie, being able to customize my software appealed to me in a big way. I’ve been a user of TMG since 1995—way back in the DOS days.

At the time TMG was the most sophisticated, advanced, user-customizable genealogy software of all the applications I tested. But in the last sixteen years several other programs have caught up with TMG. Some even surpass TMG in certain areas. I started using RootsMagic when a client wanted the research I performed to be entered into a database they could get at the end of the project. I didn’t want to have to teach a non-techie how to use TMG. While I really liked TMG, too many had trouble understanding it when I tried to teach it to my local society members.

I found features in RootsMagic that I really liked. RootsMagic version 4 has most of the advanced features I use in TMG. RootsMagic version 5 has even more. And both versions have some features that aren’t in TMG—some that are important to me.

Switching from one genealogical software program to another can be an excruciating ordeal. What convinced me to endure the pain of converting my data? Two big things and a few small ones:

• Twice in the last ten years there have been long delays for new releases of TMG.
  
  • TMGv5 was delayed for over a year, in part, while features were added to entice customers of another software program that had folded (UFT). My personal research was at a point where I really needed one of the new features in TMGv5. I can’t blame the company for making a decision that brings them many new customers and more income. But it did impact my use of the program.
    
  • Since Microsoft 64-bit operating systems became widely available and the default installed on new computers, about 2006 or 2007, TMG users with a 64-bit OS have not been able to export genealogy reports to a word processor file. Suggested workarounds included exporting to a PDF or plain text file or keeping an old computer running Windows XP 32-bit version. As a techie I had a 32-bit XP laptop so this worked for me, until it died and had to be replaced with a Windows 7 system. But you can’t tell Grandma she has to keep an old computer when she just got her first computer and it has 64-bit Windows Vista on it. TMGv8 was finally released a few months ago, four years after TMGv7 and two-plus years after TMGv7.04. TMGv8 supports word processor output on a 64-bit operating system, but it comes months and months after I decided to convert.
    
• TMG is based on a database application that Microsoft is dropping support for. At some point a complete rewrite of TMG will be needed to port it to a modern database engine. My programming experience tells me this is likely to result in another long delay before a usable product is available for users. For all I know this port has been in the works for years and could be released tomorrow. But I have to make my decisions based on what I know, not what I hope has been happening behind the scenes.
  
• Some TMG users are so loyal to the TMG way that they don’t seem to see better options offered by other programs. There was a TMG-list discussion a while back about the new “Research Notes” report in RootsMagic. What some call a complicated process was documented to create a similar report in TMG. This demonstrated that the customization features of TMG can allow almost anything to be done. It also demonstrated how hard it is for a less-experienced user to do some things. In my opinion.
  
• By switching to RootsMagic for my personal research, I only have to know how to use one program, not two, for myself and my clients.
  

So, other than the “Research Notes” report, what convinced me to switch to RootsMagic? It is based on an open-source, cross-platform database engine (SQLite 3, according to Wikipedia). RootsMagic has already done the port to a modern database that I think TMG will do in the future.

As with TMG, techie RootsMagic users can write their own database queries (using SQL tools) to access the database. This should allow me to implement my own add-on to pull DNA data and lineage from my database to help with analysis of DNA results. I don’t want to spend all of my time programming instead of performing research, but it is nice to be able to access the underlying database when I need something not yet provided by a software program.

In the future I will post some of the details of modifications to the TMG output GEDCOM so I got the cleanest import to RootsMagic. I’ll also post with other information that may be of interest to RootsMagic users and those converting from other programs.

As an aside, one of the things I will miss the most about moving from TMG is the loss of Second Site by John Cardinal. It is an add-on product that creates a website from a TMG database, better than the website that either TMG or RootsMagic create. Second Site includes some of the best DNA charting tools I have seen for comparing DNA data stored in a genealogy database. I don't put my database online, but made frequent use of Second Site files to access my data without having to start the database program.

© 2012, Debbie Parker Wayne, CG, All Rights Reserved