I think The Programming Historian sums up my wariness of data mining best:

Screen Shot 2014-10-17 at 10.09.42 AM

Sure, these are powerful tools if one takes the time to get to know them and can do so without permanently damaging the data itself. There is the beginning of a solution in making multiple backup copies of data, but I know from working with different versions of CSS and HTML for my own webpage that even when files are properly named and organized, it is very easy to lose track of which version is the most recent, leading to duplicated work and inconvenience.

It took me several tries to get my own data loaded into Voyant. I began by trying something from Google Books, Willard Glazier’s The Capture, Prison Pen, and Escape. PDF would only read the first page of the file, which is always the Google Books information sheet. EPUB did slightly better, but I could not load individual pages and all of the “most common words” were shown as individual letters, often with accent marks and other abnormalities. Because I could not see the individual pages, it is hard to say for sure that Voyant had translated it into gibberish, but I suspect that was the case.

Finally, I went to good old archive.org, because I know that they provide books in plain text. I found Alonzo Cooper’s In and Out of Rebel Prisons and got Voyant to at least show me the word cloud and list of most frequently used words–in English, no less! The internet both in my office on campus and at my apartment is apparently too slow to load some pages, although a few have loaded in the time it has taken me to write this post.


Screen Shot 2014-10-17 at 10.24.02 AM

Clearly, the text needs a great deal of help. I’m haunted by a line from an article by Ted Underwood, in which he states that he had to write 50,000 individual rules to correct OCR from 18th and 19th century manuscripts. While his material comes from a slightly earlier period and my Civil War documents don’t have the f/s problem, correcting this OCR would still require thousands of lines.

I’m still having trouble conceptualizing how text mining could help me in my own research. The applications I’ve thought of have been things that I don’t necessarily need text mining to do–for example, if I want to find “Unionist” in close proximity to “Transylvania County,” then there are tools already built into Google that can do that.

After a few more minutes of playing around and mapping frequencies, I did come up with one very basic thing that Voyant might be able to do. This is a screenshot showing the frequency of the word “prison” in the manuscript. While rudimentary, it does show that there is a significant drop in the use of the word starting around a third of the way through and continuing past the halfway point of the text. By clicking on the lowest point and skimming the text, I could see that this section of the text described Cooper’s escape from a Confederate prison in Columbia, SC. I knew that–that’s why I’m interested in the book–but if I were working with an unfamiliar text and wanted to know if it could be useful in my research, then a similar technique might allow me to rule out books that have little or no relevance.

Screen Shot 2014-10-17 at 10.33.45 AM

One thought on “10/17/2014

  1. Thanks for the link! To avoid undeserved imputations of heroism: I didn’t really *write* 50,000 correction rules by hand. I mined them, in a variety of ways. (Very common “words” that aren’t in the dictionary and can be transformed into dictionary words by f/s substitution, for instance.)

    But I’m not contesting your point that text mining is hard, and it may not be worth inventing a huge process to do something we could already do in other ways. (Of course, searching for “Unionist” in proximity to “Transylvania County” is itself, arguably, text mining …)


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s