Output

After a great day yesterday at Code4Lib North, and playing with some of Nick Ruest’s WARC files from the ‘Free Dale Askey’ collection, I’ve put everything together with the bash and Mathematica script.

I’ll be playing with some of this stuff in two weeks at the Canadian Historical Association‘s annual meeting in Victoria, in my presentation entitled “The Internet Archive and Social Historians: Challenge and Potential Amidst the WebARChive Files.” Slides and text’ll be up later, and I will also make the full text paper available to interested parties.

I see a potential use for this in collections of WARC files where there are no finding aids, just a bunch of files: which I think can be relatively common in ‘just-in-time’ grabs or in other case studies. In the best case, I think the script used for the Dale Askey collection is great – each file has an attached screenshot and PDF in addition to the WARC – but that this has a role for cases where little information is attached (i.e. more useful for cases like this one than this one with more data).

Here’s how it works. Unfortunately, you need Mathematica.

Two files:
- All-Together-MMA.sh: Takes a WARC file passed to it from the command line, generates full-text, topic models it, and then invokes…
- WARC-to-Analysis-single-file.m: a Mathematica script that generates the PDF file discussed in the last post.

How to run it:

On command line, once made executable (otherwise prepend sh):
./all-together-mma.sh 00016-2013_02_23.warc

This takes the WARC file, one of the Dale Askey collection, and runs it through the script. With proper directories set in the files, it generates output as above in one step. A big benefit of this is that I can now automate this across a ton of WARC files.

Work to do:

- Need to refine stop words
- Topic models are set up for large corpuses, so running 50 topics on a single page is overkill.
- Sparklines are set up for large corpuses as well, so output is weird on one page. But still can be moderately useful.
- Integrate with AlchemyAPI for sentiment analysis? Multiple KWICs?

This program creates a PDF like this from a WARC file, building on previous work.

I also have the ulterior motive of getting historians more involved in this topic (there are certainly a few already working in the field) – we were around to design the first generations of archival boxes (historians were fundamental in the early days of the archival profession), and want us around as we tackle the second generation of the web archive box.

Today, using Mathematica, I developed my script further. It can be called from the command line if you have Mathematica, and so can be built into the aforementioned workflow. It does the following:

In Mathematica, you can change the search word and see KWIC change dynamically. The final PDF output requires some pre-defined keywords, however.

- takes the fulltext file and topic modelling data generated by the previous script;
- generates word frequency information and displays them in a word cloud;
- provides Keyword-in-Context of specific words (or, if run from Mathematica, can provide dynamic information as seen at right);
- and visualizes the topic models, in declining order of overall prominence within the WARC file, using sparklines to demonstrate whether it is evenly spread throughout the file or just in a few files.

It then generates a long PDF file that you could store alongside the file, or use – with the specific keywords that you’re using – in an attempt to ascertain whether a given WARC file is handy for your research. It also shows how we have moved from large, ungainly WARC files into an area where we can apply text mining tools to them. Click through to see the final output:

Download a trial-1 here, or see below for a graphical version. Code is at github:

Work still remains to be done, of course, especially on cleaning up the text that’s going into the system. That’s for tomorrow, however.

If anybody has anything they think would be a good addition, or other things you would like to see, feel free to comment, e-mail, or tweet me.

I submitted it, and after further pruning by their staff (an aside to this aside: great editorial staff, from the contact people to the copy editors), a very short version appeared in today’s issue.

It’s almost certainly the shortest thing I’ve ever written, but I like to think that it’ll get some readers and hopefully encourage more  researchers to hop on the digital preservation bandwagon.

Thanks to all my readers who reblogged and retweeted my earlier post, which helped spread it around the Internet.

Regular readers will know that I’m interested in how historians can approach web archives, as discussed in a three part series in late 2012 (see part one, two, and three). As I’ve stressed, in both tweets and in some draft writing: Historians need to understand web archives, however, as we will be professional end users of these archives.  We played a critical role in shaping the modern practice of traditional archiving. Let us make sure that historians are present for the next step. There’s a conversation, but its largely amongst people involved in web archiving as creators rather than as users.

[if you want to skip to my code, it's here]

So here’s me positing a problem: Some web archives do not have description, so you aren’t sure what you’re going to find inside. This includes some just-in-time web saves, like this mirror of the Montreal Mirror’s website. There’s always an item listing, automatically generated, that lets you know what exactly is in the website. When dealing with wide web crawl data, part of that massive 80TB dataset, this is a life saver. Very briefly: Web ARChive files are complicated containers of multiple files – ActiveHistory.ca, for example, is made up of over 18,000 files. That’d choke a file system, but you can turn it into one Web ARChive that you can play with later.

Furthermore, these WARC files are too big. Wouldn’t it be nice if you could, at a glance, see what ianmilligan.ca is about without having to read what I’m writing here? (yes, but also imagine if you were just looking at a bunch of visualizations – would be invaluable in a research project)

But for the historian, it’s not terribly useful in and of itself. What if we had a lot of these files, how could we quickly see what related to our topic, and what didn’t? Would we be able to automate it?

Here’s my idea, which I cooked up as a way to learn some more technical skills and invent a tool to help me in my workflow. What if we could take a Web ARChive file (WARC) and then hook it up to a topic modelling visualizer, like Stanford’s Termite [see their paper here]? This flow would ideally take a WARC file and then give us an output like this:

Here is Termite running on the text of the 1933 Canadian Commonwealth Federation’s Regina Manifesto. For more information on topic modelling, you can go check out our Programming Historian piece.

Termite is great, but requires its data formatted in a particular way: a file number and then a document on one single line, in tab-separated format. It isn’t terribly friendly to just dump websites into. This is preliminary work, started two days ago, simultaneous with me learning how to script in bash. It works for me, but there’s no sense in me working in a bubble.

Here is my attempt to introduce a workflow, written in Bash, drawing on various Python tools, written on MAC OS X and presumably compatible with Linux. What it does is the following:

(1) Finds a website, such as http://ianmilligan.ca, and mirrors it all into a single WARC file

(2) Transforms that WARC file into a full-text searchable index, which is far more usable in terms of size. Downsides: we lose some of the context provided by images, etc. If images have alt-image tags, those are filtered in with the text.

(3) Takes that full-text searchable index and transforms into Termite format, each individual file that makes up a WARC being given a single line.

How does it work?

Step One: Install Initial Tools

WGET: It requires wget, with WARC functionality (so no old versions). If you don’t have that, you’ll need to install it. If you’re on a MAC, you need to download xCode, command line tools, and compile it. Follow the specific instructions in my Programming Historian piece.

LYNX: These tools use Lynx to read the WARCed webpages and display them. Newer versions use Beautiful Soup 4, but I’m sticking with Lynx for now (I’ve been working with it for a while and am fairly impressed). Follow the detailed instructions here.

Step Two: Download and Install Stanford/Termite

Remarkably for such a project, the documentation is exceptional. They’re on github, so check it out there.

Step Three: Download and Install Historian-WARC-1 Files

For these, check out my github repository here.

Download them all to a directory (click the zip button above for convenience). The easiest way is to place all these files into the same directory that you have Stanford-Termite running. 

You’ll need to initialize the WARC-Tools. In your terminal window, in the /WARC/Hanzo-Warc directory, run the following commands:

./setup.py build

sudo ./setup.py build install

You’ll be asked for your root password on the latter.

Step Four: Point all the Directories at the Right Place

For the Historian-WARC-1 files, you’ll need to change a few things.

Editing All-Together.sh, make sure to:

Change the URLTOGET value to the website you’re interested in looking at. You might want to change the OUTPUT field too.

Then you’ll need to change, in line 38, the path to the warc-tools-mandel directory.

Then, editing TRIAL.CFG, make sure to:

Change the path to the directory you’ve installed everything in. There are three of these. You may also need to alter the number_of_seriated terms variable as well.

Otherwise, try to read the comments, and feel free to leave comments below. If it all works, you can run this script – it grabs a website, WARCs it, and then visualizes it in termite on localhost:8888.

Output from ianmilligan.ca – you can get a sense of who I am pretty quickly, without having to read all the content. (thank god, eh?)

SSHRC Postdoctoral Fellowship Success Rates, 1995-2013.

The above includes the 2013-14 results. We’re seeing a blip upwards in the success rate: there are more awards and a slight decline in the number of applicants. We’ll need a few more years to see whether this is a statistical hiccup or the beginning of a new trend. There might be some chill factor with the impending tightening restrictions around the number of applications, announced last summer. In any case, this success rate still is low: although it is an uptake, it is still lower than at any time between the 1996-97 competition year and the 2008-09 competition year.

An additional thing to keep in mind: The “History Wars” have been getting a lot of attention lately, most recently due to the federal government’s decision to begin a “comprehensive review of significant aspects in Canadian history.” The specifics are best left to others, notably Active History co-editor Thomas Peace. I don’t want to get into it, but certainly there’s a tendency for the Conservatives to want to encourage scholarship that they find politically agreeable (military history, Diefenbaker) – just as there was for the Liberals (peacekeeping, multiculturalism). But keep in mind, as we discuss historians, that it’s quite frankly getting harder to do any kind of history as a professional undertaking. 

There’s a war on junior scholars in Canada. That’s being a bit provocative, I know, and the federal government doesn’t deserve all the blame. We have seen declines in total postdoctoral fellowships awarded in several years as seen above, which we can attribute to funding issues. Universities and provincial governments also deserve some blame, as they’ve flooded the market. But whatever the root causes, we’re seeing a major issue of human capital in the data above.

This affects all of us: whether you’re a military historian, a social historian, a Conservative, a Liberal, or a New Democrat. If we really care about history, let’s put our money where our mouth is and help fund the next generation of historians. It affects junior scholars most of all, but also senior scholars. This is the future of the profession.

A return to 2004-05 success rates would mean 62 more postdoctoral fellowships would have been awarded this year (244 out of 903 would have been a 27.02% success rate, which we had in 2004-05).

It’s getting harder to commit history.

I love exploratory research. A forthcoming article grew out of exploratory research (blogged about here), when I was messing around with dissertations and citation counts. Now, on the other hand, that obscures out the days and days that I’ve literally spent hitting up against dead ends, batting my wall up against bad data, technical limitations, or sources that never really went everywhere. I have a terabytes of external hard drives, filled up with datasets, some of them representing a few days of work that won’t soon see the light of day.

So it’s nice to have these guilt free days to just play, in a constructive way. I think of it as akin to the Google 20% time. To check out what’s new at the Internet Archive. To take an abstract problem and play with it in a programming language. To listen to a CBC debate on Canadian history.

So what have I discovered today?

- Well, Jim Clifford and I spent a good hour figuring out how to do a pattern match for a list made up of mixed integers and strings in Mathematica. Sounds boring, eh? Next time this comes up, it won’t take an hour – it’ll take a second. That’s the joy of programming. And I’ve hopefully made just a small, minor contribution to the understanding of global historical commodity flows.

And because it took so damn long: Cases[test, {x_ /; 1620 <= x <= 1629, _, _}] got us the dates we needed.

- You can download Internet Archive collections en masse, thanks to their openness to using wget on their collections! Check out the blog post here, and try generating your own list here. I’m downloading a massive collection of magazines right now. Once again, there’s a good chance that it’ll just sit on my external hard drive and collect digital dust. But who knows. (p.s. you can check wget out at the Programming Historian)

This command, which you can see broken down on the Internet Archive blog, is downloading an entire item list for me – just the text files and some of the images.

wget -r -H -nc -np -nH --cut-dirs=2 -t 1 -A .txt,.jpg -e robots=off -l1 -i ./itemlist.txt -B 'http://archive.org/download/'


You could use that to download cookbooks en masse, or Statistical reports, or really anything you might want.

- I discovered a new OS X command, textutil. There’s probably a reason I’ve never heard of it before, but it worked on some Lynx-formatted web archives where html2text balked. This is probably more my fault than html2text’s, but it’s still nice to have a built-in terminal command for this stuff.

- Also, the coffee over at Engineering’s appropriately named “Coffee and Donuts” has more caffeine than the President’s Choice stuff I normally drink here. Gah!

Come Monday morning, I’ll be back at the main projects: wrangling these articles together, preparing a conference presentation, responding to the e-mails that are stacking up, and so forth. But for the rest of the day, it’s play time.

Active history is history that listens, is responsive and encourages a broad range of forms of public engagement. As the accessibility and volume of digital content increases, so do possibilities for digital outreach activities. These opportunities bring challenges, benefits, and new methods of approaching the past. This panel focuses on the intersection of active history and digital technologies; with an emphasis on community involvement, alternate reality games, digital vs. physical engagement, and the possibilities of engaging disparate audiences.

What will we be talking about?

I may be kicking things off with an introduction to the website ActiveHistory.ca, highlighting some of our successes, challenges, and the errors we made along the way. Engaging with the public isn’t just as simple as throwing up your shingle. I’m going to argue that “blogging can and needs to be considered a central component of a university-based historian who wants to engage the public.” We’ll see if I’m preaching to the crowd.

Things will get more interesting as others move beyond my often utopian visions of the Internet and provide their perspectives from perspectives of very involved public historians! Krista McCracken (an editor at ActiveHistory.ca) is going to be drawing on a case study from some very moving residential school photographs, talking about the shift from the “real world” to “digital space.” What challenges are presented? How can you engage with communities digitally? The question of audience?

Devon Elliott, of DH Maker Bus fame , is looking at DIY in the sense of all the making he does! What does it mean to make objects that speak to the past? What do they do at the Fab Lab at Western? How can there be an Active History beyond text? Devon came to speak to my class at Waterloo, and I’m not sucking up to him when I note that it was an overwhelming favourite ‘hit’ of the course.

And then Tom Peace – who helped come up with the original Active History idea with Jim Clifford and a few others way back in 2008 – highlights the issue with just relying on the web. As an engaged community member, leading Jane’s Walks, grasping the socio-economic complexities of cities like his hometown of Hamilton, Ontario, Tom brings a good cautionary note. Whenever I’m getting too utopian, he’s usually there to bring reality to bear.

All of this is moderated by Nathan Smith from the University of Northern British Columbia, who co-leads the Active History Canadian Historical Association group and is a really engaged scholar who brings it together.

It’s funny, as we often tend to put forward imposing, confident faces on social media: the researcher, teacher, and administrator with his/her stuff together, progressing forward. Even in a few months where I’ve designed and taught two new courses while moving a pretty aggressive research agenda forward, that voice is always there: “Is this enough?” “Shouldn’t you work just a bit later on the weekend?” “How dare you take a break when others fight for full-time work?”

These articles helped remind me that I’m not alone. Hopefully others find them helpful too.

Back to work, though! Tomorrow, I’m off to Ottawa for NCPH 2013, and am looking forward to a glass of wine on the train.

Topic modelling, as the review and Programming Historian 2 piece notes, have been critical parts of my digital toolkit over the last year or so. It’s enabled me to make cursory explorations of very large datasets, from musical lyrics, Canadian history dissertations, dead websites, and even more. Some of the initial euphoria around it being a ‘magic bullet’ for everything have worn off (honestly, the first time you try it, you’ll be hopping off the walls) but it’s an incredible program to use.

If you ever want to chat topic modelling, shoot me an e-mail.

It made me realize how much history has probably been written because somebody didn’t throw something out. Jonathan Vance, a historian at Western University, said as much at a keynote I attended last month – that his greatest fear was garbage day. The war records lost when somebody passes away and a son or daughter doesn’t know what they are. All those diskettes and hard drives that might just get tossed because “who cares about that old data.” The invaluable building blocks of a future social historian.

So I laid down a challenge to my students: that they were historians now, and that they’d always be historians.

And that would be true if they didn’t go into academia, as most will not. They will still be historians if they go into the private sector, if go into primary/secondary education, if they start working for the Ministry of Education.

And, probably getting increasingly animated as I paced around our lecture room, I tried to explain what that means to me. That beyond the usual things that we know a history education gets them (great reading/writing skills, an ability to evaluate competing narratives, evaluate sources, cut through the bullshit, etc.), that there was a responsibility – a responsibility to always think like a historian.

Because I was wondering – what if a historian had been on the senior staff at Yahoo! What if a historian had been part of Google when they planned on deleting Google Video? (And by this I don’t mean somebody who might happen to have a History BA, picked up along the way – but somebody who thinks like a historian)

That they should (and would, I argue):

• Question the destruction of sources: raise a flag when somebody suggests shredding some documents, or throwing them out en masse;
• Think ahead to the future, especially when thinking of the digital: keep those old disks, keep those hard drives, keep backing up and making available to future generations (and to themselves, should they want to look back on their own past down the line);
• Keep it Open; I was so happy to see several of my students slapping CreativeCommons licenses on their work, and moving out of the realm of Facebook and into websites. Facebook might not be preserved, but their WordPress site certainly will – as will the data;
• If they see something important, keep it or at least be mindful of its potential. I made a joke about how it might not even be legal. Obviously, I don’t want my students breaking the law. How much of our history, especially around the contentious grey areas of our past, is possible because somehow something became available. Whether its advocating for declassification of materials, or in some cases, maybe even saving something for the distant future…. I dunno.

I guess my challenge to them was that they’re hopefully going to get a lot out of a history degree. And that they are historians, forever and ever. You don’t need a PhD to be a historian, let alone a BA. It’s a frame of mind, and one part of that ought to be preserving sources.