If you already have Warcbase installed, check out the guide here!
Installing Warcbase
OS X
On OS X, requires dependencies. You should begin by installing homebrew. On the command line, you can then run the following:
brew install git brew install maven brew cask install java
Now we need to set JAVA_HOME. On my machine, it is
export JAVA_HOME=/Library/Java/JavaVirtualMachines/jdk1.8.0_74.jdk/Contents/Home
You might want to test to see where yours is by typing
ls /Library/Java/JavaVirtualMachines/
And then take the resulting version and plug it into your JAVA_HOME variable, such as if you saw jdk1.8.0_131.jdk in the list, typing:export JAVA_HOME=/Library/Java/JavaVirtualMachines/jdk1.8.0_131.jdk/Contents/Home.
Now with Git, Maven, and JAVA_HOME set, you want to install Warcbase. Clone the repo by:
git clone http://github.com/lintool/warcbase.git
Now go to the warcbase directory (cd warcbase) and build just warcbase-core:
mvn clean package -pl warcbase-core -DskipTests
Now we need to install “Spark Shell” so we can interface with Warcbase. As of April 20th, install this version in a different directory.
wget http://d3kbcqa49mib13.cloudfront.net/spark-1.6.1-bin-hadoop2.6.tgz
and then un-tar it using:
tar -xvf spark-1.6.1-bin-hadoop2.6.tgz
We are almost ready! Change to the Spark-Shell directory, and then run this following command (make sure to point the —jars at the warcbase jar).
./bin/spark-shell --jars ~/penn/warcbase/warcbase-core/target/warcbase-core-0.1.0-SNAPSHOT-fatjar.jar
You should now be at a prompt.
Type :paste and press enter.
Now paste the following script and then press Ctrl+D. Make sure to change the path appropriately!
import org.warcbase.spark.matchbox._
import org.warcbase.spark.rdd.RecordRDD._
val r = RecordLoader.loadArchives("/Users/ianmilligan1/penn/warcbase/warcbase-core/src/test/resources/arc/example.arc.gz", sc)
.keepValidPages()
.map(r => ExtractDomain(r.getUrl))
.countItems()
.take(10)
Success?
Now you’re ready to move on to the guide.
Linux
This is similar to above, but with some differences.
First, install git:
sudo apt-get install git
Download Apache Maven manually
wget http://apache.mirror.gtcomm.net/maven/maven-3/3.5.0/binaries/apache-maven-3.5.0-bin.tar.gz
and untar
tar -xvf apache-maven-3.5.0-bin.tar.gz
Now we need to set maven variables so we can use it from anywhere
export M2_HOME=/home/ubuntu/apache-maven-3.5.0 export M2=$M2_HOME/bin export PATH=$M2:$PATH
You may need to change your paths accordingly (in M2_HOME).
Now we need to install Java. The following works for Ubuntu 14:
sudo apt-get install openjdk-7-jdk export JAVA_HOME=/usr/lib/jvm/java-7-openjdk-amd64
And the following for Ubuntu 16:
sudo add-apt-repository ppa:openjdk-r/ppa sudo apt-get update sudo apt-get install openjdk-7-jdk export JAVA_HOME=/usr/lib/jvm/java-7-openjdk-amd64
Phew, now we’re ready to install warcbase. Go back to where you want to now install Warcbase.
git clone http://github.com/lintool/warcbase.git
Now let’s build
cd warcbase mvn clean package -pl warcbase-core -DskipTests
And now we can build Spark shell! As of April 20th, install this version in a different directory.
wget http://d3kbcqa49mib13.cloudfront.net/spark-1.6.1-bin-hadoop2.6.tgz
and then un-tar it using:
tar -xvf spark-1.6.1-bin-hadoop2.6.tgz
We are almost ready! Change to the Spark-Shell directory, and then run this following command (make sure to point the —jars at the warcbase jar).
./bin/spark-shell --jars ~/penn/warcbase/warcbase-core/target/warcbase-core-0.1.0-SNAPSHOT-fatjar.jar
You should now be at a prompt.
Type :paste and press enter.
Now paste the following script and then press Ctrl+D. Make sure to change the path appropriately!
import org.warcbase.spark.matchbox._
import org.warcbase.spark.rdd.RecordRDD._
val r = RecordLoader.loadArchives("/Users/ianmilligan1/penn/warcbase/warcbase-core/src/test/resources/arc/example.arc.gz", sc)
.keepValidPages()
.map(r => ExtractDomain(r.getUrl))
.countItems()
.take(10)
Success?
