Hadoop: How to add all JARs onto the Classpath

ByTomas Zezula

I’ve just begun with Apache Hadoop. Getting started isn’t as straightforward as one might wish. There is quite a bit of work to do after the initial installation (“brew install hadoop” in my case). Expect tweaks and workarounds. Fortunately, there is plenty of advice available, such as this excellent blog post.

After having installed the latest stable version (2.7.1) and going through the initial setup I was still unable to use the damn thing! Strange as it sounds, some of the key modules were not present on the classpath. See the failure I got when trying to format HDFS:

$ hdfs namenode -format
Could not find or load main class org.apache.hadoop.hdfs.server.namenode.Namenode

I tried altering the hadoop-env.sh, but after a while I realised it was too much of a hassle. I ended up looking for a way of how to add ALL of the JARs in the Hadoop package to the classpath. It didn’t take too long to find the answer at Stackoverflow.

So, here is my take on how to add all Hadoop JARs on the classpath.

Open your bash profile (~/.profile or ~/.bash_profile) for editing and add the following:

export HADOOP_HOME="/usr/local/Cellar/hadoop" # Replace with your own path
export HADOOP_CLASSPATH=$(find $HADOOP_HOME -name '*.jar' | xargs echo | tr ' ' ':')

Save the changes and reload:

source ~/.profile

Finally, check the classpath. I’ve truncated the output, but you get the idea.

$ hadoop classpath
hadoop/yarn/hadoop-yarn-server-common-2.7.1.jar:/usr/local/Cellar/hadoop/2.7.1/libexec/share/hadoop/yarn/hadoop-yarn-server-nodemanager-2.7.1.jar:/usr/local/Cellar/hadoop/2.7.1/libexec/share/hadoop/yarn/hadoop-yarn-server-resourcemanager-2.7.1.jar:/usr/local/Cellar/hadoop/2.7.1/libexec/share/hadoop/yarn/hadoop-yarn-server-sharedcachemanager-2.7.1.jar:...

From this point on, all commands should work. Let’s get back to hdfs:

$ hdfs namenode -format
SLF4J: Class path contains multiple SLF4J bindings.
...
15/08/28 09:53:33 INFO namenode.NameNode: STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting NameNode
STARTUP_MSG: host = toms-macbook-pro.home/192.168.1.99
STARTUP_MSG: args = [-format]
STARTUP_MSG: version = 2.7.1
...

I would love to say that this is it, but you are likely to run into other issues. As usual, patience and Google are your best friends.

Similar Posts

Featured image
|

Stripe Payment Links and Custom Parameters

ByTomas Zezula

As of now, Stripe’s Payment Links don’t offer the flexibility of adding custom fields beyond those available in the API. The fields that you can directly pass into the Payment Links endpoint are complementary identifiers like prefilled_email or client_reference_id. This post shows some of the workarounds you can use to enrich your payment links with additional custom parameters.

| |

Spring MVC, TeeOutputStream, grep4j and How it All Fits Together

ByTomas Zezula

Grep4j caters for an easy search in log files. Since it is really easy to use and simple enough  I could not resist the temptation to try it out. I came up with a simple scenario: Let’s use Spring MVC to conduct a small JSON-based web app, capture the exchanged requests and responses in a…

Failed Gradle build
| |

Gradle Multi-Module Builds: Setting Up Shared Dependencies

ByTomas Zezula

When you are managing a multi-module Gradle project, more often than not you’d want to share certain dependencies across different modules. This is particularly true for test dependencies. You can achieve this by defining common dependencies in the root build file and applying them to all subprojects. However, a problem often encountered is the dreaded Unresolved reference: testImplementation error. This blog post suggests a simple trick that resolves the issue. Give it a go and happy testing ever after!

|

On Branching

ByTomas Zezula

Branching is a common practice in code versioning. To my experience, a branch is usually seen as a playground for experiments with brand new features. As far as I can tell, branches have typically been used as a natural shield against tight deadlines. Let’s branch out, see how it goes and take a qualified decision…

|

Hibernate LazyInitialization Exception

ByTomas Zezula

Would you agree that loading only as little data as needed, a.k.a lazy loading, is a good practice? You probably  would. To prevent lengthy object graphs be loaded into memory Hibernate uses lazy loading by default. The downside is that sooner or later you are destined to see something like this:   org.hibernate.LazyInitializationException: could not…