Install Hadoop 2.6.0 on Ubuntu 14.04 LTS in one go

Hi,

I have created a shell script to install hadoop 2.6.0 on ubuntu 14.04 LTS. By using this shell script you can install you can install hadoop and configure it in one go.

This script will install and configure hadoop for single node cluster only.

#!/bin/bash

echo -n "Enter the new 'GroupName' for new Hadoop user : "
read hdGroup
echo -n "Enter the new 'UserName' for Hadoop: "
read hdUserName


echo "...................Updating your System.................."

sudo apt-get update

JAVA_VERSION=`echo "$(java -version 2>&1)" | grep "java version" | awk '{ print substr($3, 4, length($3)-9); }'`

#checking for java 7, In this script we are using default jdk from ubuntu  
if [ $JAVA_VERSION -eq "7" ] ; then
   echo "Java 7 is installed in your system "

else 
   echo "-----------------Removing older version of Java and installing default JDK of Ubuntu--------------"
   sudo apt-get autoremove java-common
   sudo apt-get install default-jdk
fi 

#Getting JAVA_HOME value and storing in java_home variable

java_home=`echo $JAVA_HOME`

echo "-----------------Adding a dedicated HADOOP user---------------------------"

sudo addgroup $hdGroup

echo "------------------Enter the Details for new HADOOP user---------------------------"
sudo adduser -ingroup $hdGroup $hdUserName

echo "--------New $hdGroup  GROUP is created and $hdUserName USER is assaigned to Group------------------"

echo "------------------------Installing SSH------------------------------- "

sudo apt-get install ssh


echo "--------Please press Enter when asking for file to save RSA keys -------------------------------"

sudo -u $hdUserName ssh-keygen -t rsa -P ""
sudo -u $hdUserName cat /home/$hdUserName/.ssh/id_rsa.pub >> /home/$hdUserName/.ssh/authorized_keys

#Download hadoop2.6.0 from apache
sudo wget http://archive.apache.org/dist/hadoop/core/hadoop-2.6.0/hadoop-2.6.0.tar.gz  

sudo tar xvfz hadoop-2.6.0.tar.gz

sudo  mv hadoop-2.6.0 /home/$hdUserName/hadoop

sudo chown -R $hdUserName:$hdGroup /home/$hdUserName/hadoop

#Creating tmp folder for hadoop under hadoop/app/hadoop/tmp 

sudo -u $hdUserName mkdir -p /home/$hdUserName/hadoop/app/hadoop/tmp
sudo -u $hdUserName chown -R $hdUserName:$hdGroup /home/$hdUserName/hadoop/app/hadoop/tmp

#Re naming  mapred-site.xml.template file to mapred-site.xml

sudo -u $hdUserName cp /home/$hdUserName/hadoop/etc/hadoop/mapred-site.xml.template /home/$hdUserName/hadoop/etc/hadoop/mapred-site.xml

#Creating hadoop storage location for NameNode and DataNode under hadoop/hadoop_store/hdfs
sudo -u $hdUserName mkdir -p /home/$hdUserName/hadoop/hadoop_store/hdfs/namenode
sudo -u $hdUserName mkdir -p /home/$hdUserName/hadoop/hadoop_store/hdfs/datanode

sudo -u $hdUserName chown -R $hdUserName:$hdGroup /home/$hdUserName/hadoop/hadoop_store

#Giving permission to write the .bashrc, hadoop-env.sh core-site.xml, mapred-site.xml, hdfs-site.xml, yarn-site.xml.

sudo -u hduser chmod o+w /home/$hdUserName/.bashrc
sudo -u hduser chmod o+w /home/$hdUserName/hadoop/etc/hadoop/hadoop-env.sh
sudo -u hduser chmod o+w /home/$hdUserName/hadoop/etc/hadoop/core-site.xml
sudo -u hduser chmod o+w /home/$hdUserName/hadoop/etc/hadoop/mapred-site.xml
sudo -u hduser chmod o+w /home/$hdUserName/hadoop/etc/hadoop/hdfs-site.xml
sudo -u hduser chmod o+w /home/$hdUserName/hadoop/etc/hadoop/yarn-site.xml

#hadoop-env.sh

sudo sed -i "s|\${JAVA_HOME}|$java_home|g" /home/$hdUserName/hadoop/etc/hadoop/hadoop-env.sh

echo -e '\n\n #Hadoop Variable START \n export HADOOP_HOME=/home/'$hdUserName'/hadoop \n export HADOOP_INSTALL=$HADOOP_HOME \n export HADOOP_MAPRED_HOME=$HADOOP_HOME \n export HADOOP_COMMON_HOME=$HADOOP_HOME \n export HADOOP_HDFS_HOME=$HADOOP_HOME \n export YARN_HOME=$HADOOP_HOME \n export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native \n export PATH=$PATH:$HADOOP_HOME/sbin/:$HADOOP_HOME/bin \n #Hadoop Variable END\n\n' >> /home/$hdUserName/.bashrc

source /home/$hdUserName/.bashrc

#core-site.xml
sudo sed -i '/<configuration>/a <property>\n\t\t<name>hadoop.tmp.dir</name>\n\t\t<value>/home/hduser/hadoop/app/hadoop/tmp</value>\n</property>\n<property>\n\t\t<name>fs.default.name</name>\n\t\t<value>hdfs://localhost:54310</value>\n</property>' /home/$hdUserName/hadoop/etc/hadoop/core-site.xml

#mapred-site.xml
sudo sed -i '/<configuration>/a <property>\n\t\t <name>mapreduce.framework.name</name>\n\t\t <value>yarn</value>\n</property>' /home/$hdUserName/hadoop/etc/hadoop/mapred-site.xml

#Yarn-site.xml
sudo sed -i '/<configuration>/a <property>\n\t\t<name>yarn.nodemanager.aux-services</name>\n\t\t<value>mapreduce_shuffle</value>\n</property>' /home/$hdUserName/hadoop/etc/hadoop/yarn-site.xml


#revoking write permission for .bashrc, hadoop-env.sh core-site.xml, mapred-site.xml, hdfs-site.xml, yarn-site.xml files. 
sudo -u hduser chmod o-w /home/$hdUserName/.bashrc
sudo -u hduser chmod o-w /home/$hdUserName/hadoop/etc/hadoop/hadoop-env.sh
sudo -u hduser chmod o-w /home/$hdUserName/hadoop/etc/hadoop/core-site.xml
sudo -u hduser chmod o-w /home/$hdUserName/hadoop/etc/hadoop/mapred-site.xml
sudo -u hduser chmod o-w /home/$hdUserName/hadoop/etc/hadoop/hdfs-site.xml
sudo -u hduser chmod o-w /home/$hdUserName/hadoop/etc/hadoop/yarn-site.xml

echo "--------------------HADOOP DIRECTORY------------------------- "
sudo ls /home/$hdUserName/hadoop/
echo "--------------------Proceed with 'yes' for continue connecting---------------------------- "
sudo -u hduser ssh localhost

Now after running the shell script, you have configured ssh, hadoop and its configuration file.

Format the name and start the hadoop

$hdfs namenode -format

 Start the hadoop

#start-all file is in sbin
$cd hadoop/sbin
$./start-all.sh 

#check hadoop 
$jps

Now you are done with hadoop installation and configured single node.

please find the code in github