588
Hadoop 2.x 集群環境搭建
基礎環境設置：
1.配置/etc/sysconfig/network-scripts/ifcfg-ens33 綁定ip
2.配置主機名ip解析，編輯 /etc/hosts
3.修改主機名，編輯/etc/sysconfig/network ,添加一行HOSTNAME=hostname
4.關閉iptables，selinux，firewalld
5.安裝jdk，配置$JAVA_HOME
6.解壓hadoop2.x 至/opt/app下，配置$HADOOP_HOME
7.所有主機之間設置ssh免驗證登陸,包括本機自己ssh也要配置 (3台機器都有同一個用戶，beifeng)
hadoop 2.x 分布式部署方案
HOSTNAME IPADDR HDFS YARN MAPREDUCE
hadoop-master 192.168.1.129 NameNode,DataNode NodeManager Job_History_server
hadoop-slave1 192.168.1.130 DataNode ResourceManager,NodeManager
hadoop-slave2 192.168.1.131 SecondaryNameNode,DataNode NodeManager
hadoop 2.x 各守護進程相關配置文件
hdfs：
hadoop-env.sh --> 配置$JAVA_HOME
core-site.xml --> 配置NameNode節點（fs.defaultFS)
配置Hadoop的臨時目錄(tmp.dir)
hdfs-site.xml --> 配置SecondaryNameNode(dfs.namenode.secondary.http-address)
slaves --> 配置DataNode節點的ip/hostname
yarn:
yarn-env.sh --> 配置$JAVA_HOME
yarn-site.xml --> 配置ResourceManager節點
配置日誌聚集（yarn.log-aggregetion-enable）
配置MapReduce的shuffle(yarn.nodemanager.aux-services----mapreduce_shuffle )
slaves --> 配置NodeManager節點的ip/hostname
mapreduce:
mapred-site.xml --> 配置job history
配置在yarn上運行mapreduce
在hadoop-master節點配置hdfs、yarn及mapreduce
1.配置hdfs
(一般配置好javahome不用再配置hadoop-env.sh)
a.$HADOOP_HOME/etc/hadoop/core-site.xml
fs.defaultFS
hdfs://hadoop-master:8020
hadoop.tmp.dir
/opt/data/tmp
b.$HADOOP_HOME/etc/hadoop/hdfs-site.xml
不需要配置分片
dfs.namenode.secondary.http-address
https://hadoop-slave2:50090
c.$HADOOP_HOME/etc/hadoop/slaves
同時配置了NodeManager的節點地址
hadoop-master
hadoop-slave1
hadoop-slave2
2.配置yarn
a.yarn-site.xml
yarn.resourcemanager.hostname
hadoop-slave1
yarn.nodemanager.aux-services
mapreduce_shuffle
yarn.log-aggregation-enable
true
yarn.log-aggregation.retain-seconds
640800
3.配置MapReduce
a.mapred-site.xml
mapreduce.framework.name
yarn
mapreduce.jobhistory.address
hadoop-master:10020
mapreduce.jobhistory.webapp.address
hadoop-master:19888
拷貝hadoop到hadoop-slave1，slave2
scp -r $HADOOP_HOME hadoop-slave1:/opt/app
scp -r $HADOOP_HOME hadoop-slave2:/opt/app
啟動hadoop 集群
1.在hadoop-master上首次初始格式化namenode節點
hdfs namenode -format
2.啟動hdfs集群
start-dfs.sh
3.啟動yarn集群
start-yarn.sh
4.啟動job-history server
mr-jobhistory-daemon.sh start historyserver
5.各節點查看運行狀態
jps
END
最後更新：2017-07-13 07:28:08