cent os 6.5+ambari+HDP集群安装-白红宇

cent os 6.5+ambari+HDP集群安装

阅读量：5146 次

发布时间：2019-06-13

本文共 7296 字，大约阅读时间需要 24 分钟。

1. 搭建一个测试集群，集群有4台机器，配置集群中每一台机器的/etc/hosts文件：

[root@nn .ssh]# cat /etc/hosts127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4::1         localhost localhost.localdomain localhost6 localhost6.localdomain6172.28.3.40  nn  nn.hadoop.plat172.28.3.41  dn1  dn0.hadoop.plat172.28.3.42  dn2  dn1.hadoop.plat 172.28.3.43  dn3  dn2.hadoop.plat

2. 配置namenode到datanode ssh免密码登陆：

在nn上执行，ssh-keygen –t rsa

cd ~/.ssh

cat id_rsa.put >> authorized_keys

对集群中每一台data node执行: ssh-copy-id ssh-copy-id ssh-copy-id

这样就可以保证，nn节点可以免密码登陆到dn1, dn2, dn3

[root@nn .ssh]# ifconfigeth0      Link encap:Ethernet  HWaddr 00:1A:4A:C6:6B:A0            inet addr:172.28.3.40  Bcast:172.28.7.255  Mask:255.255.248.0          inet6 addr: fe80::21a:4aff:fec6:6ba0/64 Scope:Link          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1          RX packets:1064845 errors:0 dropped:0 overruns:0 frame:0          TX packets:557212 errors:0 dropped:0 overruns:0 carrier:0          collisions:0 txqueuelen:1000           RX bytes:1578655986 (1.4 GiB)  TX bytes:647178854 (617.1 MiB)lo        Link encap:Local Loopback            inet addr:127.0.0.1  Mask:255.0.0.0          inet6 addr: ::1/128 Scope:Host          UP LOOPBACK RUNNING  MTU:16436  Metric:1          RX packets:103276 errors:0 dropped:0 overruns:0 frame:0          TX packets:103276 errors:0 dropped:0 overruns:0 carrier:0          collisions:0 txqueuelen:0           RX bytes:58108687 (55.4 MiB)  TX bytes:58108687 (55.4 MiB)[root@nn .ssh]# cat /etc/hosts127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4::1         localhost localhost.localdomain localhost6 localhost6.localdomain6172.28.3.40  nn  nn.hadoop.plat172.28.3.41  dn1  dn0.hadoop.plat172.28.3.42  dn2  dn1.hadoop.plat 172.28.3.43  dn3  dn2.hadoop.plat[root@nn .ssh]# ssh dn1SIOCADDRT: File exists

3. 关闭iptables

chkconfig iptables off

/etc/init.d/iptables stop

4. 关闭seLinux

查看selinux状态：

[root@localhost ~]# /usr/sbin/sestatus –v

/usr/sbin/setenforce 0 #使SELinux工作模式变成permissive模式

/usr/sbin/setenforce 1 #使SELinux工作模式变成enforcing模式

这样就可以实时控制SELinux的启用和不启用了。

三个参数介绍介绍

enforcing — The SELinux security policy is enforced.

permissive — The SELinux system prints warnings but does not enforce policy.

disabled — SELinux is fully disabled. SELinux hooks are disengaged from the kernel and the pseudo-file system is unregistered.

永久关闭SELinux

编辑/etc/selinux/config，找到SELINUX 行修改成为：SELINUX=disabled:

# This file controls the state of SELinux on the system.

# SELINUX= can take one of these three values:

# enforcing - SELinux security policy is enforced.

# permissive - SELinux prints warnings instead of enforcing.

# disabled - No SELinux policy is loaded.
SELINUX=disabled

# SELINUXTYPE= can take one of these two values:

# targeted - Only targeted network daemons are protected.

# strict - Full SELinux protection.

SELINUXTYPE=targeted

如果重启系统，就会发现SELinux的状态变成disabled

5. 关闭linux内核huge_page:

Add the following lines in /etc/rc.local and reboot the server:echo never > /sys/kernel/mm/redhat_transparent_hugepage/enabledecho never > /sys/kernel/mm/redhat_transparent_hugepage/defrag

6. 安装java，配置JAVA_HOME

mkdir /usr/javacd /usr/javarz -be/usr/java/jdk1.7.0_75ln -s /usr/java/jdk1.7.0_75 /usr/java/defaultvim /etc/profileroute add default gw 172.28.0.1export JAVA_HOME=/usr/java/defaultexport PATH=$JAVA_HOME/bin:$PATH

编辑好/etc/profile，执行source /etc/profile使配置生效，保证每一台机器上的java版本都是一致的，并且JAVA_HOME环境变量是有效的：

7. 每台机器上安装ntpd

rpm -aq | grep ntpdyum install ntpdchkconfig ntpd onservice ntpd start

确保每台机器的ntpd服务都处于运行状态：

8. 确保机器上安装了openssh-server，并且升级openssl到最新：

rpm -qa | grep sshyum install openssh-serverservice sshd restartchkconfig sshd on

确保openssl最新：

yum install openssl-devel-1.0.1e-42.el6.x86_64

9. yum源确保可以用，本次安装采用了163的yum源，先将/etc/yum.repos.d/中，所有的文件都删掉，然后新建文件CentOS6-Base-163.repo，填入如下内容：

# CentOS-Base.repo## The mirror system uses the connecting IP address of the client and the# update status of each mirror to pick mirrors that are updated to and# geographically close to the client.  You should use this for CentOS updates# unless you are manually picking other mirrors.## If the mirrorlist= does not work for you, as a fall back you can try the# remarked out baseurl= line instead.##[base]name=CentOS-$releasever - Base - 163.combaseurl=http://mirrors.163.com/centos/$releasever/os/$basearch/#mirrorlist=http://mirrorlist.centos.org/?release=$releasever&arch=$basearch&repo=osgpgcheck=1gpgkey=http://mirror.centos.org/centos/RPM-GPG-KEY-CentOS-6#released updates[updates]name=CentOS-$releasever - Updates - 163.combaseurl=http://mirrors.163.com/centos/$releasever/updates/$basearch/#mirrorlist=http://mirrorlist.centos.org/?release=$releasever&arch=$basearch&repo=updatesgpgcheck=1gpgkey=http://mirror.centos.org/centos/RPM-GPG-KEY-CentOS-6#additional packages that may be useful[extras]name=CentOS-$releasever - Extras - 163.combaseurl=http://mirrors.163.com/centos/$releasever/extras/$basearch/#mirrorlist=http://mirrorlist.centos.org/?release=$releasever&arch=$basearch&repo=extrasgpgcheck=1gpgkey=http://mirror.centos.org/centos/RPM-GPG-KEY-CentOS-6#additional packages that extend functionality of existing packages[centosplus]name=CentOS-$releasever - Plus - 163.combaseurl=http://mirrors.163.com/centos/$releasever/centosplus/$basearch/#mirrorlist=http://mirrorlist.centos.org/?release=$releasever&arch=$basearch&repo=centosplusgpgcheck=1enabled=0gpgkey=http://mirror.centos.org/centos/RPM-GPG-KEY-CentOS-6#contrib - packages by Centos Users[contrib]name=CentOS-$releasever - Contrib - 163.combaseurl=http://mirrors.163.com/centos/$releasever/contrib/$basearch/#mirrorlist=http://mirrorlist.centos.org/?release=$releasever&arch=$basearch&repo=contribgpgcheck=1enabled=0gpgkey=http://mirror.centos.org/centos/RPM-GPG-KEY-CentOS-6

配置ambari的yum源，本配置源是部署在本地局域网中的一台apache服务器上的：

[root@hdp159 yum.repos.d]# cat ambari.repo [Updates-ambari-2.0.1]name=ambari-2.0.1 - Updatesbaseurl=http://172.28.4.159/ambari-test/centos6gpgcheck=1gpgkey=http://172.28.4.159/ambari-test/centos6/RPM-GPG-KEY/RPM-GPG-KEY-Jenkinsenabled=1priority=1[root@hdp159 yum.repos.d]#

将yum源配置好好了后，执行如下命令：

yum clean allyum repolist

10. 在nn机器上，安装ambari-servier，执行如下命令，之所以加—nogpgcheck参数，是因为此处安装的是公司修改后的ambari，如果是安装原生的ambari，不用加该选项：

yum install --nogpgcheck ambari-server

11.配置和启动ambari-server, setup –j 配置ambari-server要使用的java环境：

ambari-server setup -j /usr/java/defaultambari-server start

12. 在浏览器中，输入nn:8080进入ambari的登陆页面，用户名和密码都是admin:

13. 配置HDP的redhat6的baseURL,此处使用的是本地局域网中的HDP安装源：

14. 将nn节点下，/root/.ssh/id_rsa文件上传到ambari中，配置好Target Hosts:

15. 发现一个警告消息，在每一台机器上执行如下命令，消除警告：

[root@dn3 yum.repos.d]# python /usr/lib/python2.6/site-packages/ambari_agent/HostCleanup.py --silent --skip=usersINFO:HostCleanup:Killing pid's: ['']INFO:HostCleanup:Deleting packages: ['']INFO:HostCleanup:Deleting directories: ['']INFO:HostCleanup:Path doesn't exists:INFO:HostCleanup:Deleting additional directories: ['']INFO:HostCleanup:Deleting repo files: []INFO:HostCleanup:Erasing alternatives:{
    'symlink_list': [''], 'target_list': ['']}INFO:HostCleanup:Path doesn't exists:INFO:HostCleanup:Clean-up completed. The output is at /var/lib/ambari-agent/data/hostcleanup.result

16. 分配slaves和clients:

设置好，hive和oozie数据库的用户名和密码：

查看总结信息：

17. 进入安装流程，最后安装成功：