這篇文章將為大家詳細(xì)講解有關(guān)Hadoop如何部署偽分布模式，小編覺(jué)得挺實(shí)用的，因此分享給大家做個(gè)參考，希望大家閱讀完這篇文章后可以有所收獲。

十年的德宏州網(wǎng)站建設(shè)經(jīng)驗(yàn)，針對(duì)設(shè)計(jì)、前端、開(kāi)發(fā)、售后、文案、推廣等六對(duì)一服務(wù)，響應(yīng)快，48小時(shí)及時(shí)工作處理。營(yíng)銷(xiāo)型網(wǎng)站建設(shè)的優(yōu)勢(shì)是能夠根據(jù)用戶設(shè)備顯示端的尺寸不同，自動(dòng)調(diào)整德宏州建站的顯示方式，使網(wǎng)站能夠適用不同顯示終端，在瀏覽器中調(diào)整網(wǎng)站的寬度，無(wú)論在任何一種瀏覽器上瀏覽網(wǎng)站，都能展現(xiàn)優(yōu)雅布局與設(shè)計(jì)，從而大程度地提升瀏覽體驗(yàn)。成都創(chuàng)新互聯(lián)公司從事“德宏州網(wǎng)站設(shè)計(jì)”,“德宏州網(wǎng)站推廣”以來(lái)，每個(gè)客戶項(xiàng)目都認(rèn)真落實(shí)執(zhí)行。

 部署方式:
1.單機(jī)模式standalone   1個(gè)java進(jìn)程
2.偽分布模式Pseudo-Distributed Mode  開(kāi)發(fā)|學(xué)習(xí)  多個(gè)java進(jìn)程
3.集群模式Cluster Mode   ：生產(chǎn) 多臺(tái)機(jī)器多個(gè)java進(jìn)程

偽分布式部署: HDFS
1.創(chuàng)建hadoop服務(wù)的一個(gè)用戶
[root@hadoop02 software]# useradd hadoop
[root@hadoop02 software]# id hadoop
uid=515(hadoop) gid=515(hadoop) groups=515(hadoop)
[root@rzdatahadoop02 software]#  [root@hadoop02 software]# vi /etc/sudoers
hadoop  ALL=(root)      NOPASSWD:ALL

2.部署JAVA
Oracle jdk1.8(Open JDK盡量不要使用)
[root@hadoop02 jdk1.8.0_45]# which java
/usr/java/jdk1.8.0_45/bin/java
[root@hadoop02 jdk1.8.0_45]#

3.部署ssh服務(wù)是運(yùn)行
[root@hadoop02 ~]# service sshd status
openssh-daemon (pid  1386) is running...
[root@hadoop02 ~]# 

4.解壓hadoop
[root@hadoop02 software]# tar -xzvf hadoop-2.8.1.tar.gz
chown -R hadoop:hadoop 文件夾 -->文件夾和文件夾的里面的 
chown -R hadoop:hadoop 軟連接文件夾 --> 只修改軟連接文件夾，不會(huì)修改文件夾里面的
chown -R hadoop:hadoop 軟連接文件夾/* --> 軟連接文件夾不修改，只會(huì)修改文件夾里面的
chown -R hadoop:hadoop hadoop-2.8.1 --> 修改原文件夾
[root@hadoop02 software]# ln -s hadoop-2.8.1 hadoop

[root@hadoop02 software]# cd hadoop
[root@hadoop02 hadoop]# rm -f *.txt
[root@hadoop02 hadoop]# ll
total 28
drwxr-xr-x. 2 hadoop hadoop 4096 Dec 10 11:54 bin
drwxr-xr-x. 3 hadoop hadoop 4096 Dec 10 11:54 etc
drwxr-xr-x. 2 hadoop hadoop 4096 Dec 10 11:54 include
drwxr-xr-x. 3 hadoop hadoop 4096 Dec 10 11:54 lib
drwxr-xr-x. 2 hadoop hadoop 4096 Dec 10 11:54 libexec
drwxr-xr-x. 2 hadoop hadoop 4096 Dec 10 11:54 sbin
drwxr-xr-x. 3 hadoop hadoop 4096 Dec 10 11:54 share
[root@hadoop02 hadoop]#  bin: 命令
etc：配置文件
sbin: 用來(lái)啟動(dòng)關(guān)閉hadoop進(jìn)程

5.切換hadoop用戶和配置
[root@hadoop02 hadoop]# su - hadoop
[hadoop@hadoop02 ~]$ ll
total 0
[hadoop@hadoop02 ~]$ cd /opt/software/hadoop
[hadoop@hadoop02 hadoop]$ ll
total 28
drwxr-xr-x. 2 hadoop hadoop 4096 Dec 10 11:54 bin
drwxr-xr-x. 3 hadoop hadoop 4096 Dec 10 11:54 etc
drwxr-xr-x. 2 hadoop hadoop 4096 Dec 10 11:54 include
drwxr-xr-x. 3 hadoop hadoop 4096 Dec 10 11:54 lib
drwxr-xr-x. 2 hadoop hadoop 4096 Dec 10 11:54 libexec
drwxr-xr-x. 2 hadoop hadoop 4096 Dec 10 11:54 sbin
drwxr-xr-x. 3 hadoop hadoop 4096 Dec 10 11:54 share
[hadoop@hadoop02 hadoop]$ cd etc/hadoop hadoop-env.sh : hadoop配置環(huán)境
core-site.xml : hadoop 核心配置文件
hdfs-site.xml : hdfs服務(wù)的 --> 會(huì)起進(jìn)程
[mapred-site.xml : mapred計(jì)算所需要的配置文件] 只當(dāng)在jar計(jì)算時(shí)才有
yarn-site.xml : yarn服務(wù)的 --> 會(huì)起進(jìn)程
slaves: 集群的機(jī)器名稱 [hadoop@hadoop02 hadoop]$ vi core-site.xml 
<configuration>
<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?> <configuration>
    <property>
            <name>fs.defaultFS</name>
            <value>hdfs://localhost:9000</value>
    </property>
</configuration> [hadoop@hadoop02 hadoop]$ vi hdfs-site.xml 
<configuration>
<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
    <property>
        <name>dfs.replication</name>
        <value>1</value>
    </property>
</configuration>

6.配置hadoop用戶的ssh的信任關(guān)系
[hadoop@hadoop02 ~]$ ssh-keygen -t rsa -P '' -f ~/.ssh/id_rsa
Generating public/private rsa key pair.
Created directory '/home/hadoop/.ssh'.
Your identification has been saved in /home/hadoop/.ssh/id_rsa.
Your public key has been saved in /home/hadoop/.ssh/id_rsa.pub.
The key fingerprint is:
5b:07:ff:e5:82:85:f3:41:32:f3:80:05:c9:57:0f:e9 hadoop@rzdatahadoop002
The key's randomart image is:
+--[ RSA 2048]----+
|         ..o..o. |
|          oo. .o |
|          o.=.. .|
|           o OE  |
|        S . = + .|
|         o . * + |
|        .   . + .|
|               . |
|                 |
+-----------------+

[hadoop@hadoop02 ~]$ cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
[hadoop@hadoop02 ~]$ chmod 0600 ~/.ssh/authorized_keys

7.格式化
[hadoop@hadoop002 hadoop]$ bin/hdfs namenode -format
17/12/13 22:22:04 INFO common.Storage: Storage directory /tmp/hadoop-hadoop/dfs/name has been successfully formatted.
17/12/13 22:22:04 INFO namenode.FSImageFormatProtobuf: Saving image file /tmp/hadoop-hadoop/dfs/name/current/fsimage.ckpt_0000000000000000000 using no compression
17/12/13 22:22:04 INFO namenode.FSImageFormatProtobuf: Image file /tmp/hadoop-hadoop/dfs/name/current/fsimage.ckpt_0000000000000000000 of size 323 bytes saved in 0 seconds.
17/12/13 22:22:04 INFO namenode.NNStorageRetentionManager: Going to retain 1 images with txid >= 0
17/12/13 22:22:04 INFO util.ExitUtil: Exiting with status 0
17/12/13 22:22:04 INFO namenode.NameNode: SHUTDOWN_MSG: 
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at rzdatahadoop002/192.168.137.201
************************************************************/ Storage directory： /tmp/hadoop-hadoop/dfs/name 
1.默認(rèn)的存儲(chǔ)路徑哪個(gè)配置？
2.hadoop-hadoop指的什么意思？
core-site.xml
hadoop.tmp.dir: /tmp/hadoop-${user.name}
hdfs-site.xml
dfs.namenode.name.dir : file://${hadoop.tmp.dir}/dfs/name

8.啟動(dòng)HDFS服務(wù)
[hadoop@hadoop02 sbin]$ ./start-dfs.sh
Starting namenodes on [localhost]
The authenticity of host 'localhost (::1)' can't be established.
RSA key fingerprint is 9a:ea:f5:06:bf:de:ca:82:66:51:81:fe:bf:8a:62:36.
Are you sure you want to continue connecting (yes/no)? yes
localhost: Warning: Permanently added 'localhost' (RSA) to the list of known hosts.
localhost: Error: JAVA_HOME is not set and could not be found.
localhost: Error: JAVA_HOME is not set and could not be found.
Starting secondary namenodes [0.0.0.0]
The authenticity of host '0.0.0.0 (0.0.0.0)' can't be established.
RSA key fingerprint is 9a:ea:f5:06:bf:de:ca:82:66:51:81:fe:bf:8a:62:36.
Are you sure you want to continue connecting (yes/no)? yes
0.0.0.0: Warning: Permanently added '0.0.0.0' (RSA) to the list of known hosts.
0.0.0.0: Error: JAVA_HOME is not set and could not be found.
[hadoop@hadoop02 sbin]$ ps -ef|grep hadoop
root     11292 11085  0 21:59 pts/1    00:00:00 su - hadoop
hadoop   11293 11292  0 21:59 pts/1    00:00:00 -bash
hadoop   11822 11293  0 22:34 pts/1    00:00:00 ps -ef
hadoop   11823 11293  0 22:34 pts/1    00:00:00 grep hadoop
[hadoop@rzdatahadoop002 sbin]$ echo $JAVA_HOME
/usr/java/jdk1.8.0_45
發(fā)現(xiàn)JAVA_HOME變量是存在的，無(wú)法啟動(dòng)HDFS服務(wù)

[hadoop@hadoop02 sbin]$ vi ../etc/hadoop/hadoop-env.sh
# The java implementation to use.
export JAVA_HOME=/usr/java/jdk1.8.0_45 [hadoop@hadoop02 sbin]$ ./start-dfs.sh 
Starting namenodes on [localhost]
localhost: starting namenode, logging to /opt/software/hadoop-2.8.1/logs/hadoop-hadoop-namenode-rzdatahadoop002.out
localhost: starting datanode, logging to /opt/software/hadoop-2.8.1/logs/hadoop-hadoop-datanode-rzdatahadoop002.out
Starting secondary namenodes [0.0.0.0]
0.0.0.0: starting secondarynamenode, logging to /opt/software/hadoop-2.8.1/logs/hadoop-hadoop-secondarynamenode-rzdatahadoop002.out namenode(名稱節(jié)點(diǎn)) : localhost  
datanode(數(shù)據(jù)節(jié)點(diǎn)) : localhost
secondary namenode(第二名稱節(jié)點(diǎn)): 0.0.0.0

http://localhost:50070/
默認(rèn)的端口:50070

9.使用命令(hadoop、hdfs)
[hadoop@hadoop02 bin]$ ./hdfs dfs -mkdir /user
[hadoop@hadoop02 bin]$ ./hdfs dfs -mkdir /user/hadoop [hadoop@hadoop02 bin]$ echo "123456" > rz.log
[hadoop@hadoop02 bin]$ ./hadoop fs -put rz.log hdfs://localhost:9000/
[hadoop@hadoop02 bin]$ 
[hadoop@hadoop02 bin]$ ./hadoop fs -ls hdfs://localhost:9000/
Found 2 items
-rw-r--r--   1 hadoop supergroup          7 2017-12-13 22:56 hdfs://localhost:9000/rz.log
drwxr-xr-x   - hadoop supergroup          0 2017-12-13 22:55 hdfs://localhost:9000/user [hadoop@hadoop02 bin]$ ./hadoop fs -ls /
Found 2 items
-rw-r--r--   1 hadoop supergroup          7 2017-12-13 22:56 hdfs://localhost:9000/rz.log
drwxr-xr-x   - hadoop supergroup          0 2017-12-13 22:55 hdfs://localhost:9000/user 10.想要修改hdfs://localhost:9000為hdfs://192.168.137.201:9000
[hadoop@hadoop02 bin]$ ../sbin/stop-dfs.sh  [hadoop@hadoop02 bin]$ vi ../etc/hadoop/core-site.xml 
<configuration>
<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?> <configuration>
    <property>
            <name>fs.defaultFS</name>
            <value>hdfs://192.168.137.201:9000</value>
    </property>
</configuration>

[hadoop@hadoop02 bin]$ ./hdfs namenode -format
[hadoop@hadoop02 bin]$ ../sbin/start-dfs.sh 
Starting namenodes on [hadoop002]
rzdatahadoop002: starting namenode, logging to /opt/software/hadoop-2.8.1/logs/hadoop-hadoop-namenode-rzdatahadoop002.out
localhost: starting datanode, logging to /opt/software/hadoop-2.8.1/logs/hadoop-hadoop-datanode-rzdatahadoop002.out
Starting secondary namenodes [0.0.0.0]
0.0.0.0: starting secondarynamenode, logging to /opt/software/hadoop-2.8.1/logs/hadoop-hadoop-secondarynamenode-rzdatahadoop002.out [hadoop@hadoop02 bin]$ netstat -nlp|grep 9000
(Not all processes could be identified, non-owned process info
 will not be shown, you would have to be root to see it all.)
tcp        0      0 192.168.137.201:9000        0.0.0.0:*                   LISTEN      14974/java          
[hadoop@hadoop02 bin]$ 
11.修改HDFS的服務(wù)以hadoop02啟動(dòng)
namenode: hadoop02
datanode: localhost  
secondarynamenode: 0.0.0.0  針對(duì)于datanode修改:
[hadoop@hadoop002 hadoop]$ vi slaves
hadoop02 針對(duì)于secondarynamenode修改:
[hadoop@hadoop02 hadoop]$ vi hdfs-site.xml 
<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!-- Put site-specific property overrides in this file. -->
<configuration>
    <property>
            <name>dfs.replication</name>
            <value>1</value>
    </property>         <property>
                 <name>dfs.namenode.secondary.http-address</name>
                 <value>rzdatahadoop002:50090</value>
        </property>
        <property>
                 <name>dfs.namenode.secondary.https-address</name>
                 <value>rzdatahadoop002:50091</value>
        </property> "hdfs-site.xml" 35L, 1173C written  [hadoop@hadoop02 hadoop]$ cd ../../sbin
[hadoop@hadoop02 sbin]$ ./stop-dfs.sh
[hadoop@hadoop02 sbin]$ ./start-dfs.sh 
Starting namenodes on [hadoop02]
hadoop02: starting namenode, logging to /opt/software/hadoop-2.8.1/logs/hadoop-hadoop-namenode-rzdatahadoop002.out
hadoop02: starting datanode, logging to /opt/software/hadoop-2.8.1/logs/hadoop-hadoop-datanode-rzdatahadoop002.out
Starting secondary namenodes [rzdatahadoop002]
hadoop02: starting secondarynamenode, logging to /opt/software/hadoop-2.8.1/logs/hadoop-hadoop-secondarynamenode-rzdatahadoop002.out

補(bǔ)充: 
某個(gè)服務(wù)數(shù)據(jù)目錄在A盤(pán)(500G),還剩10G。/a/dfs/data
添加B盤(pán)2T。
1.A盤(pán)：mv /a/dfs /b/
2.B盤(pán)：ln -s /b/dfs /a
3.檢查(修改)A,B盤(pán)的文件夾的用戶和用戶組的權(quán)限

關(guān)于“Hadoop如何部署偽分布模式”這篇文章就分享到這里了，希望以上內(nèi)容可以對(duì)大家有一定的幫助，使各位可以學(xué)到更多知識(shí)，如果覺(jué)得文章不錯(cuò)，請(qǐng)把它分享出去讓更多的人看到。

網(wǎng)站欄目：Hadoop如何部署偽分布模式
網(wǎng)頁(yè)路徑：http://aaarwkj.com/article40/ihhdho.html

成都網(wǎng)站建設(shè)公司_創(chuàng)新互聯(lián)，為您提供定制網(wǎng)站、Google、品牌網(wǎng)站建設(shè)、手機(jī)網(wǎng)站建設(shè)、動(dòng)態(tài)網(wǎng)站、網(wǎng)站維護(hù)

廣告

聲明：本網(wǎng)站發(fā)布的內(nèi)容（圖片、視頻和文字）以用戶投稿、用戶轉(zhuǎn)載內(nèi)容為主，如果涉及侵權(quán)請(qǐng)盡快告知，我們將會(huì)在第一時(shí)間刪除。文章觀點(diǎn)不代表本網(wǎng)站立場(chǎng)，如需處理請(qǐng)聯(lián)系客服。電話：028-86922220；郵箱：631063699@qq.com。內(nèi)容未經(jīng)允許不得轉(zhuǎn)載，或轉(zhuǎn)載時(shí)需注明來(lái)源： 創(chuàng)新互聯(lián)