OS X10.10安装hadoop2.6.0

配置ssh localhost无密码访问

  1. 首先在系统偏好设置->共享->打开远程登录服务->右侧选择允许所有用户访问
  2. 生成密钥对 ssh-keygen -t rsa
  3. 拷贝进验证文件
    cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
  4. 测试无密码访问
    ssh localhost

安装hadoop

  1. brew install hadoop
  2. Hadoop会被安装在/usr/local/Cellar/hadoop目录下

配置hadoop

  1. Hadoop会被安装在/usr/local/Cellar/hadoop目录下可以找到需要配置的5个文件
  2. 先去修改path , sudo nano /etc/profile
  3. 修改

    HADOOP_HOME=/usr/local/Cellar/hadoop/2.6.0/libexec/etc/hadoop
    PATH=$PATH:/usr/local/Cellar/hadoop/2.6.0/sbin
    source /etc/profile
  4. hadoop-env.sh

    export HADOOP_OPTS="-Djava.security.krb5.realm=OX.AC.UK -Djava.security.krb5.kdc=kdc0.ox.ac.uk:kdc1.ox.ac.uk"
    export JAVA_HOME=/Library/Java/JavaVirtualMachines/jdk1.8.0_40.jdk/Contents/Home
  5. core-site.xml

    <configuration>   
    <property>
    <name>hadoop.tmp.dir</name>
    <value>/usr/local/Cellar/hadoop/hdfs/tmp</value>
    <description>A base for other temporary directories.</description>
    </property>
    <property>
    <name>fs.default.name</name>
    <value>hdfs://localhost:9000</value>
    </property>
    </configuration>
  6. yarn-site.xml

    <configuration>
    <!-- Site specific YARN configuration properties -->
    <property>
    <name>yarn.nodemanager.aux-services</name>
    <value>mapreduce_shuffle</value>
    </property>
    <property>
    <name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
    <value>org.apache.hadoop.mapred.ShuffleHandler</value>
    </property>
    </configuration>
  7. mapred-site.xml

    <configuration> 
    <property>
    <name>mapreduce.framework.name</name>
    <value>yarn</value>
    </property>
    </configuration>
  8. hdfs-site.xml

    <configuration>   
    <property>
    <name>dfs.replication</name>
    <value>3</value>
    </property>
    <property>
    <name>dfs.namenode.name.dir</name>
    <value>file:/usr/local/Cellar/hadoop/hdfs/name</value>
    </property>
    <property>
    <name>dfs.datanode.data.dir</name>
    <value>file:/usr/local/Cellar/hadoop/hdfs/data</value>
    </property>
    </configuration>
  9. 启动hadoop

    1. 格式化namenode hadoop namenode -format
    2. start-dfs.sh
    3. start-yarn.sh
    4. 查看hadoop管理界面
      可以在http://localhost:50070 看到hdfs管理页面,localhost 看到hadoop进程管理页面。
    5. jps 查看所有启动的hadoop进程
  10. wordcount验证

    1. dfs上创建input目录:hadoop fs -mkdir -p input
    2. 把hadoop目录下的README.txt拷贝到dfs新建的input里

      hadoop fs -copyFromLocal /Users/Gao/Documents/学习/01-Hadoop/01-hadoop安装/hadoop-2.6.0/README.txt input
    3. 运行WordCount

      hadoop jar /Users/Gao/Documents/学习/01-Hadoop/01-hadoop安装/hadoop-2.6.0/share/hadoop/mapreduce/sources/hadoop-mapreduce-examples-2.6.0-sources.jar org.apache.hadoop.examples.WordCount input output
    4. 运行完毕后,查看单词统计结果hadoop fs -cat output/*

参考页面

×

纯属好玩

扫码支持
扫码打赏,你说多少就多少

打开支付宝扫一扫,即可进行扫码打赏哦

文章目录
  1. 1. 配置ssh localhost无密码访问
  2. 2. 安装hadoop
  3. 3. 配置hadoop
  4. 4. 参考页面
,