配置ssh localhost无密码访问

1. 首先在系统偏好设置->共享->打开远程登录服务->右侧选择允许所有用户访问
2. 生成密钥对 ssh-keygen -t rsa
3. 拷贝进验证文件
   cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
4. 测试无密码访问
   ssh localhost

安装hadoop

1. brew install hadoop
2. Hadoop会被安装在/usr/local/Cellar/hadoop目录下

配置hadoop

1. Hadoop会被安装在/usr/local/Cellar/hadoop目录下可以找到需要配置的5个文件
2. 先去修改path , sudo nano /etc/profile
3. 修改

   HADOOP_HOME=/usr/local/Cellar/hadoop/2.6.0/libexec/etc/hadoop
   PATH=$PATH:/usr/local/Cellar/hadoop/2.6.0/sbin
   source  /etc/profile 
   

4. hadoop-env.sh

   export HADOOP_OPTS="-Djava.security.krb5.realm=OX.AC.UK -Djava.security.krb5.kdc=kdc0.ox.ac.uk:kdc1.ox.ac.uk"
   export JAVA_HOME=/Library/Java/JavaVirtualMachines/jdk1.8.0_40.jdk/Contents/Home
   

5. core-site.xml

   <configuration>   
   <property>     
       <name>hadoop.tmp.dir</name>     
       <value>/usr/local/Cellar/hadoop/hdfs/tmp</value>     
       <description>A base for other temporary directories.</description>   
   </property>   
   <property>     
       <name>fs.default.name</name>     
       <value>hdfs://localhost:9000</value>             
   </property> 
   </configuration>
   

6. yarn-site.xml

   <configuration>
   <!-- Site specific YARN configuration properties -->
       <property>
           <name>yarn.nodemanager.aux-services</name>
           <value>mapreduce_shuffle</value>
       </property>
       <property>
           <name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
           <value>org.apache.hadoop.mapred.ShuffleHandler</value>
       </property>
   </configuration>
   

7. mapred-site.xml

   <configuration> 
       <property>
           <name>mapreduce.framework.name</name>
           <value>yarn</value>
       </property>
   </configuration>
   

8. hdfs-site.xml

   <configuration>   
       <property>     
           <name>dfs.replication</name>     
           <value>3</value>   
       </property> 
       <property>
           <name>dfs.namenode.name.dir</name>
           <value>file:/usr/local/Cellar/hadoop/hdfs/name</value>
       </property>
       <property>
           <name>dfs.datanode.data.dir</name>
           <value>file:/usr/local/Cellar/hadoop/hdfs/data</value>
       </property>
   </configuration>
   

9. 启动hadoop
   1. 格式化namenode hadoop namenode -format
   2. start-dfs.sh
   3. start-yarn.sh
   4. 查看hadoop管理界面
      可以在http://localhost:50070 看到hdfs管理页面，localhost 看到hadoop进程管理页面。
   5. jps 查看所有启动的hadoop进程
10. wordcount验证
    1. dfs上创建input目录：hadoop fs -mkdir -p input
    2. 把hadoop目录下的README.txt拷贝到dfs新建的input里

       hadoop fs -copyFromLocal /Users/Gao/Documents/学习/01-Hadoop/01-hadoop安装/hadoop-2.6.0/README.txt input
       

    3. 运行WordCount

       hadoop jar /Users/Gao/Documents/学习/01-Hadoop/01-hadoop安装/hadoop-2.6.0/share/hadoop/mapreduce/sources/hadoop-mapreduce-examples-2.6.0-sources.jar org.apache.hadoop.examples.WordCount input output
       

    4. 运行完毕后，查看单词统计结果hadoop fs -cat output/*

参考页面