For setting up Hadoop on a cluster of machines, the master should be
able to do a password-less ssh to start the daemons on all the slaves.
Class MR - master starts TaskTracker and the DataNode on all the slaves.
MRv2 (next generation MR) - master starts NodeManager and the DataNode on all the slaves.
Here are the steps to setup password-ssh. Ensure that port 22 is open on all the slave (`telnet slave-hostname 22` should connect).
1) Install openssh-client on the master
2) Install openssh-server on all the slaves
3) Generate the ssh key
4) Copy the key to all the slaves (replace username appropriately as the
user starting the Hadoop daemons). Will be prompted for the password.
5) If the master also acts a slave (`ssh localhost` should work without a password)
If hdfs/mapreduce are run as different users then the steps (3,4 and 5) have to be repeated for all the users.
How to test ?
1) Run `ssh user@slave-hostname`. It should get connected without prompting for a password.
Class MR - master starts TaskTracker and the DataNode on all the slaves.
MRv2 (next generation MR) - master starts NodeManager and the DataNode on all the slaves.
Here are the steps to setup password-ssh. Ensure that port 22 is open on all the slave (`telnet slave-hostname 22` should connect).
1) Install openssh-client on the master
1
| sudo apt-get install openssh-client |
1
| sudo apt-get install openssh-server |
1
| ssh-keygen -t rsa -P "" -f ~/.ssh/id_rsa |
1
| ssh-copy-id -i $HOME/.ssh/id_rsa.pub username @slave -hostname |
1
| cat $HOME/.ssh/id_rsa.pub >> $HOME/.ssh/authorized_keys |
If hdfs/mapreduce are run as different users then the steps (3,4 and 5) have to be repeated for all the users.
How to test ?
1) Run `ssh user@slave-hostname`. It should get connected without prompting for a password.
No comments:
Post a Comment