Passwordless SSH from Master to Slave on AWS EMR

Upload pem file to master node and SSH to master

scp -i admin-key.pem admin-key.pem ec2-user@ecIP.eu-west-1.compute.amazonaws.com:~/
aws emr ssh --cluster-id JOB_ID

Configure passwordless SSH from master to slaves

# Identify IPs of slave nodes, 
hdfs dfsadmin -report | grep ^Name | cut -f2 -d:| cut -f2 -d' '
ssh-agent bash
ssh-add YOUR_AWS_IAM.pem
ssh-keygen [enter enter]
ssh-copy-id -i ~/.ssh/id_rsa.pub hadoop@IP_OF_SLAVE

Reference for ssh-agent.

passwordless ssh – ssh without a password

“ssh without password” is a must for many cluster solutions. I encountered this when I started to play with apache spark. The following is how:

From master=>

ssh-keygen [enter enter]

This will generate id_rsa and id_rsa.pub. Next, we will put the public key to a slave. Luckily, ssh-copy-id is the tool to do the job.

ssh-copy-id -i ~/.ssh/id_rsa.pub [slaveIP e.g. 1. 2. 3. 4.]

Now we should be able to ssh ‘remote IP’ without ip using current username.

Reference