Managing SSH Key Pairs

Published

2026-07-14

WarningCaution

This section is being revised. Thank you for your patience.

Working across multiple machines makes it easy to get sloppy with keys, so a few rules are worth stating plainly.

Security considerations

  • Keep the AWS .pem key on one machine only. It’s the recovery key for the instance. There’s no reason to copy it to a laptop we carry around.

  • Give each machine its own key when you can. Per-machine keys mean a lost laptop is a one-line fix on the server, not a full key rotation.

  • Never open port 22 to the world for convenience. In the security group, scope the SSH inbound rule to the IP addresses we actually connect from. Using 0.0.0.0/0 is fine for a quick test, but leaving it open is an invitation.

  • Rotate keys you can’t account for. If a private key might have leaked, or a machine is gone, remove its public key from ~/.ssh/authorized_keys on the server and generate a replacement.

TipTip: one key per machine

Naming keys after the machine that holds them (<your-key-name>-macbook, <your-key-name>-desktop) makes the entries in authorized_keys self-documenting. When you need to revoke access for a single device, you know exactly which line to remove.

Logging in from another machine

So far we’ve connected from the same computer that created the key pair. But at some point we’ll want to reach the instance from a second machine, like a MacBook Pro we use when we’re away from our main desktop. The problem is that our keys live on the original computer, and the server only trusts the keys it already knows about. This section covers how to get set up on a new machine without weakening the security of the instance.

SSH authentication is based on a key pair: the private key stays on our computer, and the matching public key lives on the server in ~/.ssh/authorized_keys. The server has no idea which machine we’re sitting at. It only checks whether the private key we present matches a public key it trusts. This means a new machine can’t connect until it holds a private key the server recognizes.

%%{init: {'theme': 'base', 'themeVariables': {'fontFamily': 'monospace'}}}}%%

graph LR
    subgraph Client["Your Computer"]
        priv["Private Key<br/>~/.ssh/id_ed25519<br/>(secret)"]
    end
    subgraph Server["EC2 Instance"]
        auth["authorized_keys<br/>(trusted public keys)"]
    end
    priv -->|"presents key"| auth
    auth -.-|"match?<br/>grant / deny"| priv

    style Client fill:#E8E8E8,stroke:#333,stroke-width:2px,color:#000
    style Server fill:#1B2A41,stroke:#fff,stroke-width:2px,color:#fff
    style priv fill:#D2562B,stroke:#fff,stroke-width:1px,color:#fff
    style auth fill:#2A6F77,stroke:#fff,stroke-width:1px,color:#fff

SSH Key Pair Authentication

We have two options:

  1. Generate a fresh key pair on the new machine and add its public key to the server

  2. Copy an existing private key to the new machine.

The first option is safer, so we’ll cover it below. Copying private keys around is convenient, but every copy is another file that can leak.

Replacing key pairs

Key pair names are immutable in AWS, which means they can’t be renamed; we must create a new one and replace the old.

NoteNote

Key pairs are not tied to security groups. Security groups are attached to the instance separately and won’t change. We’re just swapping the SSH public key on the instance.

Step 1. Create the new key pair

aws ec2 create-key-pair \
  --key-name <my-better-name> \
  --key-type rsa \
  --key-format pem \
  --query 'KeyMaterial' \
  --output text > ~/.ssh/<my-better-name>.pem
chmod 400 ~/.ssh/<my-better-name>.pem

Or via Console: EC2 > Key Pairs > Create key pair

Step 2. Get the public key

ssh-keygen -y -f ~/.ssh/<my-better-name>.pem > ~/.ssh/<my-better-name>.pub
cat ~/.ssh/<my-better-name>.pub

Step 3. Add the new public key to the instance

SSH in using the <old-key>.pem key:

ssh -i ~/.ssh/<old-key>.pem ec2-user@<instance-ip>

Then append the new public key:

echo "ssh-rsa AAAA...new-key-content..." >> ~/.ssh/authorized_keys

Step 4. Test the new key in a new terminal (keep the old session open)

ssh -i ~/.ssh/<my-better-name>.pem ec2-user@<instance-ip>

Step 5: Remove the old public key from authorized_keys

Edit ~/.ssh/authorized_keys and delete the old key line.

Step 6: Delete the old key pair from AWS

aws ec2 delete-key-pair --key-name <old-key>

Then delete the old .pem locally.

Replacing key pairs (notes)

  • The “Key pair name” shown in the EC2 console for an instance reflects the key used at launch and won’t update, but access is controlled by ~/.ssh/authorized_keys on the instance itself.

  • Security groups, IAM roles, EBS volumes, and Elastic IP all remain unchanged.

  • If you want the console to reflect the new key name too, you’d need to create an AMI and launch a new instance with the new key, which is usually overkill.

Setting up an SSH config

Typing the full ssh -i ~/.ssh/<your-key-name> ec2-user@YOUR_PUBLIC_IP_ADDRESS command every time gets old quickly. SSH reads a config file at ~/.ssh/config that lets us assign a short alias to a connection. On the MacBook, we open (or create) the file:

touch ~/.ssh/config
chmod 600 ~/.ssh/config

Then add an entry describing the connection:

Host my-ec2
    HostName YOUR_PUBLIC_IP_ADDRESS
    User ec2-user
    IdentityFile ~/.ssh/<your-key-name>-macbook

The Host line is the alias we type, HostName is the public IP (or Public DNS name), User is the login user for the AMI, and IdentityFile points to the private key for this machine. With that saved, connecting is now a single short command:

ssh my-ec2

We can confirm the config is being used by adding -v to the command, which prints the identity file and hostname SSH resolved from the alias. If the public IP changes, we only have to update one line in the config rather than remember a new address.

Recap

  1. Prefer generating a fresh key per machine over copying private keys around. It keeps revocation simple.

  2. An ~/.ssh/config entry turns a long connection command into a short alias and gives you one place to update when the instance’s IP changes.