Running Jenkins on Docker Container as Master node and AWS EC2 Instance as Worker Node

In this post, I’ll walk you through setting up Jenkins in a Docker container as the master node, and configuring an AWS EC2 instance as the worker (or agent) node. This setup allows Jenkins to distribute and execute jobs on remote machines, which is especially useful when scaling out workloads.

Let’s get started with a step-by-step guide that uses simple explanations, Docker for Jenkins master, and EC2 for the worker node.

Prerequisites:

1) Basics of Docker and installed on local machine
2) Basics of Jenkins
3) Some Basics of AWS
4) SSH Connection

Steps :

1) Set up Jenkins in a Docker container as the master node.
2) Configure an EC2 instance to act as the worker node.
3) Establish communication between the Jenkins master and worker using SSH
4) Test the setup by running a job on the worker node.

Step 1 : Running Jenkins in a Container

1) Pulling the Image From DockerHub

docker pull jenkins/jenkins:lts

2) Run the Jenkins Container

docker run -d -p 8080:8080 -p 50000:50000 -v jenkins_home:/var/jenkins_home --name jenkins-master jenkins/jenkins:lts

Overview of the Command:

-p 8080:8080: Exposes Jenkins UI on port 8080 (Port Binding).
-p 50000:50000: Allows communication between Jenkins master and agent.
-v jenkins_home:/var/jenkins_home: Mounts a volume for data persistence.

You can docker ps to verify if the container is up or not

3) Initial Setup:

Open your browser and go to http://localhost:8080. Jenkins will prompt you to unlock it by providing an admin password, which you can find by running:

docker exec -it jenkins-master bash  // It lets you inside the container
cat /var/jenkins_home/secrets/initialAdminPassword

Copy the password, log in, and complete the initial setup by installing the recommended plugins and creating an admin user.

Step 2 : Launching AWS EC2 Instance as Worker Node

1) Launch an EC2 instance:

Head to the AWS console and launch an EC2 instance with Ubuntu.
Make sure to open port 22 (SSH) and any necessary ports in the security group for your Jenkins setup.
You can expose HTTPS or HTTP also (optional)/

2) SSH Into the Instance

ssh -i "your-key.pem" ubuntu@<EC2-public-IP>

💡

“your-key.pem” file should be in the same directory where you want to execute this command.

3) Installing Java on EC2 Instance

Jenkins require java to run its agent/worker node as it is build on java

sudo apt update
sudo apt install openjdk-11-jdk -y

4) Set up SSH Keys:

Generate SSH keys on your Jenkins master container if you haven't already:
```
  ssh-keygen -t rsa
```
Copy the public key from the master container to the EC2 instance:
```
  ssh-copy-id -i ~/.ssh/id_rsa.pub ubuntu@<EC2-public-IP>
```
This enables password-less authentication between Jenkins and the EC2 worker.

Step 3: Connect the EC2 Worker to Jenkins Master

Access Jenkins Dashboard:

Go to Jenkins' UI at http://localhost:8080 and log in.
Add a New Node:
- Navigate to Manage Jenkins > Manage Nodes and Clouds.
- Click New Node, give it a name (e.g., AWS-EC2-Worker), select Permanent Agent, and proceed.
Configure the Worker Node:
- Remote root directory: /home/ubuntu (this is the home directory of the Ubuntu user on EC2).
- Labels: Add a label if needed (e.g., aws-agent).
- Usage: Use this node as much as possible.
- Launch method: Select Launch agents via SSH.
- Host: Enter your EC2 public IP address.
- Credentials: Add the private SSH key that Jenkins will use to connect to the EC2 instance (you can paste the contents of the id_rsa file).
Test the Connection:

Once you’ve entered all the information, click Save and Launch agent. If everything is set up correctly, Jenkins will establish a connection with the EC2 instance, and the node will appear as online.

Step 4: Test the Setup

To test the setup, create a new Jenkins job and configure it to run on the EC2 worker node.
1. Create a new Freestyle Project:
  - Go to Jenkins dashboard, click New Item, and select Freestyle Project.
  - In the Restrict where this project can be run field, add the label of the EC2 agent (e.g., aws-agent).
2. Add a simple build step:
  - Add an Execute shell step and write a simple shell command like:
```
  echo "Hello From Agent"
```