From 8c8c8f002fc489b30eadbbf03c21fb0be57091db Mon Sep 17 00:00:00 2001 From: Madhup Pandey <121779953+DevMadhup@users.noreply.github.com> Date: Tue, 18 Mar 2025 09:55:56 +0530 Subject: [PATCH 001/133] Create solution.md --- DailyChallenges/Day7/solution.md | 160 +++++++++++++++++++++++++++++++ 1 file changed, 160 insertions(+) create mode 100644 DailyChallenges/Day7/solution.md diff --git a/DailyChallenges/Day7/solution.md b/DailyChallenges/Day7/solution.md new file mode 100644 index 0000000..2be0d81 --- /dev/null +++ b/DailyChallenges/Day7/solution.md @@ -0,0 +1,160 @@ +## Flask app deployment + +### Steps to deploy using python virtual environment: + +- Create an EC2 instance with the following details: + + - **AMI:** Ubuntu + - **Instance type:** t2.micro + - **Volume:** 10GB +--- +- Update the system +```bash +sudo apt update +``` +--- +- Install python virtual environment: +```bash +sudo apt install python3-venv +```` +--- +- Create virtual environment: +```bash +python3 -m venv venv +source venv/bin/activate +``` +--- +- Clone your source code: +```bash +git clone https://github.com/DevMadhup/flask.git +``` +--- +- Move to source code directory +```bash +cd flask/examples/tutorial +``` +--- +- Install the required dependencies: +```bash +pip install -e . +``` +--- +- Run tests: +```bash +pip install '.[test]' +pytest +``` +--- +- Initialize database: +```bash +flask --app flaskr init-db +``` +--- +- Run the application: +```bash +flask --app flaskr run --host=0.0.0.0 --debug +``` +> Note: Access the application on http://\:5000 + +--- +### Steps to deploy using Jenkins & Docker: +- Install docker: +```bash +sudo apt update +sudo apt install docker.io -y +sudo usermod -aG docker $USER && newgrp docker +``` +--- +- Create a Dockerfile and paste the below content: +```bash +vim Dockerfile +``` +```bash +# Use Alpine-based Python image +FROM python:3.12-alpine + +# Working directory +WORKDIR /app + +# Copy code +COPY . . + +# Run Tests +RUN pip install --upgrade flask +RUN pip install '.[test]' +RUN pytest + +# Install Dependencies +RUN pip install -e . + +# Expose port +EXPOSE 5000 + +# Run the app +RUN flask --app flaskr init-db +ENTRYPOINT ["flask", "--app", "flaskr", "run", "--host=0.0.0.0", "--debug"] +``` +--- +- Install Jenkins: +```bash +sudo apt update -y +sudo apt install fontconfig openjdk-17-jre -y + +sudo wget -O /usr/share/keyrings/jenkins-keyring.asc \ + https://pkg.jenkins.io/debian-stable/jenkins.io-2023.key + +echo "deb [signed-by=/usr/share/keyrings/jenkins-keyring.asc]" \ + https://pkg.jenkins.io/debian-stable binary/ | sudo tee \ + /etc/apt/sources.list.d/jenkins.list > /dev/null + +sudo apt-get update -y +sudo apt-get install jenkins -y +``` +--- +- Access Jenkins from the browser and create a declarative pipeline and paste the below declarative pipeline: +```bash +pipeline{ + agent any + stages{ + stage("Cleanup Workspace"){ + steps{ + cleanWs() + } + } + + stage("Clone code"){ + steps{ + git branch: "main", url: "https://github.com/DevMadhup/flask.git" + } + } + + stage("Build Docker Image"){ + steps{ + sh "ls -lrt" + dir("examples/tutorial/"){ + sh "ls -lrt" + sh "docker build -t flaskapp ." + } + } + } + + stage("Push Image to DockerHub"){ + steps{ + withCredentials([usernamePassword(credentialsId: 'Docker-cred', passwordVariable: 'DOCKERHUB_PASS', usernameVariable: 'DOCKERHUB_USER')]) { + sh "docker login -u $DOCKERHUB_USER -p $DOCKERHUB_PASS" + sh "docker tag flaskapp madhupdevops/flaskapp-getfitwithsagar" + sh "docker push madhupdevops/flaskapp-getfitwithsagar" + } + } + } + + stage("Deploy docker container"){ + steps{ + sh "docker run -itd --name flaskapp -p 5000:5000 madhupdevops/flaskapp-getfitwithsagar" + } + } + } +} +``` +--- +- Now, run the pipeline and after it get successfull access it on public ip and port 5000 From 77753518f36a6f3113cba31104ca398384828a42 Mon Sep 17 00:00:00 2001 From: Madhup Pandey <121779953+DevMadhup@users.noreply.github.com> Date: Fri, 21 Mar 2025 19:34:37 +0530 Subject: [PATCH 002/133] Create solution.md --- .../Day3/solution_by_madhup/solution.md | 111 ++++++++++++++++++ 1 file changed, 111 insertions(+) create mode 100644 DailyChallenges/Day3/solution_by_madhup/solution.md diff --git a/DailyChallenges/Day3/solution_by_madhup/solution.md b/DailyChallenges/Day3/solution_by_madhup/solution.md new file mode 100644 index 0000000..a7b3667 --- /dev/null +++ b/DailyChallenges/Day3/solution_by_madhup/solution.md @@ -0,0 +1,111 @@ +## Launch EC2 Instances +- Create Two EC2 Instances - One for Grafana, One for Jenkins. +- Open required ports in the EC2 security group: +- Grafana: Port 3000. +- Jenkins: Port 8080. +- Nginx: Ports 80 and 443. + +## Update the system and install necessary packages on both instances: +``` +sudo apt update && sudo apt upgrade -y +``` + +### Install Grafana on the Grafana EC2 instance: +``` +sudo apt-get install -y apt-transport-https software-properties-common wget +sudo mkdir -p /etc/apt/keyrings/ +wget -q -O - https://apt.grafana.com/gpg.key | gpg --dearmor | sudo tee /etc/apt/keyrings/grafana.gpg > /dev/null +echo "deb [signed-by=/etc/apt/keyrings/grafana.gpg] https://apt.grafana.com stable main" | sudo tee -a /etc/apt/sources.list.d/grafana.list +echo "deb [signed-by=/etc/apt/keyrings/grafana.gpg] https://apt.grafana.com beta main" | sudo tee -a /etc/apt/sources.list.d/grafana.list +# Updates the list of available packages +sudo apt-get update +sudo apt-get install -y grafana +sudo systemctl start grafana-server +sudo systemctl enable grafana-server +sudo systemctl status grafana-server +``` + +## Install Java +``` +sudo apt update +sudo apt install fontconfig openjdk-17-jre +java -version +``` + +### Install Jenkins on the Jenkins EC2 instance: +``` +sudo wget -O /usr/share/keyrings/jenkins-keyring.asc \ + https://pkg.jenkins.io/debian-stable/jenkins.io-2023.key +echo "deb [signed-by=/usr/share/keyrings/jenkins-keyring.asc]" \ + https://pkg.jenkins.io/debian-stable binary/ | sudo tee \ + /etc/apt/sources.list.d/jenkins.list > /dev/null +sudo apt-get update +sudo apt-get install jenkins +sudo systemctl status jenkins +``` + +### Install and Configure Nginx as a Reverse Proxy +- Install Nginx on one EC2 instance (e.g., Grafana EC2): + +```bash +sudo apt-get install -y nginx +``` +- Configure Nginx: Edit the Nginx configuration file: + +```bash +sudo vim /etc/nginx/sites-available/default +``` + +Replace the file contents with: +``` +server { + listen 80; + server_name ; + + location / { + proxy_pass http://127.0.0.1:8080; + proxy_set_header Host $host; + proxy_set_header X-Real-IP $remote_addr; + proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; + proxy_set_header X-Forwarded-Proto $scheme; + } +} + +server { + listen 80; + server_name ; + + auth_basic "Restricted Access"; + auth_basic_user_file /etc/nginx/.htpasswd; + + location / { + proxy_pass http://127.0.0.1:3000; + proxy_set_header Host $host; + proxy_set_header X-Real-IP $remote_addr; + proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; + proxy_set_header X-Forwarded-Proto $scheme; + } +} +``` + +- Create a .htpasswd file for Grafana basic authentication: +``` +sudo apt-get install -y apache2-utils +sudo htpasswd -c /etc/nginx/.htpasswd admin +``` + +- Test and restart Nginx: +``` +sudo nginx -t +sudo systemctl restart nginx +``` + +- Generate a self-signed SSL certificate using certbot: + - Install python-certbot + ```bash + sudo apt install certbot python3-certbot-nginx + ``` + - Obtain an SSL Certificate + ```bash + sudo certbot --nginx -d -d + ``` From 7ed5b1cbec596089d923ac914059e882b65bc194 Mon Sep 17 00:00:00 2001 From: Madhup Pandey <121779953+DevMadhup@users.noreply.github.com> Date: Sat, 22 Mar 2025 10:40:11 +0530 Subject: [PATCH 003/133] Create solution.md --- DailyChallenges/Day8/solution.md | 69 ++++++++++++++++++++++++++++++++ 1 file changed, 69 insertions(+) create mode 100644 DailyChallenges/Day8/solution.md diff --git a/DailyChallenges/Day8/solution.md b/DailyChallenges/Day8/solution.md new file mode 100644 index 0000000..315cac1 --- /dev/null +++ b/DailyChallenges/Day8/solution.md @@ -0,0 +1,69 @@ +## Build a Distributed Logging System (Part1) + +### Steps to implement (with RabbitMQ): + +- Create an EC2 instance with the following details: + + - **AMI:** Ubuntu + - **Instance type:** t2.micro + - **Volume:** 15GB + - **Security group ports**: 22, 443, 15672 +--- + +- Install RabbitMQ +```bash +sudo apt update && sudo apt install rabbitmq-server -y +``` + +- Enable management plugin +```bash +sudo rabbitmq-plugins enable rabbitmq_management +sudo systemctl restart rabbitmq-server +``` +--- + +- Set Up a User and Queue +```bash +sudo rabbitmqctl add_user admin admin +sudo rabbitmqctl set_permissions -p / admin ".*" ".*" ".*" +``` +--- + +- Restart RabbitMQ +```bash +sudo systemctl restart rabbitmq-server +``` +--- + +- Change created user to admin user +```bash +sudo rabbitmqctl set_user_tags admin administrator +sudo systemctl restart rabbitmq-server +``` +--- + +- Access the RabbitMQ UI at `http://:15672`. Default credentials: `admin/admin`. +--- + +- Add logs queue to rabbitMQ +```bash +sudo apt install -y curl +curl -o rabbitmqadmin http://localhost:15672/cli/rabbitmqadmin +chmod +x rabbitmqadmin +sudo mv rabbitmqadmin /usr/local/bin/ +``` +--- + +- Create queue +```bash +rabbitmqadmin declare queue name=logs_queue +``` +--- + +- Check created queues +```bash +rabbitmqadmin list queues +``` +`Example output:` + +![image](https://github.com/user-attachments/assets/487533b7-7c04-4a5e-a977-988a15193c42) From 9dee140a3340d911a055f3a42032b92f43d51e8b Mon Sep 17 00:00:00 2001 From: mdsraihaniqbal1999 <43779921+mdsraihaniqbal1999@users.noreply.github.com> Date: Sun, 23 Mar 2025 07:56:50 +0530 Subject: [PATCH 004/133] Update challenge.md --- DailyChallenges/Season2/DAY6/challenge.md | 333 +++++++++++----------- 1 file changed, 172 insertions(+), 161 deletions(-) diff --git a/DailyChallenges/Season2/DAY6/challenge.md b/DailyChallenges/Season2/DAY6/challenge.md index 7a693d4..7819bec 100644 --- a/DailyChallenges/Season2/DAY6/challenge.md +++ b/DailyChallenges/Season2/DAY6/challenge.md @@ -1,164 +1,175 @@ -Challenge Overview -Theory: Explain the key differences between GitHub and GitLab, the migration process, and the importance of migrating all objects. +# GitHub Actions: Theory & Practical Challenge + +## **1. What is GitHub Actions?** +GitHub Actions is a CI/CD automation tool that enables developers to automate workflows within their GitHub repositories. It allows you to automate software development workflows, such as building, testing, and deploying applications. + +## **2. When Should GitHub Actions Be Used? When Should It Not Be Used?** +### **When to Use GitHub Actions:** +- Automating CI/CD pipelines. +- Running tests automatically on code commits. +- Deploying applications to cloud services (AWS, Azure, GCP, etc.). +- Managing infrastructure as code (Terraform, Ansible, etc.). +- Automating security scans and linting. +- Scheduling periodic jobs (e.g., nightly builds, cron jobs). + +### **When Not to Use GitHub Actions:** +- When you need on-premises-only solutions. +- If strict security compliance requires hosting the CI/CD pipeline on internal infrastructure. +- If you need advanced enterprise features available only in other CI/CD tools like Jenkins. + +| Feature | GitHub Actions | Jenkins | +|------------------|---------------|---------| +| Hosting | Cloud-based (GitHub-hosted runners) | Self-hosted (or cloud-based with effort) | +| Ease of Use | Simple YAML-based workflows | Requires setup & configuration | +| Integration | Native GitHub integration | Supports multiple VCS (Git, SVN, etc.) | +| Cost | Free for public repos, limited free usage for private repos | Requires dedicated infrastructure | +| Plugins | Built-in Marketplace | Extensive plugin ecosystem | +| Scalability | Managed by GitHub | Requires manual scaling | + +## **3. Steps to Create a GitHub Action** +1. Navigate to your GitHub repository. +2. Click on the `Actions` tab. +3. Choose a predefined template or create a `.github/workflows/main.yml` file. +4. Define your workflow steps using YAML. +5. Commit and push the file to trigger the workflow. + +## **4. GitHub Actions Workflow Structure** +A typical GitHub Actions workflow consists of: +- **name**: Name of the workflow. +- **on**: Defines when the workflow should run (e.g., push, pull_request, schedule, workflow_dispatch). +- **jobs**: Contains the tasks to be executed. +- **steps**: Defines individual steps in a job. +- **uses**: Calls an external GitHub Action. +- **run**: Executes custom commands or scripts. + +Example: +```yaml +name: Example Workflow +on: push +jobs: + build: + runs-on: ubuntu-latest + steps: + - name: Checkout code + uses: actions/checkout@v4 + - name: Run a script + run: echo "Hello, GitHub Actions!" +``` + +## **5. What is a Job? What is an Action?** +- **Job**: A job is a set of steps that execute in the same runner. + ```yaml + jobs: + build: + runs-on: ubuntu-latest + steps: + - run: echo "This is a job" + ``` + +- **Action**: A reusable unit of work within GitHub Actions. + ```yaml + steps: + - name: Use a Docker-based Action + uses: actions/hello-world-docker-action@v1 + ``` + +## **6. What Are Inputs and Outputs in GitHub Actions?** +- **Inputs** allow workflows to accept dynamic values. +- **Outputs** store results from one step/job and pass them to another. + +Example: +```yaml +jobs: + example: + runs-on: ubuntu-latest + steps: + - id: step1 + run: echo "::set-output name=myoutput::Hello" + - run: echo "Output from step1: ${{ steps.step1.outputs.myoutput }}" +``` + +## **7. General Steps in a GitHub Action** +1. **Checkout Code** → Use `actions/checkout` to fetch repository code. +2. **Authenticate & Authorize** → Configure AWS, Azure, or any required credentials. +3. **Perform Necessary Operations** → Run scripts, deploy applications, execute tests, etc. + +## **8. Why Should Permissions Be Declared at the Top?** +Permissions should be set at the workflow level to ensure security by restricting access to only required resources. +Example: +```yaml +permissions: + id-token: write + contents: read +``` +This prevents unauthorized actions from executing within the workflow. + +## **9. Theory Challenge** +### **Answer the following questions:** +1. What are the key differences between GitHub Actions and Jenkins? +2. Explain the purpose of inputs and outputs in GitHub Actions. +3. Describe the general structure of a GitHub Actions workflow. +4. Why is it important to declare permissions at the top of a workflow file? + +## **10. Practical Challenge** +### **Challenge Scenario:** +You are required to create a GitHub Actions workflow that: +1. **Builds and pushes a Docker image** to Docker Hub. +2. **Deploys the application on an AWS EC2 instance** using AWS Systems Manager (SSM). +3. **Validates that the application is running correctly**. +4. **Sends an email notification** with the deployment status. + +### **Provided Code:** +#### **app.py** +```python +from flask import Flask + +app = Flask(__name__) + +@app.route('/') +def home(): + return "Hello, Welcome to Season 2! You are learning GitHub Actions." + +if __name__ == '__main__': + app.run(host='0.0.0.0', port=5000) +``` + +#### **Dockerfile** +```dockerfile +# Use official Python image +FROM python:3.9-slim + +# Set working directory +WORKDIR /app + +# Copy files +COPY app.py /app + +# Install Flask +RUN pip install flask + +# Expose port 5000 +EXPOSE 5000 + +# Run Flask app +CMD ["python", "app.py"] +``` + +### **Submission Guidelines:** +- Make a **GitHub repository** for the challenge. +- Create necessary cloud services required (e.g., **AWS EC2**, **Gmail App Password**, **AWS IAM Roles**, etc.). +- Take **screenshots** of significant steps. +- Document the **solution** in a `README.md` file, explaining each step clearly. +- Mention **challenges faced** and how you overcame them. +- Ensure that your repository is well-structured and follows best practices. + +### **Additional Notes:** +- Make sure to use proper authentication methods while pushing Docker images. +- Structure your workflow efficiently by separating build, deployment, and notification steps. +- Follow security best practices, such as setting permissions and using secrets. +- Ensure logging and error handling are in place to debug issues easily. + +Kind Regards +Sagar Utekar & Raihan Iqbal -Subtasks: -Migrate repositories and their content. - -Migrate users, permissions, and teams. - -Migrate issues, pull requests, and projects. - -Migrate GitHub Actions workflows to GitLab CI/CD. - -Migrate repository settings, webhooks, and integrations. - -Process: Provide a step-by-step guide for the candidate to follow. - -Evaluation: Assess the candidate's understanding, execution, and problem-solving skills. - -Detailed Steps and Plan -1. Theory (Knowledge Assessment) -Ask the candidate to answer the following questions: - -What are the key differences between GitHub and GitLab? - -What are the challenges of migrating all GitHub objects (repositories, users, workflows, etc.) to GitLab? - -How would you ensure data integrity during the migration? - -What are the best practices for migrating CI/CD pipelines and workflows? - -2. Subtasks -Subtask 1: Migrate Repositories and Content -The candidate should: - -Use GitLab's built-in GitHub importer (recommended in the GitLab blog) to migrate repositories. - -Ensure all branches, tags, and commit history are preserved. - -Verify that the repository content (code, files, etc.) is intact. - -Subtask 2: Migrate Users, Permissions, and Teams -The candidate should: - -Create corresponding users in GitLab. - -Map GitHub users to GitLab users. - -Migrate teams and permissions (e.g., admin, maintainer, developer) using GitLab groups and access controls. - -Subtask 3: Migrate Issues, Pull Requests, and Projects -The candidate should: - -Use GitLab's import tools or third-party tools (e.g., github-gitlab-importer) to migrate issues, pull requests, and projects. - -Ensure all metadata (e.g., labels, milestones, comments) is preserved. - -Subtask 4: Migrate GitHub Actions Workflows to GitLab CI/CD -The candidate should: - -Convert GitHub Actions workflows (.github/workflows/*.yml) to GitLab CI/CD pipelines (.gitlab-ci.yml). - -Replicate all steps (e.g., build, test, deploy) in GitLab CI/CD. - -Use GitLab CI/CD features like caching, artifacts, and environments to optimize the pipeline. - -Subtask 5: Migrate Repository Settings and Webhooks -The candidate should: - -Migrate repository settings (e.g., branch protection rules, merge request settings). - -Recreate webhooks in GitLab for integrations (e.g., Slack, Jira). - -3. Process -Provide the candidate with the following step-by-step instructions: - -Prepare for Migration: - -Audit the GitHub organization to identify all objects to be migrated (repositories, users, projects, workflows, etc.). - -Create a GitLab group or namespace to mirror the GitHub organization structure. - -Migrate Repositories: - -Use GitLab's GitHub importer (as described in the GitLab blog) to migrate repositories. - -Verify that all branches, tags, and commit history are preserved. - -Migrate Users, Permissions, and Teams: - -Create users in GitLab and map them to GitHub users. - -Assign appropriate roles and permissions in GitLab using groups and access controls. - -Migrate Issues, Pull Requests, and Projects: - -Use GitLab's import tools or third-party tools to migrate issues, pull requests, and projects. - -Verify that all metadata (labels, milestones, comments) is preserved. - -Migrate GitHub Actions Workflows: - -Convert GitHub Actions workflows to GitLab CI/CD pipelines. - -Test the pipelines to ensure they work as expected. - -Migrate Repository Settings and Webhooks: - -Recreate repository settings (e.g., branch protection rules) in GitLab. - -Recreate webhooks for integrations. - -Validate the Migration: - -Run the GitLab CI/CD pipelines and verify that all steps pass. - -Deploy the application (if applicable) and ensure it works as expected. - -Verify that all issues, pull requests, and projects are intact. - -Document the Process: - -Write a detailed report explaining the migration steps, challenges faced, and solutions implemented. - -4. Evaluation Criteria -Evaluate the candidate based on the following: - -Knowledge: Understanding of GitHub, GitLab, and migration concepts. - -Execution: Ability to migrate all objects and ensure data integrity. - -Problem-Solving: Handling errors and optimizing the migration process. - -Documentation: Clarity and completeness of the migration report. - -Tools and Resources -GitHub Organization: Provide access to a sample GitHub organization with repositories, users, projects, and workflows. - -GitLab Group: Provide access to a GitLab group for migration. - -Documentation: - -GitLab's GitHub Importer - -GitLab CI/CD Documentation - -Third-Party Migration Tools - -Timeline -Preparation: 1 hour (audit GitHub organization and plan migration). - -Migration: 4 hours (migrate repositories, users, projects, workflows, etc.). - -Validation and Documentation: 2 hours. - -Bonus Tasks (Optional) -Migrate GitHub Wikis and Pages to GitLab. - -Implement advanced GitLab CI/CD features (e.g., environments, artifacts). - -Set up monitoring and alerting for the migrated pipelines. - -This challenge will test the candidate's ability to perform a complete migration while ensuring data integrity, minimal downtime, and adherence to best practices. It also provides insight into their problem-solving and documentation skills. The referenced guides will serve as excellent resources for the candidate to follow during the challenge. From 2e5a98f61c403cfb23e150f3746826c6a8e18e5e Mon Sep 17 00:00:00 2001 From: mdsraihaniqbal1999 <43779921+mdsraihaniqbal1999@users.noreply.github.com> Date: Sun, 23 Mar 2025 07:57:10 +0530 Subject: [PATCH 005/133] Update challenge.md --- DailyChallenges/Season2/DAY6/challenge.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/DailyChallenges/Season2/DAY6/challenge.md b/DailyChallenges/Season2/DAY6/challenge.md index 7819bec..8adb763 100644 --- a/DailyChallenges/Season2/DAY6/challenge.md +++ b/DailyChallenges/Season2/DAY6/challenge.md @@ -1,4 +1,4 @@ -# GitHub Actions: Theory & Practical Challenge +# Day 6:GitHub Actions: Theory & Practical Challenge ## **1. What is GitHub Actions?** GitHub Actions is a CI/CD automation tool that enables developers to automate workflows within their GitHub repositories. It allows you to automate software development workflows, such as building, testing, and deploying applications. From 79a13659df4af21bff384cc542b52488a66c0043 Mon Sep 17 00:00:00 2001 From: mdsraihaniqbal1999 <43779921+mdsraihaniqbal1999@users.noreply.github.com> Date: Sun, 23 Mar 2025 07:57:35 +0530 Subject: [PATCH 006/133] Update challenge.md --- DailyChallenges/Season2/DAY6/challenge.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/DailyChallenges/Season2/DAY6/challenge.md b/DailyChallenges/Season2/DAY6/challenge.md index 8adb763..2eb93f2 100644 --- a/DailyChallenges/Season2/DAY6/challenge.md +++ b/DailyChallenges/Season2/DAY6/challenge.md @@ -1,4 +1,4 @@ -# Day 6:GitHub Actions: Theory & Practical Challenge +# Day 6 : GitHub Actions Theory & Practical Challenge ## **1. What is GitHub Actions?** GitHub Actions is a CI/CD automation tool that enables developers to automate workflows within their GitHub repositories. It allows you to automate software development workflows, such as building, testing, and deploying applications. From 5cc5c95ae04e4f859a71f92a9d90eb066662d38e Mon Sep 17 00:00:00 2001 From: mdsraihaniqbal1999 <43779921+mdsraihaniqbal1999@users.noreply.github.com> Date: Sun, 23 Mar 2025 07:58:02 +0530 Subject: [PATCH 007/133] Update challenge.md --- DailyChallenges/Season2/DAY6/challenge.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/DailyChallenges/Season2/DAY6/challenge.md b/DailyChallenges/Season2/DAY6/challenge.md index 2eb93f2..fc70441 100644 --- a/DailyChallenges/Season2/DAY6/challenge.md +++ b/DailyChallenges/Season2/DAY6/challenge.md @@ -168,7 +168,7 @@ CMD ["python", "app.py"] - Follow security best practices, such as setting permissions and using secrets. - Ensure logging and error handling are in place to debug issues easily. -Kind Regards +Kind Regards
Sagar Utekar & Raihan Iqbal From e7bcb22bb5b4835cedf4b2fac845f7fb4b70054c Mon Sep 17 00:00:00 2001 From: mdsraihaniqbal1999 <43779921+mdsraihaniqbal1999@users.noreply.github.com> Date: Sun, 23 Mar 2025 07:58:34 +0530 Subject: [PATCH 008/133] Create Dockerfile --- DailyChallenges/Season2/DAY6/Dockerfile | 16 ++++++++++++++++ 1 file changed, 16 insertions(+) create mode 100644 DailyChallenges/Season2/DAY6/Dockerfile diff --git a/DailyChallenges/Season2/DAY6/Dockerfile b/DailyChallenges/Season2/DAY6/Dockerfile new file mode 100644 index 0000000..13b31b0 --- /dev/null +++ b/DailyChallenges/Season2/DAY6/Dockerfile @@ -0,0 +1,16 @@ +FROM python:3.9-slim + +# Set working directory +WORKDIR /app + +# Copy files +COPY app.py /app + +# Install Flask +RUN pip install flask + +# Expose port 5000 +EXPOSE 5000 + +# Run Flask app +CMD ["python", "app.py"] From f238a602fa3e7cffee4458e77c98ffa2b092579d Mon Sep 17 00:00:00 2001 From: mdsraihaniqbal1999 <43779921+mdsraihaniqbal1999@users.noreply.github.com> Date: Sun, 23 Mar 2025 07:58:55 +0530 Subject: [PATCH 009/133] Create app.py --- DailyChallenges/Season2/DAY6/app.py | 10 ++++++++++ 1 file changed, 10 insertions(+) create mode 100644 DailyChallenges/Season2/DAY6/app.py diff --git a/DailyChallenges/Season2/DAY6/app.py b/DailyChallenges/Season2/DAY6/app.py new file mode 100644 index 0000000..f9a9aff --- /dev/null +++ b/DailyChallenges/Season2/DAY6/app.py @@ -0,0 +1,10 @@ +from flask import Flask + +app = Flask(__name__) + +@app.route('/') +def home(): + return "Hello, Welcome to Season 2! You are learning GitHub Actions." + +if __name__ == '__main__': + app.run(host='0.0.0.0', port=5000) From 009e34f95e6549cf2d2e23dcaa8fe28faa6a360f Mon Sep 17 00:00:00 2001 From: A-ravi <55378835+A-ravi@users.noreply.github.com> Date: Sun, 23 Mar 2025 22:26:05 +0530 Subject: [PATCH 010/133] Create module1.md --- DailyChallenges/Season2/Linux/module1.md | 108 +++++++++++++++++++++++ 1 file changed, 108 insertions(+) create mode 100644 DailyChallenges/Season2/Linux/module1.md diff --git a/DailyChallenges/Season2/Linux/module1.md b/DailyChallenges/Season2/Linux/module1.md new file mode 100644 index 0000000..4ef7cec --- /dev/null +++ b/DailyChallenges/Season2/Linux/module1.md @@ -0,0 +1,108 @@ + +## What is OS? + +An Operating System (OS) is a software that manages everything your computer does and makes sure all the parts work together properly. Without it, your computer wouldn’t know how to run programs, connect to the internet, or even display anything on the screen. + +### Key Functions of an Operating System: +- Manages Hardware: It controls the CPU (processor), memory, and other devices (like printers, keyboards, and screens). +- Runs Applications: It allows you to open apps like browsers, games, or word processors. +- Provides User Interface: It gives you a way to interact with the computer, like the Windows desktop or your smartphone’s screen. +- Manages Files and Folders: It helps you store and organize your files properly. + +### Examples of Operating Systems: +- Windows – Used on many PCs. +- macOS – Used on Apple computers. +- Linux – Often used by developers and servers. +- Android/iOS – Operating systems for smartphones. + +In simple terms, without an operating system, your computer is just a box of hardware with no idea how to do anything! + +## Operating System Architecture + +An Operating System (OS) is built using several layers, each with its own purpose. + +### Here’s a breakdown of the architecture from top to bottom: +- User Interface (UI): + - This is what users see and interact with. + - It can be a Graphical User Interface (GUI) (like icons, windows, and buttons) or a Command-Line Interface (CLI) (where you type commands). + - Example: The desktop screen in Windows, the home screen on Android and shell on Linux. + +- Application Layer: + - This layer runs applications like browsers, media players, or text editors. + - These apps send requests to the operating system when they need to access hardware or files. + - Example: When you save a document in Word, the app sends a request to the OS to store it on your hard disk. + +- Kernel: + - This is the core part of the operating system. + - It manages essential tasks like: + - Process Management – Handling running apps. + - Memory Management – Allocating memory to processes. + - Device Management – Managing input/output devices like printers, keyboards, and hard drives. + - File System Management – Handling how files are read, stored, and accessed. + - The kernel acts as a bridge between applications and hardware. + +- Hardware: + - This includes the physical components of your computer, like the CPU (processor), RAM (memory), hard disk, and other devices. + - The OS controls and coordinates how these components function. + +### Flow of Communication: +- Top to Bottom: When you open an app, the request flows from the user interface to the kernel, which manages hardware resources to complete the task. +- Bottom to Top: When hardware generates data (like keyboard input), it passes through the kernel and reaches the user via the user interface. + +## What is Unix? +Unix is a family of operating systems that began at AT&T in the late 1960s. It was designed to be a powerful, multiuser, and multitasking system. + +Some members of Unix family are: BSD, Xenix, Solaris, HP-UX and AIX. + +**Key points about Unix include:** + +- **Multiuser and Multitasking**: Multiple users can work on the same system at the same time. +- **Portability**: Unix was written in the C programming language, making it easier to run on different hardware. +- **Influence**: Many modern operating systems, including Linux, trace their ideas and design principles back to Unix. + +**History:** +- 1969-1970: Unix was developed at AT&T by Ken Thompson, Dennis Ritchie, and others. +- 1970s: It spread to universities and became popular for its simplicity and powerful tools. +- Impact: Unix’s design principles laid the groundwork for many modern operating systems. + +## What is Linux? + +Linux is a _Unix-like_ operating system kernel originally created in 1991 by Linus Torvalds while he was a student in Helsinki. When combined with a vast collection of GNU software and other applications, it forms a complete operating system—often called GNU/Linux or simply Linux. + +**Here are some key points:** +- **Open Source**: Linux is developed and maintained by a global community under the GNU General Public License (GPL), meaning its source code is freely available. +- **Flexibility**: It is highly modular, allowing users to customize everything from the kernel to the desktop interface. +- **Wide Usage**: Linux powers everything from smartphones (via Android) and personal computers to servers, supercomputers, and embedded systems. + +**History:** +- 1991: Linus Torvalds begins a project to create a free kernel for personal computers, announcing Linux in a famous Usenet post. +- 1992: The Linux kernel is relicensed under the GNU GPL, encouraging community contributions. +- Growth: Over the years, Linux has grown from a small project into a robust, full-featured operating system used worldwide. +- Collaboration with GNU: By combining the Linux kernel with GNU tools and libraries, users get a complete free operating system. + +## Different Linux Distros + +Because Linux is free and open source, various groups—both community-driven and commercial—have built their own “flavors” of Linux. These are called distributions or distros. They package the Linux kernel along with different sets of software to suit particular needs. Some examples include: + +- Ubuntu: + Known for its user-friendly design, Ubuntu is popular for desktops and servers. It’s based on Debian and is widely supported by a strong community and Canonical Ltd. + +- Red Hat Enterprise Linux (RHEL): + It is a commercial open-source Linux distribution developed by Red Hat for the commercial market. It is designed to provide a stable, secure, and high-performance platform for various applications, from servers to desktops. + +- openSUSE: + openSUSE is a community-driven project that provides two main Linux distributions:​ openSUSE Leap & openSUSE Tumbleweed. + +- Fedora: + Backed by Red Hat, Fedora is a cutting-edge distro that showcases the latest free software and innovations, often serving as a testing ground for features that later appear in Red Hat Enterprise Linux. + +- Linux Mint: + Based on Ubuntu or Debian, Linux Mint is designed to be easy for beginners with a familiar desktop interface and multimedia support out-of-the-box. + +- Arch Linux: + A minimalist, rolling-release distro for experienced users. Arch Linux offers extensive customizability and a “build it yourself” approach using the Pacman package manager. + +- Others: + There are hundreds of distributions targeting specific use cases—from lightweight distros for older hardware to specialized distros for security testing (like Kali Linux) or multimedia production. + +Each distro has its own package management, default desktop environment, and tools that cater to different user preferences and system requirements. From e98bcbcd8af7876c9126a4496389ff3233709537 Mon Sep 17 00:00:00 2001 From: Sagar Utekar Date: Tue, 25 Mar 2025 10:17:07 +0530 Subject: [PATCH 011/133] Update solution.md --- DailyChallenges/Day3/solution.md | 23 +++++++++++++++++++++++ 1 file changed, 23 insertions(+) diff --git a/DailyChallenges/Day3/solution.md b/DailyChallenges/Day3/solution.md index a2f5bdf..8d5ee39 100644 --- a/DailyChallenges/Day3/solution.md +++ b/DailyChallenges/Day3/solution.md @@ -1,3 +1,26 @@ +``` +1. Core Concepts +Client: Requests a service (e.g., browser accessing a website). + +Server: Responds to requests (e.g., Google’s servers). + +Proxy: Acts as an intermediary. + +Forward Proxy: Sits on the client side (e.g., corporate VPNs to monitor employee internet access). + +Reverse Proxy: Sits on the server side (e.g., Nginx routing traffic to backend services like Grafana/Jenkins). + +Load Balancer: Distributes traffic across servers (e.g., AWS ALB/NLB). + +2. Why Nginx? +Reverse Proxy + Load Balancing: Routes requests to multiple backend services. + +Security: SSL/TLS termination, IP whitelisting, rate limiting. + +Performance: Caching static content, reducing server load. +``` + + ## Launch EC2 Instances - Create Two EC2 Instances - One for Grafana, One for Jenkins. - Open required ports in the EC2 security group: From dd196a1be2cf075937413b83cc6691526c226460 Mon Sep 17 00:00:00 2001 From: A-ravi <55378835+A-ravi@users.noreply.github.com> Date: Tue, 25 Mar 2025 19:06:36 +0530 Subject: [PATCH 012/133] add os architecture image --- DailyChallenges/Season2/Linux/module1.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/DailyChallenges/Season2/Linux/module1.md b/DailyChallenges/Season2/Linux/module1.md index 4ef7cec..e6794d7 100644 --- a/DailyChallenges/Season2/Linux/module1.md +++ b/DailyChallenges/Season2/Linux/module1.md @@ -21,6 +21,8 @@ In simple terms, without an operating system, your computer is just a box of har An Operating System (OS) is built using several layers, each with its own purpose. +![image](https://github.com/user-attachments/assets/26678a2d-b4c6-4180-83d8-256daef70eab) + ### Here’s a breakdown of the architecture from top to bottom: - User Interface (UI): - This is what users see and interact with. From 5abc718a5b983808f455efc02d834ec843b1967c Mon Sep 17 00:00:00 2001 From: Sagar Utekar Date: Wed, 26 Mar 2025 07:28:34 +0530 Subject: [PATCH 013/133] Update solution.md --- DailyChallenges/Day3/solution.md | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/DailyChallenges/Day3/solution.md b/DailyChallenges/Day3/solution.md index 8d5ee39..3e6b3ae 100644 --- a/DailyChallenges/Day3/solution.md +++ b/DailyChallenges/Day3/solution.md @@ -56,6 +56,12 @@ java -version openjdk version "17.0.8" 2023-07-18 ``` +``` +curl -fL -o corretto.rpm https://corretto.aws/downloads/latest/amazon-corretto-21-x64-linux-jdk.rpm +yum localinstall -y corretto.rpm +export JAVA_HOME=/usr/lib/jvm/java-21-amazon-corretto +``` + ### Install Jenkins on the Jenkins EC2 instance: ``` sudo wget -O /usr/share/keyrings/jenkins-keyring.asc \ From 962ca18dcd92ebd2fbce5d8ee9d3a9fc8cb5f2ac Mon Sep 17 00:00:00 2001 From: Sagar Utekar Date: Wed, 26 Mar 2025 07:37:57 +0530 Subject: [PATCH 014/133] Update solution.md --- DailyChallenges/Day3/solution.md | 50 ++++++++++++++++++-------------- 1 file changed, 29 insertions(+), 21 deletions(-) diff --git a/DailyChallenges/Day3/solution.md b/DailyChallenges/Day3/solution.md index 3e6b3ae..023835a 100644 --- a/DailyChallenges/Day3/solution.md +++ b/DailyChallenges/Day3/solution.md @@ -48,30 +48,38 @@ sudo systemctl enable grafana-server sudo systemctl status grafana-server ``` -## Install Java -``` -sudo apt update -sudo apt install fontconfig openjdk-17-jre -java -version -openjdk version "17.0.8" 2023-07-18 +## Install Java & Jenkins ``` +Downloading and installing Jenkins +Completing the previous steps enables you to download and install Jenkins on AWS. To download and install Jenkins: -``` -curl -fL -o corretto.rpm https://corretto.aws/downloads/latest/amazon-corretto-21-x64-linux-jdk.rpm -yum localinstall -y corretto.rpm -export JAVA_HOME=/usr/lib/jvm/java-21-amazon-corretto -``` +Ensure that your software packages are up to date on your instance by using the following command to perform a quick software update: + +[ec2-user ~]$ sudo yum update –y +Add the Jenkins repo using the following command: + +[ec2-user ~]$ sudo wget -O /etc/yum.repos.d/jenkins.repo \ + https://pkg.jenkins.io/redhat-stable/jenkins.repo +Import a key file from Jenkins-CI to enable installation from the package: + +[ec2-user ~]$ sudo rpm --import https://pkg.jenkins.io/redhat-stable/jenkins.io-2023.key +[ec2-user ~]$ sudo yum upgrade +Install Java (Amazon Linux 2023): + +[ec2-user ~]$ sudo dnf install java-17-amazon-corretto -y +Install Jenkins: + +[ec2-user ~]$ sudo yum install jenkins -y +Enable the Jenkins service to start at boot: + +[ec2-user ~]$ sudo systemctl enable jenkins +Start Jenkins as a service: + +[ec2-user ~]$ sudo systemctl start jenkins +You can check the status of the Jenkins service using the command: + +[ec2-user ~]$ sudo systemctl status jenkins -### Install Jenkins on the Jenkins EC2 instance: -``` -sudo wget -O /usr/share/keyrings/jenkins-keyring.asc \ - https://pkg.jenkins.io/debian-stable/jenkins.io-2023.key -echo "deb [signed-by=/usr/share/keyrings/jenkins-keyring.asc]" \ - https://pkg.jenkins.io/debian-stable binary/ | sudo tee \ - /etc/apt/sources.list.d/jenkins.list > /dev/null -sudo apt-get update -sudo apt-get install jenkins -sudo status jenkins ``` ### Install and Configure Nginx as a Reverse Proxy From 701332766103ab50196ddd1e09cb4df7e2cdd12e Mon Sep 17 00:00:00 2001 From: Sagar Utekar Date: Wed, 26 Mar 2025 07:41:38 +0530 Subject: [PATCH 015/133] Update solution.md --- DailyChallenges/Day3/solution.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/DailyChallenges/Day3/solution.md b/DailyChallenges/Day3/solution.md index 023835a..0551b3f 100644 --- a/DailyChallenges/Day3/solution.md +++ b/DailyChallenges/Day3/solution.md @@ -47,6 +47,7 @@ sudo systemctl start grafana-server sudo systemctl enable grafana-server sudo systemctl status grafana-server ``` +Official Doc for Grafana Installation: https://grafana.com/grafana/download?edition=oss ## Install Java & Jenkins ``` @@ -81,6 +82,7 @@ You can check the status of the Jenkins service using the command: [ec2-user ~]$ sudo systemctl status jenkins ``` +Official Doc for Jenkins Installation: https://www.jenkins.io/doc/tutorials/tutorial-for-installing-jenkins-on-AWS/#prerequisites ### Install and Configure Nginx as a Reverse Proxy - Install Nginx on one EC2 instance (e.g., Grafana EC2): ```sudo apt-get install -y nginx``` From 88016f67ead037cbe3b233b0a53d9c2e0c34f329 Mon Sep 17 00:00:00 2001 From: Madhup Pandey <121779953+DevMadhup@users.noreply.github.com> Date: Wed, 26 Mar 2025 15:14:04 +0530 Subject: [PATCH 016/133] Create solution.md for Day10 --- DailyChallenges/DAY10/solution.md | 248 ++++++++++++++++++++++++++++++ 1 file changed, 248 insertions(+) create mode 100644 DailyChallenges/DAY10/solution.md diff --git a/DailyChallenges/DAY10/solution.md b/DailyChallenges/DAY10/solution.md new file mode 100644 index 0000000..00abc22 --- /dev/null +++ b/DailyChallenges/DAY10/solution.md @@ -0,0 +1,248 @@ +## 3 Node rabbitMQ cluster setup and monitoring using Prometheus and Grafana + +### Steps to implement: + +- Create 3 EC2 instance with the following details: + + - **AMI:** Ubuntu + - **Instance type:** t2.medium + - **Volume:** 15GB + - **Security group ports**: + + - 5672 (AMQP) for RabbitMQ messaging + - 15672 (RabbitMQ management UI) + - 15692 (Prometheus metrics endpoint) + - 9090 (Prometheus) + - 3000 (Grafana) +--- + +- Update system (All nodes): +```bash +sudo su +apt update +``` +--- + +- create a shell script to install erlang and rabbitmq-server (All nodes): +```bash +#!/bin/sh + +sudo apt-get install curl gnupg apt-transport-https -y + +## Team RabbitMQ's main signing key +curl -1sLf "https://keys.openpgp.org/vks/v1/by-fingerprint/0A9AF2115F4687BD29803A206B73A36E6026DFCA" | sudo gpg --dearmor | sudo tee /usr/share/keyrings/com.rabbitmq.team.gpg > /dev/null +## Community mirror of Cloudsmith: modern Erlang repository +curl -1sLf https://github.com/rabbitmq/signing-keys/releases/download/3.0/cloudsmith.rabbitmq-erlang.E495BB49CC4BBE5B.key | sudo gpg --dearmor | sudo tee /usr/share/keyrings/rabbitmq.E495BB49CC4BBE5B.gpg > /dev/null +## Community mirror of Cloudsmith: RabbitMQ repository +curl -1sLf https://github.com/rabbitmq/signing-keys/releases/download/3.0/cloudsmith.rabbitmq-server.9F4587F226208342.key | sudo gpg --dearmor | sudo tee /usr/share/keyrings/rabbitmq.9F4587F226208342.gpg > /dev/null + +## Add apt repositories maintained by Team RabbitMQ +sudo tee /etc/apt/sources.list.d/rabbitmq.list < Node-2: vim /var/lib/rabbitmq/.erlang.cookie (PASTE)
+ Node-3: vim /var/lib/rabbitmq/.erlang.cookie (PASTE) + +--- + +- Restart rabbitmq-server (Node1, Node2 and Node3): +```bash +systemctl restart rabbitmq-server +``` +--- + +- Stop application to join cluster (Node2 and Node3): +```bash +rabbitmqctl stop_app +``` +--- + +- Join cluster (Node2 and Node3): +```bash +rabbitmqctl join_cluster rabbit@ip- +``` +--- + +- Start application (Node2 and Node3): +```bash +rabbitmqctl start_app +``` +--- + +- Go to Node-1, and check cluster status: +```bash +rabbitmqctl cluster_status +``` +--- + +- Enable the RabbitMQ Management Plugin to access UI: +```bash +rabbitmq-plugins enable rabbitmq_management +systemctl restart rabbitmq-server +``` + +`Congratulations, you have successfully setupped RabbitMQ 3 node cluster.` + +--- + +- Now go to the Node-1, and setup prometheus server: +```bash +wget https://github.com/prometheus/prometheus/releases/download/v2.47.0/prometheus-2.47.0.linux-amd64.tar.gz +tar zxvf prometheus-2.47.0.linux-amd64.tar.gz +``` +--- + +- Create a prometheus.service inside the /etc/systemd/system directory and paste the below content: `vim /etc/systemd/system/prometheus.service` +```bash +[Unit] + +Description=Prometheus Server + +Documentation=https://prometheus.io/docs/introduction/overview/ + +After=network-online.target + +[Service] + +User=root + +Restart=on-failure + +ExecStart=/home/ubuntu/prometheus-2.47.0.linux-amd64/prometheus --config.file=/home/ubuntu/prometheus-2.47.0.linux-amd64/prometheus.yml + +[Install] + +WantedBy=multi-user.target +``` +--- + +- Reload system daemon: +```bash +systemctl daemon-reload +``` +--- + +- Restart prometheus server: +```bash +systemctl restart prometheus +``` + +> [!Important] +> node exporter runs on 9090 port no. + +--- + +- Now, enable prometheus plugin (Node2 and Node3): +```bash +rabbitmq-plugins enable rabbitmq_prometheus +systemctl restart rabbitmq-server +``` +--- + +- Go to Node1, go to `prometheus-2.47.0.linux-amd64` directory and edit `prometheus.yml` file to scrape the metrics which are exposed from other cluster nodes +```bash +cd prometheus-2.47.0.linux-amd64 +vim prometheus.yml +``` +Add the below code to `prometheus.yml` +```bash + - job_name: "Node2" + + # metrics_path defaults to '/metrics' + # scheme defaults to 'http'. + + static_configs: + - targets: [":15692"] + + - job_name: "Node3" + + # metrics_path defaults to '/metrics' + # scheme defaults to 'http'. + + static_configs: + - targets: [":15692"] +``` +--- + +- Restart your prometheus server: +```bash +systemctl restart prometheus +``` +--- + +- Now, go to web browser, access your prometheus server and click on `Status` and then `Targets` and you will see all nodes metrics: + +![image](https://github.com/user-attachments/assets/98631c4e-3a3b-4009-954a-602c7af0e6bc) + +--- + +- Now, setup grafana on Node1: +```bash +sudo apt-get update +sudo apt-get install -y software-properties-common +sudo add-apt-repository "deb https://packages.grafana.com/oss/deb stable main" +sudo apt-get update +sudo apt-get install -y grafana +sudo systemctl start grafana-server +sudo systemctl enable grafana-server +``` + +> Grafana UI: http://\:3000 + +--- + +- Go to your grafana server, add prometheus as a datasource and go to dashboard and click on import Dashboard + - **Dashboard ID**: 10991 + +You will see beautiful RabbitMQ cluster Dashboard like this + +![image](https://github.com/user-attachments/assets/295083ed-7a31-4c56-b890-09cbbc59ed82) From 2115bd49d472bf40f91cf96dd289950b8d8acd7d Mon Sep 17 00:00:00 2001 From: Sagar Utekar Date: Thu, 27 Mar 2025 07:35:30 +0530 Subject: [PATCH 017/133] Update challenge.md --- DailyChallenges/Season2/DAY6/challenge.md | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/DailyChallenges/Season2/DAY6/challenge.md b/DailyChallenges/Season2/DAY6/challenge.md index fc70441..b94ba34 100644 --- a/DailyChallenges/Season2/DAY6/challenge.md +++ b/DailyChallenges/Season2/DAY6/challenge.md @@ -106,9 +106,8 @@ This prevents unauthorized actions from executing within the workflow. ## **9. Theory Challenge** ### **Answer the following questions:** 1. What are the key differences between GitHub Actions and Jenkins? -2. Explain the purpose of inputs and outputs in GitHub Actions. -3. Describe the general structure of a GitHub Actions workflow. -4. Why is it important to declare permissions at the top of a workflow file? +2. Describe the general structure of a GitHub Actions workflow. +3. How to manage variables and secrets in GitHub Actions ## **10. Practical Challenge** ### **Challenge Scenario:** From 8e73765ba4b981206d1f080a9008c11548f16059 Mon Sep 17 00:00:00 2001 From: Sagar Utekar Date: Thu, 27 Mar 2025 07:37:47 +0530 Subject: [PATCH 018/133] Update challenge.md --- DailyChallenges/Season2/DAY6/challenge.md | 17 +++-------------- 1 file changed, 3 insertions(+), 14 deletions(-) diff --git a/DailyChallenges/Season2/DAY6/challenge.md b/DailyChallenges/Season2/DAY6/challenge.md index b94ba34..fd06445 100644 --- a/DailyChallenges/Season2/DAY6/challenge.md +++ b/DailyChallenges/Season2/DAY6/challenge.md @@ -93,27 +93,17 @@ jobs: 2. **Authenticate & Authorize** → Configure AWS, Azure, or any required credentials. 3. **Perform Necessary Operations** → Run scripts, deploy applications, execute tests, etc. -## **8. Why Should Permissions Be Declared at the Top?** -Permissions should be set at the workflow level to ensure security by restricting access to only required resources. -Example: -```yaml -permissions: - id-token: write - contents: read -``` -This prevents unauthorized actions from executing within the workflow. - -## **9. Theory Challenge** +## **8. Theory Challenge** ### **Answer the following questions:** 1. What are the key differences between GitHub Actions and Jenkins? 2. Describe the general structure of a GitHub Actions workflow. 3. How to manage variables and secrets in GitHub Actions -## **10. Practical Challenge** +## **9. Practical Challenge** ### **Challenge Scenario:** You are required to create a GitHub Actions workflow that: 1. **Builds and pushes a Docker image** to Docker Hub. -2. **Deploys the application on an AWS EC2 instance** using AWS Systems Manager (SSM). +2. **Deploys the application on GitHub Runners** 3. **Validates that the application is running correctly**. 4. **Sends an email notification** with the deployment status. @@ -155,7 +145,6 @@ CMD ["python", "app.py"] ### **Submission Guidelines:** - Make a **GitHub repository** for the challenge. -- Create necessary cloud services required (e.g., **AWS EC2**, **Gmail App Password**, **AWS IAM Roles**, etc.). - Take **screenshots** of significant steps. - Document the **solution** in a `README.md` file, explaining each step clearly. - Mention **challenges faced** and how you overcame them. From 193292831486e691f9b935faae7df5ed7f8e64b5 Mon Sep 17 00:00:00 2001 From: Sagar Utekar Date: Thu, 27 Mar 2025 07:39:37 +0530 Subject: [PATCH 019/133] Update challenge.md --- DailyChallenges/Season2/DAY6/challenge.md | 29 ++++++++++++++++++++--- 1 file changed, 26 insertions(+), 3 deletions(-) diff --git a/DailyChallenges/Season2/DAY6/challenge.md b/DailyChallenges/Season2/DAY6/challenge.md index fd06445..09e2f79 100644 --- a/DailyChallenges/Season2/DAY6/challenge.md +++ b/DailyChallenges/Season2/DAY6/challenge.md @@ -146,9 +146,11 @@ CMD ["python", "app.py"] ### **Submission Guidelines:** - Make a **GitHub repository** for the challenge. - Take **screenshots** of significant steps. -- Document the **solution** in a `README.md` file, explaining each step clearly. +- Document the **solution** in a `solution.md` file, explaining each step clearly. - Mention **challenges faced** and how you overcame them. - Ensure that your repository is well-structured and follows best practices. +- Post your experience on social media with #getfitwithsagar #SRELife #DevOpsForAll. +- Tag us in your posts for visibility and networking. ### **Additional Notes:** - Make sure to use proper authentication methods while pushing Docker images. @@ -156,8 +158,29 @@ CMD ["python", "app.py"] - Follow security best practices, such as setting permissions and using secrets. - Ensure logging and error handling are in place to debug issues easily. -Kind Regards
-Sagar Utekar & Raihan Iqbal +--- + +## **Join Our Community** + +To make the most of this journey, connect with fellow learners: +- **Discord** – Ask questions and collaborate: [Join here](https://discord.gg/mNDm39qB8t) +- **Google Group** – Get updates and discussions: [Join here](https://groups.google.com/forum/#!forum/daily-devops-sre-challenge-series/join) +- **YouTube** – Watch solution videos and tutorials: [Subscribe here](https://www.youtube.com/@Sagar.Utekar) + +--- + +--- + +## **Stay Motivated!** + +Every challenge you complete brings you closer to mastering Git, DevOps, and SRE practices. Keep pushing your limits and collaborating with the community. + +Happy Learning! + +----------- + +**Best regards,** +Sagar Utekar & Raihan From 4a851c213e084ada3f01c66c9d1df969b8a69192 Mon Sep 17 00:00:00 2001 From: Sagar Utekar Date: Thu, 27 Mar 2025 08:27:28 +0530 Subject: [PATCH 020/133] Update challenge.md --- DailyChallenges/Season2/DAY6/challenge.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/DailyChallenges/Season2/DAY6/challenge.md b/DailyChallenges/Season2/DAY6/challenge.md index 09e2f79..11ea3d7 100644 --- a/DailyChallenges/Season2/DAY6/challenge.md +++ b/DailyChallenges/Season2/DAY6/challenge.md @@ -1,5 +1,7 @@ # Day 6 : GitHub Actions Theory & Practical Challenge +![Untitled design (1)](https://github.com/user-attachments/assets/085eac7f-92a3-4af4-a406-e94fced1e84a) + ## **1. What is GitHub Actions?** GitHub Actions is a CI/CD automation tool that enables developers to automate workflows within their GitHub repositories. It allows you to automate software development workflows, such as building, testing, and deploying applications. From fdfa9bf047094cab4f022d0c37dfbcdb81bd528c Mon Sep 17 00:00:00 2001 From: mdsraihaniqbal1999 <43779921+mdsraihaniqbal1999@users.noreply.github.com> Date: Thu, 27 Mar 2025 17:14:05 +0530 Subject: [PATCH 021/133] Create Challenge.md --- DailyChallenges/Season2/DAY-07/Challenge.md | 123 ++++++++++++++++++++ 1 file changed, 123 insertions(+) create mode 100644 DailyChallenges/Season2/DAY-07/Challenge.md diff --git a/DailyChallenges/Season2/DAY-07/Challenge.md b/DailyChallenges/Season2/DAY-07/Challenge.md new file mode 100644 index 0000000..6f58bd8 --- /dev/null +++ b/DailyChallenges/Season2/DAY-07/Challenge.md @@ -0,0 +1,123 @@ +# Day 07: GitHub Self-Hosted Runners + +## Introduction +Welcome to Day 07 of the DevOps SRE Daily Challenge! 🎉 + +Today, we focus on **GitHub Self-Hosted Runners**, a powerful way to customize GitHub Actions workflows by running jobs on your own infrastructure. + +By the end of this challenge, you will: +- Understand what GitHub runners are and their types. +- Compare **hosted** vs. **self-hosted** runners with advantages and disadvantages. +- Set up a **self-hosted runner** on an **AWS EC2 instance**. +- **Dockerize and deploy** the Snake game from [this repo](https://github.com/clear-code-projects/Snake) using the self-hosted runner. + +--- + +## 1️⃣ What Are Runners? + +GitHub **runners** are machines that execute workflows defined in **GitHub Actions**. These runners can be: +- Hosted by **GitHub** (default). +- **Self-hosted** (your own machine or cloud instance). + + +## 2️⃣ Types of GitHub Runners + +| Runner Type | Description | Use Cases | Advantages | Disadvantages | +|-----------------|-------------|-----------|-------------|---------------| +| **GitHub-Hosted** | Managed by GitHub with predefined environments. | General CI/CD workflows with minimal setup. | No maintenance, built-in scaling. | Limited customization, restricted software access. | +| **Self-Hosted** | User-managed runner on a personal machine or cloud instance. | Workflows needing custom dependencies or private network access. | Full control, cost-effective for large workloads. | Requires maintenance, security responsibility. | + +### When to Use What? + +| Scenario | Use **GitHub-Hosted** | Use **Self-Hosted** | +|----------|---------------------|---------------------| +| Simple CI/CD workflows | ✅ | ❌ | +| Need custom software/dependencies | ❌ | ✅ | +| Need access to private networks | ❌ | ✅ | +| Cost-effective for large workloads | ❌ | ✅ | +| Security and compliance control | ❌ | ✅ | + +--- + +## 3️⃣ Challenge Breakdown + +### **📝 Theoretical Questions** +Answer the following questions: +1. What is a GitHub runner? +2. How do self-hosted runners differ from GitHub-hosted runners? +3. What security considerations should you take when using self-hosted runners? +4. How can you scale self-hosted runners? +5. Can a single self-hosted runner be used for multiple repositories? Why or why not? + +--- + +### **⚙️ Practical Challenge: Setting Up a Self-Hosted Runner** + +#### **Step 1: Create an EC2 Instance** +1. Launch an AWS EC2 instance (Ubuntu 22.04 recommended). +2. Allow inbound traffic on ports **22 (SSH)** and **80 (HTTP)**. + +#### **Step 2: Configure GitHub Self-Hosted Runner** +1. Navigate to your **GitHub repository** → **Settings** → **Actions** → **Runners**. +2. Click **"New self-hosted runner"**, select **Linux**, and follow the instructions to set it up. +3. SSH into the EC2 instance and install the runner using the provided commands. +4. Start the runner: + ```bash + ./run.sh + ``` + +#### **Step 3: Deploy the Snake Game** +1. Install **Docker** on your EC2 instance: + ```bash + sudo apt update + sudo apt install docker.io -y + ``` +2. Clone the **Snake Game repository**: + ```bash + git clone https://github.com/clear-code-projects/Snake + cd Snake + ``` +3. Build and run the application as a Docker container: + ```bash + docker build -t snake-game . + docker run -d -p 80:5000 snake-game + ``` +4. Confirm the deployment by accessing `http://` in a browser. + +#### **Step 4: Take Screenshots for Submission** +- Running EC2 instance (`aws ec2 describe-instances`). +- GitHub Actions workflow logs showing execution on the self-hosted runner. +- Webpage running the Snake Game. + +--- + +## 4️⃣ Submission Guidelines + +✅ **Proof of Completion:** +- Screenshot of **EC2 instance running**. +- Screenshot of **GitHub Actions logs showing execution on self-hosted runner**. +- Screenshot of **Snake Game running in the browser**. + +✅ **Documentation:** +- Steps taken to configure the self-hosted runner. +- Learnings and challenges faced. + +--- + +## 5️⃣ Bonus Tasks 🎯 +- Use **GitHub Actions** to automate the deployment. +- Auto-scale self-hosted runners with **multiple EC2 instances**. +- Implement **load balancing** using **Nginx**. + +### **Share Your Progress!** +Post your experience on social media with the hashtags: **#getfitwithsagar, #SRELife, #DevOpsForAll** + +--- + +### **Why This Matters?** +Understanding **self-hosted GitHub runners** is critical for enterprises managing private infrastructure, optimizing costs, and ensuring workflow security. + +Keep learning and happy automating! 🚀 + +Kind Regards, +Sagar Utekar and Raihan From 1b6bf801fd1fd5d033fede93c2e85eebfefc7718 Mon Sep 17 00:00:00 2001 From: Sagar Utekar Date: Fri, 28 Mar 2025 05:31:53 +0530 Subject: [PATCH 022/133] Create challenge.md --- DailyChallenges/Season2/Day7/challenge.md | 138 ++++++++++++++++++++++ 1 file changed, 138 insertions(+) create mode 100644 DailyChallenges/Season2/Day7/challenge.md diff --git a/DailyChallenges/Season2/Day7/challenge.md b/DailyChallenges/Season2/Day7/challenge.md new file mode 100644 index 0000000..915cd81 --- /dev/null +++ b/DailyChallenges/Season2/Day7/challenge.md @@ -0,0 +1,138 @@ +# Day7: GitHub Self-Hosted Runners + +## Introduction +Welcome to Day7 of the Daily DevOps SRE Daily Challenge Series - Season2! 🎉 + +Today, we focus on **GitHub Self-Hosted Runners**, a powerful way to customize GitHub Actions workflows by running jobs on your own infrastructure. + +By the end of this challenge, you will: +- Understand what GitHub runners are and their types. +- Compare **hosted** vs. **self-hosted** runners with advantages and disadvantages. +- Set up a **self-hosted runner** on an **AWS EC2 instance**. +- **Dockerize and deploy** the Snake game from [this repo](https://github.com/clear-code-projects/Snake) using the self-hosted runner. + +--- + +### **Why This Matters?** +Understanding **self-hosted GitHub runners** is critical for enterprises managing private infrastructure, optimizing costs, and ensuring workflow security. + +## 1️⃣ What Are Runners? + +GitHub **runners** are machines that execute workflows defined in **GitHub Actions**. These runners can be: +- Hosted by **GitHub** (default). +- **Self-hosted** (your own machine or cloud instance). + + +## 2️⃣ Types of GitHub Runners + +| Runner Type | Description | Use Cases | Advantages | Disadvantages | +|-----------------|-------------|-----------|-------------|---------------| +| **GitHub-Hosted** | Managed by GitHub with predefined environments. | General CI/CD workflows with minimal setup. | No maintenance, built-in scaling. | Limited customization, restricted software access. | +| **Self-Hosted** | User-managed runner on a personal machine or cloud instance. | Workflows needing custom dependencies or private network access. | Full control, cost-effective for large workloads. | Requires maintenance, security responsibility. | + +### When to Use What? + +| Scenario | Use **GitHub-Hosted** | Use **Self-Hosted** | +|----------|---------------------|---------------------| +| Simple CI/CD workflows | ✅ | ❌ | +| Need custom software/dependencies | ❌ | ✅ | +| Need access to private networks | ❌ | ✅ | +| Cost-effective for large workloads | ❌ | ✅ | +| Security and compliance control | ❌ | ✅ | + +--- + +## 3️⃣ Challenge Breakdown + +### **📝 Theoretical Questions** +Answer the following questions: +1. What is a GitHub runner? +2. How do self-hosted runners differ from GitHub-hosted runners? +3. What security considerations should you take when using self-hosted runners? +4. How can you scale self-hosted runners? +5. Can a single self-hosted runner be used for multiple repositories? Why or why not? + +--- + +### **⚙️ Practical Challenge: Setting Up a Self-Hosted Runner** + +#### **Step 1: Create an EC2 Instance** +1. Launch an AWS EC2 instance (Ubuntu 22.04 recommended). +2. Allow inbound traffic on ports **22 (SSH)** and **80 (HTTP)**. + +#### **Step 2: Configure GitHub Self-Hosted Runner** +1. Navigate to your **GitHub repository** → **Settings** → **Actions** → **Runners**. +2. Click **"New self-hosted runner"**, select **Linux**, and follow the instructions to set it up. +3. SSH into the EC2 instance and install the runner using the provided commands. +4. Start the runner: + ```bash + ./run.sh + ``` + +#### **Step 3: Deploy the Snake Game** +1. Install **Docker** on your EC2 instance: + ```bash + sudo apt update + sudo apt install docker.io -y + ``` +2. Clone the **Snake Game repository**: + ```bash + git clone https://github.com/clear-code-projects/Snake + cd Snake + ``` +3. Build and run the application as a Docker container: + ```bash + docker build -t snake-game . + docker run -d -p 80:5000 snake-game + ``` +4. Confirm the deployment by accessing `http://` in a browser. + +5. You are required to create a GitHub Actions workflow that: + **Builds and pushes a Docker image** to Docker Hub. + **Deploys the application on Self Hosted GitHub Runners** + **Validates that the application is running correctly**. + **Sends an email notification** with the deployment status. + +#### **Step 4: Take Screenshots for Submission** +- Running EC2 instance (`aws ec2 describe-instances`). +- GitHub Actions workflow logs showing execution on the self-hosted runner. +- Webpage running the Snake Game. + +--- + +## 4️⃣ Submission Guidelines + +✅ **Proof of Completion:** +- Screenshot of **EC2 instance running**. +- Screenshot of **GitHub Actions logs showing execution on self-hosted runner**. +- Screenshot of **Snake Game running in the browser**. + +✅ **Documentation:** +- Steps taken to configure the self-hosted runner. +- Learnings and challenges faced. + +--- + +## 5️⃣ Bonus Tasks 🎯 +- Use **GitHub Actions** to automate the deployment. +- Auto-scale self-hosted runners with **multiple EC2 instances**. +- Implement **load balancing** using **Nginx**. + +### **Share Your Progress!** +Post your experience on social media with the hashtags: **#getfitwithsagar, #SRELife, #DevOpsForAll** + +--- + +## **Join Our Community** + +To make the most of this journey, connect with fellow learners: +- **Discord** – Ask questions and collaborate: [Join here](https://discord.gg/mNDm39qB8t) +- **Google Group** – Get updates and discussions: [Join here](https://groups.google.com/forum/#!forum/daily-devops-sre-challenge-series/join) +- **YouTube** – Watch solution videos and tutorials: [Subscribe here](https://www.youtube.com/@Sagar.Utekar) + +--- + +Keep learning and happy automating! 🚀 + +Kind Regards, +Sagar Utekar and Raihan From da3eacf57622aa39a337e209bb9f43d0e94123f9 Mon Sep 17 00:00:00 2001 From: Sagar Utekar Date: Fri, 28 Mar 2025 05:32:23 +0530 Subject: [PATCH 023/133] Update challenge.md --- DailyChallenges/Season2/Day7/challenge.md | 1 - 1 file changed, 1 deletion(-) diff --git a/DailyChallenges/Season2/Day7/challenge.md b/DailyChallenges/Season2/Day7/challenge.md index 915cd81..8fd19dc 100644 --- a/DailyChallenges/Season2/Day7/challenge.md +++ b/DailyChallenges/Season2/Day7/challenge.md @@ -114,7 +114,6 @@ Answer the following questions: --- ## 5️⃣ Bonus Tasks 🎯 -- Use **GitHub Actions** to automate the deployment. - Auto-scale self-hosted runners with **multiple EC2 instances**. - Implement **load balancing** using **Nginx**. From d674a059a0387130ed59b2ef62854d516d1b2afd Mon Sep 17 00:00:00 2001 From: Sagar Utekar Date: Fri, 28 Mar 2025 05:36:38 +0530 Subject: [PATCH 024/133] day7 action --- .github/workflows/manual.yml | 43 ++++++++++++++++++++++++++++++++++++ 1 file changed, 43 insertions(+) create mode 100644 .github/workflows/manual.yml diff --git a/.github/workflows/manual.yml b/.github/workflows/manual.yml new file mode 100644 index 0000000..7888b0e --- /dev/null +++ b/.github/workflows/manual.yml @@ -0,0 +1,43 @@ +name: Deploy Snake Game to EC2 + +on: + push: + branches: + - main + +jobs: + deploy: + runs-on: ubuntu-latest + + steps: + - name: Checkout repository + uses: actions/checkout@v4 + + - name: Deploy to EC2 + uses: appleboy/ssh-action@v0.1.10 + with: + host: ${{ secrets.EC2_HOST }} + username: ${{ secrets.EC2_USER }} + key: ${{ secrets.EC2_SSH_KEY }} + script: | + # Update packages + sudo apt update + + # Install Docker if not installed + if ! command -v docker &> /dev/null + then + sudo apt install docker.io -y + sudo systemctl start docker + sudo systemctl enable docker + fi + + # Remove existing Snake Game if exists + sudo rm -rf Snake + + # Clone the repository + git clone https://github.com/clear-code-projects/Snake + cd Snake + + # Build and run the Docker container + sudo docker build -t snake-game . + sudo docker run -d -p 80:5000 --name snake-container --restart always snake-game From 332ee2863dee38233f008da50b5b0fbc3dfa0ff5 Mon Sep 17 00:00:00 2001 From: Sagar Utekar Date: Fri, 28 Mar 2025 05:49:45 +0530 Subject: [PATCH 025/133] Update manual.yml --- .github/workflows/manual.yml | 2 ++ 1 file changed, 2 insertions(+) diff --git a/.github/workflows/manual.yml b/.github/workflows/manual.yml index 7888b0e..785dd95 100644 --- a/.github/workflows/manual.yml +++ b/.github/workflows/manual.yml @@ -37,7 +37,9 @@ jobs: # Clone the repository git clone https://github.com/clear-code-projects/Snake cd Snake + ls -al # Build and run the Docker container sudo docker build -t snake-game . + sudo docker images sudo docker run -d -p 80:5000 --name snake-container --restart always snake-game From 7cf725dbd589bdf674364e1e532e57a7751086ce Mon Sep 17 00:00:00 2001 From: Sagar Utekar Date: Fri, 28 Mar 2025 05:55:22 +0530 Subject: [PATCH 026/133] Update challenge.md --- DailyChallenges/Season2/Day7/challenge.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/DailyChallenges/Season2/Day7/challenge.md b/DailyChallenges/Season2/Day7/challenge.md index 8fd19dc..e1d8c9f 100644 --- a/DailyChallenges/Season2/Day7/challenge.md +++ b/DailyChallenges/Season2/Day7/challenge.md @@ -131,7 +131,7 @@ To make the most of this journey, connect with fellow learners: --- -Keep learning and happy automating! 🚀 +Keep learning and happy automating! Kind Regards, Sagar Utekar and Raihan From 2d716c8871c1e8b508124e6ec054dd85f8fcdabc Mon Sep 17 00:00:00 2001 From: Sagar Utekar Date: Fri, 28 Mar 2025 06:11:09 +0530 Subject: [PATCH 027/133] Update manual.yml --- .github/workflows/manual.yml | 37 +++++++++++++++++++----------------- 1 file changed, 20 insertions(+), 17 deletions(-) diff --git a/.github/workflows/manual.yml b/.github/workflows/manual.yml index 785dd95..afc6c5f 100644 --- a/.github/workflows/manual.yml +++ b/.github/workflows/manual.yml @@ -20,26 +20,29 @@ jobs: username: ${{ secrets.EC2_USER }} key: ${{ secrets.EC2_SSH_KEY }} script: | - # Update packages - sudo apt update + # Update system packages + sudo apt update -y - # Install Docker if not installed - if ! command -v docker &> /dev/null + # Install Python and pip if not already installed + if ! command -v python3 &> /dev/null then - sudo apt install docker.io -y - sudo systemctl start docker - sudo systemctl enable docker + sudo apt install python3 python3-pip -y fi - # Remove existing Snake Game if exists - sudo rm -rf Snake + # Remove old application if it exists + sudo rm -rf ~/SnakeGame # Clone the repository - git clone https://github.com/clear-code-projects/Snake - cd Snake - ls -al - - # Build and run the Docker container - sudo docker build -t snake-game . - sudo docker images - sudo docker run -d -p 80:5000 --name snake-container --restart always snake-game + git clone https://github.com/Sagar2366/season2-snake-game ~/SnakeGame + cd ~/SnakeGame + + # Install dependencies manually (modify as needed) + pip3 install --user flask gunicorn + + # Kill any previous instance of the game + pkill -f "gunicorn" || true + + # Run the application using Gunicorn (or Flask for testing) + nohup gunicorn --bind 0.0.0.0:5000 app:app > snake_game.log 2>&1 & + + echo "Snake Game deployed successfully" From 3f478116dcf8dd2be97d77f14e7550ff18f56f73 Mon Sep 17 00:00:00 2001 From: Sagar Utekar Date: Fri, 28 Mar 2025 06:12:23 +0530 Subject: [PATCH 028/133] Update manual.yml --- .github/workflows/manual.yml | 18 +++++++++--------- 1 file changed, 9 insertions(+), 9 deletions(-) diff --git a/.github/workflows/manual.yml b/.github/workflows/manual.yml index afc6c5f..c84f207 100644 --- a/.github/workflows/manual.yml +++ b/.github/workflows/manual.yml @@ -23,11 +23,11 @@ jobs: # Update system packages sudo apt update -y - # Install Python and pip if not already installed - if ! command -v python3 &> /dev/null - then - sudo apt install python3 python3-pip -y - fi + # Install Python and pip properly + sudo apt install -y python3 python3-pip + + # Ensure pip3 is available in PATH + export PATH=$HOME/.local/bin:$PATH # Remove old application if it exists sudo rm -rf ~/SnakeGame @@ -36,13 +36,13 @@ jobs: git clone https://github.com/Sagar2366/season2-snake-game ~/SnakeGame cd ~/SnakeGame - # Install dependencies manually (modify as needed) - pip3 install --user flask gunicorn + # Manually install dependencies (modify if needed) + python3 -m pip install --user flask gunicorn - # Kill any previous instance of the game + # Stop any previous instance pkill -f "gunicorn" || true - # Run the application using Gunicorn (or Flask for testing) + # Run the app with Gunicorn nohup gunicorn --bind 0.0.0.0:5000 app:app > snake_game.log 2>&1 & echo "Snake Game deployed successfully" From 83fe8d4eee63a4be60fc7efd84d7d15ea5ccce58 Mon Sep 17 00:00:00 2001 From: Sagar Utekar Date: Fri, 28 Mar 2025 06:15:18 +0530 Subject: [PATCH 029/133] Update manual.yml --- .github/workflows/manual.yml | 19 ++++++++++--------- 1 file changed, 10 insertions(+), 9 deletions(-) diff --git a/.github/workflows/manual.yml b/.github/workflows/manual.yml index c84f207..ecf211a 100644 --- a/.github/workflows/manual.yml +++ b/.github/workflows/manual.yml @@ -23,11 +23,8 @@ jobs: # Update system packages sudo apt update -y - # Install Python and pip properly - sudo apt install -y python3 python3-pip - - # Ensure pip3 is available in PATH - export PATH=$HOME/.local/bin:$PATH + # Install Python and required tools + sudo apt install -y python3 python3-pip python3-venv # Remove old application if it exists sudo rm -rf ~/SnakeGame @@ -36,13 +33,17 @@ jobs: git clone https://github.com/Sagar2366/season2-snake-game ~/SnakeGame cd ~/SnakeGame - # Manually install dependencies (modify if needed) - python3 -m pip install --user flask gunicorn + # Create a virtual environment + python3 -m venv venv + source venv/bin/activate + + # Install dependencies inside the virtual environment + pip install flask gunicorn # Stop any previous instance pkill -f "gunicorn" || true - # Run the app with Gunicorn - nohup gunicorn --bind 0.0.0.0:5000 app:app > snake_game.log 2>&1 & + # Run the app with Gunicorn inside the virtual environment + nohup venv/bin/gunicorn --bind 0.0.0.0:5000 app:app > snake_game.log 2>&1 & echo "Snake Game deployed successfully" From 0d4455cd17b8ac55ec4abbf4fee40ba6fd80cd97 Mon Sep 17 00:00:00 2001 From: Sagar Utekar Date: Fri, 28 Mar 2025 06:27:30 +0530 Subject: [PATCH 030/133] Update manual.yml --- .github/workflows/manual.yml | 42 +++++++++++++++++++----------------- 1 file changed, 22 insertions(+), 20 deletions(-) diff --git a/.github/workflows/manual.yml b/.github/workflows/manual.yml index ecf211a..edd6d36 100644 --- a/.github/workflows/manual.yml +++ b/.github/workflows/manual.yml @@ -22,28 +22,30 @@ jobs: script: | # Update system packages sudo apt update -y - - # Install Python and required tools - sudo apt install -y python3 python3-pip python3-venv - - # Remove old application if it exists - sudo rm -rf ~/SnakeGame - - # Clone the repository - git clone https://github.com/Sagar2366/season2-snake-game ~/SnakeGame - cd ~/SnakeGame - - # Create a virtual environment + sudo apt install -y python3 python3-pip python3-venv git + + # Navigate to the deployment directory + if [ -d "~/SnakeGame" ]; then + cd ~/SnakeGame + git reset --hard + git pull origin main + else + git clone https://github.com/Sagar2366/season2-snake-game ~/SnakeGame + cd ~/SnakeGame + fi + + # Set up a virtual environment python3 -m venv venv source venv/bin/activate + pip install --upgrade pip + pip install -r requirements.txt - # Install dependencies inside the virtual environment - pip install flask gunicorn - - # Stop any previous instance - pkill -f "gunicorn" || true + # Stop previous instance cleanly + if pgrep -f "gunicorn"; then + pkill -f "gunicorn" + fi - # Run the app with Gunicorn inside the virtual environment - nohup venv/bin/gunicorn --bind 0.0.0.0:5000 app:app > snake_game.log 2>&1 & + # Start the application using Gunicorn + nohup venv/bin/gunicorn --workers 3 --bind 0.0.0.0:5000 app:app > snake_game.log 2>&1 & - echo "Snake Game deployed successfully" + echo "✅ Snake Game deployed successfully!" From 0f0a6f1f4a540abbba820414a4db604c8907d4e3 Mon Sep 17 00:00:00 2001 From: Sagar Utekar Date: Fri, 28 Mar 2025 06:32:25 +0530 Subject: [PATCH 031/133] Update manual.yml --- .github/workflows/manual.yml | 12 ++++-------- 1 file changed, 4 insertions(+), 8 deletions(-) diff --git a/.github/workflows/manual.yml b/.github/workflows/manual.yml index edd6d36..eba61e7 100644 --- a/.github/workflows/manual.yml +++ b/.github/workflows/manual.yml @@ -26,14 +26,10 @@ jobs: # Navigate to the deployment directory if [ -d "~/SnakeGame" ]; then - cd ~/SnakeGame - git reset --hard - git pull origin main - else - git clone https://github.com/Sagar2366/season2-snake-game ~/SnakeGame - cd ~/SnakeGame - fi - + git clone https://github.com/Sagar2366/season2-snake-game + cd season2-snake-game + fi + ls -al season2-snake-game # Set up a virtual environment python3 -m venv venv source venv/bin/activate From 0b9247f67a27708700a2513929b92d91715bc2d4 Mon Sep 17 00:00:00 2001 From: Sagar Utekar Date: Fri, 28 Mar 2025 06:34:15 +0530 Subject: [PATCH 032/133] Update manual.yml --- .github/workflows/manual.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/manual.yml b/.github/workflows/manual.yml index eba61e7..de181af 100644 --- a/.github/workflows/manual.yml +++ b/.github/workflows/manual.yml @@ -26,7 +26,7 @@ jobs: # Navigate to the deployment directory if [ -d "~/SnakeGame" ]; then - git clone https://github.com/Sagar2366/season2-snake-game + git clone https://github.com/Sagar2366/season2-snake_game.git cd season2-snake-game fi ls -al season2-snake-game From 074b68d62ff2218aa2ffc87513336293074e398a Mon Sep 17 00:00:00 2001 From: Sagar Utekar Date: Fri, 28 Mar 2025 06:35:12 +0530 Subject: [PATCH 033/133] Update manual.yml --- .github/workflows/manual.yml | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/.github/workflows/manual.yml b/.github/workflows/manual.yml index de181af..685512f 100644 --- a/.github/workflows/manual.yml +++ b/.github/workflows/manual.yml @@ -25,11 +25,11 @@ jobs: sudo apt install -y python3 python3-pip python3-venv git # Navigate to the deployment directory - if [ -d "~/SnakeGame" ]; then + if [ -d "~/season2-snake_game" ]; then git clone https://github.com/Sagar2366/season2-snake_game.git - cd season2-snake-game + cd season2-snake_game fi - ls -al season2-snake-game + ls -al ~/season2-snake_game # Set up a virtual environment python3 -m venv venv source venv/bin/activate From dc162be04203790ea51599bcfc7a4b69aedb39f8 Mon Sep 17 00:00:00 2001 From: Sagar Utekar Date: Fri, 28 Mar 2025 06:36:10 +0530 Subject: [PATCH 034/133] Update manual.yml --- .github/workflows/manual.yml | 10 ++++------ 1 file changed, 4 insertions(+), 6 deletions(-) diff --git a/.github/workflows/manual.yml b/.github/workflows/manual.yml index 685512f..1d194e0 100644 --- a/.github/workflows/manual.yml +++ b/.github/workflows/manual.yml @@ -24,12 +24,10 @@ jobs: sudo apt update -y sudo apt install -y python3 python3-pip python3-venv git - # Navigate to the deployment directory - if [ -d "~/season2-snake_game" ]; then - git clone https://github.com/Sagar2366/season2-snake_game.git - cd season2-snake_game - fi - ls -al ~/season2-snake_game + git clone https://github.com/Sagar2366/season2-snake_game.git + cd season2-snake_game + ls -al + # Set up a virtual environment python3 -m venv venv source venv/bin/activate From f6bf7974ffd172e8b62de95055e352110948c03e Mon Sep 17 00:00:00 2001 From: Sagar Utekar Date: Fri, 28 Mar 2025 06:38:06 +0530 Subject: [PATCH 035/133] Update manual.yml --- .github/workflows/manual.yml | 9 +-------- 1 file changed, 1 insertion(+), 8 deletions(-) diff --git a/.github/workflows/manual.yml b/.github/workflows/manual.yml index 1d194e0..e94364b 100644 --- a/.github/workflows/manual.yml +++ b/.github/workflows/manual.yml @@ -33,13 +33,6 @@ jobs: source venv/bin/activate pip install --upgrade pip pip install -r requirements.txt - - # Stop previous instance cleanly - if pgrep -f "gunicorn"; then - pkill -f "gunicorn" - fi - - # Start the application using Gunicorn - nohup venv/bin/gunicorn --workers 3 --bind 0.0.0.0:5000 app:app > snake_game.log 2>&1 & + python app.py& echo "✅ Snake Game deployed successfully!" From ef19cab46dc29daf80519cf36c20c25efac2031a Mon Sep 17 00:00:00 2001 From: Sagar Utekar Date: Fri, 28 Mar 2025 06:39:50 +0530 Subject: [PATCH 036/133] Update manual.yml --- .github/workflows/manual.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/manual.yml b/.github/workflows/manual.yml index e94364b..e00f4dc 100644 --- a/.github/workflows/manual.yml +++ b/.github/workflows/manual.yml @@ -33,6 +33,6 @@ jobs: source venv/bin/activate pip install --upgrade pip pip install -r requirements.txt - python app.py& + sudo python3 app.py & echo "✅ Snake Game deployed successfully!" From 135861618f7752f05f976f9a3c45ba7c817d432d Mon Sep 17 00:00:00 2001 From: Sagar Utekar Date: Fri, 28 Mar 2025 06:43:36 +0530 Subject: [PATCH 037/133] Update manual.yml --- .github/workflows/manual.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/manual.yml b/.github/workflows/manual.yml index e00f4dc..d5e4d32 100644 --- a/.github/workflows/manual.yml +++ b/.github/workflows/manual.yml @@ -33,6 +33,6 @@ jobs: source venv/bin/activate pip install --upgrade pip pip install -r requirements.txt - sudo python3 app.py & + sudo nohup python3 app.py > snake_game.log 2>&1 & echo "✅ Snake Game deployed successfully!" From 7db85479ffc36e6c58cd615f6fefe2b1a3b3d89f Mon Sep 17 00:00:00 2001 From: Sagar Utekar Date: Fri, 28 Mar 2025 06:46:05 +0530 Subject: [PATCH 038/133] Update manual.yml --- .github/workflows/manual.yml | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/.github/workflows/manual.yml b/.github/workflows/manual.yml index d5e4d32..37b062d 100644 --- a/.github/workflows/manual.yml +++ b/.github/workflows/manual.yml @@ -24,6 +24,13 @@ jobs: sudo apt update -y sudo apt install -y python3 python3-pip python3-venv git + # Stop any running instance of app.py + pkill -f "python3 app.py" || true + + # Remove old directory if it exists + sudo rm -rf season2-snake_game + + # Clone fresh git clone https://github.com/Sagar2366/season2-snake_game.git cd season2-snake_game ls -al From 3271740bdc363202b2c7a5cd9a76e9ba53fb7ca9 Mon Sep 17 00:00:00 2001 From: Sagar Utekar Date: Fri, 28 Mar 2025 06:46:58 +0530 Subject: [PATCH 039/133] Update manual.yml --- .github/workflows/manual.yml | 3 --- 1 file changed, 3 deletions(-) diff --git a/.github/workflows/manual.yml b/.github/workflows/manual.yml index 37b062d..c4399e0 100644 --- a/.github/workflows/manual.yml +++ b/.github/workflows/manual.yml @@ -24,9 +24,6 @@ jobs: sudo apt update -y sudo apt install -y python3 python3-pip python3-venv git - # Stop any running instance of app.py - pkill -f "python3 app.py" || true - # Remove old directory if it exists sudo rm -rf season2-snake_game From f4e454087aa8651de43a66893c8feef5d8807432 Mon Sep 17 00:00:00 2001 From: Sagar Utekar Date: Fri, 28 Mar 2025 06:52:36 +0530 Subject: [PATCH 040/133] Create day7-challenge.yaml --- .github/workflows/day7-challenge.yaml | 42 +++++++++++++++++++++++++++ 1 file changed, 42 insertions(+) create mode 100644 .github/workflows/day7-challenge.yaml diff --git a/.github/workflows/day7-challenge.yaml b/.github/workflows/day7-challenge.yaml new file mode 100644 index 0000000..c4399e0 --- /dev/null +++ b/.github/workflows/day7-challenge.yaml @@ -0,0 +1,42 @@ +name: Deploy Snake Game to EC2 + +on: + push: + branches: + - main + +jobs: + deploy: + runs-on: ubuntu-latest + + steps: + - name: Checkout repository + uses: actions/checkout@v4 + + - name: Deploy to EC2 + uses: appleboy/ssh-action@v0.1.10 + with: + host: ${{ secrets.EC2_HOST }} + username: ${{ secrets.EC2_USER }} + key: ${{ secrets.EC2_SSH_KEY }} + script: | + # Update system packages + sudo apt update -y + sudo apt install -y python3 python3-pip python3-venv git + + # Remove old directory if it exists + sudo rm -rf season2-snake_game + + # Clone fresh + git clone https://github.com/Sagar2366/season2-snake_game.git + cd season2-snake_game + ls -al + + # Set up a virtual environment + python3 -m venv venv + source venv/bin/activate + pip install --upgrade pip + pip install -r requirements.txt + sudo nohup python3 app.py > snake_game.log 2>&1 & + + echo "✅ Snake Game deployed successfully!" From 9337d4e86baa15b37f795bd12d1d8b90ac54446f Mon Sep 17 00:00:00 2001 From: Sagar Utekar Date: Fri, 28 Mar 2025 06:52:46 +0530 Subject: [PATCH 041/133] Delete .github/workflows/manual.yml --- .github/workflows/manual.yml | 42 ------------------------------------ 1 file changed, 42 deletions(-) delete mode 100644 .github/workflows/manual.yml diff --git a/.github/workflows/manual.yml b/.github/workflows/manual.yml deleted file mode 100644 index c4399e0..0000000 --- a/.github/workflows/manual.yml +++ /dev/null @@ -1,42 +0,0 @@ -name: Deploy Snake Game to EC2 - -on: - push: - branches: - - main - -jobs: - deploy: - runs-on: ubuntu-latest - - steps: - - name: Checkout repository - uses: actions/checkout@v4 - - - name: Deploy to EC2 - uses: appleboy/ssh-action@v0.1.10 - with: - host: ${{ secrets.EC2_HOST }} - username: ${{ secrets.EC2_USER }} - key: ${{ secrets.EC2_SSH_KEY }} - script: | - # Update system packages - sudo apt update -y - sudo apt install -y python3 python3-pip python3-venv git - - # Remove old directory if it exists - sudo rm -rf season2-snake_game - - # Clone fresh - git clone https://github.com/Sagar2366/season2-snake_game.git - cd season2-snake_game - ls -al - - # Set up a virtual environment - python3 -m venv venv - source venv/bin/activate - pip install --upgrade pip - pip install -r requirements.txt - sudo nohup python3 app.py > snake_game.log 2>&1 & - - echo "✅ Snake Game deployed successfully!" From 6f259da3769a8e637e8b91adbc515c558b381fdf Mon Sep 17 00:00:00 2001 From: Sagar Utekar Date: Fri, 28 Mar 2025 06:53:46 +0530 Subject: [PATCH 042/133] Delete DailyChallenges/Season2/DAY-07 directory --- DailyChallenges/Season2/DAY-07/Challenge.md | 123 -------------------- 1 file changed, 123 deletions(-) delete mode 100644 DailyChallenges/Season2/DAY-07/Challenge.md diff --git a/DailyChallenges/Season2/DAY-07/Challenge.md b/DailyChallenges/Season2/DAY-07/Challenge.md deleted file mode 100644 index 6f58bd8..0000000 --- a/DailyChallenges/Season2/DAY-07/Challenge.md +++ /dev/null @@ -1,123 +0,0 @@ -# Day 07: GitHub Self-Hosted Runners - -## Introduction -Welcome to Day 07 of the DevOps SRE Daily Challenge! 🎉 - -Today, we focus on **GitHub Self-Hosted Runners**, a powerful way to customize GitHub Actions workflows by running jobs on your own infrastructure. - -By the end of this challenge, you will: -- Understand what GitHub runners are and their types. -- Compare **hosted** vs. **self-hosted** runners with advantages and disadvantages. -- Set up a **self-hosted runner** on an **AWS EC2 instance**. -- **Dockerize and deploy** the Snake game from [this repo](https://github.com/clear-code-projects/Snake) using the self-hosted runner. - ---- - -## 1️⃣ What Are Runners? - -GitHub **runners** are machines that execute workflows defined in **GitHub Actions**. These runners can be: -- Hosted by **GitHub** (default). -- **Self-hosted** (your own machine or cloud instance). - - -## 2️⃣ Types of GitHub Runners - -| Runner Type | Description | Use Cases | Advantages | Disadvantages | -|-----------------|-------------|-----------|-------------|---------------| -| **GitHub-Hosted** | Managed by GitHub with predefined environments. | General CI/CD workflows with minimal setup. | No maintenance, built-in scaling. | Limited customization, restricted software access. | -| **Self-Hosted** | User-managed runner on a personal machine or cloud instance. | Workflows needing custom dependencies or private network access. | Full control, cost-effective for large workloads. | Requires maintenance, security responsibility. | - -### When to Use What? - -| Scenario | Use **GitHub-Hosted** | Use **Self-Hosted** | -|----------|---------------------|---------------------| -| Simple CI/CD workflows | ✅ | ❌ | -| Need custom software/dependencies | ❌ | ✅ | -| Need access to private networks | ❌ | ✅ | -| Cost-effective for large workloads | ❌ | ✅ | -| Security and compliance control | ❌ | ✅ | - ---- - -## 3️⃣ Challenge Breakdown - -### **📝 Theoretical Questions** -Answer the following questions: -1. What is a GitHub runner? -2. How do self-hosted runners differ from GitHub-hosted runners? -3. What security considerations should you take when using self-hosted runners? -4. How can you scale self-hosted runners? -5. Can a single self-hosted runner be used for multiple repositories? Why or why not? - ---- - -### **⚙️ Practical Challenge: Setting Up a Self-Hosted Runner** - -#### **Step 1: Create an EC2 Instance** -1. Launch an AWS EC2 instance (Ubuntu 22.04 recommended). -2. Allow inbound traffic on ports **22 (SSH)** and **80 (HTTP)**. - -#### **Step 2: Configure GitHub Self-Hosted Runner** -1. Navigate to your **GitHub repository** → **Settings** → **Actions** → **Runners**. -2. Click **"New self-hosted runner"**, select **Linux**, and follow the instructions to set it up. -3. SSH into the EC2 instance and install the runner using the provided commands. -4. Start the runner: - ```bash - ./run.sh - ``` - -#### **Step 3: Deploy the Snake Game** -1. Install **Docker** on your EC2 instance: - ```bash - sudo apt update - sudo apt install docker.io -y - ``` -2. Clone the **Snake Game repository**: - ```bash - git clone https://github.com/clear-code-projects/Snake - cd Snake - ``` -3. Build and run the application as a Docker container: - ```bash - docker build -t snake-game . - docker run -d -p 80:5000 snake-game - ``` -4. Confirm the deployment by accessing `http://` in a browser. - -#### **Step 4: Take Screenshots for Submission** -- Running EC2 instance (`aws ec2 describe-instances`). -- GitHub Actions workflow logs showing execution on the self-hosted runner. -- Webpage running the Snake Game. - ---- - -## 4️⃣ Submission Guidelines - -✅ **Proof of Completion:** -- Screenshot of **EC2 instance running**. -- Screenshot of **GitHub Actions logs showing execution on self-hosted runner**. -- Screenshot of **Snake Game running in the browser**. - -✅ **Documentation:** -- Steps taken to configure the self-hosted runner. -- Learnings and challenges faced. - ---- - -## 5️⃣ Bonus Tasks 🎯 -- Use **GitHub Actions** to automate the deployment. -- Auto-scale self-hosted runners with **multiple EC2 instances**. -- Implement **load balancing** using **Nginx**. - -### **Share Your Progress!** -Post your experience on social media with the hashtags: **#getfitwithsagar, #SRELife, #DevOpsForAll** - ---- - -### **Why This Matters?** -Understanding **self-hosted GitHub runners** is critical for enterprises managing private infrastructure, optimizing costs, and ensuring workflow security. - -Keep learning and happy automating! 🚀 - -Kind Regards, -Sagar Utekar and Raihan From de18be5e0b0e453e547c2e62e7d38f516e1fb580 Mon Sep 17 00:00:00 2001 From: Sagar Utekar Date: Fri, 28 Mar 2025 07:29:52 +0530 Subject: [PATCH 043/133] Update challenge.md --- DailyChallenges/Season2/Day7/challenge.md | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/DailyChallenges/Season2/Day7/challenge.md b/DailyChallenges/Season2/Day7/challenge.md index e1d8c9f..649cd5b 100644 --- a/DailyChallenges/Season2/Day7/challenge.md +++ b/DailyChallenges/Season2/Day7/challenge.md @@ -70,6 +70,11 @@ Answer the following questions: ``` #### **Step 3: Deploy the Snake Game** + + +https://github.com/user-attachments/assets/3c0e0fc7-a5ef-4285-9b3a-4cdb57915f0e + + 1. Install **Docker** on your EC2 instance: ```bash sudo apt update From dc5fa6b6250d58de78622d8a87c2b6546304900a Mon Sep 17 00:00:00 2001 From: Sagar Utekar Date: Fri, 28 Mar 2025 07:45:56 +0530 Subject: [PATCH 044/133] Update challenge.md --- DailyChallenges/Season2/Day7/challenge.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/DailyChallenges/Season2/Day7/challenge.md b/DailyChallenges/Season2/Day7/challenge.md index 649cd5b..d7c1b65 100644 --- a/DailyChallenges/Season2/Day7/challenge.md +++ b/DailyChallenges/Season2/Day7/challenge.md @@ -9,7 +9,7 @@ By the end of this challenge, you will: - Understand what GitHub runners are and their types. - Compare **hosted** vs. **self-hosted** runners with advantages and disadvantages. - Set up a **self-hosted runner** on an **AWS EC2 instance**. -- **Dockerize and deploy** the Snake game from [this repo](https://github.com/clear-code-projects/Snake) using the self-hosted runner. +- **Dockerize and deploy** the Snake game from [this repo](https://github.com/Sagar2366/season2-snake_game.git) using the self-hosted runner. --- From 4969e907d0ba6c1b66aa436d4a1ad360150f8db0 Mon Sep 17 00:00:00 2001 From: Sagar Utekar Date: Mon, 7 Apr 2025 05:57:59 +0530 Subject: [PATCH 045/133] Create readme.md --- DailyChallenges/Season2/Day8/readme.md | 203 +++++++++++++++++++++++++ 1 file changed, 203 insertions(+) create mode 100644 DailyChallenges/Season2/Day8/readme.md diff --git a/DailyChallenges/Season2/Day8/readme.md b/DailyChallenges/Season2/Day8/readme.md new file mode 100644 index 0000000..a801568 --- /dev/null +++ b/DailyChallenges/Season2/Day8/readme.md @@ -0,0 +1,203 @@ +# Daily DevOps + SRE Challenge Series – Season 2 +## Day 8: Kickstart Your Journey with Linux & Virtualization + +## Introduction +Welcome to Day 8 of the Daily DevOps SRE Challenge Series - Season 2! 🎉 + +Today, we’re starting strong with **Linux and Virtualization**—the bedrock of DevOps and SRE workflows. By the end of this challenge, you will: +- Understand what Unix and Linux are and why they matter. +- Explore **virtualization** and its role in modern infrastructure. +- Compare popular **Linux distributions** and their use cases. +- Set up **AWS EC2 instances** with different Linux distros and install **RHEL** on VirtualBox locally. + +--- + +### **Why This Matters?** +Linux powers the cloud, containers, and automation tools that DevOps and SRE engineers rely on daily. Virtualization is the foundation of scalable infrastructure—think AWS, testing environments, and cost savings. Together, they’re your launchpad to mastering modern IT, unlocking roles like Cloud Engineer, SRE, and System Admin! + +--- + +## 1️⃣ What Are Unix and Linux? + +- **Unix:** A powerful, multi-user, multitasking OS born in the 1970s at AT&T Bell Labs. It’s the ancestor of many modern systems, built for stability and modularity. +- **Linux:** A Unix-inspired, open-source OS—free, customizable, and community-driven. It runs everywhere: servers, clouds, even your phone! + +**Key Difference:** Unix is often proprietary (e.g., Solaris), while Linux is open-source and adaptable (e.g., Ubuntu, RHEL). + +**Fun Fact:** The `ls` command in Linux? It’s a Unix legacy, reborn for today’s systems! + +--- + +## 2️⃣ Virtualization Basics + +**What is Virtualization?** +Virtualization lets you run multiple **virtual machines (VMs)** on one physical host, splitting resources like CPU, RAM, and storage. It’s like having a dozen computers in one box! + +**How It Works:** +A **hypervisor** manages VMs, abstracting hardware to create isolated environments. + +**Types of Hypervisors:** +| Type | Description | Examples | +|---------|---------------------|---------------------| +| Type 1 | Runs on bare metal | VMware ESXi, Hyper-V | +| Type 2 | Runs on a host OS | VirtualBox, VMware Workstation | + +**Benefits:** +- **Cost Savings:** One server hosts multiple VMs. +- **Flexibility:** Snapshot and restore VMs for backups. +- **Dev/Test/QA:** Spin up isolated environments fast. +- **Cloud Power:** AWS, GCP, and Azure rely on virtualization. + +**Quick Quiz:** Which hypervisor type is better for production servers? *(Hint: Think bare metal!)* + +--- + +## 3️⃣ Types of Linux Distributions + +Linux comes in flavors called **distributions**, each tailored to specific needs. Here’s a rundown: + +| Distribution | Description | Use Cases | Advantages | Disadvantages | +|--------------|------------------------------|---------------------|------------------------|------------------------| +| Ubuntu | User-friendly Linux distro | Beginners, Developers | Easy to use, LTS support | Slightly bloated for servers | +| Debian | Stable, free software-focused| Servers, Advanced Users | Rock-solid, vast repos | Slower release cycle | +| Fedora | Cutting-edge tech showcase | Developers | Latest features, modular | Frequent updates | +| CentOS | Enterprise-grade stability | Servers | RHEL-like, free, SELinux | End-of-life for CentOS 7 | +| Kali Linux | Security and pentesting toolkit | Security Pros | 600+ tools preloaded | Not for general use | +| RHEL | Supported enterprise solution | Enterprises | Secure, scalable, support | Subscription cost | + +### **When to Use What?** +| Scenario | Ubuntu | RHEL | Kali | +|---------------------|--------|------|------| +| Learning Linux | ✅ | ❌ | ❌ | +| Enterprise production | ❌ | ✅ | ❌ | +| Ethical hacking | ❌ | ❌ | ✅ | + +**Pro Tip:** Match your distro to your goal—Ubuntu for learning, RHEL for enterprise skills! + +--- + +## 4️⃣ Why Learn Linux? +Linux is *the* skill for DevOps, SRE, and Cloud Engineering. Here’s why it’s non-negotiable for DevOps and SRE engineers: + +### Why Linux Matters for DevOps Engineers +- **Automation Foundation:** DevOps is all about automating workflows—Linux’s command-line tools (e.g., `bash`, `awk`, `sed`) and scripting capabilities make it ideal for building CI/CD pipelines (Jenkins, GitLab CI). +- **Containerization:** Docker and Kubernetes run on Linux kernels. Understanding Linux (e.g., namespaces, cgroups) is key to managing containers at scale. +- **Cloud Dominance:** Most cloud providers (AWS, GCP, Azure) use Linux VMs. Deploying apps or infrastructure-as-code (Terraform, Ansible) requires Linux fluency. +- **Efficiency:** Linux’s lightweight nature and customization let DevOps engineers optimize systems for performance and cost—critical for microservices and serverless architectures. + +### Why Linux Matters for SRE Engineers +- **Reliability at Scale:** SREs ensure systems are up 99.99% of the time. Linux’s stability (e.g., Debian, RHEL) and tools like `systemd` and `SELinux` help enforce uptime and security SLAs. +- **Monitoring & Debugging:** Tools like Prometheus, Grafana, and `strace` run natively on Linux, letting SREs trace issues (e.g., memory leaks, network latency) in production. +- **Incident Response:** When systems fail, SREs use Linux commands (`top`, `netstat`, `journalctl`) to diagnose and fix problems fast—often under pressure. +- **Infrastructure Mastery:** SREs manage fleets of servers or clusters (e.g., Kubernetes). Linux’s kernel-level control is essential for tuning performance and handling failures gracefully. + +### Broader Impact +- **Cloud:** AWS, Azure, and GCP run on Linux VMs. +- **Containers:** Kubernetes and Docker are Linux-native. +- **Automation:** CI/CD tools thrive on Linux. +- **Monitoring:** Prometheus and Grafana are Linux-friendly. + +### Roles It Unlocks: +- Site Reliability Engineer (SRE) +- Cloud/DevOps Engineer +- System Administrator + +**Challenge Question:** Can you name a DevOps tool that relies on Linux? *(Hint: Think containers!)* + +--- + +## 5️⃣ Challenge Breakdown + +### **📝 Theoretical Questions** +Answer these to solidify your understanding: +1. What is Unix, and how does it relate to Linux? +2. How does Linux’s open-source nature benefit DevOps workflows? +3. What’s the role of the Linux kernel in virtualization? +4. What’s the difference between Type 1 and Type 2 hypervisors? +5. Why might an SRE prefer Linux for monitoring production systems? + +--- + +### **⚙️ Practical Challenge: Mastering Linux & Virtualization** + +#### **Step 1: Launch EC2 Instances with Linux Distros** +**Goal:** Deploy three Linux distros on AWS EC2 to explore their differences. +1. Log into AWS → EC2 → "Launch Instance." +2. Deploy these AMIs (use `t2.micro` for free tier): + - **Ubuntu 22.04 LTS**: Search "Ubuntu 22.04" in AMI catalog. + - **Amazon Linux 2**: Default EC2 option. + - **RHEL 9**: Search "Red Hat Enterprise Linux 9." +3. Set up a key pair (e.g., `mykey.pem`). +4. Configure security group: Allow port 22 (SSH) from your IP. +5. SSH into each: + ```bash + ssh -i mykey.pem ubuntu@ # Ubuntu + ssh -i mykey.pem ec2-user@ # Amazon Linux + ssh -i mykey.pem ec2-user@ # RHEL + +6. Run this on each: + ```bash + cat /etc/os-release + + +**Troubleshooting Tip:** SSH failing? Double-check your security group and key permissions (`chmod 400 mykey.pem`). + +--- + +### **Step 2: Install RHEL on VirtualBox Locally** +**Goal:** Set up a local RHEL VM to grasp virtualization hands-on. + +**Downloads:** +- VirtualBox: [virtualbox.org](https://www.virtualbox.org) +- RHEL ISO: [redhat.com](https://developers.redhat.com/products/rhel/download) (free developer account). + +1. Open VirtualBox → "New" → Name: "RHEL9" → Type: Linux → Version: Red Hat (64-bit). +2. Allocate: + - RAM: 2 GB+ + - Disk: 20 GB+ (dynamic) +3. Mount the RHEL ISO in Settings → Storage. +4. Start the VM and follow the installer (default settings are fine). +5. Post-install, log in and run: + ```bash + sudo dnf update -y + cat /etc/os-release + + +**Pro Tip:** VM won’t boot? Enable VT-x/AMD-V in your BIOS settings. + +--- + +## 6️⃣ Submission Guidelines +✅ **Proof of Completion:** +- Screenshot of AWS EC2 dashboard with all three instances running. +- Screenshot of RHEL VM terminal after login. +- Output of `cat /etc/os-release` from each system (EC2 + VM). + +✅ **Documentation:** +- Steps you took to launch EC2 instances and set up the VM. +- Any challenges faced (e.g., SSH issues, VM crashes) and how you solved them. + +**Example Submission:** +- Ubuntu output: `NAME="Ubuntu" VERSION="22.04.1 LTS (Jammy Jellyfish)"` +- Screenshot showing all instances in "Running" state. + +--- + +## 7️⃣ Bonus Tasks 🎯 +- **Explore:** Install `htop` on your Ubuntu EC2 (`sudo apt install htop -y`) and screenshot the process list. +- **Compare:** Run `uname -r` on all systems to compare kernel versions. What differences do you spot? + +### **Share Your Progress!** +Post your experience on social media with the hashtags: **#getfitwithsagar, #SRELife, #DevOpsForAll** + +--- + +## **Join Our Community** +Connect with fellow learners: +- **Discord:** [Join here](https://discord.gg/mNDm39qB8t) +- **Google Group:** [Join here](https://groups.google.com/forum/#!forum/daily-devops-sre-challenge-series/join) +- **YouTube:** [Subscribe here](https://www.youtube.com/@Sagar.Utekar) + +--- + +Keep learning and happy exploring! From 13110740cf3963e819cbc51e68af74a52b87652d Mon Sep 17 00:00:00 2001 From: Sagar Utekar Date: Mon, 7 Apr 2025 06:09:35 +0530 Subject: [PATCH 046/133] Update readme.md --- DailyChallenges/Season2/Day8/readme.md | 3 +++ 1 file changed, 3 insertions(+) diff --git a/DailyChallenges/Season2/Day8/readme.md b/DailyChallenges/Season2/Day8/readme.md index a801568..001b527 100644 --- a/DailyChallenges/Season2/Day8/readme.md +++ b/DailyChallenges/Season2/Day8/readme.md @@ -18,6 +18,7 @@ Linux powers the cloud, containers, and automation tools that DevOps and SRE eng --- ## 1️⃣ What Are Unix and Linux? +![0_Qqqd7UsfFDPL7WXh](https://github.com/user-attachments/assets/126f72c0-6d92-440a-a42d-c758394f7046) - **Unix:** A powerful, multi-user, multitasking OS born in the 1970s at AT&T Bell Labs. It’s the ancestor of many modern systems, built for stability and modularity. - **Linux:** A Unix-inspired, open-source OS—free, customizable, and community-driven. It runs everywhere: servers, clouds, even your phone! @@ -37,6 +38,8 @@ Virtualization lets you run multiple **virtual machines (VMs)** on one physical A **hypervisor** manages VMs, abstracting hardware to create isolated environments. **Types of Hypervisors:** +![611e5d7e-953a-41db-9171-c7ba1cd6df9c](https://github.com/user-attachments/assets/817b944b-848a-426d-a6a3-1c030ff3adb2) + | Type | Description | Examples | |---------|---------------------|---------------------| | Type 1 | Runs on bare metal | VMware ESXi, Hyper-V | From ad8bb70d7a3131cc5fa8d2fdccf8c98b8b80eac6 Mon Sep 17 00:00:00 2001 From: Sagar Utekar Date: Mon, 7 Apr 2025 06:16:41 +0530 Subject: [PATCH 047/133] Update readme.md --- DailyChallenges/Season2/Day8/readme.md | 83 ++++++++++++++++++++++++++ 1 file changed, 83 insertions(+) diff --git a/DailyChallenges/Season2/Day8/readme.md b/DailyChallenges/Season2/Day8/readme.md index 001b527..c61ee05 100644 --- a/DailyChallenges/Season2/Day8/readme.md +++ b/DailyChallenges/Season2/Day8/readme.md @@ -138,10 +138,91 @@ Answer these to solidify your understanding: ssh -i mykey.pem ec2-user@ # Amazon Linux ssh -i mykey.pem ec2-user@ # RHEL +### Connect via PuTTY (Windows) + +**Step-by-step:** + +1. **Install [PuTTY and PuTTYgen](https://www.chiark.greenend.org.uk/~sgtatham/putty/latest.html)** + Download and install both tools on your Windows machine. + +2. **Convert `.pem` to `.ppk` using PuTTYgen** + - Open **PuTTYgen** + - Click **"Load"** and select your `mykey.pem` file + - Click **"Save private key"** + - Save it as `mykey.ppk` (you can skip the passphrase if not required) + +3. **Open PuTTY and configure connection** + - In **Host Name**, enter: + ``` + ubuntu@ # For Ubuntu + ec2-user@ # For Amazon Linux / RHEL + ``` + - **Port:** `22` + - In the left panel, go to: + `Connection` → `SSH` → `Auth` + - Click **Browse**, and attach your `mykey.ppk` file + +4. **Connect** + - Click **"Open"** + - Accept any security alert + - You should now be logged into your EC2 instance 🎉 + +**You are now connected to your VM via PuTTY!** + + 6. Run this on each: ```bash cat /etc/os-release +--- + +### Repeat the Same on Azure & GCP + +--- + +#### **Azure – Launch a Linux VM** + +1. Go to **[Azure Portal](https://portal.azure.com)** → `Virtual Machines` → **Create VM** +2. Choose your desired Linux distribution: + - Ubuntu + - RHEL + - CentOS +3. For free-tier eligible setup: + - **Size:** `Standard_B1s` +4. **Authentication type:** + - Select **SSH public key** + - Upload or generate a new key pair +5. Under **Networking**, open **Port 22 (SSH)** +6. Click **Review + Create**, then **Create** + +**Connect Options:** +- Use Azure Cloud Shell's SSH option +- Or use **PuTTY / terminal** with your `.pem` or converted `.ppk` key (same as AWS SSH steps) + +--- + +#### **GCP – Launch a Linux VM (Compute Engine)** + +1. Go to **[GCP Console](https://console.cloud.google.com)** → `Compute Engine` → `VM Instances` → **Create Instance** +2. Select your preferred OS: + - Ubuntu + - RHEL + - Debian +3. Choose machine type: + - **e2-micro** (Free tier eligible) +4. Scroll to **Security** section: + - Add your **SSH public key** +5. Under **Firewall**, check: + - ✅ Allow HTTP traffic + - ✅ Allow SSH traffic +6. Click **Create** + +**Connect Options:** +- Use GCP’s built-in browser-based SSH +- Or use **external SSH client / PuTTY**: + ```bash + ssh -i your-key.pem username@ + **Troubleshooting Tip:** SSH failing? Double-check your security group and key permissions (`chmod 400 mykey.pem`). @@ -204,3 +285,5 @@ Connect with fellow learners: --- Keep learning and happy exploring! +Sagar Utekar + From e82de1d14fc71b14d77853b3b52ccd4c11aa3e59 Mon Sep 17 00:00:00 2001 From: Sagar Utekar Date: Tue, 8 Apr 2025 00:28:19 +0530 Subject: [PATCH 048/133] Create readme.md --- DailyChallenges/Season2/Day9/readme.md | 280 +++++++++++++++++++++++++ 1 file changed, 280 insertions(+) create mode 100644 DailyChallenges/Season2/Day9/readme.md diff --git a/DailyChallenges/Season2/Day9/readme.md b/DailyChallenges/Season2/Day9/readme.md new file mode 100644 index 0000000..6632eb1 --- /dev/null +++ b/DailyChallenges/Season2/Day9/readme.md @@ -0,0 +1,280 @@ +# 🚀 Daily DevOps + SRE Challenge Series – Season 2 +## 🗓️ Day 2: Mastering the Linux Shell + +## Introduction +Welcome to Day 2 of the Daily DevOps SRE Challenge Series - Season 2! 🎉 + +Today, we’re diving into **the Linux Shell**, blending *Chapter 3: Using the Shell* (*Linux Bible*) and *Chapter 2: Using Essential Tools* (RHCSA prep). You’ll explore OS architecture, master Bash commands, edit with Vim, and harness help tools. By the end, you’ll: +- Grasp Linux’s layered structure and the shell’s critical role. +- Master commands, redirection, piping, history, variables, aliases, and Vim. +- Apply these on your Day 1 EC2 instances and RHEL VM. + +--- + +### **Why This Matters?** +The shell is the backbone of Linux system administration, DevOps, and SRE. It’s not just a tool—it’s a philosophy of control and automation. Here’s why: +- **Universal Interface:** Every Linux system—cloud VMs, containers, or bare metal—has a shell. It’s your consistent entry point, unlike GUIs that vary or may not exist. +- **Automation Powerhouse:** Shell scripts drive CI/CD pipelines (e.g., `bash deploy.sh` in Jenkins), cron jobs (`0 0 * * * backup.sh`), and container startups (`CMD ["/bin/bash", "start.sh"]`). +- **SRE Debugging:** During outages, `tail -f log | grep error` or `ps aux | grep nginx` isolates issues in seconds, bypassing slow dashboards. +- **RHCSA Requirement:** The exam tests shell fluency—redirection (`2>&1`), syntax (`ls -l`), file editing (`vim`), and help usage (`man`). +- **Efficiency Multiplier:** One-liners like `find / -name "*.conf" | xargs grep "port"` save hours over manual searches. +- **Career Edge:** Shell mastery signals expertise—recruiters ask, “How’s your Bash?” because it’s foundational to cloud (AWS CLI), Kubernetes (`kubectl exec`), and Ansible (`command` module). + +**Real-World Example:** An SRE at a major cloud provider used `for i in $(cat servers.txt); do ssh $i 'uptime'; done` to check 100 servers in seconds, averting a downtime escalation. + +--- + +## 1️⃣ Linux OS Architecture + +Linux is a modular, layered system, with each layer abstracting complexity for the next: + +| **Layer** | **Role** | **Example** | +|---------------------|---------------------------------------|-----------------------| +| **Hardware** | Physical devices | CPU, RAM, SSDs | +| **Kernel** | Core OS, manages resources | `sched`, `ext4` | +| **Libraries** | Standard functions for apps | `glibc`, `libpthread` | +| **Shell** | Command interpreter | `bash`, `zsh` | +| **Applications** | User tools and services | `ls`, `nginx` | + +- **Hardware:** Raw compute—CPUs execute instructions, disks store data. +- **Kernel:** The brain, handling system calls (e.g., `open()` for files), process scheduling, and device drivers. It’s a translator between software and hardware. +- **Libraries:** Provide reusable code—e.g., `printf()` in `glibc` lets `ls` format output without reinventing string handling. +- **Shell:** Your CLI gateway, parsing commands like `cat file | grep "error"` into kernel instructions via libraries. It’s both interactive (type `whoami`) and scriptable (`#!/bin/bash`). +- **Applications:** Tools like `vim` or `docker`, built on libraries, invoked via the shell. + +**How It Flows:** Typing `echo "hi" > file` in Bash: +1. Shell parses `echo` (internal command) and `>` (redirection). +2. Libraries handle string output (`hi`). +3. Kernel writes to disk via filesystem drivers. + +**Why It’s Elegant:** This separation lets you swap shells (Bash to Zsh), upgrade kernels, or run apps on different hardware without rewriting everything. It’s why Linux powers phones, servers, and supercomputers. + +--- + +## 2️⃣ Why Learn Shell Commands? + +Shell commands are the Swiss Army knife for DevOps and SRE: +- **Granular Control:** `kill -HUP $(pidof nginx)` reloads configs without downtime, unlike restarting via GUI. +- **Automation Foundation:** Scripts like `while true; do curl -s server; sleep 10; done` monitor health, feeding into CI/CD or alerting systems. +- **Incident Response:** `netstat -tulpn | grep 80` finds port conflicts during outages—faster than logs. +- **Data Processing:** `awk '{print $2}' access.log | sort | uniq -c` counts unique IPs in seconds, no Python needed. +- **RHCSA Must-Have:** Exam tasks demand `>`, `|`, `2>`, and correct syntax (e.g., `ls -lR /etc`). +- **Cloud Dominance:** AWS CLI (`aws s3 cp`), Terraform (`local-exec`), and Kubernetes (`kubectl`) lean on shell for execution and scripting. +- **Minimal Footprint:** Shell runs on tiny systems where GUIs can’t—think IoT or containerized apps. + +**Historical Context:** Shells evolved from 1970s Unix, prioritizing text-based efficiency. Today, Bash is ubiquitous because it’s powerful yet simple—learn it once, use it everywhere. + +**Career Impact:** Shell fluency cuts task time (e.g., `find / -mtime -1` vs. manual search), impresses in interviews, and unlocks advanced tools like `jq` or `yq` for JSON/YAML parsing. + +--- + +## 3️⃣ Shell Essentials + +This section unifies *Linux Bible* Chapter 3 and RHCSA Chapter 2, diving deep into Bash’s core features. + +#### **Shell Access** +The shell is your interface to Linux, accessible via: +- **Terminals:** GUI apps like GNOME Terminal or Konsole. +- **SSH:** Remote access (`ssh user@host`). +- **Virtual Consoles:** Ctrl+Alt+F2–F6 on physical machines. +- **TTY Devices:** `/dev/tty1` for console, `/dev/pts/0` for SSH sessions. +**Theory:** Shells like Bash interpret commands, sending them to the kernel. They’re lightweight, running even on minimal systems, unlike resource-heavy GUIs. Bash’s dominance stems from its GNU roots, balancing interactivity and scripting. + +#### **Command Structure** +Commands follow: `command -options arguments`. +- **Examples:** `ls -la /etc` (`-la` options, `/etc` argument), `grep -i "error" log` (`-i` ignores case). +- **Options:** Short (`-l`), long (`--all`), or combined (`-la`). +- **Arguments:** Files, dirs, or text the command acts on. +**Theory:** This structure is universal across Unix-like systems, rooted in 1970s design for flexibility. Options modify behavior (e.g., `ls -R` for recursive), while arguments specify targets. Bash parses these, resolving aliases first, then built-ins, then PATH. + +#### **Command Types** +Bash distinguishes three command types: +- **Aliases:** User-defined shortcuts, e.g., `alias ll='ls -la'`—checked first, overriding others. Stored in memory, reset on logout unless in `~/.bashrc`. +- **Internal (Built-in):** Part of Bash, e.g., `cd`, `echo`. Fast, no disk access. `type cd` shows “builtin”. +- **External:** Executables in `$PATH`, e.g., `/bin/ls`. Slower due to disk I/O. `which ls` finds them. +**Theory:** Aliases prioritize customization, built-ins optimize speed, and externals enable extensibility. This hierarchy ensures flexibility—e.g., `type ls` might show an alias, but `/bin/ls` forces the binary. `$PATH` dictates external command lookup, excluding `.` for security (use `./script`). + +#### **PATH** +`$PATH` lists directories for external commands: `echo $PATH` (`/bin:/usr/bin`). +- **Mechanics:** Bash searches left to right for matches. +- **Security:** Current dir (`.`) isn’t included—prevents accidental script execution. Use `./myscript`. +**Theory:** `$PATH` is a core environment variable, balancing accessibility and safety. Root’s `$PATH` includes `/sbin` for admin tools. Misconfigured `$PATH` can break commands (`command not found`), critical for troubleshooting. + +#### **I/O Redirection** +Redirection manages input/output streams: +- **STDIN (0):** `< file` (e.g., `sort < data.txt`). +- **STDOUT (1):** `> file` (overwrite), `>> file` (append), e.g., `ls > out.txt`. +- **STDERR (2):** `2> file`, `2>&1` (merge with STDOUT), e.g., `ls nofile 2> err.txt`. +- **Examples:** `ls nofile /etc > out.txt 2>&1` (all output to one file), `cat < input.txt > output.txt`. +**Theory:** File descriptors (0, 1, 2) are kernel abstractions, inherited by processes. Redirection lets programs operate agnostically—`ls` doesn’t know it’s writing to `/dev/null` vs. a file. `/dev/null` discards output, `/dev/tty` targets terminals. Redirection is key for automation, logging, and error suppression (e.g., `2> /dev/null` in cron). + +#### **Pipes** +Pipes (`|`) connect STDOUT of one command to STDIN of another: `ls -l | grep ".txt" | sort`. +- **Chaining:** `cat log | grep "error" | wc -l` counts error lines. +- **Power:** Combines simple tools into complex workflows. +**Theory:** Pipes embody Unix’s “do one thing well” philosophy—each command is a filter, transforming data streams. Unlike redirection (file-based), pipes operate in memory, enabling real-time processing (e.g., `tail -f log | grep "fail"`). They’re foundational for log analysis and data munging. + +#### **History Tricks** +Bash stores ~1,000 commands in `~/.bash_history`, updated on session close. +- **View:** `history` (all), `history 10` (last 10). +- **Recall:** `!5` (run #5), `!!` (last), `!ls` (last starting with `ls`). +- **Search:** Ctrl+R, type `ls`, cycle with Ctrl+R. +- **Edit:** `history -d 299` (delete line), `history -c` (clear memory), `history -c; history -w` (clear file too). +**Theory:** History enhances productivity, letting you reuse complex commands (e.g., `!curl` for API tests). Clearing sensitive entries (`history -d`) is crucial for security—passwords typed in cleartext (e.g., `mysql -psecret`) persist otherwise. Saved to disk only on exit, so `history -c` alone isn’t enough. + +#### **Bash Completion** +Tab completion works for commands, files, and variables: +- **Examples:** `ec` → `echo`, `cat /et` → `/etc/`, `$PA` → `$PATH`. +- **Multi-Tab:** `ls --` lists options like `--all`, `--color`. +**Theory:** Completion leverages Bash’s parsing, querying `$PATH`, filesystem, and env vars. It’s programmable—packages like `bash-completion` extend it to tools like `git` or `kubectl`. Speeds up workflows, reduces typos, and reveals options dynamically. + +#### **Variables & Environment** +Variables store dynamic values: `NAME=value`, `echo $NAME`. +- **Types:** + - **Shell Vars:** Local, e.g., `MYDIR=/tmp`. + - **Env Vars:** Exported, e.g., `export PATH`, inherited by child processes. +- **Key Vars:** `PATH`, `HOME`, `LANG` (e.g., `en_US.UTF-8` for locale), `PS1` (prompt). +- **Config Files:** + - `/etc/profile`: Global login shell settings. + - `/etc/bashrc`: Global subshell settings. + - `~/.bash_profile`: User login shell (calls `~/.bashrc`). + - `~/.bashrc`: User subshells, aliases, custom vars. +- **Messages:** `/etc/motd` (post-login notice), `/etc/issue` (pre-login banner). +**Theory:** Variables create a tailored environment—`LANG` affects date formats, `PATH` enables commands. Config files build a hierarchy: system-wide (`/etc`) then user-specific (`~`). Subshells (e.g., scripts) inherit login shell vars unless overridden. `/etc/motd` is ideal for admin alerts (e.g., “Maintenance at 2AM”). + +#### **Aliases** +Shortcuts for commands: `alias ll='ls -la'`. +- **Scope:** Session unless in `~/.bashrc`. +- **Check:** `alias` lists all. +**Theory:** Aliases override built-ins/externals, speeding repetitive tasks (e.g., `alias k='kubectl'`). They’re checked first in command resolution, making them powerful but risky—`alias rm='rm -i'` adds safety, but `alias ls='rm'` could be disastrous. + +#### **Vim Basics** +Vim is RHCSA-essential for file editing: +- **Modes:** Command (Esc), Input (`i`, `a`, `o`). +- **Commands:** `:wq` (save+quit), `:q!` (discard), `dd` (delete line), `yy` (copy), `p` (paste), `:%s/old/new/g` (replace all), `u` (undo). +- **Navigation:** `gg` (top), `G` (bottom), `/text` (search). +**Theory:** Vim’s modal design optimizes keystrokes—Command mode for navigation/ops, Input for typing. It’s universal, lightweight, and scriptable (e.g., `vim +:%s/foo/bar/g file`). RHCSA expects fluency—editing `/etc/fstab` under pressure demands `i`, `:wq`, and `:q!`. Syntax highlighting (`vim-enhanced`) aids debugging. + +#### **Help Tools** +Finding command info is critical: +- **`man`:** `man ls` (details), `man -k disk` (search summaries), `man 5 passwd` (file formats). +- **`--help`:** `ls --help` (quick options). +- **`info`:** `info coreutils` (in-depth, hierarchical), better with `pinfo`. +- **`/usr/share/doc`:** Service docs, e.g., `/usr/share/doc/rsyslog`. +- **`mandb`:** Updates `man -k` database (`sudo mandb`). +**Theory:** `man` pages are structured—NAME, SYNOPSIS, DESCRIPTION, EXAMPLES—rooted in Unix tradition. `man -k` searches `mandb`, but stale databases return “nothing appropriate.” `info` suits complex tools (e.g., `gdb`), while `/usr/share/doc` holds configs/samples. RHCSA tests help navigation—know `man -k`, `/example`, and `G` to jump to examples. + +**Pro Tip:** `type -a ls` shows all matches (alias, built-in, external) vs. `which ls` (just `$PATH`)! + +--- + + +## 4️⃣ Challenge Breakdown + +### **📝 Theoretical Questions** +1. How does the shell bridge applications and the kernel? +2. Explain `2>&1` vs. `2> file` with an example. +3. Why does `myscript` fail but `./myscript` work? +4. How does `history -c; history -w` differ from `history -c` alone? +5. Why is `/etc/motd` useful for admins but not GUI users? + +--- + +### **⚙️ Practical Challenge: Shell Skills in Action** + +#### **Step 1: Shell & Redirection on EC2 Instances** +**Goal:** Master commands and I/O on Day 1 EC2s (Ubuntu, Amazon Linux, RHEL). + +1. SSH into each: +``` + ssh -i mykey.pem ubuntu@ # Ubuntu + ssh -i mykey.pem ec2-user@ # Amazon Linux + ssh -i mykey.pem ec2-user@ # RHEL +``` + +2. Run these: +``` +type ls # Alias or binary? +ls -l /etc > out.txt # Redirect STDOUT +ls nofile 2> err.txt # Redirect STDERR +cat out.txt err.txt > combined.txt 2>&1 # Merge outputs +echo "Host: $(hostname)" >> combined.txt # Append +ls -R / | grep "bin" | less # Pipe chain +history -d 2 # Delete 2nd command +``` + +3. Capture combined.txt content. +**Troubleshooting Tip:** ls: command not found? Check $PATH with echo $PATH. +```bash +#!/bin/bash +REPORT="sys_$(date +%Y%m%d).txt" +echo "Check - $(date)" > "$REPORT" +echo "User: $USER" >> "$REPORT" +uptime -p | tee -a "$REPORT" # Pipe and append +df -h / | tail -n 1 >> "$REPORT" 2>/dev/null +``` + +#### **Step 2: Vim & Scripting on RHEL VM** +**Goal:** Edit and automate with Vim on your Day 1 RHEL VM. + +1. Boot RHEL VM, log in. +2. Create/edit a script: +``` +vim ~/monitor.sh +``` + +3. Input (press i): +``` +#!/bin/bash +REPORT="sys_$(date +%Y%m%d).txt" +echo "Check - $(date)" > "$REPORT" +echo "User: $USER" >> "$REPORT" +uptime -p | tee -a "$REPORT" # Pipe and append +df -h / | tail -n 1 >> "$REPORT" 2>/dev/null +``` + +4. Edit: Esc, :3 (line 3), dd (delete), u (undo), :wq (save+quit). +5. Run: +``` +chmod +x monitor.sh +./monitor.sh +cat "$REPORT" +``` + +6. Persist an alias: +``` +echo "alias m='./monitor.sh'" >> ~/.bashrc +source ~/.bashrc +``` + +**Pro Tip:** Vim stuck? Esc, :q! forces exit! + + +5️⃣ Submission Guidelines +✅ Proof of Completion: +Screenshot of EC2 terminal with combined.txt content. +Screenshot of RHEL VM terminal showing sys_*.txt content. +Text output of both files. +✅ Documentation: +Steps for both tasks. +Issues (e.g., Permission denied) and fixes (e.g., sudo). +Example Submission: + +EC2 combined.txt: +``` +dir1 dir2 file1 +ls: nofile: No such file +Host: ip-10-0-0-1 +``` + +RHEL VM sys_20250407.txt: +``` +Check - Mon Apr 07 2025 +User: ec2-user +up 1 hour +/dev/sda1 20G 5G 15G 25% / +``` + +6️⃣ Bonus Tasks 🎯 +Help Hunt: Run man -k disk | grep 8—screenshot admin commands. +Vim Swap: In monitor.sh, use :%s/User/USER/g—screenshot result. From 76de0125485bbbc4bcdf91fc01cbbcb5447742a3 Mon Sep 17 00:00:00 2001 From: Sagar Utekar Date: Tue, 8 Apr 2025 00:30:56 +0530 Subject: [PATCH 049/133] Update readme.md --- DailyChallenges/Season2/Day9/readme.md | 47 ++++++++++++++++++-------- 1 file changed, 32 insertions(+), 15 deletions(-) diff --git a/DailyChallenges/Season2/Day9/readme.md b/DailyChallenges/Season2/Day9/readme.md index 6632eb1..12f1050 100644 --- a/DailyChallenges/Season2/Day9/readme.md +++ b/DailyChallenges/Season2/Day9/readme.md @@ -250,24 +250,25 @@ source ~/.bashrc **Pro Tip:** Vim stuck? Esc, :q! forces exit! -5️⃣ Submission Guidelines -✅ Proof of Completion: -Screenshot of EC2 terminal with combined.txt content. -Screenshot of RHEL VM terminal showing sys_*.txt content. -Text output of both files. -✅ Documentation: -Steps for both tasks. -Issues (e.g., Permission denied) and fixes (e.g., sudo). -Example Submission: - -EC2 combined.txt: +## 5️⃣ Submission Guidelines +✅ **Proof of Completion:** +- Screenshot of EC2 terminal with combined.txt content. +- Screenshot of RHEL VM terminal showing sys_*.txt content. +- Text output of both files. +✅ **Documentation:** +- Steps for both tasks. +- Issues (e.g., Permission denied) and fixes (e.g., sudo). + +**Example Submission:** + +**EC2 combined.txt:** ``` dir1 dir2 file1 ls: nofile: No such file Host: ip-10-0-0-1 ``` -RHEL VM sys_20250407.txt: +**RHEL VM sys_20250407.txt:** ``` Check - Mon Apr 07 2025 User: ec2-user @@ -275,6 +276,22 @@ up 1 hour /dev/sda1 20G 5G 15G 25% / ``` -6️⃣ Bonus Tasks 🎯 -Help Hunt: Run man -k disk | grep 8—screenshot admin commands. -Vim Swap: In monitor.sh, use :%s/User/USER/g—screenshot result. +## 6️⃣ Bonus Tasks 🎯 +- Help Hunt: Run man -k disk | grep 8—screenshot admin commands. +- Vim Swap: In monitor.sh, use :%s/User/USER/g—screenshot result. + +### **Share Your Progress!** +Post your experience on social media with the hashtags: **#getfitwithsagar, #SRELife, #DevOpsForAll** + +--- + +## **Join Our Community** +Connect with fellow learners: +- **Discord:** [Join here](https://discord.gg/mNDm39qB8t) +- **Google Group:** [Join here](https://groups.google.com/forum/#!forum/daily-devops-sre-challenge-series/join) +- **YouTube:** [Subscribe here](https://www.youtube.com/@Sagar.Utekar) + +--- + +Keep learning and happy exploring! +Sagar Utekar From d3ec44abf4da74b9a0dc90753c936a23eb43edbd Mon Sep 17 00:00:00 2001 From: Sagar Utekar Date: Tue, 8 Apr 2025 00:41:34 +0530 Subject: [PATCH 050/133] Update readme.md --- DailyChallenges/Season2/Day9/readme.md | 3 +++ 1 file changed, 3 insertions(+) diff --git a/DailyChallenges/Season2/Day9/readme.md b/DailyChallenges/Season2/Day9/readme.md index 12f1050..20c0a77 100644 --- a/DailyChallenges/Season2/Day9/readme.md +++ b/DailyChallenges/Season2/Day9/readme.md @@ -28,6 +28,9 @@ The shell is the backbone of Linux system administration, DevOps, and SRE. It’ Linux is a modular, layered system, with each layer abstracting complexity for the next: +![linux](https://github.com/user-attachments/assets/4bba243a-b552-451b-9891-3fdf6896bc6a) + + | **Layer** | **Role** | **Example** | |---------------------|---------------------------------------|-----------------------| | **Hardware** | Physical devices | CPU, RAM, SSDs | From 132d633f0e88b2ff4412245be7d1332993e64b6e Mon Sep 17 00:00:00 2001 From: Sagar Utekar Date: Tue, 8 Apr 2025 07:48:18 +0530 Subject: [PATCH 051/133] Create 1_shell_terminal_windows.md --- .../Season2/Day9/1_shell_terminal_windows.md | 109 ++++++++++++++++++ 1 file changed, 109 insertions(+) create mode 100644 DailyChallenges/Season2/Day9/1_shell_terminal_windows.md diff --git a/DailyChallenges/Season2/Day9/1_shell_terminal_windows.md b/DailyChallenges/Season2/Day9/1_shell_terminal_windows.md new file mode 100644 index 0000000..a986bf7 --- /dev/null +++ b/DailyChallenges/Season2/Day9/1_shell_terminal_windows.md @@ -0,0 +1,109 @@ +## About Shells and Terminal Windows + +Linux offers multiple ways to access a shell interface, essential for interacting with the system. Here are three common methods: the shell prompt, terminal windows, and virtual consoles. + +--- + +### **Using the Shell Prompt** +If your Linux system boots without a graphical user interface (GUI) or the GUI isn’t working, you’ll see a plaintext login prompt after starting the system, like this: +``` +Red Hat Enterprise Linux Workstation Release 6.1 (Santiago) +Kernel 2.6.32-131... on X86 +joe login: +``` + +- **After Login:** Enter your username and password. You’ll get a shell prompt—`$` for regular users, `#` for root. +- **Example Prompt:** `[jake@pine share]$` shows username (`jake`), hostname (`pine`), and current directory (`/usr/share`). +- **Customization:** You can tweak the prompt to show info like date or directory (see “Setting your prompt” later in *Linux Bible* Ch. 3). +- **Usage:** Type commands (e.g., `ls`, `whoami`) and press Enter. This is your primary interaction method without a GUI. + +**Note:** `$` means any user can run the command; `#` indicates root privileges are needed (common for admin tasks). + +--- + +### **Using a Terminal Window** +With a GUI running, access a shell via a terminal emulator (Terminal window) from the desktop: +- **Launch Methods:** + - **Right-Click Desktop:** Look for “Open in Terminal” or “Terminal Window” in the context menu (if enabled). + - **Panel Menu:** E.g., in GNOME, go to `Applications ➪ System Tools ➪ Terminal`. +- **Default Emulator:** In Fedora/RHEL with GNOME, it’s GNOME Terminal (`gnome-terminal`). +- **Features:** Beyond basic shell access, GNOME Terminal offers: + - Cut/paste text. + - Custom fonts/colors (Edit ➪ Profiles ➪ General/Colors tabs). + - Background images or transparency. + - Scrollback buffer settings. + +**Quick Demo (Fedora/RHEL):** +1. Open `Applications ➪ System Tools ➪ Terminal`. +2. Edit profile: Change font (General tab), tweak colors (Colors tab), or set a background image (Profile window). +3. Close when done—ready to use! + +**Why Use It?** Most GUI users access shells this way for convenience. + +--- + +### **Using Virtual Consoles** +Virtual consoles provide multiple shell sessions alongside the GUI: +- **Access:** Press `Ctrl+Alt+F1` to `F7`. + - **Fedora Example:** `Ctrl+Alt+F1` = GUI, `F2`–`F7` = text-based consoles. + - **Variation:** Some systems use `F5` or `F7` for GUI (e.g., `Ctrl+Alt+F7` to return). +- **Try It:** + 1. Press `Ctrl+Alt+F3` → see a text login prompt. + 2. Log in, run commands (e.g., `whoami`), then `exit`. + 3. Return to GUI with `Ctrl+Alt+F1`. + +**Benefit:** Run multiple shells without a GUI window, ideal for troubleshooting or multi-tasking. + +--- + +### **Key Takeaways** +- **Shell Prompt:** Direct, no-GUI access—perfect for minimal systems. +- **Terminal Window:** GUI-friendly shell with extra features. +- **Virtual Consoles:** Multiple text-based shells, accessible via shortcuts. +Start exploring by booting your Linux system and trying these methods! + + +In most Linux systems, the default shell is **Bash** (`/bin/bash`). +## 🐚 Trying Other Shells + +Other shells (`ksh`, `tcsh`, `csh`, `sh`, `dash`, etc.) might be installed on your system. + +--- + +### 🔍 Test One +- Open a terminal and type the name of the shell (e.g., `ksh`). +- Try a few commands. +- Type `exit` to return to your default shell (e.g., Bash). + +--- + +### 🤔 Why Switch? + +- **Familiarity**: + Users from different UNIX backgrounds may prefer different shells. + - *System V* users → `ksh` + - *Berkeley UNIX* users → `csh` + +- **Scripts**: + Some scripts are written specifically for a certain shell (e.g., `csh` scripts need `csh` to run properly). + +- **Features**: + Certain shells have unique features that may appeal to specific users. + Example: `ksh` offers more flexible alias handling than Bash. + +--- + +### 🧠 Why Bash? + +- **Default**: + Bash is the default shell on most Linux distributions (e.g., Fedora, RHEL). + Exceptions include embedded systems using lighter shells like `dash`. + +- **Features**: + Bash blends features from `sh`, `ksh`, and `csh`, making it powerful and versatile. + +- **Learnability**: + Learning one shell—especially Bash—helps you easily transition to others. + Use the manual pages to explore: + ```bash + man bash From 025ef43d0ba285428ea7ea2c231cf7a9b67343be Mon Sep 17 00:00:00 2001 From: Sagar Utekar Date: Tue, 8 Apr 2025 08:04:22 +0530 Subject: [PATCH 052/133] Create 2_recalling_commands_history.md --- .../Day9/2_recalling_commands_history.md | 71 +++++++++++++++++++ 1 file changed, 71 insertions(+) create mode 100644 DailyChallenges/Season2/Day9/2_recalling_commands_history.md diff --git a/DailyChallenges/Season2/Day9/2_recalling_commands_history.md b/DailyChallenges/Season2/Day9/2_recalling_commands_history.md new file mode 100644 index 0000000..cd8a763 --- /dev/null +++ b/DailyChallenges/Season2/Day9/2_recalling_commands_history.md @@ -0,0 +1,71 @@ +## Recalling Commands Using Command History + +Repeating or modifying commands from your shell session is a powerful way to save time and effort. The Bash shell provides a **command history** feature that stores commands you’ve entered, allowing you to recall, edit, and reuse them. This is especially handy for correcting typos in long commands or reusing complex pipelines without retyping. + +--- + +### **Understanding the History List** +The shell maintains a list of your previous commands, accessible via the `history` command: +**View History:** + ```bash + history + ``` +Displays all stored commands (up to a default of 1,000, configurable via HISTSIZE). + +**Limit Output:** +```bash +history 10 +``` +Shows the last 10 commands, each prefixed with a number (e.g., 382 date). + +**Storage:** Commands are held in memory during the session and saved to ~/.bash_history upon exit. +**Example Output:** + + ```bash +382 date +383 ls /usr/bin | sort -f | less +384 history 10 +``` + +### **Command-Line Editing in Bash** +Bash lets you edit recalled commands instead of starting from scratch. +By default, it uses Emacs-style key bindings, but you can switch to Vi-style for a different experience. + +**Set Vi Mode** +Add the following to your ~/.bashrc file: + + ```bash +set -o vi +Then, reload your shell: + + ```bash +source ~/.bashrc +``` +**In Vi mode:** Press Esc to enter command mode, then use Vi keys (e.g., i to insert). + +**Emacs Editing Example** +**Modify this command:** + + ```bash +ls /usr/bin | sort -f | less +``` + +**To:** + + ```bash +ls /bin | sort -f | less +``` + +**Steps:** +Press ↑ (up arrow) to recall the previous command. + +Press Ctrl+A to move to the start of the line. + +Press Ctrl+F (or →) to move under the first /. + +Press Ctrl+D four times to delete /usr. + +Type bin, then press Enter. + + +Screenshot 2025-04-08 at 8 03 36 AM From a92def90fd84e6985562c3e28b37c0abc086d92b Mon Sep 17 00:00:00 2001 From: Sagar Utekar Date: Tue, 8 Apr 2025 08:16:26 +0530 Subject: [PATCH 053/133] Update readme.md --- DailyChallenges/Season2/Day9/readme.md | 17 ++++++++++++++--- 1 file changed, 14 insertions(+), 3 deletions(-) diff --git a/DailyChallenges/Season2/Day9/readme.md b/DailyChallenges/Season2/Day9/readme.md index 20c0a77..3fd4b4d 100644 --- a/DailyChallenges/Season2/Day9/readme.md +++ b/DailyChallenges/Season2/Day9/readme.md @@ -124,6 +124,7 @@ Bash stores ~1,000 commands in `~/.bash_history`, updated on session close. - **Search:** Ctrl+R, type `ls`, cycle with Ctrl+R. - **Edit:** `history -d 299` (delete line), `history -c` (clear memory), `history -c; history -w` (clear file too). **Theory:** History enhances productivity, letting you reuse complex commands (e.g., `!curl` for API tests). Clearing sensitive entries (`history -d`) is crucial for security—passwords typed in cleartext (e.g., `mysql -psecret`) persist otherwise. Saved to disk only on exit, so `history -c` alone isn’t enough. +Screenshot 2025-04-08 at 8 08 03 AM #### **Bash Completion** Tab completion works for commands, files, and variables: @@ -135,13 +136,23 @@ Tab completion works for commands, files, and variables: Variables store dynamic values: `NAME=value`, `echo $NAME`. - **Types:** - **Shell Vars:** Local, e.g., `MYDIR=/tmp`. - - **Env Vars:** Exported, e.g., `export PATH`, inherited by child processes. + - **Env Vars:** Exported, e.g., `export PATH`, inherited by child processes. + - You can display the value of a specific environment variable by using the echo command followed by the variable name preceded by a dollar sign ($). For example: +```bash +echo $USER +``` +- To refer to the value of an environment variable in a command simply prefix the variable name with a dollar sign ($). For example: +```bash +echo $USER +``` - **Key Vars:** `PATH`, `HOME`, `LANG` (e.g., `en_US.UTF-8` for locale), `PS1` (prompt). -- **Config Files:** +- **Config Files:** - `/etc/profile`: Global login shell settings. - `/etc/bashrc`: Global subshell settings. - `~/.bash_profile`: User login shell (calls `~/.bashrc`). - - `~/.bashrc`: User subshells, aliases, custom vars. + - `~/.bashrc`: User subshells, aliases, custom vars. +Screenshot 2025-04-08 at 8 14 04 AM + - **Messages:** `/etc/motd` (post-login notice), `/etc/issue` (pre-login banner). **Theory:** Variables create a tailored environment—`LANG` affects date formats, `PATH` enables commands. Config files build a hierarchy: system-wide (`/etc`) then user-specific (`~`). Subshells (e.g., scripts) inherit login shell vars unless overridden. `/etc/motd` is ideal for admin alerts (e.g., “Maintenance at 2AM”). From 8e282eb1c4eb8029ca2167e7ab7460677faeddf7 Mon Sep 17 00:00:00 2001 From: Sagar Utekar Date: Tue, 8 Apr 2025 08:16:45 +0530 Subject: [PATCH 054/133] Update readme.md --- DailyChallenges/Season2/Day9/readme.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/DailyChallenges/Season2/Day9/readme.md b/DailyChallenges/Season2/Day9/readme.md index 3fd4b4d..dadf5b5 100644 --- a/DailyChallenges/Season2/Day9/readme.md +++ b/DailyChallenges/Season2/Day9/readme.md @@ -1,5 +1,5 @@ -# 🚀 Daily DevOps + SRE Challenge Series – Season 2 -## 🗓️ Day 2: Mastering the Linux Shell +# Daily DevOps + SRE Challenge Series – Season 2 +## Day 2: Mastering the Linux Shell ## Introduction Welcome to Day 2 of the Daily DevOps SRE Challenge Series - Season 2! 🎉 From 238b728fa5137382b3149fb392868abd976400b9 Mon Sep 17 00:00:00 2001 From: Sagar Utekar Date: Tue, 8 Apr 2025 08:18:44 +0530 Subject: [PATCH 055/133] Update readme.md --- DailyChallenges/Season2/Day9/readme.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/DailyChallenges/Season2/Day9/readme.md b/DailyChallenges/Season2/Day9/readme.md index dadf5b5..1f06d7b 100644 --- a/DailyChallenges/Season2/Day9/readme.md +++ b/DailyChallenges/Season2/Day9/readme.md @@ -1,10 +1,10 @@ # Daily DevOps + SRE Challenge Series – Season 2 -## Day 2: Mastering the Linux Shell +## Day 9: Mastering the Linux Shell ## Introduction Welcome to Day 2 of the Daily DevOps SRE Challenge Series - Season 2! 🎉 -Today, we’re diving into **the Linux Shell**, blending *Chapter 3: Using the Shell* (*Linux Bible*) and *Chapter 2: Using Essential Tools* (RHCSA prep). You’ll explore OS architecture, master Bash commands, edit with Vim, and harness help tools. By the end, you’ll: +Today, we’re diving into **the Linux Shell**, You’ll explore OS architecture, master Bash commands, edit with Vim, and harness help tools. By the end, you’ll: - Grasp Linux’s layered structure and the shell’s critical role. - Master commands, redirection, piping, history, variables, aliases, and Vim. - Apply these on your Day 1 EC2 instances and RHEL VM. From c1b6fd39ee0167bf31e874033fc12393fb15c12e Mon Sep 17 00:00:00 2001 From: Sagar Utekar Date: Tue, 8 Apr 2025 08:18:54 +0530 Subject: [PATCH 056/133] Update readme.md --- DailyChallenges/Season2/Day9/readme.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/DailyChallenges/Season2/Day9/readme.md b/DailyChallenges/Season2/Day9/readme.md index 1f06d7b..ddb593d 100644 --- a/DailyChallenges/Season2/Day9/readme.md +++ b/DailyChallenges/Season2/Day9/readme.md @@ -2,7 +2,7 @@ ## Day 9: Mastering the Linux Shell ## Introduction -Welcome to Day 2 of the Daily DevOps SRE Challenge Series - Season 2! 🎉 +Welcome to Day 9 of the Daily DevOps SRE Challenge Series - Season 2! 🎉 Today, we’re diving into **the Linux Shell**, You’ll explore OS architecture, master Bash commands, edit with Vim, and harness help tools. By the end, you’ll: - Grasp Linux’s layered structure and the shell’s critical role. From d744ce7c58c2b91b3ae858acb4f3a22c36ef09ad Mon Sep 17 00:00:00 2001 From: Sagar Utekar Date: Wed, 9 Apr 2025 06:09:34 +0530 Subject: [PATCH 057/133] Create readme.md --- DailyChallenges/Season2/Day10/readme.md | 59 +++++++++++++++++++++++++ 1 file changed, 59 insertions(+) create mode 100644 DailyChallenges/Season2/Day10/readme.md diff --git a/DailyChallenges/Season2/Day10/readme.md b/DailyChallenges/Season2/Day10/readme.md new file mode 100644 index 0000000..6b1436c --- /dev/null +++ b/DailyChallenges/Season2/Day10/readme.md @@ -0,0 +1,59 @@ +# Daily DevOps + SRE Challenge Series – Season 2 +## Day 10: Navigating and Managing the Linux Filesystem + +### Introduction +Welcome to Day 10 of the Daily DevOps + SRE Challenge Series - Season 2! 🎉 + +Today, we’re exploring the Linux filesystem—its structure, navigation, and essential file management tools. You’ll master commands to move around, create, copy, move, and delete files, all while building skills critical for DevOps and SRE workflows. By the end, you’ll: +- Understand the filesystem hierarchy and path navigation. +- Gain fluency in tools like `cd`, `mkdir`, `cp`, `mv`, and `rm`. +- Apply these on your Day 1 EC2 instances or RHEL VM. + +#### Why This Matters? +The filesystem is the foundation of Linux operations. Whether you’re deploying apps, troubleshooting outages, or prepping for RHCSA, these skills are non-negotiable. Here’s why: +- **Universal Skill**: Every Linux system—from cloud instances to Kubernetes pods—relies on the filesystem. Commands like `cd /var/log` or `rm -rf /tmp/cache` work everywhere. +- **Automation Backbone**: Scripts like `mv *.log /archive/$(date +%Y%m%d)` organize logs daily in cron jobs or CI/CD pipelines. +- **SRE Efficiency**: During incidents, `cp -r /etc/nginx /backup` or `find / -name core.*` pinpoints crash dumps fast, saving downtime. +- **RHCSA Must-Know**: The exam tests navigation (`/etc` vs `../`), file ops (`touch`, `mkdir -p`), and cleanup (`rm -rf`). +- **Career Boost**: Fluency in filesystem tasks impresses in interviews—think “How do you restore a config?” or “How do you free disk space?” +- **Real-World Example**: An SRE at a fintech firm used `mkdir -p /data/backups/$(date +%F) && cp -r /prod/data/* /data/backups/$(date +%F)` to snapshot critical data before a risky upgrade, averting disaster when it failed. + +Let’s dive in and take control of the filesystem! + +--- + + +### Linux Filesystem Overview + +The Linux filesystem is a hierarchical structure used to store all information on a computer. It is based on the UNIX system, where nearly everything, including data, commands, symbolic links, devices, and directories, is represented within the filesystem. + +### Structure and Navigation +Screenshot 2025-04-09 at 5 57 43 AM + +1. **Hierarchy**: The filesystem is organized like an upside-down tree with the root directory (`/`) at the top. Below the root are common directories such as `/bin`, `/dev`, `/home`, `/lib`, and `/tmp`. These directories can contain subdirectories, creating a hierarchical structure. +2. **Paths**: + * **Full Path**: Specifies the complete route from the root directory to a file or directory (e.g., `/home/joe/Documents/memos/memo1.doc`). + * **Relative Path**: Specifies the path relative to the current directory (e.g., `memo1.doc` if the current directory is `/home/joe/Documents/memos`). + +### Common Linux Directories + +#### Key Directories + +- **`/bin`**: Contains common Linux user commands like `ls`, `sort`, `date`, and `chmod`. +- **`/boot`**: Contains the bootable Linux kernel and boot loader configuration files (GRUB). +- **`/dev`**: Contains files representing access points to devices (e.g., `tty*`, `fd*`, `hd*`, `sd*`, `ram*`, `cd*`). +- **`/etc`**: Contains administrative configuration files, which are typically plaintext files that can be edited with a text editor. +- **`/home`**: Contains directories for each regular user account (except root, which uses `/root`). +- **`/media`**: Standard location for automatically mounting devices, particularly removable media. +- **`/lib`**: Contains shared libraries required by applications in `/bin` and `/sbin` to boot the system. +- **`/mnt`**: A common mount point for devices, often used for temporarily mounting local or remote filesystems. +- **`/misc`**: Sometimes used to automatically mount filesystems upon request. +- **`/opt`**: Used to store add-on application software. +- **`/proc`**: Contains information about system resources. +- **`/root`**: The home directory for the root user, located outside `/home` for security reasons. +- **`/sbin`**: Contains administrative commands and daemon processes. +- **`/tmp`**: Contains temporary files used by applications. +- **`/usr`**: Contains user documentation, games, graphical files (X11), libraries, and other commands and files not needed during the boot process. +- **`/var`**: Contains directories of data used by various applications, including FTP server files (`/var/ftp`), web server files (`/var/www`), system log files (`/var/log`), and spool files (`/var/spool`). + + From 9b647bb52c6be8d87d020bb68e3717de2fb65e78 Mon Sep 17 00:00:00 2001 From: Sagar Utekar Date: Wed, 9 Apr 2025 06:10:23 +0530 Subject: [PATCH 058/133] Create mounts_in_linux.md --- .../Season2/Day10/mounts_in_linux.md | 121 ++++++++++++++++++ 1 file changed, 121 insertions(+) create mode 100644 DailyChallenges/Season2/Day10/mounts_in_linux.md diff --git a/DailyChallenges/Season2/Day10/mounts_in_linux.md b/DailyChallenges/Season2/Day10/mounts_in_linux.md new file mode 100644 index 0000000..8fbc2fe --- /dev/null +++ b/DailyChallenges/Season2/Day10/mounts_in_linux.md @@ -0,0 +1,121 @@ +### Understanding Mounts in the Linux Filesystem + +#### 1. What is a Mount? + +A **mount** is the process of connecting a storage device (e.g., a disk partition, logical volume, or network share) to a specific directory in the Linux filesystem. Once mounted, that directory—called the **mount point**—serves as the access point to the device’s contents. + +**Core Concept**: Linux presents a single, unified filesystem hierarchy starting at the root directory (`/`). Mounts integrate various devices into this hierarchy, making them appear as part of the same tree, even if they reside on physically separate hardware or remote systems. + +**How It Works**: When a device is mounted to a directory (e.g., `/mnt/data`), accessing that directory allows you to interact with the device’s filesystem as if it were part of the main structure. + +**Example**: Mounting `/dev/sda1` to `/boot` makes the partition’s boot files accessible under `/boot`. + +#### 2. Why Use Mounts? + +Mounting devices provides flexibility and addresses limitations of storing all files on a single filesystem. Using multiple mounts avoids several pitfalls: + +- **Preventing Overflows**: High activity in one area (e.g., log files in `/var/log`) could fill a single filesystem, disrupting services. Separate mounts isolate space usage. +- **Enhancing Security**: A single filesystem limits security customization. By mounting a device to a specific directory, you can apply tailored mount options (e.g., read-only or no execution) to meet security needs. +- **Improving Scalability**: If a single filesystem runs out of space, expanding it is complex. Separate mounts make it easier to add or replace storage devices without affecting the entire system. +- **Organizational Clarity**: Mounts allow admins to dedicate devices to specific purposes (e.g., a partition for `/home` vs. `/var`), improving management and maintenance. + + +#### 3. Common Mounted Directories + +Certain directories are frequently mounted on dedicated devices to optimize performance, security, or system reliability. The choice depends on the system’s needs and the administrator’s discretion, but common examples include: + +- **`/boot`**: + - **Purpose**: Stores essential boot files (e.g., the Linux kernel). + - **Why Separate?**: Often placed on a dedicated partition because the root filesystem (`/`) may use Logical Volume Manager (LVM), which isn’t bootable by default. A separate `/boot` ensures the system can start. +- **`/boot/EFI`**: + - **Purpose**: Holds files for systems using Extensible Firmware Interface (EFI) for booting. + - **Why Separate?**: EFI requires a dedicated filesystem (e.g., FAT32) for early boot processes, distinct from other filesystems. +- **`/var`**: + - **Purpose**: Contains dynamic data like logs (`/var/log`), mail, or runtime files. + - **Why Separate?**: `/var` grows unpredictably (e.g., due to log writes). A dedicated device prevents it from filling the root filesystem and crashing services. +- **`/home`**: + - **Purpose**: Stores user home directories. + - **Why Separate?**: Enhances security (e.g., mount with `noexec` to block executable files or `nodev` to prevent device access). Also preserves user data during OS reinstalls, as `/home` can remain untouched. +- **`/usr`**: + - **Purpose**: Contains operating system files and programs, typically read-only for users. + - **Why Separate?**: Can be mounted as read-only to protect system files from unauthorized changes. +- **Custom Mounts**: Administrators may mount other directories (e.g., `/data` or `/backup`) based on specific requirements, such as isolating application data or backups. + + +#### 4. Mount Options + +When mounting a device, administrators can specify **mount options** to control how the filesystem behaves. These options enhance security, performance, or functionality. Examples include: + +- **`ro`**: Mounts the filesystem as read-only, preventing modifications (e.g., for `/usr`). +- **`noexec`**: Prevents execution of binaries, reducing security risks (e.g., for `/home`). +- **`nodev`**: Blocks device files, limiting potential exploits (e.g., for `/home`). +- **`rw`**: Allows read-write access (default for most mounts). + +**Use Case**: Mounting `/home` with `noexec,nodev` ensures users can’t run malicious scripts or access raw devices. + +#### 5. Exploring Mounts with Commands + +Several commands help administrators view and understand the current mount configuration: + +- **`mount`**: + - **Purpose**: Displays all currently mounted devices by reading `/proc/mounts`, a kernel-maintained file listing active mounts. + - **Output**: Includes device, mount point, filesystem type, and options (e.g., `/dev/nvme0n1p1` on `/boot` type `xfs` (`rw,relatime,seclabel`)). + - **Note**: Output can be lengthy, as it includes kernel interfaces (e.g., `sysfs`, `proc`) alongside physical devices. + - **Use Case**: Quick check of what’s mounted and where. +- **`df -Th`**: + - **Purpose**: Shows disk space usage, mount points, and filesystem types for mounted devices. + - **Options**: + - **`-T`**: Displays the filesystem type (e.g., `xfs`, `ext4`). + - **`-h`**: Presents sizes in human-readable units (e.g., GiB instead of kibibytes). + - **Output Columns**: + - **Filesystem**: Device name (e.g., `/dev/sda1`) or virtual device (e.g., `tmpfs`). + - **Type**: Filesystem type. + - **Size**: Total capacity. + - **Used**: Space in use. + - **Avail**: Free space. + - **Use%**: Percentage used. + - **Mounted on**: Mount point (e.g., `/boot`). + - **Example**: + +```plaintext +Filesystem Type Size Used Avail Use% Mounted on +/dev/sda1 xfs 197M 131M 67M 67% /boot +``` + + - **Use Case**: Checking available space or confirming mount points. +- **`findmnt`**: + - **Purpose**: Presents mounts in a tree-like structure, highlighting relationships between mount points. + - **Advantage**: Cleaner and more readable than `mount`, focusing on physical and key virtual filesystems. + - **Use Case**: Visualizing how mounts fit into the hierarchy (e.g., `/` vs. `/boot` vs. `/var`). + + +#### 6. Practical Context + +Mounts in Administration: Mounts are configured in `/etc/fstab` to persist across reboots, ensuring devices are automatically attached to their mount points. + +**Real-World Example**: A server might have: + +- `/` on an LVM volume for flexibility. +- `/boot` on a separate partition for booting. +- `/var` on a dedicated disk to handle log growth. +- `/home` on a network share for centralized user data. + +**Dynamic Management**: Admins can manually mount devices (e.g., `mount /dev/sdb1 /mnt`) for temporary access or troubleshooting. + +#### 7. Why Mounts Matter + +- **System Reliability**: Separate mounts prevent one area (e.g., `/var`) from crashing the entire system by filling up. +- **Security**: Mount options like `noexec` or `ro` protect critical directories. +- **Flexibility**: Mounts allow admins to mix local disks, network shares, and virtual filesystems seamlessly. +- **RHCSA Relevance**: Understanding mounts is critical for tasks like configuring `/etc/fstab`, troubleshooting disk issues, or setting up secure filesystems. +- **DevOps/SRE Impact**: Mounts underpin storage management in cloud VMs, containers, and clusters, ensuring data availability and performance. + +**Key Takeaways for Learners**: + +- A mount connects a device to a directory, integrating it into the `/` hierarchy. +- Multiple mounts avoid space, security, and scalability issues of a single filesystem. +- Common mount points like `/boot`, `/var`, and `/home` serve specific purposes. +- Mount options (e.g., `ro`, `noexec`) customize behavior for security or functionality. +- Use `mount`, `df -Th`, and `findmnt` to inspect the mount structure. + + From 3a5622df81fcc8ec55672018408b5fc3f8a53481 Mon Sep 17 00:00:00 2001 From: Sagar Utekar Date: Wed, 9 Apr 2025 06:20:55 +0530 Subject: [PATCH 059/133] Create core_filesystem_commands.md --- .../Season2/Day10/core_filesystem_commands.md | 178 ++++++++++++++++++ 1 file changed, 178 insertions(+) create mode 100644 DailyChallenges/Season2/Day10/core_filesystem_commands.md diff --git a/DailyChallenges/Season2/Day10/core_filesystem_commands.md b/DailyChallenges/Season2/Day10/core_filesystem_commands.md new file mode 100644 index 0000000..6659a5c --- /dev/null +++ b/DailyChallenges/Season2/Day10/core_filesystem_commands.md @@ -0,0 +1,178 @@ +As a Linux user or administrator, you frequently interact with the filesystem to navigate, create, and manage files and directories. When you log into a Linux system and open a shell, you start in your **home directory** (e.g., `/home/joe`), which serves as your personal workspace for files and subdirectories. Mastering basic commands and file management techniques is essential for organizing data, securing access, and performing administrative tasks. + +#### Home Directory + +The home directory is represented by `~`, which is your default base (e.g., `/home/$USER`). You can use `~` to navigate to directories relative to your home directory. + +### Core Filesystem Commands + +These commands are the foundation for navigating and managing the filesystem. + +#### 2.1 `cd` - Change Directory + +- **Purpose**: Moves you to another directory. +- **Usage**: + - `cd`: Returns to home (e.g., `/home/joe`). + - **Absolute Path**: `cd /usr/share → /usr/share`. (Starts from the root directory `/`) + - **Relative Path**: From `/usr/share`, `cd doc → /usr/share/doc`. (Relative to the current directory) + - `cd ~`: Home directory (e.g., `/home/joe`). + - `cd ~/Music`: Subdirectory of home (e.g., `/home/joe/Music`). + - `cd ..`: Up one level (e.g., from `/usr/share/doc` to `/usr/share`). + - `cd -`: Previous directory (`$OLDPWD`). +- **Key Concept**: `..` (parent) and `~` (home) simplify navigation. +- **Example**: + +```bash +$ cd /usr/share/ +$ pwd +/usr/share +$ cd doc +$ pwd +/usr/share/doc +$ cd +$ pwd +/home/chris +``` + + +#### 2.2 `pwd` - Print Working Directory + +- **Purpose**: Displays your current absolute path. +- **Usage**: `pwd → /home/joe`. +- **Use Case**: Confirms your location. + + +#### 2.3 `mkdir` - Make Directory + +- **Purpose**: Creates directories. +- **Usage**: + - `mkdir files`: Creates `files` in the current directory (e.g., `/home/joe/files`). + - `mkdir -p /tmp/test/docs`: Creates nested directories if parents don’t exist. +- **Use Case**: Organizing files. + + +#### 2.4 `chmod` - Change Permissions + +- **Purpose**: Sets file/directory permissions. +- **Usage**: `chmod 700 test → rwx------` (owner full access, others none). +- **Permissions**: `r` (4), `w` (2), `x` (1); `700 = 7` (owner), `0` (group), `0` (others). +- **Use Case**: Securing directories. + + +#### 2.5 `ls` - List Directory Contents + +- **Purpose**: Shows files and directories. +- **Usage**: + - `ls`: Non-hidden items. + - `ls -l`: Long listing (e.g., `drwxr-xr-x 2 joe sales 1024 Jan 24 12:17 test`). + - `ls -a`: Includes hidden files (e.g., `.hidden`). + - `ls -lrt`: Sorts by time, reversed (recent last). + - `ls -R`: Recursive listing. + - `ls -d`: Directory info only (e.g., `ls -ld test`). +- **Note**: Colors may vary (e.g., blue for dirs, green for executables). + + +### File Management Tasks + +Administrators perform key tasks to manage files effectively. + +#### 3.1 Working with Wildcards + +- **Purpose**: Match multiple files efficiently. +- **Wildcards**: + + +| Wildcard | Use | +| :-- | :-- | +| `*` | Refers to an unlimited number of any characters. `ls *` shows all files in the current directory (except those that have a name starting with a dot). | +| `?` | Used to refer to one specific character that can be any character. `ls c?t` would match `cat` as well as `cut`. | +| `[auo]` | Refers to one character that may be selected from the range that is specified between square brackets. `ls c[auo]t` would match `cat`, `cut`, and `cot`. | + +- **Use Case**: `touch apple banana grape → ls *e* → apple, grape`. + + +#### 3.2 Absolute and Relative Pathnames + +- **Absolute Pathname**: Complete path from the root directory `/` (e.g., `/home/lisa/file1`). + - Always works, regardless of the current directory. +- **Relative Pathname**: Path relative to the current directory (e.g., from `/home`, `lisa/file1 → /home/lisa/file1`). +- When working with relative filenames, `..` moves up one level (e.g., from `/home/lisa`, `cp file1 ../lara` copies `/home/lisa/file1` to `/home/lara`). + + +#### 3.3 Listing Files and Directories + +- **Enhanced `ls` Options**: + - `ls -a`: Shows hidden files (e.g., `.bashrc`). Hidden files start with a dot (`.`). + - `ls -lrt`: Recent files last. + - `ls -F`: Adds indicators (`/` for dirs, `*` for executables, `@` for links). + - `ls -d`: Shows the names of directories, not the contents of all directories that match the wildcards. + - `ls -R`: Shows the contents of the current directory and all its subdirectories recursively. +- **Permissions**: `drwxr-xr-x` (dir, owner `rwx`, group `rx`, others `rx`). + + +#### 3.4 Copying Files and Directories + +- **Command**: `cp <source> <dest>`. +- **Usage**: + - `cp /etc/hosts /tmp`: Copies `hosts` to `/tmp`. + - `cp -R /etc /tmp`: Copies directory recursively. + - `cp -a ~ /tmp`: Preserves permissions (archive mode). +- **Hidden Files**: + - `cp /somedir/.* /tmp`: Copies hidden files only. + - `cp -a /somedir/. .`: Copies all (hidden + regular) to the current directory. +- **Tip**: Use trailing `/` (e.g., `/tmp/`) to ensure the destination is a directory. + + +#### 3.5 Moving Files and Directories + +- **Command**: `mv <source> <dest>`. +- **Usage**: + - `mv file1 /tmp`: Moves to `/tmp`. + - `mv file1 file2`: Renames. + - `mv dir /tmp`: Moves directory (works with contents). +- **Note**: Combines copy and delete. + + +#### 3.6 Deleting Files and Directories + +- **Command**: `rm <target>`. +- **Usage**: + - `rm file1`: Deletes a file. + - `rm -r dir`: Deletes directory recursively. + - `rm -f`: Forces without prompting. +- **Note**: On RHEL, `rm` may alias to `rm -i` (prompts for confirmation); use `-f` to override. + + +### Metacharacters and Operators + +- **Brace Expansion**: `touch memo{1,2,3} → memo1 memo2 memo3`. +- **Redirection**: + - `>`: Overwrites file (e.g., `echo "test" > file`). + - `>>`: Appends (e.g., `echo "more" >> file`). + - `<`: Input from file (e.g., `mail root < file`). + - `<<WORD`: Here document (e.g., `cat <<END`). + + +### Practical Context + +- **Workflow**: `cd ~; mkdir test; chmod 700 test; cp -a test /tmp; rm -r /tmp/test`. +- **Permissions**: `ls -l` shows owner, group, and access (e.g., `rwxr-xr-x`). +- **Hidden Files**: Start with `.` (e.g., `.hidden`), visible with `ls -a`. + + +### Why This Matters + +- **Foundation**: Essential for Linux administration and scripting. +- **Efficiency**: Wildcards and operators save time. +- **DevOps/SRE**: Critical for server management and automation. +- **RHCSA**: Tests these skills explicitly. + + +### Key Takeaways for Learners + +- Navigate with `cd`, `pwd`. +- Create with `mkdir`, `touch`. +- Manage files with `cp`, `mv`, `rm`; secure with `chmod`. +- Use wildcards (`*`, `?`, `[]`) and paths (absolute/relative) for flexibility. +- List with `ls` variations (e.g., `-l`, `-a`, `-R`, `-d`). +- Understand permissions and file types via `ls -l`. From 324bb9166420c1ec56e52ee696d42437678207fa Mon Sep 17 00:00:00 2001 From: Sagar Utekar Date: Wed, 9 Apr 2025 06:25:29 +0530 Subject: [PATCH 060/133] Update core_filesystem_commands.md --- .../Season2/Day10/core_filesystem_commands.md | 79 +++++++++++++++++-- 1 file changed, 72 insertions(+), 7 deletions(-) diff --git a/DailyChallenges/Season2/Day10/core_filesystem_commands.md b/DailyChallenges/Season2/Day10/core_filesystem_commands.md index 6659a5c..9336ab0 100644 --- a/DailyChallenges/Season2/Day10/core_filesystem_commands.md +++ b/DailyChallenges/Season2/Day10/core_filesystem_commands.md @@ -101,13 +101,78 @@ Administrators perform key tasks to manage files effectively. #### 3.3 Listing Files and Directories -- **Enhanced `ls` Options**: - - `ls -a`: Shows hidden files (e.g., `.bashrc`). Hidden files start with a dot (`.`). - - `ls -lrt`: Recent files last. - - `ls -F`: Adds indicators (`/` for dirs, `*` for executables, `@` for links). - - `ls -d`: Shows the names of directories, not the contents of all directories that match the wildcards. - - `ls -R`: Shows the contents of the current directory and all its subdirectories recursively. -- **Permissions**: `drwxr-xr-x` (dir, owner `rwx`, group `rx`, others `rx`). +##### Detailed Explanation of `ls -la` + +The command `ls -la` is a powerful tool for listing files and directories in Linux with detailed information. It combines two options: + +- `-l`: Long listing format. +- `-a`: Show all files, including hidden files. + +When you run `ls -la`, you get a detailed view of all files and directories, including those that are hidden (i.e., those whose names start with a dot `.`). This is incredibly useful for examining your system's configuration files and hidden directories. + +##### Sample Output + +```bash +$ ls -la /home/joe +total 158 +drwxrwxrwx 2 joe sales 4096 May 12 13:55 . +drwxr-xr-x 3 root root 4096 May 10 01:49 .. +-rw------- 1 joe sales 2204 May 18 21:30 .bash_history +-rw-r--r-- 1 joe sales 24 May 10 01:50 .bash_logout +-rw-r--r-- 1 joe sales 230 May 10 01:50 .bash_profile +-rw-r--r-- 1 joe sales 124 May 10 01:50 .bashrc +drw-r--r-- 1 joe sales 4096 May 10 01:50 .kde +-rw-rw-r-- 1 joe sales 149872 May 11 22:49 letter +``` + + +##### Explanation of Columns + +| Column | Description | +| :-- | :-- | +| 1 | **File type and permissions**: The first character indicates the file type (e.g., `d` for directory, `-` for file, `l` for symbolic link). The next nine characters represent permissions for the owner, group, and others. | +| 2 | **Number of hard links**: The number of hard links to the file or directory. | +| 3 | **Owner**: The owner of the file or directory. | +| 4 | **Group**: The group associated with the file or directory. | +| 5 | **Size**: The size of the file in bytes. For directories, it is typically 4096 bytes, representing the size of the directory metadata. | +| 6 | **Last modified**: The date and time when the file was last modified. | +| 7 | **Name**: The name of the file or directory. | + +##### Key Details + +* **. (Current Directory)**: Represents the current directory (`/home/joe` in this example). +* **.. (Parent Directory)**: Represents the directory above the current directory (`/home` in this example). +* **Hidden Files**: Files starting with a dot (`.`) are hidden files and are typically configuration files or directories used by applications. + + +##### Special Characters in Permissions + +* **s (SetUID or SetGID)**: If you see an `s` instead of `x` in the permissions, it indicates a SetUID or SetGID program. + * SetUID: Allows any user to run the application with the owner's permissions. + * SetGID: Allows any user to run the application with the group's permissions. +* **t (Sticky Bit)**: If a `t` appears at the end of the permissions for a directory (e.g., `drwxrwxr-t`), it indicates the sticky bit is set. + * Sticky Bit: Allows users to add files to the directory but prevents them from deleting files owned by others. +* **+ (Extended Attributes)**: If a `+` appears at the end of the permission bits (e.g., `-rw-rw-r--+`), it means that extended attributes, such as Access Control Lists (ACLs) or SELinux, are set on the file. + + +##### Examples and Best Practices + +1. **Basic Usage** + +```bash +$ ls -la ~ +``` + +This command lists all files and directories in your home directory, including hidden ones, with detailed information. +2. **Checking Permissions** + +```bash +$ ls -la /etc/passwd +``` + +Use this to check the permissions, owner, and group of the `/etc/passwd` file. +3. **Troubleshooting** +If you're missing configuration files, running `ls -la` will show you if they exist and what their permissions are. #### 3.4 Copying Files and Directories From f332873e055ef2353f3c0889b291a7809301db01 Mon Sep 17 00:00:00 2001 From: Sagar Utekar Date: Wed, 9 Apr 2025 06:26:13 +0530 Subject: [PATCH 061/133] Update core_filesystem_commands.md --- DailyChallenges/Season2/Day10/core_filesystem_commands.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/DailyChallenges/Season2/Day10/core_filesystem_commands.md b/DailyChallenges/Season2/Day10/core_filesystem_commands.md index 9336ab0..e1b58b1 100644 --- a/DailyChallenges/Season2/Day10/core_filesystem_commands.md +++ b/DailyChallenges/Season2/Day10/core_filesystem_commands.md @@ -164,6 +164,7 @@ $ ls -la ~ ``` This command lists all files and directories in your home directory, including hidden ones, with detailed information. + 2. **Checking Permissions** ```bash @@ -171,6 +172,7 @@ $ ls -la /etc/passwd ``` Use this to check the permissions, owner, and group of the `/etc/passwd` file. + 3. **Troubleshooting** If you're missing configuration files, running `ls -la` will show you if they exist and what their permissions are. From 83bf08c41085a7855a5952e70532e3afa20f6dc0 Mon Sep 17 00:00:00 2001 From: Sagar Utekar Date: Wed, 9 Apr 2025 06:30:38 +0530 Subject: [PATCH 062/133] Update core_filesystem_commands.md --- .../Season2/Day10/core_filesystem_commands.md | 96 ++++++++++++++++--- 1 file changed, 85 insertions(+), 11 deletions(-) diff --git a/DailyChallenges/Season2/Day10/core_filesystem_commands.md b/DailyChallenges/Season2/Day10/core_filesystem_commands.md index e1b58b1..941050b 100644 --- a/DailyChallenges/Season2/Day10/core_filesystem_commands.md +++ b/DailyChallenges/Season2/Day10/core_filesystem_commands.md @@ -179,20 +179,94 @@ If you're missing configuration files, running `ls -la` will show you if they ex #### 3.4 Copying Files and Directories -- **Command**: `cp <source> <dest>`. -- **Usage**: - - `cp /etc/hosts /tmp`: Copies `hosts` to `/tmp`. - - `cp -R /etc /tmp`: Copies directory recursively. - - `cp -a ~ /tmp`: Preserves permissions (archive mode). -- **Hidden Files**: - - `cp /somedir/.* /tmp`: Copies hidden files only. - - `cp -a /somedir/. .`: Copies all (hidden + regular) to the current directory. -- **Tip**: Use trailing `/` (e.g., `/tmp/`) to ensure the destination is a directory. +- **Command**: `cp source dest;`. +- **Purpose**: The `cp` command allows you to copy files and directories from one location to another. +- **Basic Usage**: + - To copy a single file: + +```bash +cp /path/to/source/file /path/to/destination/ +``` + + - Example: + +```bash +cp /etc/hosts /tmp/ +``` + +This copies the file `/etc/hosts` to the `/tmp/` directory. +- **Options**: + + +| Option | Description | +| :-- | :-- | +| `-R` | Recursive: Copies directories and their contents recursively. | +| `-a` | Archive: Preserves file attributes (permissions, ownership, timestamps). It is equivalent to `-dpR`. | +| `-i` | Interactive: Prompts before overwriting existing files. | +| `-u` | Update: Copies only when the source file is newer than the destination file or when the destination file is missing. | +| `-v` | Verbose: Shows the files being copied. | + +- **Important Considerations**: + - **Target Directory**: Always ensure the target directory exists. If you copy a file to a non-existent directory without a trailing `/`, it will create a file named as the directory instead. + +```bash +cp /etc/hosts /tmp # If /tmp doesn't exist, it creates a file named 'tmp' +cp /etc/hosts /tmp/ # Correct way, ensures /tmp is treated as a directory +``` + + - **Copying Directories Recursively**: +To copy an entire directory, use the `-R` option. + +```bash +cp -R /etc /tmp +``` + +This copies the `/etc` directory and all its contents to the `/tmp` directory. + - **Preserving Attributes**: +To keep the original file permissions and other attributes, use the `-a` option (archive mode). + +```bash +cp -a ~ /tmp +``` + +This copies your entire home directory to the `/tmp` directory while preserving all file attributes. + - **Hidden Files**: +Hidden files (files starting with `.`) are not copied by default. Here's how to handle them: + +1. **Using `.*`**: + +```bash +cp /somedir/.* /tmp +``` + +This copies hidden files from `/somedir/` to `/tmp`. Note that this will produce an error for the `.` and `..` directories unless you use the `-R` option. +2. **Copying the entire directory**: + +```bash +cp -a /somedir/ . +``` + +This creates a `somedir` subdirectory in the current directory and copies all contents, including hidden files. +3. **Copying all files, including hidden files, into the current directory**: + +```bash +cp -a /somedir/. . +``` + +Make sure there is a space between the two dots. This copies all files and directories (including hidden ones) from /somedir into the current directory. + + +### Best Practices and Tips + +- Ensure the destination directory exists to avoid creating a file instead of copying into a directory. +- Use the -a option when you want to preserve file attributes like permissions and timestamps. +- Be cautious when using wildcards, and always double-check your command before executing to avoid unintended consequences. + #### 3.5 Moving Files and Directories -- **Command**: `mv <source> <dest>`. +- **Command**: `mv source dest`. - **Usage**: - `mv file1 /tmp`: Moves to `/tmp`. - `mv file1 file2`: Renames. @@ -202,7 +276,7 @@ If you're missing configuration files, running `ls -la` will show you if they ex #### 3.6 Deleting Files and Directories -- **Command**: `rm <target>`. +- **Command**: `rm target;`. - **Usage**: - `rm file1`: Deletes a file. - `rm -r dir`: Deletes directory recursively. From e474bfeaefc6ed5623ac3b8454008ced918b82b9 Mon Sep 17 00:00:00 2001 From: Sagar Utekar Date: Wed, 9 Apr 2025 06:35:49 +0530 Subject: [PATCH 063/133] Create 3_links.md --- DailyChallenges/Season2/Day10/3_links.md | 101 +++++++++++++++++++++++ 1 file changed, 101 insertions(+) create mode 100644 DailyChallenges/Season2/Day10/3_links.md diff --git a/DailyChallenges/Season2/Day10/3_links.md b/DailyChallenges/Season2/Day10/3_links.md new file mode 100644 index 0000000..60cadef --- /dev/null +++ b/DailyChallenges/Season2/Day10/3_links.md @@ -0,0 +1,101 @@ +## 1. Using Links + +Links on Linux are like aliases assigned to a file. There are symbolic links and hard links. To understand a link, you need to know how the Linux filesystem uses inodes for filesystem administration. + +### 1.1. Understanding Hard Links + +Linux stores administrative data about files in inodes. The inode stores all administrative data about files. Every file on Linux has an inode, and important information is stored in it: + +* The data block where the file contents are stored +* The creation, access, and modification date +* Permissions +* File owners + +The filename is not stored in the inode; names are stored in the directory. Each filename knows which inode it has to address to access further file information. An inode knows how many names are associated with it; these names are referred to as **hard links**. Every file always has one hard link to start with, which is the name of the file. When you create a file, you are creating a hard link. On a Linux filesystem, multiple hard links can be created to a file, which is useful if a file with the same contents needs to be available at multiple locations. If a change is applied to any one of the hard links, it will show in all other hard links because all hard links point to the same data blocks. + +Restrictions for hard links: + +* Hard links must exist all on the same device (partition, logical volume, etc.). +* You cannot create hard links to directories. + +When the last hard link to a file is removed, access to the file’s data is also removed. There is no difference between the first hard link and the second hard link; they are both just hard links. + +#### Inode Diagram + +Here's a simple representation of how hard links and inodes work: + + +----------+ +---------+ + | Directory| --; | Inode | --; Data Blocks (File Content) + +----------+ +---------+ + ^ + | + +----------+ + |Hard Link | + +----------+ + +### 1.2. Understanding Symbolic Links + +A **symbolic link** (also referred to as a soft link) does not link directly to the inode but to the name of the file. This makes symbolic links more flexible but has some disadvantages. The advantage of symbolic links is that they can link to files on other devices and directories. The major disadvantage is that when the original file is removed, the symbolic link becomes invalid. + +#### Symbolic Link Diagram + + +----------+ +--------------+ +----------+ +---------+ + |Directory | --; | Symbolic Link| --; |Directory | --; | Inode | --; Data Blocks + +----------+ +--------------+ +----------+ +---------+ + (Points to name) + +### 1.3. Creating Links + +Use the `ln` command to create links. It uses the same order of parameters as `cp` and `mv`; first, you mention the source name, followed by the destination name. If you want to create a symbolic link, use the option `-s`, and then specify the source and target file or directory. One important restriction applies: to be able to create hard links, you must be the owner of the item that you want to link to. + + +| Command | Explanation | +| :-- | :-- | +| `ln /etc/hosts .` | Creates a hard link to the file `/etc/hosts` in the current directory | +| `ln -s /etc/hosts .` | Creates a symbolic link to the file `/etc/hosts` in the current directory | +| `ln -s /home /tmp` | Creates a symbolic link to the directory `/home` in the directory `/tmp` | + +The `ls` command will reveal whether a file is a link: + +* In the output of the `ls -l` command, the first character is an `l` if the file is a symbolic link. +* If a file is a symbolic link, the output of `ls -l` shows the name of the item it links to after the filename. +* If a file is a hard link, `ls -l` shows the hard link counter. + +[root@localhost tmp]\# \ls -l +total 3 +lrwxrwxrwx. 1 root root 5 Jan 19 04:38 home -> /home +-rw-r--r--. 3 root root 158 Jun 7 2013 hosts + +In the example above, `home` is a symbolic link pointing to `/home`, and `hosts` is a file with 3 hard links. + +### 1.4. Removing Links + +Removing links can be dangerous. Let’s consider the following procedure: + +1. Make a directory named `test` in your home directory: `mkdir ~/test` +2. Copy all files that have a name starting with a, b, c, d, or e from `/etc` to this directory: `cp /etc/[a-e]* ~/test` +3. Type `ls -l ~/test/` to verify the contents of the `test` directory. +4. Make sure that you are in your home directory, by using `cd` without arguments. +5. Type `ln -s test link` +6. Type `rm link`. This removes the symbolic link. (Do not use `-r` or `-f` to remove symbolic links, even if they are subdirectories.) +7. Type `ls -l`. You’ll see that the symbolic link has been removed. +8. Let’s do it again. Type `ln -s test link` to create the link again. +9. Type `rm -rf link/` (which is what you would get by using Bash command-line completion). +10. Type `ls`. You’ll see that the directory `link` still exists. +11. Type `ls test/`. You’ll see the directory `test` is now empty. + +### 1.5. Exercise: Working with Symbolic Links and Hard Links + +1. Open a shell as the `student` user. +2. From your home directory, type `ln /etc/passwd .` (Make sure that the command ends with a dot that has a space before it!). This command gives you an “operation not permitted” error because you are not the owner of `/etc/passwd`. +3. Type `ln -s /etc/passwd .` (Again, make sure that the command ends with a space and a dot!). This works; you do not have to be the owner to create a symbolic link. +4. Type `ln -s /etc/hosts` (this time with no dot at the end of the command). You’ll notice this command also works. If the target is not specified, the link is created in the current directory. +5. Type `touch newfile` and create a hard link to this file by using `ln newfile linkedfile`. +6. Type `ls -l` and notice the link counter for `newfile` and `linkedfile`, which is currently set to `2`. +7. Type `ln -s newfile symlinkfile` to create a symbolic link to `newfile`. +8. Type `rm newfile` +9. Type `cat symlinkfile`. You will get a “no such file or directory” error message because the original file could not be found. +10. Type `cat linkedfile`. This gives no problem. +11. Type `ls -l` and look at the way the `symlinkfile` is displayed. Also, look at `linkedfile`, which now has the link counter set to `1`. +12. Type `ln linkedfile newfile` +13. Type `ls -l` again. You’ll see that the original situation has been restored. From 829671ee74d74c76afaf084cb7ddb3d3a43dac20 Mon Sep 17 00:00:00 2001 From: Sagar Utekar Date: Wed, 9 Apr 2025 06:36:14 +0530 Subject: [PATCH 064/133] Rename core_filesystem_commands.md to 1_core_filesystem_commands.md --- ...{core_filesystem_commands.md => 1_core_filesystem_commands.md} | 0 1 file changed, 0 insertions(+), 0 deletions(-) rename DailyChallenges/Season2/Day10/{core_filesystem_commands.md => 1_core_filesystem_commands.md} (100%) diff --git a/DailyChallenges/Season2/Day10/core_filesystem_commands.md b/DailyChallenges/Season2/Day10/1_core_filesystem_commands.md similarity index 100% rename from DailyChallenges/Season2/Day10/core_filesystem_commands.md rename to DailyChallenges/Season2/Day10/1_core_filesystem_commands.md From f5fce44e2e3f8ac6db1c6202b6447edb95525f82 Mon Sep 17 00:00:00 2001 From: Sagar Utekar Date: Wed, 9 Apr 2025 06:36:31 +0530 Subject: [PATCH 065/133] Rename mounts_in_linux.md to 2_mounts_in_linux.md --- .../Season2/Day10/{mounts_in_linux.md => 2_mounts_in_linux.md} | 0 1 file changed, 0 insertions(+), 0 deletions(-) rename DailyChallenges/Season2/Day10/{mounts_in_linux.md => 2_mounts_in_linux.md} (100%) diff --git a/DailyChallenges/Season2/Day10/mounts_in_linux.md b/DailyChallenges/Season2/Day10/2_mounts_in_linux.md similarity index 100% rename from DailyChallenges/Season2/Day10/mounts_in_linux.md rename to DailyChallenges/Season2/Day10/2_mounts_in_linux.md From 822bc4d4dac72a0901b6d03d34dacd8b6fb3b727 Mon Sep 17 00:00:00 2001 From: Sagar Utekar Date: Wed, 9 Apr 2025 06:39:39 +0530 Subject: [PATCH 066/133] Create 4_compression_archive.md --- .../Season2/Day10/4_compression_archive.md | 71 +++++++++++++++++++ 1 file changed, 71 insertions(+) create mode 100644 DailyChallenges/Season2/Day10/4_compression_archive.md diff --git a/DailyChallenges/Season2/Day10/4_compression_archive.md b/DailyChallenges/Season2/Day10/4_compression_archive.md new file mode 100644 index 0000000..9c827e0 --- /dev/null +++ b/DailyChallenges/Season2/Day10/4_compression_archive.md @@ -0,0 +1,71 @@ +# Working with Archives and Compressed Files + +This document provides a guide on how to manage archives and compressed files on a Linux system using the `tar` command, `gzip`, `bzip2`, and `xz` utilities. + +## Managing Archives with tar + +The `tar` (Tape Archiver) utility is used to create, list, extract, compress, and uncompress archives. + +### Creating Archives + +- Syntax: `tar -cf archivename.tar /files-you-want-to-archive` +- `-c`: Creates an archive. +- `-v`: Shows verbose output. +- Example: `tar -cvf /root/homes.tar /home` (archives the /home directory to /root/homes.tar) +- `-r`: Adds a file to an existing archive. + - Example: `tar -rvf /root/homes.tar /etc/hosts` +- `-u`: Updates an existing archive with newer files. + - Example: `tar -uvf /root/homes.tar /home` + +### Monitoring and Extracting tar Files + +- `-t`: Lists the contents of an archive. + - Example: `tar -tvf /root/homes.tar` +- `file` command: Analyzes a file and shows its type. + - Example: `file somefile` +- `-x`: Extracts an archive. + - Syntax: `tar -xvf /archivename.tar` (extracts to the current directory) +- `-C /targetdir`: Specifies the target directory for extraction. + - Example: `tar -xvf homes.tar -C /tmp` (extracts to /tmp) +- Extracting a single file: `tar -xvf /archivename.tar file-you-want-to-extract` + - Example: `tar -xvf /root/etc.tar etc/hosts` + +## Using Compression + +- `gzip`: Compresses files. + - Example: `gzip home.tar` (creates home.tar.gz) +- `bzip2`: An alternative compression utility. +- `xz`: Another alternative for compression. +- `gunzip`: Decompresses files compressed with gzip. +- `bunzip2`: Decompresses files compressed with bzip2. + +### tar Options for Compression + +- `-z`: Compresses/decompresses using gzip. +- `-j`: Compresses/decompresses using bzip2. +- `-J`: Compresses/decompresses using xz. + +## Overview of tar Options + +| Option | Use | +| :----- | :----------------------------------------------------------------- | +| c | Creates an archive. | +| v | Shows verbose output while tar is working. | +| t | Shows the contents of an archive. | +| z | Compresses/decompresses the archive using gzip. | +| j | Compresses/decompresses the archive using bzip2. | +| J | Compresses/decompresses the archive using xz. | +| x | Extracts an archive. | +| u | Updates an archive; only newer files will be written to the archive. | +| C | Changes the working directory before performing the command. | +| r | Appends files to an archive. | + +## Exercise + +1. Archive `/etc` directory: `tar cvf etc.tar /etc` +2. Compress the tar file: `gzip etc.tar` +3. Extract a file: `tar xvf etc.tar.gz etc/hosts` +4. Decompress the file: `gunzip etc.tar.gz` +5. Extract to a specific directory: `tar xvf etc.tar -C /tmp etc/passwd` +6. Create a compressed archive of the home directory: `tar cjvf homes.tar /home` +7. Remove temporary files: `rm -f *gz *tar` From 85ab47081271022b2bdd3eb619b4b1693c30ebb8 Mon Sep 17 00:00:00 2001 From: Sagar Utekar Date: Wed, 9 Apr 2025 06:59:22 +0530 Subject: [PATCH 067/133] Update readme.md --- DailyChallenges/Season2/Day10/readme.md | 122 ++++++++++++++++++++++++ 1 file changed, 122 insertions(+) diff --git a/DailyChallenges/Season2/Day10/readme.md b/DailyChallenges/Season2/Day10/readme.md index 6b1436c..286138c 100644 --- a/DailyChallenges/Season2/Day10/readme.md +++ b/DailyChallenges/Season2/Day10/readme.md @@ -57,3 +57,125 @@ The Linux filesystem is a hierarchical structure used to store all information o - **`/var`**: Contains directories of data used by various applications, including FTP server files (`/var/ftp`), web server files (`/var/www`), system log files (`/var/log`), and spool files (`/var/spool`). +## Challenge Breakdown + +### ** Theoretical Questions** +### Scenario + +You are a Linux system administrator entrusted with managing an EC2 instance for a software development team. Your goal is to master essential Linux skills, secure the environment, and prepare for real-world challenges, using practical exercises instead of focusing on specific certifications or exam preparation. + +### I. Theory + +These questions are designed to assess your understanding of fundamental Linux concepts. + +1. You need to locate configuration files. Where do you look? +2. How do you list the newest files first in a directory? +3. Rename `myfile` to `yourfile`. +4. Wipe an entire directory structure. +5. Create a link to `/tmp` in your home directory. +6. Copy files starting with a, b, or c from `/etc` to your current directory. +7. Create a link to `/etc` in your home directory. +8. Safely remove a symbolic link to a directory. +9. Create a compressed archive of `/etc` and `/home` as `/tmp/etchome.tgz`. +10. Extract `/etc/passwd` from `/tmp/etchome.tgz`. + +### II. Practical + +These tasks require you to perform specific actions on a Linux system. + +#### A. Storage Management + +1. Attach a new EBS volume to your EC2 instance. +2. Create a partition on the new disk. +3. Create an ext4 filesystem on the partition. +4. Mount the filesystem to a directory named `/data`. +5. Verify the mount using `df -h` and `lsblk`. + +#### B. Basic File & Directory Operations (as `ec2-user`) + +1. Log in as the `ec2-user`. +2. Use `pwd` to verify you're in `/home/ec2-user`. +3. Create `newfiles` and `oldfiles` directories. +4. Create a hidden file `.hidden` and a regular file `unhidden` within `newfiles`. + +#### C. Advanced File & Directory Manipulation (as `ec2-user`) + +1. From `oldfiles`, copy `newfiles` (including hidden files) into `oldfiles` using `cp -a ../newfiles/ .`. +2. Remove `newfiles` from within `oldfiles` using `rm -rf newfiles`. +3. Copy only visible files from `../newfiles` into `oldfiles` using `cp -a ../newfiles/* .`. +4. Copy hidden files separately using `cp -a ../newfiles/. .`. +5. In your home directory, create `projects`. Within it, create nine empty files named `house1` through `house9`. List *only* those files, even with other files present. +6. Create the path `$HOME/projects/houses/doors/`. Create these files: + * `$HOME/projects/houses/bungalow.txt` + * `$HOME/projects/houses/doors/bifold.txt` + * `$HOME/projects/outdoors/vegetation/landscape.txt` +7. Copy `house1` and `house5` to `$HOME/projects/houses/`. +8. Recursively copy `/usr/share/doc/initscripts*` to `$HOME/projects/`, preserving attributes. +9. Recursively list `$HOME/projects/`, paging with `less`. +10. Remove `house6`, `house7`, and `house8` non-interactively. +11. Move `house3` and `house4` to `$HOME/projects/houses/doors/`. +12. Remove `$HOME/projects/houses/doors/` and its contents. +13. Set permissions on `$HOME/projects/house2`: owner read/write, group read-only, others no access. +14. Recursively restrict write access to `$HOME/projects/` for everyone. + +#### D. User & Group Management and Collaboration (as root, `alice`, and `bob`) + +1. As root, create a `developers` group. +2. As root, create users `alice` and `bob`, adding them to `developers`. +3. As root, create `/data/shared_space`. +4. As root, set group ownership of `/data/shared_space` to `developers` with `chgrp`. +5. As root, set permissions on `/data/shared_space` to 770. +6. As root, enable the SGID bit on `/data/shared_space`. +7. As `alice`, create `alice_file.txt` in `/data/shared_space`. +8. As `bob`, create `bob_file.txt` in `/data/shared_space`. +9. Confirm both files are group-owned by `developers`. + +#### E. Link Exploration (as `ec2-user`) + +1. Attempt to hard link `/etc/passwd` in your home directory (expect permission denied). +2. Create a symbolic link to `/etc/passwd` using `ln -s`. +3. Create a symbolic link to `/etc/hosts` (without specifying a target directory). +4. Create `newfile` using `touch`. +5. Create a hard link `linkedfile` to `newfile` using `ln`. +6. Use `ls -l` to view link counts. +7. Create a symbolic link `symlinkfile` to `newfile` using `ln -s`. +8. Remove `newfile` using `rm`. +9. Attempt `cat symlinkfile` (expect error). +10. `cat linkedfile`. +11. Use `ls -l` to examine `symlinkfile` and `linkedfile`. +12. Restore `newfile` using `ln linkedfile newfile`. +13. Verify the original state using `ls -l`. + +#### F. Archiving & Compression (as root) + +1. Create an uncompressed tar archive of `/etc` as `/root/etc.tar` using `tar cvf`. +2. Verify the archive type using `file`. +3. Compress `/root/etc.tar` to `/root/etc.tar.gz` using `gzip`. +4. List contents of `/root/etc.tar.gz` without extracting using `tar tvf`. +5. Extract `/etc/hosts` only from `/root/etc.tar.gz` to `/root`. +6. Use `ls -R` to see the extracted file in `/root/etc/`. +7. Decompress `/root/etc.tar.gz` using `gunzip`. +8. Extract `/etc/passwd` from `/root/etc.tar` to `/tmp`, preserving directory structure, using `tar xvf -C`. +9. Verify using `ls -l /tmp/etc/passwd`. +10. Create a bzip2 compressed archive of `/home` as `/root/homes.tar.bz2` using `tar cjvf`. +11. Remove archive files from `/root`. + +#### G. Root Shell Operations & File Management + +1. As user `student`, use `sudo -i` to become root. Create `/root/essentials.tar` containing `/home` and `/etc`. +2. Copy the archive to `/tmp`. Create a hard link to the archive in `/`. +3. Rename `/essentials.tar` to `/archive.tar`. +4. Create a symbolic link `link.tar` in `/root` pointing to `/archive.tar`. +5. Remove `/archive.tar`. Observe the symbolic link. Remove the symbolic link. +6. Compress `/root/essentials.tar`. + +### Deliverables + +* Answers to all theory questions. +* A script (or well-documented sequence of commands) that accomplishes all practical tasks. +* Screenshots showing successful execution of key tasks. +* Contents of `/etc/fstab` (if modified). +* Explanations of commands, reasoning, and troubleshooting. +* Demonstration of SGID functionality. +* Verification steps for links and archiving. +* Observations of symbolic link behavior after target removal. From 6c8903ec7d9d1018a7ce4bbe052d36cb3c257acd Mon Sep 17 00:00:00 2001 From: Sagar Utekar Date: Wed, 9 Apr 2025 07:11:05 +0530 Subject: [PATCH 068/133] Update readme.md --- DailyChallenges/Season2/Day10/readme.md | 269 +++++++++++------------- 1 file changed, 118 insertions(+), 151 deletions(-) diff --git a/DailyChallenges/Season2/Day10/readme.md b/DailyChallenges/Season2/Day10/readme.md index 286138c..a3f560f 100644 --- a/DailyChallenges/Season2/Day10/readme.md +++ b/DailyChallenges/Season2/Day10/readme.md @@ -7,14 +7,12 @@ Welcome to Day 10 of the Daily DevOps + SRE Challenge Series - Season 2! 🎉 Today, we’re exploring the Linux filesystem—its structure, navigation, and essential file management tools. You’ll master commands to move around, create, copy, move, and delete files, all while building skills critical for DevOps and SRE workflows. By the end, you’ll: - Understand the filesystem hierarchy and path navigation. - Gain fluency in tools like `cd`, `mkdir`, `cp`, `mv`, and `rm`. -- Apply these on your Day 1 EC2 instances or RHEL VM. #### Why This Matters? -The filesystem is the foundation of Linux operations. Whether you’re deploying apps, troubleshooting outages, or prepping for RHCSA, these skills are non-negotiable. Here’s why: +Whether you’re deploying apps, troubleshooting outages, or writing scripts these skills are non-negotiable. Here’s why: - **Universal Skill**: Every Linux system—from cloud instances to Kubernetes pods—relies on the filesystem. Commands like `cd /var/log` or `rm -rf /tmp/cache` work everywhere. - **Automation Backbone**: Scripts like `mv *.log /archive/$(date +%Y%m%d)` organize logs daily in cron jobs or CI/CD pipelines. - **SRE Efficiency**: During incidents, `cp -r /etc/nginx /backup` or `find / -name core.*` pinpoints crash dumps fast, saving downtime. -- **RHCSA Must-Know**: The exam tests navigation (`/etc` vs `../`), file ops (`touch`, `mkdir -p`), and cleanup (`rm -rf`). - **Career Boost**: Fluency in filesystem tasks impresses in interviews—think “How do you restore a config?” or “How do you free disk space?” - **Real-World Example**: An SRE at a fintech firm used `mkdir -p /data/backups/$(date +%F) && cp -r /prod/data/* /data/backups/$(date +%F)` to snapshot critical data before a risky upgrade, averting disaster when it failed. @@ -22,160 +20,129 @@ Let’s dive in and take control of the filesystem! --- - -### Linux Filesystem Overview - -The Linux filesystem is a hierarchical structure used to store all information on a computer. It is based on the UNIX system, where nearly everything, including data, commands, symbolic links, devices, and directories, is represented within the filesystem. - -### Structure and Navigation -Screenshot 2025-04-09 at 5 57 43 AM - -1. **Hierarchy**: The filesystem is organized like an upside-down tree with the root directory (`/`) at the top. Below the root are common directories such as `/bin`, `/dev`, `/home`, `/lib`, and `/tmp`. These directories can contain subdirectories, creating a hierarchical structure. -2. **Paths**: - * **Full Path**: Specifies the complete route from the root directory to a file or directory (e.g., `/home/joe/Documents/memos/memo1.doc`). - * **Relative Path**: Specifies the path relative to the current directory (e.g., `memo1.doc` if the current directory is `/home/joe/Documents/memos`). - -### Common Linux Directories - -#### Key Directories - -- **`/bin`**: Contains common Linux user commands like `ls`, `sort`, `date`, and `chmod`. -- **`/boot`**: Contains the bootable Linux kernel and boot loader configuration files (GRUB). -- **`/dev`**: Contains files representing access points to devices (e.g., `tty*`, `fd*`, `hd*`, `sd*`, `ram*`, `cd*`). -- **`/etc`**: Contains administrative configuration files, which are typically plaintext files that can be edited with a text editor. -- **`/home`**: Contains directories for each regular user account (except root, which uses `/root`). -- **`/media`**: Standard location for automatically mounting devices, particularly removable media. -- **`/lib`**: Contains shared libraries required by applications in `/bin` and `/sbin` to boot the system. -- **`/mnt`**: A common mount point for devices, often used for temporarily mounting local or remote filesystems. -- **`/misc`**: Sometimes used to automatically mount filesystems upon request. -- **`/opt`**: Used to store add-on application software. -- **`/proc`**: Contains information about system resources. -- **`/root`**: The home directory for the root user, located outside `/home` for security reasons. -- **`/sbin`**: Contains administrative commands and daemon processes. -- **`/tmp`**: Contains temporary files used by applications. -- **`/usr`**: Contains user documentation, games, graphical files (X11), libraries, and other commands and files not needed during the boot process. -- **`/var`**: Contains directories of data used by various applications, including FTP server files (`/var/ftp`), web server files (`/var/www`), system log files (`/var/log`), and spool files (`/var/spool`). - - ## Challenge Breakdown +Today, you’re not just learning—you’re on a mission! Imagine you’re an SRE tasked with securing a critical server after a chaotic deployment. You’ll navigate treacherous filesystem paths, manage secret files, attach and secure new storage, and even collaborate with rogue agents (users). By the end, you’ll: +- Navigate and manipulate the filesystem like a stealth operative. +- Secure files, directories, and mounts under pressure. +- Execute advanced ops—linking, archiving, and user management—to outsmart the chaos. -### ** Theoretical Questions** -### Scenario +--- -You are a Linux system administrator entrusted with managing an EC2 instance for a software development team. Your goal is to master essential Linux skills, secure the environment, and prepare for real-world challenges, using practical exercises instead of focusing on specific certifications or exam preparation. +## Challenge Description +You’re securing a server after a botched update. Your mission: navigate the filesystem, manage critical files, secure a new storage vault, collaborate with agents, and archive secrets—all while leaving no trace. Ready? -### I. Theory +## Theory Questions +Before you dive into the heist, test your knowledge with these questions. They’re based on the filesystem concepts you’ll need to succeed. Write your answers in a file `~/heist_vault/theory_answers.txt` and check them against the solutions later. -These questions are designed to assess your understanding of fundamental Linux concepts. +1. **Filesystem Hierarchy**: What is the purpose of the `/var` directory, and why might it be mounted on a separate device? +2. **Paths**: Explain the difference between an absolute path and a relative path. Give an example of each that reaches `/home/user/docs` from `/usr/share`. +3. **Wildcards**: If you run `ls f[io]le*` in a directory containing `file1`, `fole2`, `file3`, and `flea`, what files will be listed? +4. **Permissions**: What does `chmod 640 file.txt` do to the permissions of `file.txt`? Describe the resulting access for owner, group, and others. +5. **Mounts**: Why might an administrator mount `/home` with the `noexec` option? What does this prevent? +6. **Copying Files**: How does `cp -a /dir1 /dir2` differ from `cp -r /dir1 /dir2` when copying a directory? +7. **Links**: What’s the difference between a hard link and a symbolic link? Why might a hard link to `/etc/passwd` fail when created in your home directory? +8. **Redirection**: What does `echo "Log entry" >> /var/log/mylog` do? How is it different from using `>`? +9. **Mount Commands**: Compare the output of `mount`, `df -h`, and `findmnt`. Which would you use to check available disk space? +10. **SGID**: What happens to the group ownership of files created in a directory with the SGID bit set (e.g., `chmod g+s`)? -1. You need to locate configuration files. Where do you look? -2. How do you list the newest files first in a directory? -3. Rename `myfile` to `yourfile`. -4. Wipe an entire directory structure. -5. Create a link to `/tmp` in your home directory. -6. Copy files starting with a, b, or c from `/etc` to your current directory. -7. Create a link to `/etc` in your home directory. -8. Safely remove a symbolic link to a directory. -9. Create a compressed archive of `/etc` and `/home` as `/tmp/etchome.tgz`. -10. Extract `/etc/passwd` from `/tmp/etchome.tgz`. +--- ### II. Practical -These tasks require you to perform specific actions on a Linux system. - -#### A. Storage Management - -1. Attach a new EBS volume to your EC2 instance. -2. Create a partition on the new disk. -3. Create an ext4 filesystem on the partition. -4. Mount the filesystem to a directory named `/data`. -5. Verify the mount using `df -h` and `lsblk`. - -#### B. Basic File & Directory Operations (as `ec2-user`) - -1. Log in as the `ec2-user`. -2. Use `pwd` to verify you're in `/home/ec2-user`. -3. Create `newfiles` and `oldfiles` directories. -4. Create a hidden file `.hidden` and a regular file `unhidden` within `newfiles`. - -#### C. Advanced File & Directory Manipulation (as `ec2-user`) - -1. From `oldfiles`, copy `newfiles` (including hidden files) into `oldfiles` using `cp -a ../newfiles/ .`. -2. Remove `newfiles` from within `oldfiles` using `rm -rf newfiles`. -3. Copy only visible files from `../newfiles` into `oldfiles` using `cp -a ../newfiles/* .`. -4. Copy hidden files separately using `cp -a ../newfiles/. .`. -5. In your home directory, create `projects`. Within it, create nine empty files named `house1` through `house9`. List *only* those files, even with other files present. -6. Create the path `$HOME/projects/houses/doors/`. Create these files: - * `$HOME/projects/houses/bungalow.txt` - * `$HOME/projects/houses/doors/bifold.txt` - * `$HOME/projects/outdoors/vegetation/landscape.txt` -7. Copy `house1` and `house5` to `$HOME/projects/houses/`. -8. Recursively copy `/usr/share/doc/initscripts*` to `$HOME/projects/`, preserving attributes. -9. Recursively list `$HOME/projects/`, paging with `less`. -10. Remove `house6`, `house7`, and `house8` non-interactively. -11. Move `house3` and `house4` to `$HOME/projects/houses/doors/`. -12. Remove `$HOME/projects/houses/doors/` and its contents. -13. Set permissions on `$HOME/projects/house2`: owner read/write, group read-only, others no access. -14. Recursively restrict write access to `$HOME/projects/` for everyone. - -#### D. User & Group Management and Collaboration (as root, `alice`, and `bob`) - -1. As root, create a `developers` group. -2. As root, create users `alice` and `bob`, adding them to `developers`. -3. As root, create `/data/shared_space`. -4. As root, set group ownership of `/data/shared_space` to `developers` with `chgrp`. -5. As root, set permissions on `/data/shared_space` to 770. -6. As root, enable the SGID bit on `/data/shared_space`. -7. As `alice`, create `alice_file.txt` in `/data/shared_space`. -8. As `bob`, create `bob_file.txt` in `/data/shared_space`. -9. Confirm both files are group-owned by `developers`. - -#### E. Link Exploration (as `ec2-user`) - -1. Attempt to hard link `/etc/passwd` in your home directory (expect permission denied). -2. Create a symbolic link to `/etc/passwd` using `ln -s`. -3. Create a symbolic link to `/etc/hosts` (without specifying a target directory). -4. Create `newfile` using `touch`. -5. Create a hard link `linkedfile` to `newfile` using `ln`. -6. Use `ls -l` to view link counts. -7. Create a symbolic link `symlinkfile` to `newfile` using `ln -s`. -8. Remove `newfile` using `rm`. -9. Attempt `cat symlinkfile` (expect error). -10. `cat linkedfile`. -11. Use `ls -l` to examine `symlinkfile` and `linkedfile`. -12. Restore `newfile` using `ln linkedfile newfile`. -13. Verify the original state using `ls -l`. - -#### F. Archiving & Compression (as root) - -1. Create an uncompressed tar archive of `/etc` as `/root/etc.tar` using `tar cvf`. -2. Verify the archive type using `file`. -3. Compress `/root/etc.tar` to `/root/etc.tar.gz` using `gzip`. -4. List contents of `/root/etc.tar.gz` without extracting using `tar tvf`. -5. Extract `/etc/hosts` only from `/root/etc.tar.gz` to `/root`. -6. Use `ls -R` to see the extracted file in `/root/etc/`. -7. Decompress `/root/etc.tar.gz` using `gunzip`. -8. Extract `/etc/passwd` from `/root/etc.tar` to `/tmp`, preserving directory structure, using `tar xvf -C`. -9. Verify using `ls -l /tmp/etc/passwd`. -10. Create a bzip2 compressed archive of `/home` as `/root/homes.tar.bz2` using `tar cjvf`. -11. Remove archive files from `/root`. - -#### G. Root Shell Operations & File Management - -1. As user `student`, use `sudo -i` to become root. Create `/root/essentials.tar` containing `/home` and `/etc`. -2. Copy the archive to `/tmp`. Create a hard link to the archive in `/`. -3. Rename `/essentials.tar` to `/archive.tar`. -4. Create a symbolic link `link.tar` in `/root` pointing to `/archive.tar`. -5. Remove `/archive.tar`. Observe the symbolic link. Remove the symbolic link. -6. Compress `/root/essentials.tar`. - -### Deliverables - -* Answers to all theory questions. -* A script (or well-documented sequence of commands) that accomplishes all practical tasks. -* Screenshots showing successful execution of key tasks. -* Contents of `/etc/fstab` (if modified). -* Explanations of commands, reasoning, and troubleshooting. -* Demonstration of SGID functionality. -* Verification steps for links and archiving. -* Observations of symbolic link behavior after target removal. +### Task 1: Infiltrate the Filesystem (as `ec2-user`) +- Confirm your entry point with `pwd` (expect `/home/ec2-user`). +- Infiltrate `/var/log` using an absolute path. +- Retreat to `/var` with a relative move. +- Slip back to base (`~`) using a shortcut. +- In one command, hit `/tmp` then bounce back to base, verifying with `pwd`. + +### Task 2: Set Up the Hideout (as `ec2-user`) +- Establish `heist_vault` in `~`. +- Create subdirs `newfiles` and `oldfiles` in one command. +- In `newfiles`, plant a hidden file `.secret` and a decoy `decoy.txt` using `touch`. +- Secure `newfiles` with `chmod 700` and verify with `ls -ld`. + +### Task 3: Secure the New Vault - Storage Management (Root) +- **Agent Brief**: A new 1GB EBS volume arrives (or simulate with a loop device). + 1. As root: If on AWS, attach a 1GB EBS volume via the console; note the device (e.g., `/dev/xvdf`). If not, create a 1GB loop device: `dd if=/dev/zero of=/root/disk.img bs=1M count=1000; losetup /dev/loop0 /root/disk.img`. + 2. Partition it: `fdisk /dev/xvdf` (or `/dev/loop0`) → `n` (new), `p` (primary), `1` (partition 1), defaults, `w` (write). + 3. Format as ext4: `mkfs.ext4 /dev/xvdf1` (or `/dev/loop0p1`). + 4. Mount at `/data`: `mkdir /data; mount /dev/xvdf1 /data` (or `/dev/loop0p1`). + 5. Verify: `df -h` (shows `/data` usage) and `lsblk` (lists device tree). +- **Tip**: If `lsblk` shows no partitions, recheck `fdisk` steps. + +### Task 4: Advanced File Ops Under Pressure (as `ec2-user`) +- From `heist_vault/oldfiles`: + 1. Copy `newfiles` (including `.secret`) into `oldfiles` with `cp -a ../newfiles/ .`. + 2. Remove the nested `newfiles` with `rm -rf newfiles`. + 3. Copy only visible files from `../newfiles` using `cp -a ../newfiles/* .`. + 4. Copy hidden files separately with `cp -a ../newfiles/. .` and verify with `ls -a`. +- In `~`: + 5. Create `projects` with `house1` to `house9` using brace expansion. + 6. List only `house*` files (exclude others like `heist_vault`). + 7. Build `$HOME/projects/houses/doors/` and plant: + - `$HOME/projects/houses/bungalow.txt` + - `$HOME/projects/houses/doors/bifold.txt` + - `$HOME/projects/outdoors/vegetation/landscape.txt` + 8. Copy `house1` and `house5` to `houses/`. + 9. Recursively copy `/usr/share/doc/initscripts*` to `projects/` with `-a`. + 10. List `projects/` recursively, paging with `less`. + 11. Wipe `house6` to `house8` non-interactively. + 12. Move `house3` and `house4` to `doors/`. + 13. Obliterate `doors/` and its contents. + 14. Set `house2` perms to owner `rw`, group `r`, others none. + 15. Recursively block write access to `projects/` for all. + +### Task 5: Agent Collaboration (Root + Users) +- As root: + 1. Create group `agents`. + 2. Add users `alice` and `bob` to `agents` with home dirs and passwords. + 3. Establish `/data/shared_space`. + 4. Assign group `agents` with `chgrp`. + 5. Set perms to `770`. + 6. Enable SGID with `chmod g+s`. +- As `alice`: Create `alice_file.txt` in `/data/shared_space`. +- As `bob`: Create `bob_file.txt` in `/data/shared_space`. +- Verify: `ls -l` shows both files group-owned by `agents`. + +### Task 6: Cover Your Tracks with Links (as `ec2-user`) +- In `~`: + 1. Try hard-linking `/etc/passwd` to `passwd_hard` (expect denial). + 2. Soft-link `/etc/passwd` to `passwd_link`. + 3. Soft-link `/etc/hosts` without a target dir (default to current dir). +- In `heist_vault`: + 4. Create `evidence.txt`. + 5. Hard-link it to `evidence_copy` and check link count with `ls -l`. + 6. Soft-link it to `evidence_sym`. + 7. Delete `evidence.txt`. + 8. Test `cat evidence_sym` (broken) and `cat evidence_copy` (works). + 9. Restore `evidence.txt` from `evidence_copy` with `ln`. + 10. Verify with `ls -l`. + +### Task 7: Archive the Loot (Root) +- As root: + 1. Archive `/etc` to `/root/etc.tar` (uncompressed). + 2. Check type with `file`. + 3. Compress to `etc.tar.gz` with `gzip`. + 4. List contents with `tar tvf`. + 5. Extract `/etc/hosts` to `/root`. + 6. Verify with `ls -R /root/etc/`. + 7. Decompress `etc.tar.gz`. + 8. Extract `/etc/passwd` to `/tmp` with dir structure. + 9. Confirm with `ls -l /tmp/etc/passwd`. + 10. Create bzip2 archive of `/home` as `/root/homes.tar.bz2`. + 11. Clean up archives from `/root`. + +### Task 8: Root Shell Heist (as `student` → Root) +- As `student`: + 1. Escalate to root with `sudo -i`. + 2. Archive `/home` and `/etc` to `/root/essentials.tar`. + 3. Copy to `/tmp`. + 4. Hard-link it to `/essentials.tar`. + 5. Rename to `/archive.tar`. + 6. Soft-link from `/root/link.tar` to `/archive.tar`. + 7. Delete `/archive.tar` and observe `link.tar`. + 8. Remove `link.tar`. + 9. Compress `/root/essentials.tar` to `.gz`. + +--- From 28242d4db626a9f415f9c68c1d2738a942f58df1 Mon Sep 17 00:00:00 2001 From: Sagar Utekar Date: Wed, 9 Apr 2025 07:11:25 +0530 Subject: [PATCH 069/133] Create 0_filesystem.md --- DailyChallenges/Season2/Day10/0_filesystem.md | 33 +++++++++++++++++++ 1 file changed, 33 insertions(+) create mode 100644 DailyChallenges/Season2/Day10/0_filesystem.md diff --git a/DailyChallenges/Season2/Day10/0_filesystem.md b/DailyChallenges/Season2/Day10/0_filesystem.md new file mode 100644 index 0000000..4c87bc9 --- /dev/null +++ b/DailyChallenges/Season2/Day10/0_filesystem.md @@ -0,0 +1,33 @@ +### Linux Filesystem Overview + +The Linux filesystem is a hierarchical structure used to store all information on a computer. It is based on the UNIX system, where nearly everything, including data, commands, symbolic links, devices, and directories, is represented within the filesystem. + +### Structure and Navigation +Screenshot 2025-04-09 at 5 57 43 AM + +1. **Hierarchy**: The filesystem is organized like an upside-down tree with the root directory (`/`) at the top. Below the root are common directories such as `/bin`, `/dev`, `/home`, `/lib`, and `/tmp`. These directories can contain subdirectories, creating a hierarchical structure. +2. **Paths**: + * **Full Path**: Specifies the complete route from the root directory to a file or directory (e.g., `/home/joe/Documents/memos/memo1.doc`). + * **Relative Path**: Specifies the path relative to the current directory (e.g., `memo1.doc` if the current directory is `/home/joe/Documents/memos`). + +### Common Linux Directories + +#### Key Directories + +- **`/bin`**: Contains common Linux user commands like `ls`, `sort`, `date`, and `chmod`. +- **`/boot`**: Contains the bootable Linux kernel and boot loader configuration files (GRUB). +- **`/dev`**: Contains files representing access points to devices (e.g., `tty*`, `fd*`, `hd*`, `sd*`, `ram*`, `cd*`). +- **`/etc`**: Contains administrative configuration files, which are typically plaintext files that can be edited with a text editor. +- **`/home`**: Contains directories for each regular user account (except root, which uses `/root`). +- **`/media`**: Standard location for automatically mounting devices, particularly removable media. +- **`/lib`**: Contains shared libraries required by applications in `/bin` and `/sbin` to boot the system. +- **`/mnt`**: A common mount point for devices, often used for temporarily mounting local or remote filesystems. +- **`/misc`**: Sometimes used to automatically mount filesystems upon request. +- **`/opt`**: Used to store add-on application software. +- **`/proc`**: Contains information about system resources. +- **`/root`**: The home directory for the root user, located outside `/home` for security reasons. +- **`/sbin`**: Contains administrative commands and daemon processes. +- **`/tmp`**: Contains temporary files used by applications. +- **`/usr`**: Contains user documentation, games, graphical files (X11), libraries, and other commands and files not needed during the boot process. +- **`/var`**: Contains directories of data used by various applications, including FTP server files (`/var/ftp`), web server files (`/var/www`), system log files (`/var/log`), and spool files (`/var/spool`). + From 7a7cfed6b087df1ed5d62e2c3132f575afeec304 Mon Sep 17 00:00:00 2001 From: Sagar Utekar Date: Wed, 9 Apr 2025 07:12:15 +0530 Subject: [PATCH 070/133] Update 3_links.md --- DailyChallenges/Season2/Day10/3_links.md | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/DailyChallenges/Season2/Day10/3_links.md b/DailyChallenges/Season2/Day10/3_links.md index 60cadef..1de3fde 100644 --- a/DailyChallenges/Season2/Day10/3_links.md +++ b/DailyChallenges/Season2/Day10/3_links.md @@ -1,8 +1,8 @@ -## 1. Using Links +# Using Links Links on Linux are like aliases assigned to a file. There are symbolic links and hard links. To understand a link, you need to know how the Linux filesystem uses inodes for filesystem administration. -### 1.1. Understanding Hard Links +### Understanding Hard Links Linux stores administrative data about files in inodes. The inode stores all administrative data about files. Every file on Linux has an inode, and important information is stored in it: @@ -33,7 +33,7 @@ Here's a simple representation of how hard links and inodes work: |Hard Link | +----------+ -### 1.2. Understanding Symbolic Links +### Understanding Symbolic Links A **symbolic link** (also referred to as a soft link) does not link directly to the inode but to the name of the file. This makes symbolic links more flexible but has some disadvantages. The advantage of symbolic links is that they can link to files on other devices and directories. The major disadvantage is that when the original file is removed, the symbolic link becomes invalid. @@ -44,7 +44,7 @@ A **symbolic link** (also referred to as a soft link) does not link directly to +----------+ +--------------+ +----------+ +---------+ (Points to name) -### 1.3. Creating Links +### Creating Links Use the `ln` command to create links. It uses the same order of parameters as `cp` and `mv`; first, you mention the source name, followed by the destination name. If you want to create a symbolic link, use the option `-s`, and then specify the source and target file or directory. One important restriction applies: to be able to create hard links, you must be the owner of the item that you want to link to. @@ -68,7 +68,7 @@ lrwxrwxrwx. 1 root root 5 Jan 19 04:38 home -> /home In the example above, `home` is a symbolic link pointing to `/home`, and `hosts` is a file with 3 hard links. -### 1.4. Removing Links +### Removing Links Removing links can be dangerous. Let’s consider the following procedure: @@ -84,7 +84,7 @@ Removing links can be dangerous. Let’s consider the following procedure: 10. Type `ls`. You’ll see that the directory `link` still exists. 11. Type `ls test/`. You’ll see the directory `test` is now empty. -### 1.5. Exercise: Working with Symbolic Links and Hard Links +### 1Exercise: Working with Symbolic Links and Hard Links 1. Open a shell as the `student` user. 2. From your home directory, type `ln /etc/passwd .` (Make sure that the command ends with a dot that has a space before it!). This command gives you an “operation not permitted” error because you are not the owner of `/etc/passwd`. From 49bb1cbdb7472058e9dd6e5adb1d15f90343ebce Mon Sep 17 00:00:00 2001 From: Sagar Utekar Date: Wed, 9 Apr 2025 07:13:54 +0530 Subject: [PATCH 071/133] Update readme.md --- DailyChallenges/Season2/Day10/readme.md | 27 +++++++++++++------------ 1 file changed, 14 insertions(+), 13 deletions(-) diff --git a/DailyChallenges/Season2/Day10/readme.md b/DailyChallenges/Season2/Day10/readme.md index a3f560f..2c5d76c 100644 --- a/DailyChallenges/Season2/Day10/readme.md +++ b/DailyChallenges/Season2/Day10/readme.md @@ -1,22 +1,23 @@ # Daily DevOps + SRE Challenge Series – Season 2 -## Day 10: Navigating and Managing the Linux Filesystem +## Day 10: The Filesystem Heist – Crack the Vault of Linux Mastery ### Introduction -Welcome to Day 10 of the Daily DevOps + SRE Challenge Series - Season 2! 🎉 +Welcome to Day 10 of the Daily DevOps + SRE Challenge Series – Season 2! 🎉 -Today, we’re exploring the Linux filesystem—its structure, navigation, and essential file management tools. You’ll master commands to move around, create, copy, move, and delete files, all while building skills critical for DevOps and SRE workflows. By the end, you’ll: -- Understand the filesystem hierarchy and path navigation. -- Gain fluency in tools like `cd`, `mkdir`, `cp`, `mv`, and `rm`. +Today, you’re stepping into the core of Linux: the filesystem. This isn’t just a tutorial—it’s a high-stakes mission to navigate, secure, and conquer the server’s deepest layers. You’ll wield essential tools to traverse directories, manipulate files, manage storage, and lock down secrets, building skills that power DevOps pipelines and SRE heroics. By the end, you’ll: +- Master filesystem navigation with precision and speed. +- Command file and directory operations like a seasoned operative. +- Secure storage mounts, links, and archives under pressure. #### Why This Matters? -Whether you’re deploying apps, troubleshooting outages, or writing scripts these skills are non-negotiable. Here’s why: -- **Universal Skill**: Every Linux system—from cloud instances to Kubernetes pods—relies on the filesystem. Commands like `cd /var/log` or `rm -rf /tmp/cache` work everywhere. -- **Automation Backbone**: Scripts like `mv *.log /archive/$(date +%Y%m%d)` organize logs daily in cron jobs or CI/CD pipelines. -- **SRE Efficiency**: During incidents, `cp -r /etc/nginx /backup` or `find / -name core.*` pinpoints crash dumps fast, saving downtime. -- **Career Boost**: Fluency in filesystem tasks impresses in interviews—think “How do you restore a config?” or “How do you free disk space?” -- **Real-World Example**: An SRE at a fintech firm used `mkdir -p /data/backups/$(date +%F) && cp -r /prod/data/* /data/backups/$(date +%F)` to snapshot critical data before a risky upgrade, averting disaster when it failed. - -Let’s dive in and take control of the filesystem! +The filesystem is the backbone of every Linux system—your key to deploying apps, debugging outages, and scaling infrastructure. Here’s why these skills are mission-critical: +- **Everywhere You Go**: From AWS EC2 to Kubernetes pods, `cd /etc` and `cp -r /data /backup` are universal moves. +- **Automation Fuel**: Commands like `mv logs/*.log /archive/$(date +%F)` keep CI/CD pipelines humming. +- **Outage Lifeline**: When chaos strikes, `df -h` or `tar -czf backup.tar.gz /var` can save the day—fast. +- **Interview Edge**: “How do you recover a deleted config?” or “How do you manage disk space?”—nail these with filesystem fluency. +- **Real-World Win**: An SRE at a streaming service used `mkdir -p /mnt/cache && mount /dev/nvme1n1 /mnt/cache` to shift live streams to a new volume during a disk failure, keeping millions watching without a glitch. + +Gear up, crack your terminal—it’s time to breach the vault! --- From ca6e45aa0d350ea87817e840f662551d0a1f259f Mon Sep 17 00:00:00 2001 From: Sagar Utekar Date: Wed, 9 Apr 2025 07:18:51 +0530 Subject: [PATCH 072/133] Update readme.md --- DailyChallenges/Season2/Day10/readme.md | 42 +++++++++++++++++++++++++ 1 file changed, 42 insertions(+) diff --git a/DailyChallenges/Season2/Day10/readme.md b/DailyChallenges/Season2/Day10/readme.md index 2c5d76c..49422a3 100644 --- a/DailyChallenges/Season2/Day10/readme.md +++ b/DailyChallenges/Season2/Day10/readme.md @@ -147,3 +147,45 @@ Before you dive into the heist, test your knowledge with these questions. They 9. Compress `/root/essentials.tar` to `.gz`. --- + +# Submission Guidelines for Day 10: The Filesystem Heist + +Congrats on cracking the vault! To claim your victory and share your heist mastery, follow these guidelines. Submit your evidence and insights to showcase your filesystem skills! + +## Proof of Completion +Submit the following to prove you’ve conquered the challenge: +- **Screenshot of EC2 Terminal**: Show `ls -l ~/heist_vault` after Task 6 (post-restore of `evidence.txt`), displaying `evidence_sym`, `evidence_copy`, and `evidence.txt`. +- **Screenshot of EC2 Terminal**: Display `ls -l /data/shared_space` from Task 5, showing `alice_file.txt` and `bob_file.txt` with `agents` group ownership. +- **Text Output**: Contents of `~/heist_vault/theory_answers.txt` (your theory responses). +- **Screenshot of EC2 Terminal**: Run `history 5` to show your last 5 commands (any Linux distro). +- **Screenshot of EC2 Terminal**: Recall and run a command from history (e.g., `!?ls?`)—capture the output. +- **Text Output**: Run `cat ~/.bash_history | tail -n 3` on both EC2 and RHEL VM to show your last 3 logged commands. + +## Documentation +Detail your heist in a text file or markdown doc: +- **Steps Taken**: Explain how you: + - Accessed shells (e.g., SSH to EC2 as `ec2-user`, switched to root with `sudo -i`, logged in as `alice`/`bob`). + - Executed key tasks (e.g., mounted `/data`, managed files, used `tar`). + - Answered theory questions (e.g., researched `/var` purpose). +- **Challenges Faced & Fixes**: Note any hiccups and solutions, e.g.: + - “`fdisk` wouldn’t save partitions” → “Forgot `w`, reran it.” + - “Permission denied on `/etc/passwd` link” → “Expected per task, used symbolic link instead.” + - “`lsblk` missing” → “Installed `util-linux` with `yum`.” +- **Optional**: Add reflections (e.g., “SGID was tricky but clicked after testing!”). + +## Share Your Progress! +Broadcast your heist success to the world: +- Post on social media with your screenshots, insights, or a quick “I cracked Day 10!” shoutout. +- Use hashtags: `#getfitwithsagar`, `#SRELife`, `#DevOpsForAll`. +- Tag me (@SagarUtekar) for a chance to get featured! + +## **Join Our Community** +Connect with fellow learners: +- **Discord:** [Join here](https://discord.gg/mNDm39qB8t) +- **Google Group:** [Join here](https://groups.google.com/forum/#!forum/daily-devops-sre-challenge-series/join) +- **YouTube:** [Subscribe here](https://www.youtube.com/@Sagar.Utekar) + +--- + +Keep learning and happy exploring! +Sagar Utekar From a1308954630f6276f37dffc2f08c8fef2b83e667 Mon Sep 17 00:00:00 2001 From: Sagar Utekar Date: Wed, 9 Apr 2025 07:31:24 +0530 Subject: [PATCH 073/133] Update readme.md --- DailyChallenges/Season2/Day10/readme.md | 43 +++++++++++++++---------- 1 file changed, 26 insertions(+), 17 deletions(-) diff --git a/DailyChallenges/Season2/Day10/readme.md b/DailyChallenges/Season2/Day10/readme.md index 49422a3..91c0726 100644 --- a/DailyChallenges/Season2/Day10/readme.md +++ b/DailyChallenges/Season2/Day10/readme.md @@ -1,5 +1,6 @@ # Daily DevOps + SRE Challenge Series – Season 2 ## Day 10: The Filesystem Heist – Crack the Vault of Linux Mastery +![ChatGPT Image Apr 9, 2025, 07_30_45 AM](https://github.com/user-attachments/assets/738656ce-b015-4ddd-8b42-fb310654cf39) ### Introduction Welcome to Day 10 of the Daily DevOps + SRE Challenge Series – Season 2! 🎉 @@ -153,25 +154,33 @@ Before you dive into the heist, test your knowledge with these questions. They Congrats on cracking the vault! To claim your victory and share your heist mastery, follow these guidelines. Submit your evidence and insights to showcase your filesystem skills! ## Proof of Completion -Submit the following to prove you’ve conquered the challenge: -- **Screenshot of EC2 Terminal**: Show `ls -l ~/heist_vault` after Task 6 (post-restore of `evidence.txt`), displaying `evidence_sym`, `evidence_copy`, and `evidence.txt`. -- **Screenshot of EC2 Terminal**: Display `ls -l /data/shared_space` from Task 5, showing `alice_file.txt` and `bob_file.txt` with `agents` group ownership. -- **Text Output**: Contents of `~/heist_vault/theory_answers.txt` (your theory responses). -- **Screenshot of EC2 Terminal**: Run `history 5` to show your last 5 commands (any Linux distro). -- **Screenshot of EC2 Terminal**: Recall and run a command from history (e.g., `!?ls?`)—capture the output. -- **Text Output**: Run `cat ~/.bash_history | tail -n 3` on both EC2 and RHEL VM to show your last 3 logged commands. +Submit these deliverables to confirm you’ve cracked the challenge: +- **Screenshot of EC2 Terminal (Task 1)**: After the final step, run `pwd` to show `/home/ec2-user`, proving you hit `/tmp` and bounced back. +- **Screenshot of EC2 Terminal (Task 2)**: Display `ls -ld ~/heist_vault/newfiles` (expect `drwx------`) and `ls -a ~/heist_vault/newfiles` (showing `.secret`, `decoy.txt`). +- **Screenshot of EC2 Terminal (Task 3)**: As root, run `df -h | grep /data` and `lsblk`, confirming `/data` is mounted with a partition (e.g., `/dev/xvdf1` or `/dev/loop0p1`). +- **Screenshot of EC2 Terminal (Task 4)**: Post-task, show `ls -l ~/projects/house2` (perms `rw-r-----`) and `ls -R ~/projects` (no `doors/` subdir present). +- **Screenshot of EC2 Terminal (Task 5)**: Run `ls -l /data/shared_space`, displaying `alice_file.txt` and `bob_file.txt` with group `agents` and SGID (`drwxrws---`). +- **Screenshot of EC2 Terminal (Task 6)**: After restoring `evidence.txt`, show `ls -l ~/heist_vault` with `evidence_sym -> evidence.txt`, `evidence_copy`, and `evidence.txt` (link count 2). +- **Screenshot of EC2 Terminal (Task 7)**: As root, display `ls -l /tmp/etc/passwd` and `ls -R /root/etc/ | grep hosts`, verifying extracted archive files. +- **Screenshot of EC2 Terminal (Task 8)**: As root, show `ls -l /root/essentials.tar.gz` and `ls -l /root/link.tar` (broken link post-removal). +- **Text Output**: Full contents of `~/heist_vault/theory_answers.txt` with your answers to all 10 theory questions. ## Documentation -Detail your heist in a text file or markdown doc: -- **Steps Taken**: Explain how you: - - Accessed shells (e.g., SSH to EC2 as `ec2-user`, switched to root with `sudo -i`, logged in as `alice`/`bob`). - - Executed key tasks (e.g., mounted `/data`, managed files, used `tar`). - - Answered theory questions (e.g., researched `/var` purpose). -- **Challenges Faced & Fixes**: Note any hiccups and solutions, e.g.: - - “`fdisk` wouldn’t save partitions” → “Forgot `w`, reran it.” - - “Permission denied on `/etc/passwd` link” → “Expected per task, used symbolic link instead.” - - “`lsblk` missing” → “Installed `util-linux` with `yum`.” -- **Optional**: Add reflections (e.g., “SGID was tricky but clicked after testing!”). +Detail your heist in a text file or markdown doc (e.g., `heist_notes.md`): +- **Steps Taken**: Outline your approach for each task: + - Task 1: Navigating with `cd` and `pwd`. + - Task 2: Setting up `heist_vault` with `mkdir`, `touch`, and `chmod`. + - Task 3: Mounting `/data` with `fdisk`, `mkfs.ext4`, and `mount`. + - Task 4: Managing files in `oldfiles` and `projects` with `cp`, `rm`, `mv`, etc. + - Task 5: Creating users and syncing files in `/data/shared_space`. + - Task 6: Handling hard and soft links in `heist_vault`. + - Task 7: Archiving `/etc` and `/home` with `tar`, `gzip`, and `bzip2`. + - Task 8: Root ops with linking and compression. +- **Challenges & Fixes**: Document hurdles and solutions, e.g.: + - “`fdisk` didn’t save” → “Missed `w`, reran it.” + - “Permission denied on `/etc/passwd` hard link” → “Expected, switched to soft link.” + - “`ls -R` flooded terminal” → “Piped to `less` as instructed.” +- **Optional**: Add a takeaway (e.g., “Hard links survive deletion—mind blown!”). ## Share Your Progress! Broadcast your heist success to the world: From 76a66f0d23247c339785e2fecda6f3ab24b41f08 Mon Sep 17 00:00:00 2001 From: Sagar Utekar Date: Thu, 10 Apr 2025 01:09:36 +0530 Subject: [PATCH 074/133] Create readme.md --- DailyChallenges/Season2/Day11/readme.md | 1 + 1 file changed, 1 insertion(+) create mode 100644 DailyChallenges/Season2/Day11/readme.md diff --git a/DailyChallenges/Season2/Day11/readme.md b/DailyChallenges/Season2/Day11/readme.md new file mode 100644 index 0000000..8b13789 --- /dev/null +++ b/DailyChallenges/Season2/Day11/readme.md @@ -0,0 +1 @@ + From ae3aae3caa24502d2a993c294f490befb3361c0d Mon Sep 17 00:00:00 2001 From: Sagar Utekar Date: Thu, 10 Apr 2025 01:35:45 +0530 Subject: [PATCH 075/133] Update readme.md --- DailyChallenges/Season2/Day11/readme.md | 106 ++++++++++++++++++++++++ 1 file changed, 106 insertions(+) diff --git a/DailyChallenges/Season2/Day11/readme.md b/DailyChallenges/Season2/Day11/readme.md index 8b13789..b9037c2 100644 --- a/DailyChallenges/Season2/Day11/readme.md +++ b/DailyChallenges/Season2/Day11/readme.md @@ -1 +1,107 @@ +# Daily DevOps + SRE Challenge Series – Season 2 +## Day 11: The Text File Takedown – Wrangle Logs and Configs Like a Pro +### Introduction +Welcome to Day 11 of the Daily DevOps + SRE Challenge Series – Season 2! 🎉 + +Today, you’re diving into the art of text file mastery—crucial for taming logs, tweaking configs, and slicing through data in Linux. You’ll wield powerful tools to read, filter, sort, and edit text, turning raw files into actionable insights. By the end, you’ll: +- Navigate and extract data from text files with ninja-like precision. +- Transform and analyze logs/configs using command-line magic. +- Build SRE-ready skills for debugging and automation. + +#### Why This Matters? +Text files are the lifeblood of Linux systems—logs in `/var/log`, configs in `/etc`, scripts everywhere. Mastering them is non-negotiable: +- **Universal Power**: `grep`, `awk`, and `sed` work across clouds, containers, and bare metal—your Swiss Army knife for any system. +- **Automation Edge**: `tail -f /var/log/app.log | grep ERROR` feeds live alerts into monitoring scripts. +- **SRE Superpower**: During outages, `sort | uniq -c` on logs spots patterns fast; `sed` fixes configs on the fly. +- **Interview Gold**: “How do you find errors in a 1GB log?” or “Extract IPs from a file?”—you’ll ace these. +- **Real-World Save**: An SRE at a cloud provider used `awk '{print $1}' /var/log/access.log | sort | uniq -c` to pinpoint a DDoS source, throttling it in minutes. + +Grab your terminal—it’s time to take down the text chaos! + + +In UNIX and Linux systems, configuration and information management have traditionally relied on plain-text files. Despite the availability of graphical tools today, editing these files manually with a text editor remains essential, especially for full system control and on servers without GUIs. Aspiring system administrators must learn to use text editors and tools like find (to locate files) and grep (to search within files) to manage and configure Linux systems effectively. + +# vi/vim Cheat Sheet – Quick Reference + +## Modes +| Mode | Key | Description | +|-----------------|-------------|------------------------------------| +| Command | Default | For navigation and actions | +| Insert | `i` / `a` / `o` | Enter insert mode (insert, append, open) | +| Return to Command | `Esc` | Switch from insert to command mode | + +## Insert Mode Commands +| Command | Description | +|---------|---------------------------------| +| `i` | Insert before the cursor | +| `a` | Append after the cursor | +| `I` | Insert at the beginning of the line | +| `A` | Append at the end of the line | +| `o` | Open new line below | +| `O` | Open new line above | +| `Esc` | Return to command mode | + +## Navigation +| Key | Moves Cursor To | +|-------------|-------------------------------| +| `h` / `l` | Left / Right one character | +| `j` / `k` | Down / Up one line | +| `w` / `W` | Next word (by punctuation / space) | +| `b` / `B` | Previous word | +| `0` | Start of line | +| `$` | End of line | +| `H` | Top of screen | +| `M` | Middle of screen | +| `L` | Bottom of screen | +| `G` | Last line of file | +| `1G` | First line of file | +| `35G` | Line 35 | +| `Ctrl+f` | Page forward | +| `Ctrl+b` | Page backward | +| `Ctrl+d` | Half page forward | +| `Ctrl+u` | Half page backward | + +## ✂️ Editing Text +| Command | Description | +|-------------|-----------------------------------| +| `x` / `X` | Delete character under / before cursor | +| `dd` | Delete current line | +| `dw` | Delete word after cursor | +| `db` | Delete word before cursor | +| `d$` | Delete from cursor to end of line | +| `c + move` | Change text (e.g., `c$` changes to end) | +| `cc` | Change entire line | +| `yy` | Yank (copy) current line | +| `y}` | Yank paragraph | +| `p` | Paste below/after | +| `P` | Paste above/before | +| `.` | Repeat last change | + +## Searching +| Command | Description | +|------------------|---------------------------------------| +| `/text` | Search forward for "text" | +| `?text` | Search backward for "text" | +| `n` / `N` | Repeat search forward / backward | +| `/The.*end` | Regex: match “The” followed by “end” | +| `?[Pp]rint` | Case-insensitive search for “Print” or “print” | + +## Ex Commands (use `:`) +| Command | Description | +|-------------------|---------------------------------------| +| `:w` | Save file | +| `:q` | Quit if no unsaved changes | +| `:wq` / `ZZ` | Save and quit | +| `:q!` | Quit without saving | +| `:s/old/new/` | Replace first “old” with “new” in current line | +| `:g/old/s//new/g` | Replace all “old” with “new” in file | +| `:g/old/s//new/gp`| Replace and print affected lines | + +## Pro Tips +| Command | Description | +|-----------|---------------------------------------| +| `u` | Undo last change | +| `Ctrl+r` | Redo (undo the undo) | +| `:!command`| Run shell command (e.g., `:!date`, `:!ls`) | +| `Ctrl+G` | Show filename, line number, and position | From 81c78c16c6b7073729fa4a0ac06938b473969e9d Mon Sep 17 00:00:00 2001 From: Sagar Utekar Date: Thu, 10 Apr 2025 01:42:04 +0530 Subject: [PATCH 076/133] Create 1_vi_editor_tips.md --- .../Season2/Day11/1_vi_editor_tips.md | 83 +++++++++++++++++++ 1 file changed, 83 insertions(+) create mode 100644 DailyChallenges/Season2/Day11/1_vi_editor_tips.md diff --git a/DailyChallenges/Season2/Day11/1_vi_editor_tips.md b/DailyChallenges/Season2/Day11/1_vi_editor_tips.md new file mode 100644 index 0000000..377423e --- /dev/null +++ b/DailyChallenges/Season2/Day11/1_vi_editor_tips.md @@ -0,0 +1,83 @@ +# vi/vim Cheat Sheet – Quick Reference + +## Modes +| Mode | Key | Description | +|-----------------|-------------|------------------------------------| +| Command | Default | For navigation and actions | +| Insert | `i` / `a` / `o` | Enter insert mode (insert, append, open) | +| Return to Command | `Esc` | Switch from insert to command mode | + +## Insert Mode Commands +| Command | Description | +|---------|---------------------------------| +| `i` | Insert before the cursor | +| `a` | Append after the cursor | +| `I` | Insert at the beginning of the line | +| `A` | Append at the end of the line | +| `o` | Open new line below | +| `O` | Open new line above | +| `Esc` | Return to command mode | + +## Navigation +| Key | Moves Cursor To | +|-------------|-------------------------------| +| `h` / `l` | Left / Right one character | +| `j` / `k` | Down / Up one line | +| `w` / `W` | Next word (by punctuation / space) | +| `b` / `B` | Previous word | +| `0` | Start of line | +| `$` | End of line | +| `H` | Top of screen | +| `M` | Middle of screen | +| `L` | Bottom of screen | +| `G` | Last line of file | +| `1G` | First line of file | +| `35G` | Line 35 | +| `Ctrl+f` | Page forward | +| `Ctrl+b` | Page backward | +| `Ctrl+d` | Half page forward | +| `Ctrl+u` | Half page backward | + +## ✂️ Editing Text +| Command | Description | +|-------------|-----------------------------------| +| `x` / `X` | Delete character under / before cursor | +| `dd` | Delete current line | +| `dw` | Delete word after cursor | +| `db` | Delete word before cursor | +| `d$` | Delete from cursor to end of line | +| `c + move` | Change text (e.g., `c$` changes to end) | +| `cc` | Change entire line | +| `yy` | Yank (copy) current line | +| `y}` | Yank paragraph | +| `p` | Paste below/after | +| `P` | Paste above/before | +| `.` | Repeat last change | + +## Searching +| Command | Description | +|------------------|---------------------------------------| +| `/text` | Search forward for "text" | +| `?text` | Search backward for "text" | +| `n` / `N` | Repeat search forward / backward | +| `/The.*end` | Regex: match “The” followed by “end” | +| `?[Pp]rint` | Case-insensitive search for “Print” or “print” | + +## Ex Commands (use `:`) +| Command | Description | +|-------------------|---------------------------------------| +| `:w` | Save file | +| `:q` | Quit if no unsaved changes | +| `:wq` / `ZZ` | Save and quit | +| `:q!` | Quit without saving | +| `:s/old/new/` | Replace first “old” with “new” in current line | +| `:g/old/s//new/g` | Replace all “old” with “new” in file | +| `:g/old/s//new/gp`| Replace and print affected lines | + +## Pro Tips +| Command | Description | +|-----------|---------------------------------------| +| `u` | Undo last change | +| `Ctrl+r` | Redo (undo the undo) | +| `:!command`| Run shell command (e.g., `:!date`, `:!ls`) | +| `Ctrl+G` | Show filename, line number, and position | From 575ad7e7bd729f7c45e783ba8c8d4ea8e25425aa Mon Sep 17 00:00:00 2001 From: Sagar Utekar Date: Thu, 10 Apr 2025 01:49:13 +0530 Subject: [PATCH 077/133] Create 2_ --- DailyChallenges/Season2/Day11/2_ | 67 ++++++++++++++++++++++++++++++++ 1 file changed, 67 insertions(+) create mode 100644 DailyChallenges/Season2/Day11/2_ diff --git a/DailyChallenges/Season2/Day11/2_ b/DailyChallenges/Season2/Day11/2_ new file mode 100644 index 0000000..7d9a51e --- /dev/null +++ b/DailyChallenges/Season2/Day11/2_ @@ -0,0 +1,67 @@ +## Essential Tools for Managing Text File Contents (Table 4-2) +| Command | Explanation | +|---------|-----------------------------------------------------------------------------| +| `less` | Opens a file in a pager for easy reading and scrolling | +| `cat` | Displays the entire file contents on the screen | +| `head` | Shows the first few lines of a file (default: 10) | +| `tail` | Shows the last few lines of a file (default: 10) | +| `cut` | Filters specific columns or characters from a file | +| `sort` | Sorts the file contents alphabetically or numerically | +| `wc` | Counts lines, words, and characters in a file | + +### Versatility with Pipes +These commands shine when paired with pipes: +- `less /etc/passwd`: View the `/etc/passwd` file. +- `ps aux | less`: Pipe process list output to `less` for scrollable reading. + +--- + +## Mastering `less` +The `less` utility is a go-to tool for reading text files as a Linux admin. + +### Usage +- Open a file: `less /etc/passwd` +- Navigate: Use **Page Up**/**Page Down** to scroll, press **`q`** to quit. +- Search: + - Forward: `/sometext` + - Backward: `?sometext` + - Repeat: **`n`** + +### Connection to `vim` and `man` +`less` shares navigation and search mechanics with `vim` and `man` due to their common code base. + +> **Note:** +> `less` was created to outdo `more`, a basic pager, inspiring the phrase "do more with less." Today, both tools have similar features, but `more` exits at file end (use `-p` to change this) and is standard everywhere, while `less` may require installation. + +--- + +## Exercise 1: Applying Basic `less` Skills + +### Objective + +Practice using the `less` command to view and navigate through text files. + +### Steps + +1. **Open a File:** +Open the `/etc/passwd` file using `less`: + +```bash +less /etc/passwd +``` + +2. **Search:** +Type `/root` to search for and highlight all occurrences of "root" in the file. +3. **Go to End:** +Press `G` to jump to the last line of the file. +4. **Quit:** +Press `q` to exit the `less` viewer. +5. **Pipe Output:** +Pipe the output of the `ps aux` command to `less` to browse the process list: + +```bash +ps aux | less +``` + +6. **Quit Again:** +Press `q` to exit the `less` viewer. From 58059c7cba031b8556a821e6c7e05c324430b835 Mon Sep 17 00:00:00 2001 From: Sagar Utekar Date: Thu, 10 Apr 2025 01:49:45 +0530 Subject: [PATCH 078/133] Update and rename 2_ to 2_common_text_commands.md --- DailyChallenges/Season2/Day11/{2_ => 2_common_text_commands.md} | 1 + 1 file changed, 1 insertion(+) rename DailyChallenges/Season2/Day11/{2_ => 2_common_text_commands.md} (99%) diff --git a/DailyChallenges/Season2/Day11/2_ b/DailyChallenges/Season2/Day11/2_common_text_commands.md similarity index 99% rename from DailyChallenges/Season2/Day11/2_ rename to DailyChallenges/Season2/Day11/2_common_text_commands.md index 7d9a51e..91c1164 100644 --- a/DailyChallenges/Season2/Day11/2_ +++ b/DailyChallenges/Season2/Day11/2_common_text_commands.md @@ -65,3 +65,4 @@ ps aux | less 6. **Quit Again:** Press `q` to exit the `less` viewer. + From ecb21965583dc9986379e8ad43398ae770275df9 Mon Sep 17 00:00:00 2001 From: Sagar Utekar Date: Thu, 10 Apr 2025 01:56:56 +0530 Subject: [PATCH 079/133] Update 2_common_text_commands.md --- .../Season2/Day11/2_common_text_commands.md | 198 ++++++++++++++++++ 1 file changed, 198 insertions(+) diff --git a/DailyChallenges/Season2/Day11/2_common_text_commands.md b/DailyChallenges/Season2/Day11/2_common_text_commands.md index 91c1164..7bae50c 100644 --- a/DailyChallenges/Season2/Day11/2_common_text_commands.md +++ b/DailyChallenges/Season2/Day11/2_common_text_commands.md @@ -63,6 +63,204 @@ Pipe the output of the `ps aux` command to `less` to browse the process list: ps aux | less ``` +## Showing File Contents with `cat` + +### Overview + +The `cat` command is a simple utility used to display the contents of a text file directly on the terminal screen. It is most suitable for short files, as it displays the entire file from start to finish without pausing. + +### Using `cat` + +1. **Basic Usage:** +To view the contents of a file, simply type `cat` followed by the filename: + +```bash +cat filename.txt +``` + +2. **Example:** +To display the contents of the `/etc/passwd` file, use: + +```bash +cat /etc/passwd +``` + + +### Limitations + +- **Long Files:** If the file is long, `cat` will display all contents, but only the lines that fit on the terminal screen will be visible. You might miss the beginning of the file as it scrolls by. +- **Viewing First Lines:** If you want to see the first lines of a long file, you can use the `tac` command, which displays the file in reverse order (from end to beginning). + + +### Alternatives + +For longer files, consider using `less` or `more`, which allow you to view the file one page at a time and navigate through it more easily. + +### Tip + +- **Inverse Display:** Use `tac` to display a file in reverse order: + +```bash +tac filename.txt +``` + 6. **Quit Again:** Press `q` to exit the `less` viewer. + +## Displaying First or Last Lines with `head` and `tail` + +### Overview + +The `head` and `tail` commands are used to display the beginning or end of a file, respectively. By default, they show the first or last ten lines of a file, but this can be adjusted. + +### Using `head` + +1. **Default Usage:** +To display the first ten lines of a file: + +```bash +head filename.txt +``` + +2. **Specifying Number of Lines:** +To display a specific number of lines, use the `-n` option followed by the number: + +```bash +head -n 5 filename.txt +# Or simply: +head -5 filename.txt +``` + + +### Using `tail` + +1. **Default Usage:** +To display the last ten lines of a file: + +```bash +tail filename.txt +``` + +2. **Specifying Number of Lines:** +To display a specific number of last lines: + +```bash +tail -n 2 filename.txt +# Or simply: +tail -2 filename.txt +``` + +3. **Following a File:** +The `-f` option displays the last ten lines and continues to display new lines as they are added to the file. This is useful for monitoring log files: + +```bash +tail -f /var/log/messages +``` + +Press `Ctrl-C` to stop following the file. + +### Combining `head` and `tail` + +You can combine `head` and `tail` using pipes to extract specific lines from a file. + +1. **Example:** +To display line number 11 of a file: + +```bash +head -n 11 filename.txt | tail -n 1 +``` + +This command first displays the first 11 lines and then extracts the last line from that output, which is line number 11 of the original file. + + +## Filtering Columns with `cut` and Sorting with `sort` + +### Filtering Specific Columns with `cut` + +#### Overview + +The `cut` command is used to extract specific columns (fields) from a text file based on a delimiter. + +#### Usage + +```bash +cut -d <delimiter> -f <field_number> filename.txt +``` + +- `-d <delimiter>`: Specifies the delimiter that separates the fields. +- `-f <field_number>`: Specifies the field number to extract. + + +#### Example + +To extract the first field (username) from the `/etc/passwd` file, where fields are delimited by a colon: + +```bash +cut -d : -f 1 /etc/passwd +``` + + +### Sorting File Contents with `sort` + +#### Overview + +The `sort` command is used to sort the lines of a text file. + +#### Usage + +```bash +sort filename.txt +``` + +- This sorts the file contents in byte order (ASCII order). + + +#### Sorting Output of a Command + +You can sort the output of another command using a pipe: + +```bash +cut -f 1 -d : /etc/passwd | sort +``` + +This extracts the first column from `/etc/passwd` and sorts the result. + +#### Sorting Options + +- `-n`: Sorts numerically. + +```bash +cut -f 3 -d : /etc/passwd | sort -n +``` + +This sorts the third field of `/etc/passwd` in numeric order. +- `-r`: Sorts in reverse order. + +```bash +du -h | sort -rn +``` + +This lists files sorted by size, with the largest file first. +- `-k column_number`: Specifies the column to sort. +- `-t delimiter`: Specifies the field separator. + +```bash +sort -k3 -t : /etc/passwd +``` + +This sorts the `/etc/passwd` file by the third column, using ":" as the delimiter. + + +#### Example: Finding Busiest Processes + +To sort processes by memory usage (4th column in `ps aux` output): + +```bash +ps aux | sort -k 4 -n +``` + +This command displays processes sorted by memory usage, with the process using the least memory listed first. + + + From c77411fb5bd555afeab6ef6fd456152102fa6071 Mon Sep 17 00:00:00 2001 From: Sagar Utekar Date: Thu, 10 Apr 2025 01:57:08 +0530 Subject: [PATCH 080/133] Update 2_common_text_commands.md --- DailyChallenges/Season2/Day11/2_common_text_commands.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/DailyChallenges/Season2/Day11/2_common_text_commands.md b/DailyChallenges/Season2/Day11/2_common_text_commands.md index 7bae50c..93f8335 100644 --- a/DailyChallenges/Season2/Day11/2_common_text_commands.md +++ b/DailyChallenges/Season2/Day11/2_common_text_commands.md @@ -1,4 +1,4 @@ -## Essential Tools for Managing Text File Contents (Table 4-2) +## Essential Tools for Managing Text File Contents | Command | Explanation | |---------|-----------------------------------------------------------------------------| | `less` | Opens a file in a pager for easy reading and scrolling | From 3d91d2f7b8b9e63a14c13a2f44f4f8296c0e6d56 Mon Sep 17 00:00:00 2001 From: Sagar Utekar Date: Thu, 10 Apr 2025 01:58:49 +0530 Subject: [PATCH 081/133] Create 3_extended_grep.md --- .../Season2/Day11/3_extended_grep.md | 112 ++++++++++++++++++ 1 file changed, 112 insertions(+) create mode 100644 DailyChallenges/Season2/Day11/3_extended_grep.md diff --git a/DailyChallenges/Season2/Day11/3_extended_grep.md b/DailyChallenges/Season2/Day11/3_extended_grep.md new file mode 100644 index 0000000..47ede23 --- /dev/null +++ b/DailyChallenges/Season2/Day11/3_extended_grep.md @@ -0,0 +1,112 @@ +## Using Extended Regular Expressions (EREs) + +### Overview + +Extended Regular Expressions (EREs) provide additional features compared to basic regular expressions. Tools like `grep` require the `-E` option to recognize EREs. + +### Key ERE Features + +| Regular Expression | Use | +| :-- | :-- | +| `^text` | Matches a line that starts with the specified text. | +| `text$` | Matches a line that ends with the specified text. | +| `.` | Wildcard: Matches any single character. | +| `[abc]` | Matches `a`, `b`, or `c`. | +| `?` | Matches zero or one occurrence of the preceding character. | +| `+` | Matches one or more occurrences of the preceding character. | +| `*` | Matches zero to an infinite number of occurrences of the preceding character. | +| `\{2\}` | Matches exactly two occurrences of the preceding character. | +| `\{1,3\}` | Matches a minimum of one and a maximum of three occurrences of the preceding character. | +| `colou?r` | Matches zero or one of the previous character. Matches both "color" and "colour". | +| `(...)` | Groups multiple characters so that the regular expression can be applied to the group. | + +### Examples and Usage + +1. **Enabling ERE with `grep`**: + * Use `grep -E` to interpret patterns as extended regular expressions. +2. **Example Regular Expression**: + * `"/web(/.*)?"` + * Matches `/web`, `/web/`, or `/web/filename`. + +### Demonstrating ERE with `grep` + +1. **Create a text file named `regex.txt`**: + +```text +bat +boot +boat +bt +``` + +2. **Base Regular Expression**: + +```bash +grep 'b.*t' regex.txt +``` + + * Finds any line that starts with `b` and ends with `t`. +3. **Extended Regular Expression (Incorrect)**: + +```bash +grep 'b.+t' regex.txt +``` + + * Without the `-E` option, `grep` does not correctly interpret the `+`. +4. **Extended Regular Expression (Correct)**: + +```bash +grep -E 'b.+t' regex.txt +``` + + * Finds lines that have at least one character between `b` and `t`. + +## Using `grep` to Analyze Text + +### Overview + +`grep` (Global Regular Expression Parser) is a powerful utility for searching text using regular expressions. + +### Useful `grep` Options + +| Option | Use | +| :-- | :-- | +| `-i` | Matches upper- and lowercase letters (case-insensitive). | +| `-v` | Shows only lines that do *not* contain the regular expression. | +| `-r` | Searches files in the current directory and all subdirectories. | +| `-e` | Searches for lines matching more than one regular expression. Use `-e` before each regular expression. | +| `-E` | Interprets the search pattern as an extended regular expression. | +| `-A number` | Shows `<number>` of lines *after* the matching regular expression. | +| `-B number` | Shows `<number>` of lines *before* the matching regular expression. | + +### Examples and Usage + +1. **Finding lines starting with a comment (`#`)**: + +```bash +grep ' #' /etc/services +``` + +2. **Excluding lines starting with a comment**: + +```bash +grep -v '^#' /etc/services +``` + + * Shows only lines that do not start with `#`. +3. **Showing lines before matches**: + +```bash +grep -v '^#' /etc/services -B 5 +``` + + * Shows lines that do not start with `#`, along with the 5 lines before each match. +4. **Excluding comments and blank lines**: + +```bash +grep -v -e '^#' -e '^$' /etc/services +``` + + * Excludes all blank lines and lines that start with `#`. + + From f858b9d6147ebc7da35a79e3ef2e3c2cf9667a33 Mon Sep 17 00:00:00 2001 From: Sagar Utekar Date: Thu, 10 Apr 2025 02:00:53 +0530 Subject: [PATCH 082/133] Create 4_awk_sed.md --- DailyChallenges/Season2/Day11/4_awk_sed.md | 101 +++++++++++++++++++++ 1 file changed, 101 insertions(+) create mode 100644 DailyChallenges/Season2/Day11/4_awk_sed.md diff --git a/DailyChallenges/Season2/Day11/4_awk_sed.md b/DailyChallenges/Season2/Day11/4_awk_sed.md new file mode 100644 index 0000000..fcc25f2 --- /dev/null +++ b/DailyChallenges/Season2/Day11/4_awk_sed.md @@ -0,0 +1,101 @@ +## Working with Other Useful Text Processing Utilities + +### Overview + +Besides `grep`, `awk` and `sed` are powerful utilities for text processing, particularly useful in scripted environments where direct screen interaction is limited. + +### `awk` + +#### Overview + +`awk` is a versatile utility for text processing, capable of more complex field extraction than `cut`. + +#### Usage + +```bash +awk -F delimiter '{ print $column_number }' filename +``` + +- `-F delimiter`: Specifies the field delimiter. +- `'{ print $column_number }'`: Specifies the column to print. + + +#### Examples + +1. **Printing the fourth field from `/etc/passwd`**: + +```bash +awk -F : '{ print $4 }' /etc/passwd +``` + +2. **Searching for text and printing a field**: + +```bash +awk -F : '/user/ { print $4 }' /etc/passwd +``` + + * Searches `/etc/passwd` for lines containing "user" and prints the fourth field of matching lines. + +### `sed` + +#### Overview + +`sed` (stream editor) is used for filtering and modifying text in files. + +#### Usage + +1. **Printing a specific line**: + +```bash +sed -n line_number;p filename +``` + + * `-n`: Suppresses default output. + * `p`: Print command. + +Example: + +```bash +sed -n 5p /etc/passwd +``` + + * Prints the fifth line from `/etc/passwd`. +2. **Replacing text in a file**: + +```bash +sed -i 's/old-text/new-text/g' filename +``` + + * `-i`: Modifies the file directly. + * `s/old-text/new-text/g`: Substitutes all occurrences of `old-text` with `new-text`. + +Example: + +```bash +sed -i 's/old-text/new-text/g' ~/myfile +``` + + * Replaces all occurrences of "old-text" with "new-text" in `~/myfile`. +3. **Deleting lines by line number**: + +```bash +sed -i -e 'line_number;d' filename +``` + + * `-e`: Specifies an editing command. + * `d`: Delete command. + +Example: + +```bash +sed -i -e '2d' ~/myfile +``` + + * Deletes line 2 from `~/myfile`. +4. **Deleting multiple lines**: + +```bash +sed -i -e '2d;20,25d' ~/myfile +``` + + * Deletes line 2 and lines 20 through 25 in `~/myfile`. From b578e888b86b2213aebebcdeb3a3de582ea83a59 Mon Sep 17 00:00:00 2001 From: Sagar Utekar Date: Thu, 10 Apr 2025 02:04:41 +0530 Subject: [PATCH 083/133] Create 5_locate_find.md --- .../Season2/Day11/5_locate_find.md | 57 +++++++++++++++++++ 1 file changed, 57 insertions(+) create mode 100644 DailyChallenges/Season2/Day11/5_locate_find.md diff --git a/DailyChallenges/Season2/Day11/5_locate_find.md b/DailyChallenges/Season2/Day11/5_locate_find.md new file mode 100644 index 0000000..838e2a4 --- /dev/null +++ b/DailyChallenges/Season2/Day11/5_locate_find.md @@ -0,0 +1,57 @@ +## Finding Files in Linux: Summary + +This section covers two main utilities for finding files: `locate` (for fast searches using a database) and `find` (for real-time, flexible searches). + +### 1. `locate`: Quick Database Search + +* **Method:** Searches a pre-built database of filenames. +* **Speed:** Very fast. +* **Database Updates:** The database is updated by `updatedb`, usually run daily. +* **Limitations:** + * Won't find files created since the last `updatedb` run. + * Excludes pruned paths (e.g., `/tmp`, `/media`) defined in `/etc/updatedb.conf`. +* **Permissions:** Respects user permissions. +* **Examples:** + +```bash +locate .bashrc # Finds all .bashrc files +locate -i muttrc # Case-insensitive search +``` + + +### 2. `find`: Real-Time Filesystem Search + +* **Method:** Searches the live filesystem. +* **Speed:** Slower than `locate`, but always up-to-date. +* **Search Criteria:** + * Filename (`-name`, `-iname`) + * Size (`-size +10M`, `-size -1G`) + * Ownership (`-user`, `-group`) + * Permissions (`-perm 755`, `-perm +222`) + * Modification, change, or access times (`-mtime`, `-ctime`, `-atime`, `-mmin`, `-cmin`, `-amin`) +* **Actions:** + * `-exec`: Executes a command on each result (e.g., `-exec du -sh {} \;`) +* **Error Handling:** Redirect errors with `2> /dev/null`. +* **Examples:** + +```bash +find /etc -name passwd # Exact name match +find /etc -iname '*passwd*' # Partial + case-insensitive match +find /home -user chris # Files owned by chris +find /bin -perm 755 # Files with exact permissions +find /var -size +500M # Large files +find /etc -mmin -10 # Modified in last 10 minutes +``` + + +### 3. Combining Conditions + +* `-not`: Excludes specific matches. +* `-or` (`-o`): Includes multiple conditions. +* Parentheses `$ ... $`: For grouping conditions. +* **Examples:** + +```bash +find /var/allusers \( -user joe -o -user chris \) -ls # Files owned by joe OR chris +find /var -user joe -not -group joe # Files owned by joe, NOT in group joe +``` From 9a9e94b6b8e1cf416af5fd50fff784fa185339d1 Mon Sep 17 00:00:00 2001 From: Sagar Utekar Date: Thu, 10 Apr 2025 02:08:34 +0530 Subject: [PATCH 084/133] Update and rename 3_extended_grep.md to 3_grep_egrep.md --- .../{3_extended_grep.md => 3_grep_egrep.md} | 106 ++++++++++++++++++ 1 file changed, 106 insertions(+) rename DailyChallenges/Season2/Day11/{3_extended_grep.md => 3_grep_egrep.md} (60%) diff --git a/DailyChallenges/Season2/Day11/3_extended_grep.md b/DailyChallenges/Season2/Day11/3_grep_egrep.md similarity index 60% rename from DailyChallenges/Season2/Day11/3_extended_grep.md rename to DailyChallenges/Season2/Day11/3_grep_egrep.md index 47ede23..2cbdb32 100644 --- a/DailyChallenges/Season2/Day11/3_extended_grep.md +++ b/DailyChallenges/Season2/Day11/3_grep_egrep.md @@ -1,3 +1,109 @@ +## Using `grep` to Analyze Text + +### Overview + +`grep` (Global Regular Expression Parser) is a powerful utility for searching plain-text data using regular expressions. It works on files or command outputs. + +### Key Features + +- **Case-sensitive by default** +- **Can search files, directories, and command output** +- **Supports regular and extended regular expressions** +- **Highlights matches using `--color`** + +--- + +### Commonly Used Options + +| Option | Description | +| :-- | :-- | +| `-i` | Case-insensitive matching | +| `-v` | Invert match (show lines that do **not** match) | +| `-r` | Recursively search directories | +| `-l` | Print only filenames with matches | +| `-e` | Use multiple regex patterns | +| `-E` | Interpret pattern as an extended regular expression | +| `-A N` | Show N lines *after* each match | +| `-B N` | Show N lines *before* each match | +| `--color` | Highlight the matching text | + +--- + +### Practical Examples + +#### 1. **Simple Match in File** + +```bash +grep desktop /etc/services +``` + +* Matches exact lowercase `desktop`. + + +#### 2. **Case-Insensitive Match** + +```bash +grep -i desktop /etc/services +``` + +* Matches variations like `Desktop`, `DESKTOP`, etc. + + +#### 3. **Exclude Matches** + +```bash +grep -vi tcp /etc/services +``` + +* Show all lines **except** those containing `tcp`. + + +#### 4. **Recursive Search (Case-Insensitive)** + +```bash +grep -rli peerdns /usr/share/doc/ +``` + +* Search recursively for `peerdns`, show filenames only. + + +#### 5. **Recursive Search with Highlight** + +```bash +grep -ri --color root /etc/sysconfig/ +``` + +* Highlights `root` in red (by default) wherever it appears. + + +#### 6. **Filter Output of Another Command** + +```bash +ip addr show | grep inet +``` + +* Shows only lines with `inet`, useful for filtering IP addresses. + +--- + +### Filtering Comments and Blank Lines + +```bash +grep -v -e '^#' -e '^$' /etc/services +``` + +* Removes comment lines (`#`) and empty lines. + +--- + +## Tips + +- Combine `grep` with `sort`, `uniq`, and `wc` for powerful text processing. +- Use double quotes for patterns with spaces or variables: `grep "$pattern" file.txt`. +- Use `^` and `$` to anchor searches to the beginning or end of lines. + + + ## Using Extended Regular Expressions (EREs) ### Overview From 4eaf5f6b8963711ac76e1b6c17eee2afc4785fda Mon Sep 17 00:00:00 2001 From: Sagar Utekar Date: Thu, 10 Apr 2025 02:11:48 +0530 Subject: [PATCH 085/133] Update 3_grep_egrep.md --- DailyChallenges/Season2/Day11/3_grep_egrep.md | 62 ++++++------------- 1 file changed, 20 insertions(+), 42 deletions(-) diff --git a/DailyChallenges/Season2/Day11/3_grep_egrep.md b/DailyChallenges/Season2/Day11/3_grep_egrep.md index 2cbdb32..f5cffd9 100644 --- a/DailyChallenges/Season2/Day11/3_grep_egrep.md +++ b/DailyChallenges/Season2/Day11/3_grep_egrep.md @@ -96,6 +96,26 @@ grep -v -e '^#' -e '^$' /etc/services --- +### Finding lines starting with a comment (`#`)**: + +```bash +grep ' #' /etc/services +``` + +### Excluding lines starting with a comment**: + +```bash +grep -v '^#' /etc/services +``` + +* Shows only lines that do not start with `#`. +### Showing lines before matches**: + +```bash +grep -v '^#' /etc/services -B 5 +``` +* Shows lines that do not start with `#`, along with the 5 lines before each match. + ## Tips - Combine `grep` with `sort`, `uniq`, and `wc` for powerful text processing. @@ -167,52 +187,10 @@ grep -E 'b.+t' regex.txt * Finds lines that have at least one character between `b` and `t`. -## Using `grep` to Analyze Text - -### Overview - -`grep` (Global Regular Expression Parser) is a powerful utility for searching text using regular expressions. -### Useful `grep` Options - -| Option | Use | -| :-- | :-- | -| `-i` | Matches upper- and lowercase letters (case-insensitive). | -| `-v` | Shows only lines that do *not* contain the regular expression. | -| `-r` | Searches files in the current directory and all subdirectories. | -| `-e` | Searches for lines matching more than one regular expression. Use `-e` before each regular expression. | -| `-E` | Interprets the search pattern as an extended regular expression. | -| `-A number` | Shows `<number>` of lines *after* the matching regular expression. | -| `-B number` | Shows `<number>` of lines *before* the matching regular expression. | ### Examples and Usage -1. **Finding lines starting with a comment (`#`)**: - -```bash -grep ' #' /etc/services -``` - -2. **Excluding lines starting with a comment**: - -```bash -grep -v '^#' /etc/services -``` - - * Shows only lines that do not start with `#`. -3. **Showing lines before matches**: - -```bash -grep -v '^#' /etc/services -B 5 -``` - - * Shows lines that do not start with `#`, along with the 5 lines before each match. -4. **Excluding comments and blank lines**: - -```bash -grep -v -e '^#' -e '^$' /etc/services -``` - * Excludes all blank lines and lines that start with `#`. From 9757f1c187d18758d8691aaff6e51ac0c08ac6fd Mon Sep 17 00:00:00 2001 From: Sagar Utekar Date: Thu, 10 Apr 2025 02:36:02 +0530 Subject: [PATCH 086/133] Update readme.md --- DailyChallenges/Season2/Day11/readme.md | 170 ++++++++++++------------ 1 file changed, 87 insertions(+), 83 deletions(-) diff --git a/DailyChallenges/Season2/Day11/readme.md b/DailyChallenges/Season2/Day11/readme.md index b9037c2..7fc598a 100644 --- a/DailyChallenges/Season2/Day11/readme.md +++ b/DailyChallenges/Season2/Day11/readme.md @@ -22,86 +22,90 @@ Grab your terminal—it’s time to take down the text chaos! In UNIX and Linux systems, configuration and information management have traditionally relied on plain-text files. Despite the availability of graphical tools today, editing these files manually with a text editor remains essential, especially for full system control and on servers without GUIs. Aspiring system administrators must learn to use text editors and tools like find (to locate files) and grep (to search within files) to manage and configure Linux systems effectively. -# vi/vim Cheat Sheet – Quick Reference - -## Modes -| Mode | Key | Description | -|-----------------|-------------|------------------------------------| -| Command | Default | For navigation and actions | -| Insert | `i` / `a` / `o` | Enter insert mode (insert, append, open) | -| Return to Command | `Esc` | Switch from insert to command mode | - -## Insert Mode Commands -| Command | Description | -|---------|---------------------------------| -| `i` | Insert before the cursor | -| `a` | Append after the cursor | -| `I` | Insert at the beginning of the line | -| `A` | Append at the end of the line | -| `o` | Open new line below | -| `O` | Open new line above | -| `Esc` | Return to command mode | - -## Navigation -| Key | Moves Cursor To | -|-------------|-------------------------------| -| `h` / `l` | Left / Right one character | -| `j` / `k` | Down / Up one line | -| `w` / `W` | Next word (by punctuation / space) | -| `b` / `B` | Previous word | -| `0` | Start of line | -| `$` | End of line | -| `H` | Top of screen | -| `M` | Middle of screen | -| `L` | Bottom of screen | -| `G` | Last line of file | -| `1G` | First line of file | -| `35G` | Line 35 | -| `Ctrl+f` | Page forward | -| `Ctrl+b` | Page backward | -| `Ctrl+d` | Half page forward | -| `Ctrl+u` | Half page backward | - -## ✂️ Editing Text -| Command | Description | -|-------------|-----------------------------------| -| `x` / `X` | Delete character under / before cursor | -| `dd` | Delete current line | -| `dw` | Delete word after cursor | -| `db` | Delete word before cursor | -| `d$` | Delete from cursor to end of line | -| `c + move` | Change text (e.g., `c$` changes to end) | -| `cc` | Change entire line | -| `yy` | Yank (copy) current line | -| `y}` | Yank paragraph | -| `p` | Paste below/after | -| `P` | Paste above/before | -| `.` | Repeat last change | - -## Searching -| Command | Description | -|------------------|---------------------------------------| -| `/text` | Search forward for "text" | -| `?text` | Search backward for "text" | -| `n` / `N` | Repeat search forward / backward | -| `/The.*end` | Regex: match “The” followed by “end” | -| `?[Pp]rint` | Case-insensitive search for “Print” or “print” | - -## Ex Commands (use `:`) -| Command | Description | -|-------------------|---------------------------------------| -| `:w` | Save file | -| `:q` | Quit if no unsaved changes | -| `:wq` / `ZZ` | Save and quit | -| `:q!` | Quit without saving | -| `:s/old/new/` | Replace first “old” with “new” in current line | -| `:g/old/s//new/g` | Replace all “old” with “new” in file | -| `:g/old/s//new/gp`| Replace and print affected lines | - -## Pro Tips -| Command | Description | -|-----------|---------------------------------------| -| `u` | Undo last change | -| `Ctrl+r` | Redo (undo the undo) | -| `:!command`| Run shell command (e.g., `:!date`, `:!ls`) | -| `Ctrl+G` | Show filename, line number, and position | + ## Practical Tasks Setup: Linux Text Avengers Training Camp + +Welcome, recruits! You've been selected to join the Linux Text Avengers, the elite team responsible for manipulating text files to save the world (or at least your systems). Get ready for a series of challenges that will test your text-fu and prepare you for any text-based crisis. + +### 1. Create Your Base of Operations + +* Every hero needs a base! Create your training directory: + +```bash +mkdir ~/textlab +cd ~/textlab +``` + + +### 2. The Intelligence File + +* Your first mission requires analyzing an intelligence file. Create `myfile.txt` with this vital information: + +```bash +cat << EOF > myfile.txt +Line 1: This is a test file. Initializing systems... +Line 2: Administrator access granted. Beware of rogue agents. +Line 3: Memory usage: 500MB. System stable. +Line 4: Error log entry: Unauthorized access attempt detected. +Line 5: IP address: 192.168.1.100. Target server. +Line 6: Password: secret123. DO NOT SHARE. +Line 7: End of file. Mission critical. +EOF +``` + + +### 3. Log Simulation: The Enemy's Activity + +* Simulate enemy activity by creating fake logs. The logs directory will track their actions. + +```bash +mkdir ~/textlab/logs +echo "error: disk full. System compromised!" > ~/textlab/logs/log1.log +echo "ERROR: network down. Communications disrupted!" > ~/textlab/logs/log2.log +echo "warning: low memory. System stability threatened!" > ~/textlab/logs/log3.log +# Make log1.log larger than 1MB to simulate a real log. +for i in {1..10000}; do echo "filler line $i" >> ~/textlab/logs/log1.log; done +``` + + +### 4. System Check + +* Before you begin, ensure you have the necessary tools and access. + * You should be able to read system files like `/etc/passwd` and `/var/log/messages` (some tasks might require sudo). + * Install `mailx` for communication tasks: + +```bash +sudo dnf install mailx # Or sudo apt install mailutils on Debian/Ubuntu +``` + + +### Mission Briefing: Data Analysis + +* Our world depends on your ability to analyze and manipulate text data. Every line in `myfile.txt` is critical: + * Line 1: Initialization status. + * Line 2: Administrator alert, with a hidden warning. + * Line 3: Memory info to optimize system. + * Line 4: Error log for security checks. + * Line 5: Target IP address. + * Line 6: A Secret! + + +## Operation: Text Manipulation + +### Original Tasks: Core Skills Training + +1. **Decoding the Intelligence:** Show line 5 from `~/textlab/myfile.txt` using two different methods. What does this IP address represent? +2. **Hunting the Target:** Locate text files in `~/textlab` containing `"192.168.1.100"`. Is a regex needed for this simple search? +3. **Reversing the Sabotage:** A rogue agent changed `"Administrator"` to `"root"` in `~/textlab/myfile.txt` using `sed`. Can you undo this sabotage *without* a backup? +4. **Memory Optimization:** Sort `myfile.txt` lines, simulating system performance by putting the largest memory value first (based on line 3). This tests ability to analyze numerical info within text. +5. **System Recon:** Extract the sixth column from `ps aux` to understand system usage. (This is a unchanged recon mission, unrelated to the rest of this scenario). +6. **Deleting Sensitive Info:** Remove the potentially compromised "Password" line (line 6) from `~/textlab/myfile.txt`. + +### Additional Interesting Tasks: Advanced Missions + +1. **Swapping Intel:** Swap lines 3 and 4 in `~/textlab/myfile.txt` using `vi/vim` to reorganize mission-critical info. +2. **Archiving Enemy Logs:** Find all `.log` files larger than 1MB under `~/textlab/logs`, and archive them to a `.tar.gz` in `~/textlab` for analysis. Think of this step as containing damage after an attack. +3. **Finding Patterns:** Use `grep`, `sort`, and `uniq` to count unique words with `"error"` (case-insensitive) in `~/textlab/logs/log1.log`. What does that data signify about system stability? +4. **Alerting the Team:** Search `~/textlab` for `"password"`, highlight it in color, and email this vital alert to yourself using `mail`. (Configure `mail` first if you haven't done so) + + + From a8b57407c9623d13919ca6211c4d9ae800c3fc09 Mon Sep 17 00:00:00 2001 From: Sagar Utekar Date: Thu, 10 Apr 2025 02:40:40 +0530 Subject: [PATCH 087/133] Update readme.md --- DailyChallenges/Season2/Day11/readme.md | 20 ++++++++++++++++++++ 1 file changed, 20 insertions(+) diff --git a/DailyChallenges/Season2/Day11/readme.md b/DailyChallenges/Season2/Day11/readme.md index 7fc598a..8de0b4a 100644 --- a/DailyChallenges/Season2/Day11/readme.md +++ b/DailyChallenges/Season2/Day11/readme.md @@ -108,4 +108,24 @@ sudo dnf install mailx # Or sudo apt install mailutils on Debian/Ubuntu 4. **Alerting the Team:** Search `~/textlab` for `"password"`, highlight it in color, and email this vital alert to yourself using `mail`. (Configure `mail` first if you haven't done so) +# Submission Guidelines +- Store your findings and execution steps in a markdown file (`solution.md`). +- Submit it in your GitHub repository and share the link. +- Post your experience on social media with **#getfitwithsagar #SRELife #DevOpsForAll**. +- Tag us in your posts for visibility and networking. +### **Share Your Progress!** +Post your experience on social media with the hashtags: **#getfitwithsagar, #SRELife, #DevOpsForAll** + +--- + +## **Join Our Community** +Connect with fellow learners: +- **Discord:** [Join here](https://discord.gg/mNDm39qB8t) +- **Google Group:** [Join here](https://groups.google.com/forum/#!forum/daily-devops-sre-challenge-series/join) +- **YouTube:** [Subscribe here](https://www.youtube.com/@Sagar.Utekar) + +--- + +Keep learning and happy exploring! +Sagar Utekar From 70504c5f660b6535229d3d91702c7c34434bb558 Mon Sep 17 00:00:00 2001 From: Sagar Utekar Date: Fri, 11 Apr 2025 06:16:14 +0530 Subject: [PATCH 088/133] Delete DailyChallenges/Season2/Linux/Install Linux.txt --- .../Season2/Linux/Install Linux.txt | 41 ------------------- 1 file changed, 41 deletions(-) delete mode 100644 DailyChallenges/Season2/Linux/Install Linux.txt diff --git a/DailyChallenges/Season2/Linux/Install Linux.txt b/DailyChallenges/Season2/Linux/Install Linux.txt deleted file mode 100644 index 64d637e..0000000 --- a/DailyChallenges/Season2/Linux/Install Linux.txt +++ /dev/null @@ -1,41 +0,0 @@ -Why Virtualization? -Many a times, we need to or prefer running Linux for development or for personal use. But it is not always possible to allocate a seperate machine for running Linux. There are multiple ways of running Linux without having a seperate machine for the same. One of the ways is to dual boot a system wherein you make a choise of which operating system to use when staring a machine. But if you do not want to go the dual boot way, then creating a virtual machine or creating a VM on a cloud system line AWS is alao an option. Mentioned below are the two ways on creating a linux instance for your perusal. - -1. Install using Oracle Virtual Box - 1.1 First step is to download Oracle Virtual Box. - A. Go to https://www.oracle.com/in/virtualization/technologies/vm/downloads/virtualbox-downloads.html. - B. Download the appropriate version on Virtual Box based on the operating system being used. [For this tutorial, we will assume that Windows is the base operating system being used]. - C. Since the operating system is assumed to be Windows, double click on the installer and follow the on-screen instructions for install Oracle Virtual Box. You can also follow the detailed instruction on [https://www.virtualbox.org/manual/] this page. - 1.2 Second Step is to download the latest version of Ubuntu - A. Download the latest version of Ubuntu from https://ubuntu.com/download/desktop - B. By default, on Windows, the ISO image will downloaded under the Download folder. - C. Create a separate folder on your hard disk and move the downloaded file to the same. - 1.3 Creating a New VM - [Before starting this step, please ensure that you have downloaded and installed Oracle Virtual Box as well as downloaded the latest version of Ubuntu and moved the same from the Download folder to a sepearte folder.] - A. Start Oracle Virtual Box. - B. Click on the New Icon or Select Machine -> New from the Menu Bar. - C. Give a suitable to your Virtual Machine. In case you plan to use multiple Linux OS's then naming them makes it easy to access them. In this case, name the virtual machine as Ubuntu. - D. The next step is the specify where will the VM file be stored. In case you have multiple drives, Select/Create a folder on the drive which has a reasonable amount of space. - E. Next step is to select the OS downloade by us as part of Step 1.2. Click on the Down Button and then Click on Other option. Navigate to the folder where the downloade Ubuntu ISO Image has been kept and select the same. - F. Click on Unattended Install Option. This will help us install Ubuntu unattened. - F.1 Enter an User Name. - F.2 Enter and then confirm a password. - G. Select the Hardware Option. Here you can define the amount of Ram and the number of CPU's to be used by Vurtual Box. - H. Select the Hardisk option and define the amount of space to the used on the Hard Disk based on availability. - I. Click on Finish to complete the configuration process. - J. Click on Start button the main screen to start the installation process. Note: This is a one time process that happens when you start the Ubuntu Virtual Machine the first time. - K. You can also refer to this tutorial [https://ubuntu.com/tutorials/how-to-run-ubuntu-desktop-on-a-virtual-machine-using-virtualbox#1-overview] from Ubuntu for a detailed understanding of the process. -2. Creating VM om AWS - 1.1. Create an accoount on AWS. - 1.2. Log in to your AWS account and navigate to the EC2 service in the AWS Management Console - 1.3. Give a relevant name for your instance. [eg: My Linux Machine] - 1.4. Choose an Instance Type. For this tutorial, I have selected an Ubuntu Instance. - 1.5. Choose an appropriate instance type based on your requirement. - 1.6. Create a Key Pair to securly login to your instance. For this click on the Create New Key Pair button and select RSA. Give a relevant Key Pair name to identify the same. Select PEM file for Private Key File Format. This file will be downloaded to your Download directory. Shift the same to a more secure location on your har disk. - 1.7. Under Network Setting, - 1.7.1. Select Create Security Group. This is selected by default. In case, you have already created a Security Group, you can use the same. - 1.7.2. Select the Allow SSH Traffic From Check Box. Here, select My IP from the drop down. - 1.8. Configure the storage in case your use case demands additional storage requirment. If not, go with the default. - 1.9. Click on Launch Instance to start the Ubuntu Machine. - 1.10. Click on the connect button to connect to the Ubuntu Instance. - From eea9a4d6f7282c360b0d58ba76ad45cbd67bc1d5 Mon Sep 17 00:00:00 2001 From: Sagar Utekar Date: Fri, 11 Apr 2025 06:16:47 +0530 Subject: [PATCH 089/133] Delete DailyChallenges/Season2/Linux directory --- DailyChallenges/Season2/Linux/module1.md | 110 ---------------------- DailyChallenges/Season2/Linux/syllabus.md | 1 - 2 files changed, 111 deletions(-) delete mode 100644 DailyChallenges/Season2/Linux/module1.md delete mode 100644 DailyChallenges/Season2/Linux/syllabus.md diff --git a/DailyChallenges/Season2/Linux/module1.md b/DailyChallenges/Season2/Linux/module1.md deleted file mode 100644 index e6794d7..0000000 --- a/DailyChallenges/Season2/Linux/module1.md +++ /dev/null @@ -1,110 +0,0 @@ - -## What is OS? - -An Operating System (OS) is a software that manages everything your computer does and makes sure all the parts work together properly. Without it, your computer wouldn’t know how to run programs, connect to the internet, or even display anything on the screen. - -### Key Functions of an Operating System: -- Manages Hardware: It controls the CPU (processor), memory, and other devices (like printers, keyboards, and screens). -- Runs Applications: It allows you to open apps like browsers, games, or word processors. -- Provides User Interface: It gives you a way to interact with the computer, like the Windows desktop or your smartphone’s screen. -- Manages Files and Folders: It helps you store and organize your files properly. - -### Examples of Operating Systems: -- Windows – Used on many PCs. -- macOS – Used on Apple computers. -- Linux – Often used by developers and servers. -- Android/iOS – Operating systems for smartphones. - -In simple terms, without an operating system, your computer is just a box of hardware with no idea how to do anything! - -## Operating System Architecture - -An Operating System (OS) is built using several layers, each with its own purpose. - -![image](https://github.com/user-attachments/assets/26678a2d-b4c6-4180-83d8-256daef70eab) - -### Here’s a breakdown of the architecture from top to bottom: -- User Interface (UI): - - This is what users see and interact with. - - It can be a Graphical User Interface (GUI) (like icons, windows, and buttons) or a Command-Line Interface (CLI) (where you type commands). - - Example: The desktop screen in Windows, the home screen on Android and shell on Linux. - -- Application Layer: - - This layer runs applications like browsers, media players, or text editors. - - These apps send requests to the operating system when they need to access hardware or files. - - Example: When you save a document in Word, the app sends a request to the OS to store it on your hard disk. - -- Kernel: - - This is the core part of the operating system. - - It manages essential tasks like: - - Process Management – Handling running apps. - - Memory Management – Allocating memory to processes. - - Device Management – Managing input/output devices like printers, keyboards, and hard drives. - - File System Management – Handling how files are read, stored, and accessed. - - The kernel acts as a bridge between applications and hardware. - -- Hardware: - - This includes the physical components of your computer, like the CPU (processor), RAM (memory), hard disk, and other devices. - - The OS controls and coordinates how these components function. - -### Flow of Communication: -- Top to Bottom: When you open an app, the request flows from the user interface to the kernel, which manages hardware resources to complete the task. -- Bottom to Top: When hardware generates data (like keyboard input), it passes through the kernel and reaches the user via the user interface. - -## What is Unix? -Unix is a family of operating systems that began at AT&T in the late 1960s. It was designed to be a powerful, multiuser, and multitasking system. - -Some members of Unix family are: BSD, Xenix, Solaris, HP-UX and AIX. - -**Key points about Unix include:** - -- **Multiuser and Multitasking**: Multiple users can work on the same system at the same time. -- **Portability**: Unix was written in the C programming language, making it easier to run on different hardware. -- **Influence**: Many modern operating systems, including Linux, trace their ideas and design principles back to Unix. - -**History:** -- 1969-1970: Unix was developed at AT&T by Ken Thompson, Dennis Ritchie, and others. -- 1970s: It spread to universities and became popular for its simplicity and powerful tools. -- Impact: Unix’s design principles laid the groundwork for many modern operating systems. - -## What is Linux? - -Linux is a _Unix-like_ operating system kernel originally created in 1991 by Linus Torvalds while he was a student in Helsinki. When combined with a vast collection of GNU software and other applications, it forms a complete operating system—often called GNU/Linux or simply Linux. - -**Here are some key points:** -- **Open Source**: Linux is developed and maintained by a global community under the GNU General Public License (GPL), meaning its source code is freely available. -- **Flexibility**: It is highly modular, allowing users to customize everything from the kernel to the desktop interface. -- **Wide Usage**: Linux powers everything from smartphones (via Android) and personal computers to servers, supercomputers, and embedded systems. - -**History:** -- 1991: Linus Torvalds begins a project to create a free kernel for personal computers, announcing Linux in a famous Usenet post. -- 1992: The Linux kernel is relicensed under the GNU GPL, encouraging community contributions. -- Growth: Over the years, Linux has grown from a small project into a robust, full-featured operating system used worldwide. -- Collaboration with GNU: By combining the Linux kernel with GNU tools and libraries, users get a complete free operating system. - -## Different Linux Distros - -Because Linux is free and open source, various groups—both community-driven and commercial—have built their own “flavors” of Linux. These are called distributions or distros. They package the Linux kernel along with different sets of software to suit particular needs. Some examples include: - -- Ubuntu: - Known for its user-friendly design, Ubuntu is popular for desktops and servers. It’s based on Debian and is widely supported by a strong community and Canonical Ltd. - -- Red Hat Enterprise Linux (RHEL): - It is a commercial open-source Linux distribution developed by Red Hat for the commercial market. It is designed to provide a stable, secure, and high-performance platform for various applications, from servers to desktops. - -- openSUSE: - openSUSE is a community-driven project that provides two main Linux distributions:​ openSUSE Leap & openSUSE Tumbleweed. - -- Fedora: - Backed by Red Hat, Fedora is a cutting-edge distro that showcases the latest free software and innovations, often serving as a testing ground for features that later appear in Red Hat Enterprise Linux. - -- Linux Mint: - Based on Ubuntu or Debian, Linux Mint is designed to be easy for beginners with a familiar desktop interface and multimedia support out-of-the-box. - -- Arch Linux: - A minimalist, rolling-release distro for experienced users. Arch Linux offers extensive customizability and a “build it yourself” approach using the Pacman package manager. - -- Others: - There are hundreds of distributions targeting specific use cases—from lightweight distros for older hardware to specialized distros for security testing (like Kali Linux) or multimedia production. - -Each distro has its own package management, default desktop environment, and tools that cater to different user preferences and system requirements. diff --git a/DailyChallenges/Season2/Linux/syllabus.md b/DailyChallenges/Season2/Linux/syllabus.md deleted file mode 100644 index 8b13789..0000000 --- a/DailyChallenges/Season2/Linux/syllabus.md +++ /dev/null @@ -1 +0,0 @@ - From 6c4981d01e98adadf6345dfda1a45e7dbd9c8c98 Mon Sep 17 00:00:00 2001 From: Sagar Utekar Date: Fri, 11 Apr 2025 07:01:30 +0530 Subject: [PATCH 090/133] Create readme.md --- DailyChallenges/Season2/Day12/readme.md | 468 ++++++++++++++++++++++++ 1 file changed, 468 insertions(+) create mode 100644 DailyChallenges/Season2/Day12/readme.md diff --git a/DailyChallenges/Season2/Day12/readme.md b/DailyChallenges/Season2/Day12/readme.md new file mode 100644 index 0000000..3dcb5b0 --- /dev/null +++ b/DailyChallenges/Season2/Day12/readme.md @@ -0,0 +1,468 @@ +# Daily DevOps + SRE Challenge Series – Season 2 +## Day 12: The Process Power-Up – Command Your Linux System Like a Pro + +### Introduction +Welcome to **Day 12** of the Daily DevOps + SRE Challenge Series – Season 2! 🎉 + +Today, you’re stepping into the control room of your Linux system, mastering the art of **process management**. Processes are the heartbeat of any system—every command, script, or service runs as a process. Knowing how to monitor, prioritize, and control them is like wielding a superpower for debugging, optimizing, and keeping systems humming. By the end of this challenge, you’ll: +- Monitor running processes with precision using tools like `ps` and `top`. +- Control process priorities to optimize system performance. +- Manage background jobs and terminate rogue processes like a seasoned SRE. + +#### Why This Matters? +Process management is critical for DevOps and SRE roles: +- **System Health**: Spot CPU-hogging processes before they crash your app. +- **Automation**: Script `kill` or `nice` commands to automate resource management. +- **Outage Recovery**: During incidents, `top` and `pkill` help you isolate and stop problematic services. +- **Interview Win**: Questions like “How do you find a zombie process?” or “What’s the difference between `SIGTERM` and `SIGKILL`?” are common. +- **Real-World Impact**: An SRE at a fintech firm used `htop` to trace a memory leak in a trading app, saving millions by preventing downtime. + +Grab your terminal and let’s power up your process-fu! + +--- + +### 1. Understanding Processes +- **Definition**: A process is an instance of a running program, identified by a unique **Process ID (PID)**. Every command (e.g., `ls`), script, or service (e.g., `httpd`) creates a process. +- **Attributes**: + - **PID**: Unique identifier (e.g., 1234). + - **PPID**: Parent Process ID, linking to the process that started it (e.g., `bash` spawns `ls`). + - **User**: The account running the process (e.g., `root` or `user1`). + - **State**: Current status: + - **R** (Running): Actively using CPU. + - **S** (Sleeping): Waiting for an event (e.g., I/O). + - **D** (Uninterruptible Sleep): Waiting for I/O, can’t be killed easily. + - **Z** (Zombie): Defunct, awaiting parent cleanup. + - **T** (Stopped): Paused, often via `Ctrl+Z`. +- **Types**: + - **Foreground**: Occupies the terminal (e.g., `vim`). + - **Background**: Runs without terminal control (e.g., `sleep 100 &`). + - **Daemons**: System services running continuously (e.g., `sshd`). +- **Process Creation**: Processes are forked (copied) from a parent, often `init` (PID 1) or `systemd` in RHEL. + +### 2. Monitoring Processes +To manage processes, you first need to see what’s running. RHEL provides several tools for this purpose. + +- **`ps` (Process Status)**: + - Displays a snapshot of processes. + - **Key Options**: + - `-a`: Show all processes with a terminal (excludes daemons). + - `-u user`: Filter by user (e.g., `ps -u root`). + - `-x`: Include processes without a terminal (e.g., daemons). + - `aux`: BSD-style, shows all processes (user, PID, %CPU, %MEM, command). + - `-e`: Show every process (System V style). + - `-f`: Full format (includes PPID, start time). + - `-ef`: Combines `-e` and `-f` for hierarchy view. + - `-p PID`: Select specific PID (e.g., `ps -p 1234`). + - `-C command`: Select by command name (e.g., `ps -C httpd`). + - `--sort=field`: Sort output (e.g., `ps aux --sort=-%cpu` for highest CPU first). + - `-o field1,field2`: Customize columns (e.g., `ps -o pid,comm,state`). + - `-l`: Long format (includes nice value, priority). + - Common options: + - `ps aux`: Shows all processes in BSD format (user, PID, %CPU, %MEM, command). + - `ps -ef`: Displays all processes in System V format, including PPID. + - `ps -u username`: Lists processes for a specific user. + - `ps -p 1234`: Shows details for PID 1234. + - Example: + ```bash + ps aux | grep httpd + ``` + Lists all `httpd` processes with resource usage. + - **Customization**: Use `ps -o pid,ppid,comm,state` to select specific columns (PID, PPID, command, state). + +- **`top`**: + - Real-time, interactive process viewer. + - Key features: + - Displays CPU, memory, and load averages. + - Sort by CPU (`P`), memory (`M`), or PID (`N`). + - Kill processes: Press `k`, enter PID, select signal (e.g., 15 for `SIGTERM`). + - Adjust priority: Press `r`, enter PID, set nice value. + - Example: + ```bash + top + ``` + Press `q` to quit. + - **Fields**: + - `%CPU`: CPU usage percentage. + - `%MEM`: Memory usage percentage. + - `NI`: Nice value (priority). + - `S`: Process state (R, S, Z, etc.). + +- **`uptime`**: + - Shows system load averages (1, 5, 15 minutes). + - Example: + ```bash + uptime + ``` + Output: `12:34:56 up 2 days, 3:45, 2 users, load average: 0.50, 0.75, 1.00`. + - **Interpretation**: A load average above the number of CPU cores indicates resource contention. + +### 3. Controlling Processes with Signals +Processes communicate via **signals**, which instruct them to perform actions like stopping or restarting. + +- **Common Signals**: + - `SIGTERM` (15): Requests graceful termination (default for `kill`). + - `SIGKILL` (9): Forces immediate termination (use cautiously, as it doesn’t allow cleanup). + - `SIGHUP` (1): Reloads configuration for daemons (e.g., `kill -1 1234`). + - `SIGSTOP` (19): Pauses a process (resumed with `SIGCONT`). + - `SIGCONT` (18): Resumes a paused process. +- **Commands**: + - **`kill`**: Sends a signal to a PID. + ```bash + kill -15 1234 # Graceful stop + kill -9 1234 # Force kill + ``` + - **`killall`**: Sends a signal to all processes matching a name. + ```bash + killall -15 httpd # Stops all httpd processes + ``` + - **`pkill`**: Similar to `killall`, but supports patterns. + ```bash + pkill -u user1 # Terminates all processes for user1 + ``` + +### 4. Managing Process Priorities +CPU resources are allocated based on process priority, controlled by **nice** values (-20 highest, +19 lowest). + +- **`nice`**: + - Sets priority when starting a process. + - Example: + ```bash + nice -n 10 sleep 1000 # Runs with low priority + ``` + - Range: `-20` (high priority) to `+19` (low priority, default 0). + +- **`renice`**: + - Adjusts priority of a running process. + - Example: + ```bash + renice 5 -p 1234 # Sets nice value to 5 for PID 1234 + ``` + - Requires `root` for increasing priority (lowering nice value below 0). + +- **Verification**: + - Check nice value with `ps -l` (column `NI`) or `top` (press `f`, select `NI`). + - Example: + ```bash + ps -lp 1234 + ``` + +### 5. Job Control +Job control manages foreground and background processes within a terminal session. + +- **Foreground Processes**: + - Run interactively, locking the terminal (e.g., `vim`). +- **Background Processes**: + - Run without terminal control, freeing it for other tasks. + - Start with `&`: + ```bash + sleep 1000 & + ``` +- **Key Commands**: + - **`jobs`**: Lists background/stopped jobs. + ```bash + jobs + ``` + Output: `[1]+ Running sleep 1000 &`. + - **`fg`**: Brings a job to the foreground. + ```bash + fg %1 # Brings job 1 to foreground + ``` + - **`bg`**: Resumes a stopped job in the background. + ```bash + bg %1 + ``` + - **Suspend**: Press `Ctrl+Z` to pause a foreground process. + - **Terminate**: Use `Ctrl+C` to stop a foreground process. + +- **Example Workflow**: + ```bash + sleep 1000 # Runs in foreground + Ctrl+Z # Suspends + jobs # Shows [1]+ Stopped + bg %1 # Resumes in background + fg %1 # Brings back to foreground + Ctrl+C # Terminates + ``` + +### 6. Advanced Monitoring Tools +For deeper insights, RHEL offers additional tools. + +- **`pidstat`** (part of `sysstat` package): + - Monitors CPU, memory, or I/O per process. + - Example: + ```bash + pidstat -u 1 5 # CPU usage every second for 5 iterations + ``` +- **`lsof`**: + - Lists open files associated with a process. + - Example: + ```bash + lsof -p 1234 # Files opened by PID 1234 + ``` +- **`free`**: + - Shows system memory usage, useful alongside `top` for memory-intensive processes. + ```bash + free -h + ``` + +### 7. Process Troubleshooting +- **High CPU Usage**: + - Use `top` or `ps aux --sort=-%cpu` to identify culprits. + - Lower priority with `renice` or terminate with `kill`. +- **Zombies**: + - Find with `ps aux | grep Z`. + - Kill parent process to clean up (e.g., `kill -9 `). +- **Out-of-Memory Issues**: + - Check `free -h` and `top` (%MEM column). + - Terminate non-critical processes or adjust `nice` values. + +### 8. Systemd and Processes +- Processes like `httpd` or `sshd` are managed by `systemd` units. +- Check service processes: + ```bash + systemctl status httpd + ``` + Shows main PID and recent logs. +- Stop a service to kill its processes: + ```bash + systemctl stop httpd + ``` + +## Practical Tasks: Operation Server Rescue: Diagnose, Stabilize, and Optimize +As an SRE on call, your task is to diagnose, stabilize, and optimize a production server under heavy load from rogue processes. This guide assumes a Fedora or RHEL system. + +### Setup the Crisis + +1. **Create a Workspace**: + +```bash +mkdir ~/processlab +cd ~/processlab +``` + +2. **Simulate Server Stress**: + - **CPU-intensive script**: + +```bash +echo 'while true; do :; done' > cpu_hog.sh +chmod +x cpu_hog.sh +``` + + - **Memory-heavy script**: + +```bash +echo 'python3 -c "while True: a = [0] * 1000000"' > mem_hog.sh +chmod +x mem_hog.sh +``` + + - **Sleep script**: + +```bash +echo 'sleep 3600' > sleeper.sh +chmod +x sleeper.sh +``` + +3. **Install Tools (if needed)**: + +```bash +sudo dnf install python3 procps-ng sysstat +``` + + +### Task: Stabilize the Server + +#### Assess the Load + +1. **Check Load Averages**: +Run `uptime` to check load averages and save output to `~/processlab/load.txt`. + +```bash +uptime > ~/processlab/load.txt +``` + +2. **Note CPU Cores**: +Use `lscpu` to note the number of CPU cores and save to `~/processlab/cpu_info.txt`. + +```bash +lscpu > ~/processlab/cpu_info.txt +``` + +3. **Compare Load to Cores**: +High load (>2 on 2 cores) confirms stress. + +#### Survey Running Processes + +1. **List All Processes**: +List all processes with `ps aux`, pipe to `less` for paging, and save to `~/processlab/all_processes.txt`. + +```bash +ps aux > ~/processlab/all_processes.txt +``` + +2. **List Processes Sorted by User**: +Append the list of processes sorted by user to `~/processlab/all_processes.txt`. + +```bash +ps aux --sort=user >> ~/processlab/all_processes.txt +``` + +3. **Show Custom Columns**: +Show custom columns (PID, user, group, VSZ, RSS, command) with `ps -e -o pid,user,group,vsz,rss,comm` and save to `~/processlab/custom_processes.txt`. + +```bash +ps -e -o pid,user,group,vsz,rss,comm > ~/processlab/custom_processes.txt +``` + + +#### Launch Rogue Processes + +1. **Start Background Jobs**: + +```bash +./cpu_hog.sh & +./mem_hog.sh & +./sleeper.sh & +``` + +2. **Start a Foreground Job**: + +```bash +./cpu_hog.sh +``` + +Suspend it with `Ctrl+Z`. + +#### Manage Jobs + +1. **List Jobs**: +Run `jobs` to list all jobs (expect 3 running, 1 stopped). +2. **Resume Stopped Job**: +Resume the stopped `cpu_hog.sh` job in the background with `bg`. +3. **Bring Job to Foreground and Terminate**: +Bring the first `sleeper.sh` job to the foreground with `fg %1`, then terminate it with `Ctrl+C`. +4. **Confirm Job Removal**: +Check `jobs` to confirm `sleeper.sh` is gone. + +#### Monitor in Real-Time + +1. **Run Top**: +Run `top`, sort by CPU (P), then memory (M). +2. **Identify mem_hog.sh PID**: +Identify the `mem_hog.sh` PID (high `%MEM`). +3. **Terminate mem_hog.sh**: +In `top`, press `k`, enter the `mem_hog.sh` PID, and send `SIGTERM` (15) to stop it gracefully. +4. **Quit Top**: +Quit `top` with `q`. + +#### Adjust Priorities + +1. **Find PID of cpu_hog.sh**: +Find the PID of one `cpu_hog.sh` with `ps aux | grep cpu_hog`. +2. **Set Nice Value to +5**: + +```bash +renice 5 -p <PID> +``` + +3. **Increase Priority to -5**: + +```bash +sudo renice -5 -p <PID> +``` + +4. **Verify Priority Change**: +Verify with `ps -lp <PID>` (check NI column). Save output to `~/processlab/priority.txt`. + +```bash +ps -lp <PID> > ~/processlab/priority.txt +``` + + +#### Trace Process Hierarchy + +1. **Show Process Hierarchy**: +Run `ps fax | grep -B5 cpu_hog` to show `cpu_hog.sh` processes and their parent shell. +2. **Save Hierarchy**: +Save output to `~/processlab/hierarchy.txt`. + +```bash +ps fax | grep -B5 cpu_hog > ~/processlab/hierarchy.txt +``` + + +#### Clean Up Stragglers + +1. **Start Another Rogue Process**: +From a second terminal, start another rogue process: + +```bash +./cpu_hog.sh & +exit +``` + +2. **Confirm Process Running**: +Back in the first terminal, use `ps aux | grep cpu_hog` to confirm it’s still running. +3. **Terminate All cpu_hog.sh Processes**: + +```bash +killall -15 cpu_hog.sh +``` + +4. **Verify Termination**: +Verify with `ps aux | grep cpu_hog` (no results). + +#### Final Sweep + +1. **Check Load Again**: +Check load again with `uptime`. Append to `~/processlab/load.txt`. + +```bash +uptime >> ~/processlab/load.txt +``` + +2. **Force-Stop Remaining Processes**: +If load is still high, use `killall -9 sleeper.sh` to force-stop any remaining processes. +3. **Save Final Processes**: +Save final `ps aux` output to `~/processlab/final_processes.txt`. + +```bash +ps aux > ~/processlab/final_processes.txt +``` + + +#### Document Findings + +In `~/processlab/notes.txt`, document: + +- Which process caused the most stress (`cpu_hog.sh` or `mem_hog.sh`). +- How load changed after cleanup. +- One lesson learned (e.g., “`SIGTERM` is safer than `SIGKILL`”). + + +## **Submission Guidelines** +- Store your findings and execution steps in a markdown file (`solution.md`). +- Submit it in your GitHub repository and share the link. +- Post your experience on social media with **#getfitwithsagar #SRELife #DevOpsForAll**. +- Tag us in your posts for visibility and networking. + +--- + +## **Join Our Community** + +To make the most of this journey, connect with fellow learners: +- **Discord** – Ask questions and collaborate: [Join here](https://discord.gg/mNDm39qB8t) +- **Google Group** – Get updates and discussions: [Join here](https://groups.google.com/forum/#!forum/daily-devops-sre-challenge-series/join) +- **YouTube** – Watch solution videos and tutorials: [Subscribe here](https://www.youtube.com/@Sagar.Utekar) + +--- + +--- + +## **Stay Motivated!** + +Every challenge you complete brings you closer to mastering Git, DevOps, and SRE practices. Keep pushing your limits and collaborating with the community. + +Happy Learning! + +**Best regards,** +Sagar Utekar From d1151cc436c3a7e9e2e99f41662559c20a1ce6cf Mon Sep 17 00:00:00 2001 From: Sagar Utekar Date: Sun, 6 Jul 2025 15:34:53 +0530 Subject: [PATCH 091/133] Create readme.md --- DailyChallenges/ProjectChallengeSeries/Day1/readme.md | 1 + 1 file changed, 1 insertion(+) create mode 100644 DailyChallenges/ProjectChallengeSeries/Day1/readme.md diff --git a/DailyChallenges/ProjectChallengeSeries/Day1/readme.md b/DailyChallenges/ProjectChallengeSeries/Day1/readme.md new file mode 100644 index 0000000..8b13789 --- /dev/null +++ b/DailyChallenges/ProjectChallengeSeries/Day1/readme.md @@ -0,0 +1 @@ + From dd05037116a72b492a8bce4120b737de676e22aa Mon Sep 17 00:00:00 2001 From: Sagar Utekar Date: Sun, 6 Jul 2025 15:35:26 +0530 Subject: [PATCH 092/133] Create Day1.md --- DailyChallenges/ProjectChallengeSeries/project1/Day1.md | 1 + 1 file changed, 1 insertion(+) create mode 100644 DailyChallenges/ProjectChallengeSeries/project1/Day1.md diff --git a/DailyChallenges/ProjectChallengeSeries/project1/Day1.md b/DailyChallenges/ProjectChallengeSeries/project1/Day1.md new file mode 100644 index 0000000..8b13789 --- /dev/null +++ b/DailyChallenges/ProjectChallengeSeries/project1/Day1.md @@ -0,0 +1 @@ + From 630c08804bb519a4fafe2b23b677c286d47cedcb Mon Sep 17 00:00:00 2001 From: Sagar Utekar Date: Sun, 6 Jul 2025 16:08:50 +0530 Subject: [PATCH 093/133] Update Day1.md --- .../ProjectChallengeSeries/project1/Day1.md | 158 ++++++++++++++++++ 1 file changed, 158 insertions(+) diff --git a/DailyChallenges/ProjectChallengeSeries/project1/Day1.md b/DailyChallenges/ProjectChallengeSeries/project1/Day1.md index 8b13789..f4e76d9 100644 --- a/DailyChallenges/ProjectChallengeSeries/project1/Day1.md +++ b/DailyChallenges/ProjectChallengeSeries/project1/Day1.md @@ -1 +1,159 @@ +# Enterprise-Grade AKS Platform Deployment Series: Day 1 Kickoff +## Welcome to the Mission! + +Attention, platform engineers! You’re diving into a 7-day DevOps heist to build an **enterprise-grade Azure Kubernetes Service (AKS) platform** that’s secure, scalable, and production-ready. Your mission: deploy the OpenTelemetry Demo microservices using Terraform and other cutting-edge tools. Day 1 drops you into a high-stakes scenario where you’ll master **Terraform best practices** to create a rock-solid infrastructure foundation. Each day will unveil a new challenge, building toward a world-class platform so let’s execute this heist with precision! + +## Project Overview + +Your goal is to create a **multi-environment (dev, stage, prod) AKS platform** that’s automated, resilient, and enterprise-ready. Over the week, you’ll tackle automation, security, observability, and more, with surprises at every turn. Today, you’ll handle a real-world brownfield scenario using Terraform to provision and manage a private AKS infrastructure, mastering best practices like `terraform import`, `terraform refresh`, and modular design. + +## Day 1: Milestone 1 – Infrastructure Deployment with Terraform + +**Scenario**: +CloudHeist Inc. acquired a startup that manually deployed an Azure Virtual Network (VNet) and subnet for their AKS dev environment. Your task is to bring this existing VNet under Terraform management, then provision private AKS clusters, Azure Container Registry (ACR), and Azure Key Vault for dev and prod environments. You’ll use `terraform import` to manage the existing VNet, `terraform refresh` to align state, and Terraform workspaces to manage environments, ensuring a secure, private, and compliant infrastructure. + +**Objective**: +Provision a secure, highly available, private AKS infrastructure using Terraform, incorporating a pre-existing VNet, and master best practices like `terraform import`, `terraform refresh`, modules, and workspace management. + +**Tasks**: +- **Import Existing Resources**: Import the pre-existing VNet (`cloudheist-vnet`) and subnet (`aks-subnet`) in the `cloudheist-rg` resource group into Terraform state using `terraform import`. +- **State Refresh**: Run `terraform refresh` to align Terraform state with actual Azure resources, ensuring no discrepancies. +- **Modular Terraform**: Create reusable Terraform modules for AKS, networking, ACR, and Key Vault in a `modules/` directory. +- **Workspaces**: Use Terraform workspaces (`dev`, `staging`, `prod`) to manage environment-specific configurations with a single codebase. +- **Private AKS Cluster**: + - Deploy private AKS clusters with no public IP for the API server, using a private endpoint for communication between the control plane (in AKS-managed resource group) and node pools (in `cloudheist-rg`). + - Configure Azure CNI and availability zones for high availability. +- **Managed Identity**: + - Enable system-assigned managed identity on AKS clusters. + - Grant the managed identity permissions to create Azure Load Balancers, Managed Disks, and other dependent resources. +- **Authentication & Authorization**: + - Enable Azure AD workload identity federation on AKS, configuring `kubelogin` for authentication. + - Set up Azure RBAC for Kubernetes authorization, binding roles to Azure AD users or groups. +- **State Management**: Configure a secure remote backend in Azure Blob Storage (`tfstate12345`, container `tfstate`) with encryption and state locking via Azure Table Storage. +- **Provider Configuration**: Define the `azurerm` provider with version constraints in a `terraform` block in `provider.tf`. +- **Key Vault Integration**: Deploy Azure Key Vault with private endpoints, storing sensitive data (e.g., AKS admin credentials) and referencing them via Terraform `data` sources. +- **Provisioners**: Use `local-exec` provisioners to automate post-deployment tasks, like storing AKS kubeconfig in Key Vault. +- **Compliance and Governance**: Apply Azure Policy for AKS compliance (CIS benchmarks) and add Azure tags (`Environment`, `Project`) for cost tracking. + +**Terraform Best Practices**: +- **Modularity**: Create reusable Terraform modules for AKS, networking, ACR, and Key Vault to promote DRY principles. +- **Workspaces**: Use `terraform workspace` to manage dev and prod environments, isolating state files. +- **State Security**: Store Terraform state in Azure Blob Storage with encryption and locking to prevent conflicts. +- **Provider Versioning**: Pin `azurerm` provider versions in `provider.tf` to ensure consistent builds. +- **Sensitive Data**: Store secrets in Key Vault, avoiding hardcoding in `.tf` files. +- **Idempotency**: Ensure configurations are idempotent and reusable across environments. +- **State Management**: Use `terraform import` for brownfield resources and `terraform refresh` to align state with reality. +- **Documentation**: Document module inputs/outputs, workspace setup, import process, and state management in `README.md`. + +**Deliverables**: +- `infrastructure/` directory with: + - `main.tf`: Main configuration calling modules. + - `provider.tf`: Combined backend and provider configuration with a single `terraform` block. + - `variables.tf`: Variable declarations with types and defaults. + - `outputs.tf`: Outputs like AKS cluster IDs and Key Vault URIs. + - `terraform.tfvars`: Global variables. + - `modules/aks/`, `modules/networking/`, `modules/keyvault/`, `modules/acr/`: Reusable modules with `main.tf`, `variables.tf`, and `outputs.tf`. + - `environments/dev/`, `environments/prod/`: Environment-specific `terraform.tfvars`. + - `scripts/import-vnet.sh`: Shell script to import VNet and subnet using `terraform import`. + - `scripts/refresh-state.sh`: Shell script to run `terraform refresh`. +- Architecture diagram in Mermaid format showing infrastructure components. +- `./README.md` documenting: + - Terraform setup, module structure, and workspace usage. + - Detailed steps for `terraform import` and `terraform refresh`. + - State management, Key Vault integration, workload identity setup, and compliance details. + +**Success Criteria**: +- Existing VNet (`cloudheist-vnet`) and subnet (`aks-subnet`) in `cloudheist-rg` are imported into Terraform state using `terraform import`. +- `terraform refresh` aligns state with Azure resources without errors. +- AKS clusters are private (no public API server IP), use private endpoints for control plane communication, and have node pools in `cloudheist-rg` and control plane in AKS-managed resource group. +- System-assigned managed identity is enabled and used for creating Azure Load Balancers, Managed Disks, and other resources. +- Azure AD workload identity is configured with `kubelogin` for authentication, and Kubernetes RBAC is set up with Azure AD role bindings. +- Terraform state is encrypted, locked, and stored in Azure Blob Storage. +- Workspaces manage dev and prod environments with consistent configurations. +- Key Vault stores sensitive data, accessible via private endpoints. +- Provider versions are pinned in `provider.tf`, and infrastructure is tagged for cost tracking with Azure Policy applied for CIS compliance. + +**Starter Template**: +```bash +infrastructure/ +├── main.tf +├── provider.tf +├── variables.tf +├── outputs.tf +├── terraform.tfvars +├── modules/ +│ ├── aks/ +│ │ ├── main.tf +│ │ ├── variables.tf +│ │ └── outputs.tf +│ ├── networking/ +│ │ ├── main.tf +│ │ ├── variables.tf +│ │ └── outputs.tf +│ ├── keyvault/ +│ │ ├── main.tf +│ │ ├── variables.tf +│ │ └── outputs.tf +│ ├── acr/ +│ │ ├── main.tf +│ │ ├── variables.tf +│ │ └── outputs.tf +├── environments/ +│ ├── dev/ +│ │ └── terraform.tfvars +│ ├── prod/ +│ │ └── terraform.tfvars +├── scripts/ +│ ├── import-vnet.sh +│ └── refresh-state.sh +└── README.md +``` +## Architecture Sneak Peek + +Here’s the infrastructure you’ll build and manage today, setting the stage for an epic platform: + +```mermaid +flowchart TD + subgraph Azure + ACR[Azure Container Registry] + KV[Azure Key Vault] + subgraph AKS-Clusters + DEV[AKS Dev] + PROD[AKS Prod] + end + LA[Azure Monitor] + Storage[Azure Blob Storage - Terraform State] + Cost[Azure Cost Management] + end + + Terraform[Terraform]--import/refresh-->DEV + Terraform--modules-->DEV + Terraform--modules-->PROD + Terraform--modules-->ACR + Terraform--modules-->KV + Terraform--state-->Storage + Terraform--tags-->Cost + DEV--secrets-->KV + PROD--secrets-->KV + DEV--images-->ACR + PROD--images-->ACR + DEV--logs-->LA + PROD--logs-->LA +``` + +## What’s Next? + +Tomorrow, you’ll tackle the next challenge, diving into automation, security, and resilience. Each day will unlock new tools and techniques, building toward a fully automated, enterprise-grade AKS platform. Stay sharp—the heist is just getting started! + +## Getting Started + +1. Fork the repository: [LearnWithSagar/DailyChallenges](https://github.com/Sagar2366/LearnWithSagar). +2. Navigate to the `DailyChallenges/ProjectChallengeSeries/project1` directory to understand the problem statement. +3. Create VNet and Subnet Manually, Write Terraform plan and Initialize it: `terraform init` in `infrastructure/`. +4. Set up Terraform workspaces: `terraform workspace new dev`, `terraform workspace new staging` and `terraform workspace new prod`. +5. Import the existing VNet and subnet. +6. Deploy infrastructure with `terraform apply -var-file=environments/dev/terraform.tfvars`. +7. Document your setup in `README.md`, covering module structure, workspace usage, import process, state management, workload identity setup, and compliance. + +Good luck, and let’s execute this infrastructure heist flawlessly! From f218789b97d06e77d623ffd67ed29a79467e05ad Mon Sep 17 00:00:00 2001 From: Sagar Utekar Date: Mon, 18 Aug 2025 07:17:19 +0530 Subject: [PATCH 094/133] Create challenge.md --- DailyChallenges/Season2/Day13/challenge.md | 1 + 1 file changed, 1 insertion(+) create mode 100644 DailyChallenges/Season2/Day13/challenge.md diff --git a/DailyChallenges/Season2/Day13/challenge.md b/DailyChallenges/Season2/Day13/challenge.md new file mode 100644 index 0000000..8b13789 --- /dev/null +++ b/DailyChallenges/Season2/Day13/challenge.md @@ -0,0 +1 @@ + From d5936654bc7f69e45a6cfe8bee15d8e734fb7040 Mon Sep 17 00:00:00 2001 From: Sagar Utekar Date: Mon, 18 Aug 2025 07:25:09 +0530 Subject: [PATCH 095/133] Update challenge.md --- DailyChallenges/Season2/Day13/challenge.md | 448 +++++++++++++++++++++ 1 file changed, 448 insertions(+) diff --git a/DailyChallenges/Season2/Day13/challenge.md b/DailyChallenges/Season2/Day13/challenge.md index 8b13789..225f3d8 100644 --- a/DailyChallenges/Season2/Day13/challenge.md +++ b/DailyChallenges/Season2/Day13/challenge.md @@ -1 +1,449 @@ +# Daily DevOps + SRE Challenge Series – Season 2 +## Day 13: Secure Shell Mastery – SSH, SCP, Hardening & Troubleshooting Like a Pro +### Introduction +Welcome to **Day 13** of the Daily DevOps + SRE Challenge Series – Season 2! 🔐 + +Today, you’ll master secure remote access and file transfer with **SSH** and **SCP**. You’ll also learn to harden servers, move beyond passwords to keys, and troubleshoot failures like an SRE under pressure. SSH is the backbone of Linux administration, CI/CD deploys, remote debugging, and automated operations. +Fun fact, Google ask these questions for the Cloud Engineer role. + +By the end of today, you’ll confidently connect, transfer files safely, lock down your servers, and fix tricky login issues fast. + +#### Why This Matters? +- **Security**: SSH done wrong is an open door. Harden it to reduce risk. +- **Reliability**: Key-based access avoids password lockouts and human error. +- **Scale**: Client and server configs let you manage fleets and bastions cleanly. +- **Incident Response**: During outages, fast, correct SSH troubleshooting is vital. +- **Interview Win**: “How do you harden sshd?” and “Why key-based auth?” are common. +- **Real-World Impact**: Disabling password logins and enforcing keys can reduce SSH-related incidents dramatically. + +--- + +### 1. Understanding SSH +- **Definition**: SSH (Secure Shell) is a protocol for secure remote login, command execution, and tunneling over untrusted networks. +- **How it stays secure**: + - Host keys: Identify the server; clients cache fingerprints in `~/.ssh/known_hosts`. + - Key exchange + ciphers: Negotiate an encrypted session (confidentiality + integrity). + - Authentication: Password, public key, or other methods (e.g., MFA). +- **Common use cases**: + - Admin access: `ssh user@server` + - Remote commands: `ssh user@server "systemctl status nginx"` + - File transfers: `scp`, `sftp`, `rsync -e ssh` + - Port forwarding and tunneling +- **Ports**: Default server port is 22. Many orgs change it (e.g., 2222) and update firewalls accordingly. + +--- + +### 2. Connecting with SSH +Basic commands: +```bash +# Default port +ssh user@hostname + +# Specify identity key +ssh -i ~/.ssh/id_ed25519 user@hostname + +# Non-default port +ssh -p 2222 user@hostname + +# Run a one-off command +ssh user@hostname "uptime && whoami" +``` + +Helpful client options: +- `-vvv`: Verbose output for troubleshooting (negotiation, auth) +- `-o StrictHostKeyChecking=ask`: Safer first-connection behavior +- `-J bastion`: ProxyJump via bastion (multi-hop) +- ControlMaster/ControlPersist: Multiplex connections for faster repeated SSH + +--- + +### 3. Secure File Transfers: SCP, SFTP, and rsync over SSH +SCP and SFTP both run over SSH (encrypted end-to-end). Modern OpenSSH may implement `scp` via SFTP. + +SCP examples: +```bash +# Upload a file +scp ./app.tar.gz user@server:/opt/releases/ + +# Download a file +scp user@server:/var/log/nginx/access.log ./logs/ + +# Recursive copy (directory) +scp -r ./static/ user@server:/var/www/static/ + +# Non-default port +scp -P 2222 ./file.txt user@server:/tmp/ +``` + +SFTP (interactive or batch): +```bash +sftp -P 2222 -i ~/.ssh/id_ed25519 user@server +sftp> put file.txt /opt/data/ +sftp> get /opt/data/file.txt +``` + +rsync over SSH (efficient sync with deltas): +```bash +rsync -az --progress -e "ssh -p 2222 -i ~/.ssh/id_ed25519" ./site/ user@server:/var/www/site/ +``` + +Integrity verification: +```bash +sha256sum file.txt +ssh user@server "sha256sum /opt/data/file.txt" +``` + +--- + +### 4. How SSH Works (Mermaid Diagram) +```mermaid +sequenceDiagram + participant Client + participant Server + + Client->>Server: Initiates TCP connection on port 22 + Server->>Client: Sends SSH version and server public key + Client->>Server: Negotiates encryption + session key + Client->>Server: Sends authentication request (password or key) + alt Password Auth + Client->>Server: Sends username + password + Server->>Server: Verifies credentials + else Public Key Auth + Client->>Server: Proves possession of private key (signs challenge) + Server->>Server: Verifies using authorized_keys + end + Server-->>Client: Authentication success/failure + Client->>Server: Opens shell session or executes command +``` + +--- + +### 5. How SCP Works (Mermaid Diagram) +```mermaid +sequenceDiagram + participant Client + participant Server + + Client->>Server: Establish SSH connection (port 22) + Client->>Server: Sends SCP command (upload/download) + Server->>Client: Responds with SCP process + Client->>Server: Transfers file chunks (encrypted) + Server->>Client: Sends ACK after completion + Note over Client,Server: Entire session is encrypted with SSH +``` + +--- + +### 6. sshd_config Essentials (Server) +Key directives in `/etc/ssh/sshd_config`: +- `Port 2222`: Change from 22 (remember to open firewall) +- `PermitRootLogin no`: Disable direct root login +- `PasswordAuthentication no`: Enforce key-based auth +- `PubkeyAuthentication yes`: Enable key auth +- `MaxAuthTries 3`: Throttle brute-force attempts +- `PermitEmptyPasswords no`: Disallow empty passwords +- `AllowUsers devops_user`: Restrict to allowed accounts (or `AllowGroups`) +- `ClientAliveInterval 300` + `ClientAliveCountMax 2`: Idle session control +- `LogLevel VERBOSE`: Better audit logs +- Optionally harden crypto (modern OpenSSH defaults are strong): + - `KexAlgorithms curve25519-sha256,curve25519-sha256@libssh.org` + - `Ciphers chacha20-poly1305@openssh.com,aes256-gcm@openssh.com` + - `MACs hmac-sha2-512-etm@openssh.com,hmac-sha2-256-etm@openssh.com` + +Validate and apply: +```bash +sudo sshd -t # syntax check +sudo sshd -T | sort # effective config +sudo systemctl reload sshd # safer than restart +``` + +--- + +### 7. Passwordless Access & Key Management +Generate a key pair (ed25519 recommended): +```bash +ssh-keygen -t ed25519 -C "you@example.com" +``` + +Install your public key: +```bash +ssh-copy-id user@server +# or manual: +cat ~/.ssh/id_ed25519.pub | ssh user@server 'mkdir -p ~/.ssh && chmod 700 ~/.ssh && cat >> ~/.ssh/authorized_keys && chmod 600 ~/.ssh/authorized_keys' +``` + +Login: +```bash +ssh user@server +``` + +Permissions (avoid auth failures): +```bash +chmod 700 ~/.ssh +chmod 600 ~/.ssh/id_ed25519 ~/.ssh/authorized_keys +``` + +Use `ssh-agent` to cache decrypted keys: +```bash +eval "$(ssh-agent -s)" +ssh-add ~/.ssh/id_ed25519 +``` + +Tips: +- Use `IdentitiesOnly yes` per host to avoid “Too many authentication failures” +- Always protect private keys with a passphrase and store backups securely + +--- + +### 8. Server Hardening Simulation +🔴 Buggy server config (for lab): +- `PermitRootLogin yes` +- `PasswordAuthentication yes` +- Weak password + +⚡ Exploit demonstration (lab-only): +```bash +hydra -l root -P rockyou.txt ssh://server-ip +``` + +🟢 Hardened fix: +- `PermitRootLogin no` +- `PasswordAuthentication no` +- `PubkeyAuthentication yes` +- `AllowUsers devops_user` +- Change port (e.g., 2222) & update firewall rules + +UFW example: +```bash +sudo ufw allow 2222/tcp +sudo ufw delete allow 22/tcp +sudo ufw reload +``` + +firewalld example: +```bash +sudo firewall-cmd --add-port=2222/tcp --permanent +sudo firewall-cmd --remove-service=ssh --permanent +sudo firewall-cmd --reload +``` + +--- + +### 9. Troubleshooting – Google-Style +1) Client-Side +- Correct command: `ssh -i key.pem -p 2222 user@ip` +- Debug with `ssh -vvv` +- Key file permissions: +```bash +chmod 700 ~/.ssh +chmod 600 ~/.ssh/id_* ~/.ssh/authorized_keys +``` + +2) Network +- Reachability and port: +```bash +ping -c 2 ip || true +nc -zv ip 2222 +``` +- Firewall/security group rules (local + cloud) + +3) Server +- Service and port: +```bash +sudo systemctl status sshd +sudo ss -tulpn | grep -E ':(22|2222)\s' +``` +- Config validation: +```bash +sudo sshd -t +sudo sshd -T | sort +``` + +4) Authentication +- User exists and has valid shell +- `~/.ssh` ownership and modes; `authorized_keys` present +- `AllowUsers/AllowGroups` not blocking login + +5) Logs +- Debian/Ubuntu: `/var/log/auth.log` +- RHEL/CentOS/Fedora: `/var/log/secure` +- Journal: `sudo journalctl -u sshd -e` + +6) Cloud-Specific +- AWS: Default usernames (`ec2-user`, `ubuntu`, etc.), SG inbound 2222/tcp, PEM permissions +- GCP: OS Login/IAM bindings; project-level keys +- Azure: NSGs; Just-In-Time access + +7) Edge Cases +- Fail2ban blocking IP (unban via `fail2ban-client`) +- Host key mismatch (fix `~/.ssh/known_hosts`) +- SELinux enforcing (check AVC denials; `restorecon -Rv /etc/ssh` after changes) + +--- + +### 10. Troubleshooting Flow (Mermaid Diagram) +```mermaid +flowchart TD + A[SSH Fails] --> B[Check Client Command] + B -->|Wrong user/IP/key| FixClient + B --> C[Enable ssh -vvv debug] + + C --> D[Network Reachable?] + D -->|No| FixNetwork + D -->|Yes| E[Is Port Open?] + + E -->|No| FixFirewall + E -->|Yes| F[Is sshd Running?] + + F -->|No| RestartService + F -->|Yes| G[Check sshd_config] + + G --> H[Auth Issues?] + H -->|Key mismatch| FixKeys + H -->|User restricted| UpdateAllowUsers + + G --> I[Check Logs] + I -->|Errors Found| FixConfig + I -->|Cloud Issue| CheckIAM + + I --> J[Still Broken?] + J --> Escalate +``` + +--- + +## Practical Tasks: Operation Secure Access – Harden, Validate, and Recover +As an SRE, your job is to enable secure, reliable access while minimizing risk. This lab uses Ubuntu/Debian or RHEL/Fedora. Keep a second console open before changing SSH. + +### Setup Prep +1. Create a workspace: +```bash +mkdir -p ~/ssh-lab && cd ~/ssh-lab +``` +2. Confirm current connectivity: +```bash +ssh user@server "whoami && hostname" +``` + +### Task A: Establish Key-Based Login +1. Generate key (if needed): +```bash +ssh-keygen -t ed25519 -C "you@example.com" +``` +2. Install your public key: +```bash +ssh-copy-id user@server +``` +3. Verify: +```bash +ssh -vvv user@server "echo OK && id -u && hostname" +``` +4. Save the line showing “Authentication succeeded (publickey)” to `notes.md`. + +### Task B: Harden sshd +1. Open firewall for new port (2222 shown below). +2. Edit `/etc/ssh/sshd_config` to include: +``` +Port 2222 +PermitRootLogin no +PasswordAuthentication no +PubkeyAuthentication yes +AllowUsers user +MaxAuthTries 3 +PermitEmptyPasswords no +ClientAliveInterval 300 +ClientAliveCountMax 2 +LogLevel VERBOSE +``` +3. Validate and reload: +```bash +sudo sshd -t && sudo systemctl reload sshd +``` +4. Test new port: +```bash +ssh -p 2222 user@server "echo PORT_OK" +``` +5. Remove old port (only after verifying new): +- UFW: `sudo ufw delete allow 22/tcp` +- firewalld: `sudo firewall-cmd --remove-service=ssh --permanent && sudo firewall-cmd --reload` + +Document all commands + outputs in `notes.md`. + +### Task C: Secure File Transfer Roundtrip +1. Create a file and upload/download: +```bash +echo "hello-ssh" > hello.txt +scp -P 2222 hello.txt user@server:/tmp/hello.txt +scp -P 2222 user@server:/tmp/hello.txt hello.remote.txt +``` +2. Verify integrity: +```bash +sha256sum hello.txt hello.remote.txt | tee checksums.txt +``` + +### Task D (Optional): SFTP-Only Restricted User +1. Create group and user (server): +```bash +sudo groupadd -f sftpusers +sudo useradd -m -G sftpusers -s /usr/sbin/nologin sftpuser +``` +2. Prepare chroot: +```bash +sudo mkdir -p /sftp/sftpuser/upload +sudo chown root:root /sftp /sftp/sftpuser +sudo chmod 755 /sftp /sftp/sftpuser +sudo chown sftpuser:sftpusers /sftp/sftpuser/upload +``` +3. Add to `/etc/ssh/sshd_config`: +``` +Subsystem sftp internal-sftp +Match Group sftpusers + ChrootDirectory /sftp/%u + ForceCommand internal-sftp + AllowTCPForwarding no + X11Forwarding no +``` +4. Validate + reload, then test: +```bash +sudo sshd -t && sudo systemctl reload sshd +ssh -p 2222 sftpuser@server # should fail (no shell) +sftp -P 2222 sftpuser@server # should succeed +``` + +### Task E: Induce and Fix Failures (Troubleshooting) +Pick at least one, capture `ssh -vvv` and server logs, and document your fix: +- Wrong key permissions: fix with `chmod 600 ~/.ssh/id_*` +- Blocked port: fix firewall rules and retest with `nc -zv` +- Misconfigured `AllowUsers`: correct user and reload +- Host key mismatch: remove offending line in `~/.ssh/known_hosts` + +--- + +## Submission Guidelines +- Store your findings and execution steps in a markdown file (`solution.md`). +- Include: + - Final `/etc/ssh/sshd_config` + - Commands and outputs for Tasks A–C (and D/E if attempted) + - Notes from troubleshooting (`ssh -vvv` snippets and relevant server logs) +- Submit it in your GitHub repository and share the link. +- Post your experience on social media with **#getfitwithsagar #SRELife #DevOpsForAll**. +- Tag us in your posts for visibility and networking. + +--- + +## Join Our Community +To make the most of this journey, connect with fellow learners: +- **Discord** – Ask questions and collaborate: https://discord.gg/mNDm39qB8t +- **Google Group** – Get updates and discussions: https://groups.google.com/forum/#!forum/daily-devops-sre-challenge-series/join +- **YouTube** – Watch solution videos and tutorials: https://www.youtube.com/@Sagar.Utekar + +--- + +## Stay Motivated! +Every challenge you complete brings you closer to mastering Git, DevOps, and SRE practices. Keep pushing your limits and collaborating with the community. + +Happy Learning! + +**Best regards,** +Sagar Utekar From 111eaf3193e97d8d1753f7402451833e19234cbd Mon Sep 17 00:00:00 2001 From: Sagar Utekar Date: Tue, 19 Aug 2025 15:50:10 +0530 Subject: [PATCH 096/133] Create challenge.md --- DailyChallenges/Season2/Dsy14/challenge.md | 1 + 1 file changed, 1 insertion(+) create mode 100644 DailyChallenges/Season2/Dsy14/challenge.md diff --git a/DailyChallenges/Season2/Dsy14/challenge.md b/DailyChallenges/Season2/Dsy14/challenge.md new file mode 100644 index 0000000..8b13789 --- /dev/null +++ b/DailyChallenges/Season2/Dsy14/challenge.md @@ -0,0 +1 @@ + From 0805e51704ebbd6440a5d69ec883d73a335e810c Mon Sep 17 00:00:00 2001 From: Sagar Utekar Date: Tue, 19 Aug 2025 17:22:17 +0530 Subject: [PATCH 097/133] Delete DailyChallenges/Season2/Dsy14 directory --- DailyChallenges/Season2/Dsy14/challenge.md | 1 - 1 file changed, 1 deletion(-) delete mode 100644 DailyChallenges/Season2/Dsy14/challenge.md diff --git a/DailyChallenges/Season2/Dsy14/challenge.md b/DailyChallenges/Season2/Dsy14/challenge.md deleted file mode 100644 index 8b13789..0000000 --- a/DailyChallenges/Season2/Dsy14/challenge.md +++ /dev/null @@ -1 +0,0 @@ - From dba21ba6456ed86cc73c8f656c8d11dbe3afba97 Mon Sep 17 00:00:00 2001 From: Sagar Utekar Date: Tue, 19 Aug 2025 17:22:36 +0530 Subject: [PATCH 098/133] Create challenge.md --- DailyChallenges/Season2/Day14/challenge.md | 228 +++++++++++++++++++++ 1 file changed, 228 insertions(+) create mode 100644 DailyChallenges/Season2/Day14/challenge.md diff --git a/DailyChallenges/Season2/Day14/challenge.md b/DailyChallenges/Season2/Day14/challenge.md new file mode 100644 index 0000000..418327f --- /dev/null +++ b/DailyChallenges/Season2/Day14/challenge.md @@ -0,0 +1,228 @@ +# Daily DevOps + SRE Challenge Series – Season 2 +## Day 14: Linux Users, Groups, and Permissions – Essentials for DevOps/SRE + +### Introduction +Welcome to Day 14 of the Daily DevOps + SRE Challenge Series – Season 2! + +Today you’ll learn the core skills for managing users, groups, and file permissions in Linux. We’ll stay focused on the 20% that covers 80% of real DevOps/SRE work: creating users, adding them to groups, fixing “Permission denied” issues, and setting up safe shared directories for teams. + +By the end of this challenge, you’ll be able to: +- Create and manage users and groups safely +- Read and change permissions with confidence +- Fix ownership/permission issues that break apps and deployments +- Build a team directory that “just works” for collaboration + +Why this matters +- Security: Avoid overexposed files and risky defaults +- Reliability: Services run only when ownership/modes are correct +- Collaboration: The right group setup saves time and tickets +- Troubleshooting: “Permission denied” is one of the most common production errors + +--- + +### 1) The Essentials You’ll Use Daily + +Users and groups (who can do what) +- Users have a username, UID, home, and shell +- Groups let multiple users share access without changing file owners + +Quick commands +```bash +# Users +sudo useradd -m -s /bin/bash riya +sudo passwd riya +id riya +groups riya + +# Groups +sudo groupadd devops +sudo usermod -aG devops riya # append (-a) to Group (-G) +getent group devops +``` + +Ownership (who owns a file/dir) +```bash +sudo chown user:group path +sudo chown -R www-data:www-data /var/www/html +``` + +Permissions (what actions are allowed) +- r (read), w (write), x (execute/traverse) +- Apply to: owner (u), group (g), others (o) +- Numeric: r=4, w=2, x=1 → 640 = u=rw, g=r, o=- + +Common modes you’ll set +```bash +chmod 640 file # configs readable by app group, not world +chmod 750 dir # app can enter/execute; group can traverse; others blocked +chmod 755 dir # world-readable directory (no write) +``` + +Team-friendly directories +- setgid (2) on a directory makes new files inherit the directory’s group +```bash +sudo chmod 2775 /projects/teamspace +``` + +Sticky bit (shared scratch areas like /tmp) +```bash +sudo chmod 1777 /shared # users can’t delete others’ files +``` + +Tip: For most team folders, set owner to root:team, apply 2770/2775, and add members to the team group. + +--- + +### 2) Two Short Mermaid Diagrams + +Permission evaluation (simple mental model) +```mermaid +flowchart TD + A[Request access to path] --> B{Is caller root?} + B -->|Yes| Z[Allow] + B -->|No| C{Is caller the file owner?} + C -->|Yes| D[Check owner bits] + C -->|No| E{Is caller in the file's group?} + E -->|Yes| F[Check group bits] + E -->|No| G[Check others bits] + D --> H{Bits allow action?} + F --> H + G --> H + H -->|Yes| Z + H -->|No| Y[Denied] +``` + +umask (why new files default to 644 and dirs to 755) +```mermaid +flowchart TD + A[Create] --> B{Type?} + B -->|File| C[Base 666] + B -->|Directory| D[Base 777] + C --> E[Subtract umask → mode] + D --> E[Subtract umask → mode] + E --> F[Final permissions] +``` + +--- + +### 3) Practical Scenarios (Do These) + +Scenario A: Create users and a devops group +```bash +# Users +for u in riya amit neha; do sudo useradd -m -s /bin/bash "$u" || true; done +# Group +sudo groupadd devops || true +# Add users to group +for u in riya amit neha; do sudo usermod -aG devops "$u"; done +# Verify +for u in riya amit neha; do id "$u"; done +``` + +Scenario B: Team workspace that “just works” +Goal: Everyone in devops can create/edit files in /projects/infra; others blocked. +```bash +sudo mkdir -p /projects/infra +sudo chown root:devops /projects/infra +sudo chmod 2770 /projects/infra # setgid ensures new files use group devops +# Test (from different users): create files and edit each other’s files +``` + +Scenario C: Fix a broken web app folder +Symptom: Nginx can’t read /var/www/html (403/404). +```bash +# Debian/Ubuntu typical web user +sudo chown -R www-data:www-data /var/www/html +sudo find /var/www/html -type d -exec chmod 755 {} \; +sudo find /var/www/html -type f -exec chmod 644 {} \; +``` + +Scenario D: Quick account hygiene +```bash +# Lock an unused account +sudo usermod -L guest +# Set password policy for riya +sudo chage -M 90 -m 7 -W 14 riya +# See current policy +chage -l riya +``` + +Scenario E: Troubleshoot “Permission denied” +Checklist: +- ls -ld path; stat file +- namei -l /deep/nested/path (shows each directory’s x bit) +- id user (group membership) +- If it’s a team dir, ensure setgid and group membership are correct +Commands: +```bash +namei -l /srv/app/config/app.env +id appsvc +ls -ld /srv /srv/app /srv/app/config +``` + +Optional (nice to know, not required today): +- Default ACLs for always-on group write in team dirs: +```bash +sudo apt-get install -y acl 2>/dev/null || true +sudo setfacl -m g:devops:rwx /projects/infra +sudo setfacl -d -m g:devops:rwx /projects/infra +getfacl /projects/infra +``` + +--- + +## Practical Tasks: Build, Share, Fix + +Do these and capture the commands and outputs. + +Task 1: Users and group +- Create users riya, amit, neha +- Create devops group and add the users +- Save id outputs to ~/perm-lab/users.txt + +Task 2: Team directory +- Create /projects/infra as root:devops with 2770 +- Show that files created by any member are editable by others +- Save ls -l and a short note to ~/perm-lab/teamspace.md + +Task 3: Fix permissions +- Simulate a broken app dir /opt/app owned by root:root with mode 600 +- Create appsvc user (no password needed) +- Fix to appsvc:appsvc, dirs 750, files 640 +- Prove appsvc can read configs; others cannot +- Save commands and stat outputs to ~/perm-lab/appfix.md + +Task 4: One troubleshooting case +- Break something intentionally (e.g., remove x from a parent directory) +- Use namei -l to find the missing execute bit and fix it +- Save before/after to ~/perm-lab/troubleshoot.md + +Deliverables +- A markdown file solution.md with: + - Commands and outputs for Tasks 1–4 + - 3–5 bullet notes on what you learned + +--- + +## Submission Guidelines +- Store your findings and execution steps in solution.md +- Submit it in your GitHub repository and share the link +- Post your experience on social media with #getfitwithsagar #SRELife #DevOpsForAll +- Tag us in your posts for visibility and networking + +--- + +## Join Our Community +- Discord – Ask questions and collaborate: https://discord.gg/mNDm39qB8t +- Google Group – Get updates and discussions: https://groups.google.com/forum/#!forum/daily-devops-sre-challenge-series/join +- YouTube – Watch solution videos and tutorials: https://www.youtube.com/@Sagar.Utekar + +--- + +## Stay Motivated! +Keep it simple. Master the basics. You’ll use these commands every week in real DevOps/SRE work. + +Happy Learning! + +Best regards, +Sagar Utekar From ac6a3143d9fffd466bb24304e297ef4e39041514 Mon Sep 17 00:00:00 2001 From: Swayam-Prakash-Bhuyan Date: Fri, 22 Aug 2025 20:43:56 +0530 Subject: [PATCH 099/133] Added challenge for day 15 with networking fundamentals --- DailyChallenges/Season2/Day15/challenge.md | 828 +++++++++++++++++++++ 1 file changed, 828 insertions(+) create mode 100644 DailyChallenges/Season2/Day15/challenge.md diff --git a/DailyChallenges/Season2/Day15/challenge.md b/DailyChallenges/Season2/Day15/challenge.md new file mode 100644 index 0000000..fc31ab4 --- /dev/null +++ b/DailyChallenges/Season2/Day15/challenge.md @@ -0,0 +1,828 @@ +# Day 15 — Linux Networking Challenges (Daily DevOps + SRE Challenge Series — Season 2) + +--- + +## Introduction + +Welcome to Day 15 of the Daily DevOps + SRE Challenge Series – Season 2! 🎉 + +Today, you'll dive into essential **Linux networking skills** through practical, story-based scenarios. You'll configure IP addresses, set up namespaces, simulate routing, and debug traffic issues—all on your local machine without any complex setup. By the end, you'll: + +* Understand and troubleshoot the **OSI & TCP/IP layers** in real-world failures. +* Configure **IPv4/IPv6 addressing**, resolve subnetting conflicts, and simulate enterprise VPC designs. +* Master **ARP behavior**, from cache flushing to spoofing attacks. +* Gain hands-on skills to debug outages with tools like `iptables`, `ip`, and `tcpdump`. + +--- + +## Why This Matters? + +Networking is the backbone of everything—cloud apps, Kubernetes clusters, even CI/CD pipelines. If you can't debug connectivity, you'll get stuck in outages fast. + +* **Real-World Edge:** Most outages aren't due to servers crashing—they're due to misconfigured routes, DNS failures, or firewall issues. +* **SRE Superpower:** Knowing the **why ping works but app fails** scenario makes you 10x faster in production war rooms. +* **Cloud Readiness:** VPC, subnets, and dual-stack IPv4/IPv6 configs are daily tasks for AWS/GCP/Azure engineers. +* **Interview Gold:** "Why can two hosts ping but not load HTTP?" or "How do you debug duplicate IP conflicts?"—classic SRE questions. +* **Security Awareness:** ARP spoofing and wrong subnet masks are real threats in production, not just lab theory. + +--- + +## Real-World Save 🌍 + +At a fintech startup, payments randomly failed for EU customers. Engineers could **ping servers but HTTP calls kept timing out**. Debugging revealed a firewall rule blocking port 443 on just one subnet. Fixing that restored global transactions. Knowing **layer-by-layer troubleshooting** saved millions in lost revenue. + +--- + +# Theory (detailed) — **do not skip** + +> Below: full theory for the four topics you asked for. Read fully before doing labs. + +--- + +## OSI & TCP/IP Models — Introduction + +**What is it?** +OSI (Open Systems Interconnection) is a conceptual 7-layer model that describes how data moves through networked systems. TCP/IP model is a simpler practical model used in the Internet (4 layers). These help you think logically about where failures happen. + +**Why learn it?** +Because when something breaks (web app not loading, DNS failing, packets dropping), you must ask: *which layer broke?* — physical, link (MAC), IP (routing), TCP/UDP (ports), or the app (HTTP/DNS). This narrows down troubleshooting steps. + +--- + +### OSI layers (simple, one-line each) + +1. **Physical (L1)** — cables, radio waves, NIC hardware. +2. **Data Link (L2)** — frames, MAC addresses, switches, ARP. +3. **Network (L3)** — IP addressing and routing. +4. **Transport (L4)** — TCP/UDP, ports, connection reliability. +5. **Session (L5)** — sessions, dialogs (rarely directly debugged). +6. **Presentation (L6)** — encoding/encryption (TLS sits around here). +7. **Application (L7)** — HTTP, DNS, SSH—the services users interact with. + +**TCP/IP Stack mapping (practical):** + +* Link (L1–L2) — ARP, Ethernet +* Internet (L3) — IP, routing +* Transport (L4) — TCP, UDP +* Application (L7) — HTTP, DNS, SSH + +--- + +### Key concepts & simple analogies + +* **IP = house number** — tells you *where* to go. +* **MAC = person's face** — used to deliver inside the same building (LAN). +* **Ports (TCP/UDP) = apartment door numbers** — same house, many services. +* **Firewall = security guard at door** — can block specific ports/protocols. + +--- + +### Typical tools & what they tell you + +* `ping` (ICMP) → L3 reachability test. +* `arp` / `ip neigh` → L2 IP→MAC mapping. +* `ip addr`, `ip route` → addresses & routes (L3). +* `ss -tuln` / `nc` → ports and services (L4). +* `tcpdump` → raw packet capture (L2–L4). +* `iptables` or `nft` → firewall rules (L4-L3 policies). +* `traceroute` → shows path (L3/hops), but depends on ICMP/TCP/UDP responses. + +--- + +### Troubleshooting mental model (simple process) + +1. **Is the NIC up?** (`ip link`) — Physical/L1 +2. **Can I reach IP?** (`ping`) — Network/L3 +3. **Is the service listening?** (`ss -ltnp`, `nc`) — Transport/L4 & App/L7 +4. **Is DNS resolving?** (`dig`, `nslookup`) — App/L7 +5. **Capture packets** (`tcpdump`) — inspect headers (MAC, IP, TCP flags). +6. **Check firewall rules** (`iptables -L`, `nft list ruleset`). + +--- + +### Mermaid diagrams (OSI / TCP-IP overview) + +OSI stack: + +```mermaid +graph TB + A[Application HTTP/DNS/SSH] --> B[Transport TCP/UDP] + B --> C[Network IP/Routing] + C --> D[Data Link Ethernet/ARP] + D --> E[Physical Cable/Wireless] +``` + +Packet debugging flow: + +```mermaid +flowchart LR + UserAction[User opens browser] --> App[Application: HTTP request] + App --> Transport[TCP handshake/port] + Transport --> Network[IP routing/next-hop] + Network --> Link[ARP/MAC resolution] + Link --> Physical[Wire/WiFi] +``` + +--- + +## IP Addressing (IPv4 & IPv6) — Introduction + +**What is it?** +IP addresses uniquely identify hosts on a network. IPv4 is 32-bit (e.g., `192.168.1.10`), IPv6 is 128-bit (e.g., `2001:db8::1`). + +**Why important?** +If you don't know addressing basics, you'll misconfigure masks, gateways, or try to reach the wrong network. IP addressing also affects routing, ACLs, NAT — everything SREs care about. + +--- + +### IPv4 basics + +* **Format:** dotted decimal `a.b.c.d` (32 bits). +* **CIDR prefix:** `/24` means top 24 bits are network — addresses inside are on same subnet. +* **Private ranges:** + + * `10.0.0.0/8` + * `172.16.0.0/12` + * `192.168.0.0/16` +* **Special addresses:** `0.0.0.0` (this host), `127.0.0.1` (loopback), network & broadcast addresses. + +**How to compute host counts:** + +* `/24` → 256 addresses (254 hosts usable). +* `/26` → 64 addresses (62 hosts usable). + +--- + +### IPv6 basics + +* **Format:** hex groups `2001:db8::1` (128 bits). +* **Scopes:** Link-local (`fe80::/10`), global unicast (public), loopback (`::1`). +* **Advantages:** large address space, simpler auto-configuration (SLAAC), better routing aggregation. + +--- + +### Subnet mask & network boundary (simple) + +* Mask tells whether destination IP is on your local network or needs to be sent to a router. +* Example: `192.168.1.10/24` and `192.168.1.20/24` are local; `192.168.2.1` is remote. + +--- + +### Useful commands + +* Show addresses: `ip addr show` +* Show routes: `ip route show` +* Add address: `ip addr add 10.0.0.2/24 dev eth0` +* IPv6 show: `ip -6 addr show` +* Disable IPv4 (namespace): `sysctl -w net.ipv4.conf.all.disable_ipv4=1` + +--- + +### Routing decision (simple) + +```mermaid +flowchart LR + App[App creates packet] --> IPPkt[IP packet with dest IP] + IPPkt --> Router{Is dest in local subnet?} + Router -->|Yes| ARP[Do ARP resolve MAC] + ARP --> Send[Send frame on LAN] + Router -->|No| GW[Send to default gateway] +``` + +--- + +## Subnetting & CIDR — Introduction + +**What is it?** +CIDR (Classless Inter-Domain Routing) describes networks by prefix length (`/24`, `/26`) and enables flexible splitting of IP space. Subnetting divides a big network into smaller networks. + +**Why learn it?** +To allocate IPs to teams, limit broadcast domains, design VPCs, and avoid overlaps when merging networks. + +--- + +### How CIDR works (simple) + +* `10.0.0.0/24` → addresses `10.0.0.0`–`10.0.0.255`. +* Splitting `10.0.0.0/24` into `/26` yields: + + * `10.0.0.0/26` (0–63) + * `10.0.0.64/26` (64–127) + * `10.0.0.128/26` (128–191) + * `10.0.0.192/26` (192–255) + +--- + +### Common tasks + +* **Divide network** for teams → use /26, /27 etc. +* **Plan VPC** blocks to avoid future overlap. +* **Detect overlaps** before merging nets. + +--- + +### Overlap problems (simple) + +* If Net A thinks IP is local (on-link) and Net B thinks same IP is remote, packets can be dropped or blackholed. The fix: **renumber** or **NAT** at boundary. + +--- + +### Subnet split + +```mermaid +graph TD + A[10.0.0.0/24] --> B[10.0.0.0/26] + A --> C[10.0.0.64/26] + A --> D[10.0.0.128/26] + A --> E[10.0.0.192/26] +``` + +--- + +## ARP (Address Resolution Protocol) — Introduction + +**What is it?** +ARP maps IPv4 addresses to MAC addresses inside a LAN. It's how IP packets get delivered to the correct NIC inside the same broadcast domain. + +**Why ARP matters?** +If ARP is wrong or spoofed, your traffic can go to the wrong machine, be dropped, or be intercepted. + +--- + +### ARP normal flow (simple) + +1. Host A wants to send to IP X. +2. If it has no MAC for X in ARP cache, it sends **ARP Who-has X tell A** (broadcast). +3. The owner replies **ARP is-at MAC**. +4. Host A caches and sends frames to that MAC. + +--- + +### ARP cache and states + +* Entries expire — kernel keeps freshness flags (`REACHABLE`, `STALE`, etc.). +* Commands: `ip neigh show`, `arp -n`, `arping -I ` + +--- + +### ARP security problems + +* **Duplicate IP:** Two hosts claim same IP → flapping / inconsistent replies. +* **ARP spoofing/poisoning:** Attacker sends fake ARP replies linking victim/gateway IP to attacker's MAC → MITM. +* **Mitigations:** static ARP entries (for critical hosts), switch DAI, encryption (TLS). + +--- + +### ARP sequence + +```mermaid +sequenceDiagram + participant A as Host A + participant Net as Broadcast + participant B as Host B + + A->>Net: ARP Who-has 192.168.1.20? Tell 192.168.1.10 + B-->>Net: ARP is-at 11:22:33:44:55:66 + Net-->>A: ARP reply + A->>B: Frame delivered to MAC 11:22:33:44:55:66 +``` + +--- + +## Additional Theory Concepts + +### DNS (Domain Name System) — Introduction + +**What is it?** +DNS translates human-readable domain names (like google.com) to IP addresses. It uses a hierarchical, distributed database. + +**Why learn it?** +Because most network connections start with a DNS lookup. If DNS fails, your application fails—even if the target server is reachable by IP. + +--- + +### How DNS works (simple) + +1. Application requests resolution (e.g., browser asks for google.com). +2. The request is sent to a DNS resolver (usually provided by ISP or public ones like 8.8.8.8). +3. The resolver queries the root servers, then TLD servers, then authoritative servers for the domain. +4. The IP address is returned and cached. + +--- + +### Key concepts + +- **Record types:** A (IPv4), AAAA (IPv6), CNAME (alias), MX (mail), etc. +- **Port:** UDP 53 (queries), TCP 53 (large responses or zone transfers). +- **Tools:** `dig`, `nslookup`, `host`. + +--- + +### Common issues + +- Misconfigured `/etc/resolv.conf` +- Firewall blocking UDP 53 +- Slow DNS responses (high latency) + +--- + +### DNS lookup + +```mermaid +sequenceDiagram + participant C as Client + participant R as Resolver + participant A as Authoritative Server + + C->>R: Query for google.com + R->>A: Query + A-->>R: Response (IP) + R-->>C: Response (IP) +``` + +--- + +### MTU (Maximum Transmission Unit) — Introduction + +**What is it?** +MTU is the maximum size of a packet that can be transmitted without fragmentation over a network interface. + +**Why learn it?** +If packets are too large for a link, they get fragmented (which adds overhead) or dropped (which breaks connectivity). This is common in VPNs and tunnels. + +--- + +### How MTU works (simple) + +- Each link has an MTU (e.g., Ethernet is 1500 bytes). +- If a packet is larger than the MTU, it is fragmented into smaller packets (if allowed) or an ICMP "need to fragment" error is sent. +- **Path MTU Discovery (PMTUD)** is the process of determining the smallest MTU along the path to avoid fragmentation. + +--- + +### Common issues + +- MTU mismatch between endpoints or intermediate links causes black holes for large packets. +- VPNs reduce effective MTU due to encapsulation overhead. + +--- + +### PMTUD + +```mermaid +sequenceDiagram + participant S as Sender + participant R as Router + participant D as Dest + + S->>D: Large packet (1500 bytes) + R->>S: ICMP Frag Needed (MTU=1400) + S->>D: Packet split to 1400 + D-->>S: ACK +``` + +--- + +### Routing (Multiple NICs, Policy Routing) — Introduction + +**What is it?** +Routing decides which interface to use to send a packet. Policy routing allows routing decisions based on more than just the destination IP (e.g., source IP, protocol). + +**Why learn it?** +Multi-homed hosts (multiple NICs) may have complex routing requirements. Wrong routing can cause asymmetric paths or failures. + +--- + +### How routing works (simple) + +- The kernel uses the routing table to decide the next hop. +- Default route (0.0.0.0/0) is used when no specific route matches. +- **Metric** is used to choose between multiple routes to the same destination (lower is better). +- **Policy routing** uses multiple routing tables selected by rules (e.g., from a certain source IP, use a different table). + +--- + +### Common commands + +- `ip route show` +- `ip rule list` (for policy routing) +- `ip route add ... metric ...` +- `ip rule add from table ` + +--- + +### Routing decision + +```mermaid +flowchart LR + P[Packet] --> R[Routing Table] + R -->|match| I[Interface] + R -->|no match| D[Default Gateway] +``` + +--- +--- +--- +--- + +## Challenges + +🔹 **1. OSI & TCP/IP Models** + +**Ping works, but HTTP fails** +Run python3 -m http.server 8080 in one namespace/VM. +From another, ping works but block TCP port 8080 using iptables. +👉 Mimics: "Why I can ping the server but the website isn't loading?" + +**Solution:** + +```bash +# Setup +ip netns add srv; ip netns add cli +ip link add veth-s type veth peer name veth-c +ip link set veth-s netns srv; ip link set veth-c netns cli +ip netns exec srv ip addr add 10.0.0.1/24 dev veth-s +ip netns exec cli ip addr add 10.0.0.2/24 dev veth-c +ip netns exec srv ip link set veth-s up; ip netns exec cli ip link set veth-c up +ip netns exec srv python3 -m http.server 8080 & + +# Test +ip netns exec cli ping -c2 10.0.0.1 # ✅ works +ip netns exec cli curl -m3 10.0.0.1:8080 # ✅ works + +# Break (block TCP/8080) +ip netns exec srv iptables -A INPUT -p tcp --dport 8080 -j DROP +ip netns exec cli curl -m3 10.0.0.1:8080 # ❌ timeout + +# Interpretation: L3 OK; L4/L7 blocked by firewall. +# Fix +ip netns exec srv iptables -D INPUT -p tcp --dport 8080 -j DROP +``` + +**Flow diagram (packet journey → break → fix):** + +```mermaid +flowchart LR + C[Client] -->|ICMP Echo Request| S[Server] + S -->|ICMP Echo Reply| C + C -->|TCP SYN :8080| S + S -->|iptables DROP| X[Dropped] + S -->|Fix: iptables -D| S2[Server Fixed] + C -->|TCP SYN :8080| S2 + S2 -->|TCP SYN/ACK| C + C -->|HTTP GET| S2 + S2 -->|HTTP 200| C +``` + +*Note:* ping uses ICMP (L3). curl uses TCP (L4) + HTTP (L7). iptables blocked TCP/8080 — that's why ping succeeded but web failed. + +--- + +**DNS Layer Check** +Disable /etc/resolv.conf or block UDP 53. +Test browsing (curl google.com fails, but curl 142.250.182.14 works). +👉 Mimics: "App fails because DNS is misconfigured." + +**Solution:** + +```bash +# Setup one namespace as client +ip netns add ns1; ip netns exec ns1 ip link set lo up + +# Break DNS safely (in namespace only) +ip netns exec ns1 iptables -A OUTPUT -p udp --dport 53 -j DROP + +# Test +ip netns exec ns1 curl -m3 http://google.com # ❌ fails (could not resolve) +ip netns exec ns1 curl -m3 http://142.250.182.14 # ✅ works (direct IP) + +# Interpretation: IP path fine; DNS resolution broken. +# Fix +ip netns exec ns1 iptables -D OUTPUT -p udp --dport 53 -j DROP +``` + +**Flow diagram:** + +```mermaid +flowchart LR + C[Client] -->|UDP DNS Query :53| DNS[Resolver] + DNS -->|blocked| X[Dropped] + C -->|HTTP to numeric IP| S[Remote Server] + S -->|HTTP 200| C +``` + +*Note:* block DNS (UDP/53) prevents name resolution; direct IP works. + +--- + +**MTU Mismatch** +Set MTU on one interface to 1400 and the other to 900. +Try large file transfer with scp. +👉 Mimics: "Packets drop due to MTU mismatch in VPN tunnels." + +**Solution:** + +```bash +# Setup +ip netns add a; ip netns add b +ip link add va type veth peer name vb +ip link set va netns a; ip link set vb netns b +ip netns exec a ip addr add 10.1.1.1/24 dev va +ip netns exec b ip addr add 10.1.1.2/24 dev vb +ip netns exec a ip link set va up mtu 1400 +ip netns exec b ip link set vb up mtu 900 + +# Test small vs large +ip netns exec a ping -c2 10.1.1.2 # ✅ often works +ip netns exec a ping -M do -s 1200 10.1.1.2 # ❌ fails due to MTU + +# (Optional TCP test) +ip netns exec b nc -l -p 9000 >/dev/null & +head -c 200000 |Large IP packet| Link[Link mtu=900] + Link -->|cannot forward| Router[drop or ICMP] + Router -->|ICMP frag needed| C + C -->|Fix: align MTU| Fixed[Link mtu=1400] +``` + +--- + +**Layer 2 vs Layer 3 Failure** +Assign two machines same subnet but no switch/bridge between them. +👉 Mimics: "Why hosts on same subnet can't reach each other?" + +**Solution:** + +```bash +# Setup (two separate links with no shared L2 domain) +ip netns add n1; ip netns add n2 +ip link add v1 type veth peer name v1p +ip link add v2 type veth peer name v2p +ip link set v1 netns n1; ip link set v2 netns n2 +ip netns exec n1 ip addr add 192.168.50.1/24 dev v1 +ip netns exec n2 ip addr add 192.168.50.2/24 dev v2 +ip netns exec n1 ip link set v1 up; ip netns exec n2 ip link set v2 up + +# Test +ip netns exec n1 ping -c2 192.168.50.2 # ❌ fails (no L2 connectivity) + +# Interpretation: Same subnet needs shared L2 (bridge/switch). +# Fix: create bridge and attach peers (host side v1p & v2p) +ip link add br0 type bridge; ip link set br0 up +ip link set v1p master br0; ip link set v2p master br0 +ip netns exec n1 ping -c2 192.168.50.2 # ✅ works now +``` + +**Flow diagram:** + +```mermaid +flowchart LR + A[Host A] -->|ARP Who-has| None[No shared L2 domain] + None -->|No path| X[Dropped] + B[Host B] + A -->|Fix: add bridge| BR[Bridge] + BR --> B +``` + +--- + +**Layer-wise Debugging** +Capture traffic with tcpdump. Try ICMP, TCP, UDP one by one. +👉 Helps visualize OSI/TCP-IP model in action. + +**Solution:** + +```bash +# Setup two namespaces +ip netns add c1; ip netns add c2 +ip link add t1 type veth peer name t2 +ip link set t1 netns c1; ip link set t2 netns c2 +ip netns exec c1 ip addr add 10.9.0.1/24 dev t1 +ip netns exec c2 ip addr add 10.9.0.2/24 dev t2 +ip netns exec c1 ip link set t1 up; ip netns exec c2 ip link set t2 up + +# Start capture on c2 +ip netns exec c2 tcpdump -i t2 -n & + +# Generate traffic from c1 +ip netns exec c1 ping -c2 10.9.0.2 # ICMP +ip netns exec c2 python3 -m http.server 8081 & # TCP server +ip netns exec c1 curl -m3 10.9.0.2:8081 # TCP +ip netns exec c1 sh -lc 'echo hi | nc -u 10.9.0.2 9001' # UDP + +# Interpretation: See L2 (eth), L3 (IP), L4 (ICMP/TCP/UDP) in capture. +``` + +**Flow diagram (example capture view):** + +```mermaid +flowchart LR + C[Client] -->|ICMP Echo| S[Server] + S -->|ICMP Reply| C + C -->|TCP SYN:8081| S + S -->|TCP SYN/ACK| C + C -->|UDP packet| S + S -->|no ACK required| C +``` + +--- + +🔹 **2. IP Addressing (IPv4, IPv6)** + +**Duplicate IP Conflict** +Assign 192.168.1.10 to two hosts. Ping from a third host → observe flapping replies. +👉 Mimics: "Two servers misconfigured with same IP." + +**Solution:** + +```bash +# Setup three namespaces on a bridge +ip netns add h1; ip netns add h2; ip netns add h3 +ip link add br0 type bridge; ip link set br0 up + +# Connect h1 +ip link add h1v type veth peer name h1p +ip link set h1v netns h1; ip link set h1p master br0 +ip netns exec h1 ip addr add 192.168.1.10/24 dev h1v; ip netns exec h1 ip link set h1v up + +# Connect h2 +ip link add h2v type veth peer name h2p +ip link set h2v netns h2; ip link set h2p master br0 +ip netns exec h2 ip addr add 192.168.1.10/24 dev h2v; ip netns exec h2 ip link set h2v up + +# Connect h3 +ip link add h3v type veth peer name h3p +ip link set h3v netns h3; ip link set h3p master br0 +ip netns exec h3 ip addr add 192.168.1.30/24 dev h3v; ip netns exec h3 ip link set h3v up + +# Test from h3 +ip netns exec h3 arping -I h3v -c 3 192.168.1.10 # 🔁 multiple MACs/replies (flap) +ip netns exec h3 ping -c4 192.168.1.10 # ❗ inconsistent replies + +# Interpretation: Duplicate IP causing ARP instability. +# Fix: give h2 a unique IP, e.g. +ip netns exec h2 ip addr flush dev h2v +ip netns exec h2 ip addr add 192.168.1.11/24 dev h2v +``` + +**Flow diagram:** + +```mermaid +sequenceDiagram + participant H3 as Tester (h3) + participant BR as Bridge + participant H1 as Host1 (192.168.1.10 @ MAC A) + participant H2 as Host2 (192.168.1.10 @ MAC B) + + H3->>BR: ARP Who has 192.168.1.10? + BR->>H1: Who has? + H1-->>BR: I am 192.168.1.10 at MAC A + BR-->>H3: ARP reply MAC A + BR->>H2: Who has? + H2-->>BR: I am 192.168.1.10 at MAC B + BR-->>H3: ARP reply MAC B + Note over H3: Replies flap -> connectivity inconsistent +``` + +--- + +**IPv6 Only Network** +Assign IPv6 addresses only (`2001:db8::1/64`). Disable IPv4 → try reaching services. +👉 Mimics: "Transitioning to IPv6 in enterprise." + +**Solution:** + +```bash +# Setup +ip netns add v6a; ip netns add v6b +ip link add v6aif type veth peer name v6bif +ip link set v6aif netns v6a; ip link set v6bif netns v6b +ip netns exec v6a ip -6 addr add 2001:db8::1/64 dev v6aif +ip netns exec v6b ip -6 addr add 2001:db8::2/64 dev v6bif +ip netns exec v6a ip link set v6aif up; ip netns exec v6b ip link set v6bif up + +# Disable IPv4 on both interfaces +ip netns exec v6a sysctl -w net.ipv4.conf.all.disable_ipv4=1 +ip netns exec v6b sysctl -w net.ipv4.conf.all.disable_ipv4=1 + +# Test IPv6 reachability +ip netns exec v6a ping -6 -c2 2001:db8::2 # ✅ works +# Try reaching an IPv4-only IP from v6a (will fail) +ip netns exec v6a ping -6 -c2 142.250.182.14 # ❌ cannot (v4 only target) + +# Interpretation: IPv6-only cannot reach IPv4-only services. +# Fix: deploy dual-stack or NAT64/DNS64. +``` + +**Flow diagram:** + +```mermaid +flowchart LR + A[IPv6-only Host] -->|IPv6 packet| B[IPv6 Server] + B -->|reply| A + A -->|tries IPv4 address| C[IPv4-only Server] + C -->|No route| X[Dropped] +``` + +--- + +**Wrong Subnet Mask** +Host A: 192.168.1.10/24, Host B: 192.168.1.20/16. Traffic works one way but not the other. +👉 Mimics: "Subnet mask mismatch during manual config." + +**Solution:** + +```bash +# Setup +ip netns add A; ip netns add B +ip link add av type veth peer name bv +ip link set av netns A; ip link set bv netns B +ip netns exec A ip addr add 192.168.1.10/24 dev av +ip netns exec B ip addr add 192.168.1.20/16 dev bv +ip netns exec A ip link set av up; ip netns exec B ip link set bv up + +# Test both ways +ip netns exec A ping -c2 192.168.1.20 # may ❌ (A expects router) +ip netns exec B ping -c2 192.168.1.10 # may ✅ (B thinks on-link) + +# Interpretation: Asymmetric routing due to mask mismatch. +# Fix: make both /24 or both /16 consistently. +``` + +**Flow diagram:** + +```mermaid +flowchart LR + A[Host A 192.168.1.10/24] -->|Sees 192.168.1.20 as remote| GW[Gateway] + B[Host B 192.168.1.20/16] -->|Sees 192.168.1.10 as local| A + A -->|send via gateway| X[not delivered] + B -->|ARP| A + A -->|reply| B +``` + +--- + +**Multiple NICs Routing** +One host with 192.168.1.10 (LAN) and 10.0.0.10 (WAN). Configure routing → see which NIC is chosen. +👉 Mimics: "Multi-homed servers with wrong default gateway." + +**Solution:** + +```bash +# Setup one namespace with two NICs via two bridges +ip netns add mh +ip link add br1 type bridge; ip link set br1 up +ip link add br2 type bridge; ip link set br2 up + +ip link add mhv1 type veth peer name tap1 +ip link add mhv2 type veth peer name tap2 +ip link set mhv1 netns mh; ip link set mhv2 netns mh +ip link set tap1 master br1; ip link set tap2 master br2 +ip netns exec mh ip addr add 192.168.1.10/24 dev mhv1 +ip netns exec mh ip addr add 10.0.0.10/24 dev mhv2 +ip netns exec mh ip link set mhv1 up; ip netns exec mh ip link set mhv2 up + +# Default route selection +ip netns exec mh ip route add default via 10.0.0.1 dev mhv2 metric 100 +ip netns exec mh ip route add default via 192.168.1.1 dev mhv1 metric 200 +ip netns exec mh ip route show + +# Policy routing example (force source-based egress) +ip netns exec mh ip rule add from 192.168.1.10/32 table 100 +ip netns exec mh ip route add default via 192.168.1.1 dev mhv1 table 100 +ip netns exec mh ip rule list +``` + +**Flow diagram:** + +```mermaid +flowchart TD + subgraph Multi-homed Host + NIC1[eth0: 192.168.1.10/24] + NIC2[eth1: 10.0.0.10/24] + end + + NIC1 -->|default route metric 200| GW1[192.168.1.1] + NIC2 -->|default route metric 100| GW2[10.0.0.1] + + GW1 --> Internet + GW2 --> Internet + + Note[Traffic prefers lower metric: uses GW2] +``` + +*Note:* The host will prefer the route with lower metric (100) for most traffic. Policy routing can override this based on source IP. + +--- + +This completes the full networking challenge with comprehensive theory and practical exercises. The additional theory sections cover DNS, MTU, and routing concepts that are essential for complete networking understanding. \ No newline at end of file From c0b6879a6c297b3a182beada1c06579e430a3927 Mon Sep 17 00:00:00 2001 From: Swayam-Prakash-Bhuyan Date: Fri, 22 Aug 2025 20:59:33 +0530 Subject: [PATCH 100/133] Added footer for required links --- DailyChallenges/Season2/Day15/challenge.md | 34 +++++++++++++++++++++- 1 file changed, 33 insertions(+), 1 deletion(-) diff --git a/DailyChallenges/Season2/Day15/challenge.md b/DailyChallenges/Season2/Day15/challenge.md index fc31ab4..41994f2 100644 --- a/DailyChallenges/Season2/Day15/challenge.md +++ b/DailyChallenges/Season2/Day15/challenge.md @@ -825,4 +825,36 @@ flowchart TD --- -This completes the full networking challenge with comprehensive theory and practical exercises. The additional theory sections cover DNS, MTU, and routing concepts that are essential for complete networking understanding. \ No newline at end of file +This completes the full networking challenge with comprehensive theory and practical exercises. The additional theory sections cover DNS, MTU, and routing concepts that are essential for complete networking understanding. + +--- + +Deliverables +- A markdown file solution.md with: + - Commands and outputs for Tasks 1–4 + - 3–5 bullet notes on what you learned + +--- + +## Submission Guidelines +- Store your findings and execution steps in solution.md +- Submit it in your GitHub repository and share the link +- Post your experience on social media with #getfitwithsagar #SRELife #DevOpsForAll +- Tag us in your posts for visibility and networking + +--- + +## Join Our Community +- Discord – Ask questions and collaborate: https://discord.gg/mNDm39qB8t +- Google Group – Get updates and discussions: https://groups.google.com/forum/#!forum/daily-devops-sre-challenge-series/join +- YouTube – Watch solution videos and tutorials: https://www.youtube.com/@Sagar.Utekar + +--- + +## Stay Motivated! +Keep it simple. Master the basics. You’ll use these commands every week in real DevOps/SRE work. + +Happy Learning! + +Best regards, +Sagar Utekar \ No newline at end of file From 0fe469a100b959008dbcb38de767ed0e3b7af131 Mon Sep 17 00:00:00 2001 From: Swayam-Prakash-Bhuyan Date: Fri, 22 Aug 2025 21:03:13 +0530 Subject: [PATCH 101/133] cleaned spelling mistakes --- DailyChallenges/Season2/Day15/challenge.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/DailyChallenges/Season2/Day15/challenge.md b/DailyChallenges/Season2/Day15/challenge.md index 41994f2..b7e3fb2 100644 --- a/DailyChallenges/Season2/Day15/challenge.md +++ b/DailyChallenges/Season2/Day15/challenge.md @@ -831,7 +831,7 @@ This completes the full networking challenge with comprehensive theory and pract Deliverables - A markdown file solution.md with: - - Commands and outputs for Tasks 1–4 + - Commands and outputs for Tasks - 3–5 bullet notes on what you learned --- From 761a7ef5ea43790bfaa372bae7c9bcda5e761d33 Mon Sep 17 00:00:00 2001 From: Swayam-Prakash-Bhuyan Date: Mon, 25 Aug 2025 11:23:51 +0530 Subject: [PATCH 102/133] added day 16 challenge --- DailyChallenges/Season2/Day16/challenge.md | 653 +++++++++++++++++++++ 1 file changed, 653 insertions(+) create mode 100644 DailyChallenges/Season2/Day16/challenge.md diff --git a/DailyChallenges/Season2/Day16/challenge.md b/DailyChallenges/Season2/Day16/challenge.md new file mode 100644 index 0000000..8ed6101 --- /dev/null +++ b/DailyChallenges/Season2/Day16/challenge.md @@ -0,0 +1,653 @@ +# Day 16 — Core Protocols Challenge (Daily DevOps + SRE Challenge Series — Season 2) + +--- + +## 🌟 Introduction + +Welcome to **Day 16** of the Daily DevOps + SRE Challenge Series – Season 2! + +Today, we'll explore the **core networking protocols** that form the backbone of modern infrastructure: **TCP, UDP, ICMP, DNS, DHCP, and HTTP/HTTPS**. + +Instead of just learning commands, you'll solve **real-world production-style problems** where these protocols play a critical role. These hands-on tasks will prepare you to debug outages, secure services, and explain protocol-level behavior in interviews with confidence. + +--- + +## 🚀 Why Does This Matter? + +* **TCP vs UDP:** Choosing the right transport protocol affects performance, reliability, and scalability. +* **ICMP:** Quickest way to test connectivity, routing, and latency issues. +* **DNS:** One of the biggest causes of outages in real-world systems. +* **DHCP:** Critical for dynamic IP management in data centers and cloud. +* **HTTP/HTTPS:** The face of almost every service, where security and performance meet. + +--- + +## 🔥 Real-World Save + +* A fintech company once experienced **timeouts on payment APIs**. Root cause: firewall blocked **TCP port 443**. +* A streaming platform suffered **video buffering**. Root cause: packet loss showed TCP retries, but UDP-based CDN solved it. +* A global e-commerce site went down for **4 hours** because of a **DNS misconfiguration**. +* A new VM farm booted up without IPs because of a **rogue DHCP server** in the subnet. +* A startup got flagged for **"insecure website"** by Google Chrome due to expired SSL. + +You'll now walk through these scenarios step by step. + +--- + +## 📘 Theory Section + +### 🔹 TCP (Transmission Control Protocol) + +**Deep Dive:** +TCP is a connection-oriented protocol that ensures reliable, ordered, and error-checked delivery of data between applications. It establishes a connection using a three-way handshake before data transfer begins and maintains state throughout the session. + +**Key Features:** +- Connection-oriented communication +- Error detection and correction +- Flow control (window scaling) +- Congestion control (slow start, congestion avoidance) +- Ordered data transmission +- Retransmission of lost packets + +**TCP Header Structure:** +``` + 0 1 2 3 + 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 ++-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ +| Source Port | Destination Port | ++-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ +| Sequence Number | ++-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ +| Acknowledgment Number | ++-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ +| Data | |U|A|P|R|S|F| | +| Offset| Reserved |R|C|S|S|Y|I| Window | +| | |G|K|H|T|N|N| | ++-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ +| Checksum | Urgent Pointer | ++-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ +| Options | Padding | ++-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ +| data | ++-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ +``` + +**Diagram – TCP Three-Way Handshake** + +```mermaid +sequenceDiagram + participant Client + participant Server + + Note over Client, Server: TCP Three-Way Handshake + Client->>Server: SYN (Synchronize Sequence Number) + Note right of Client: Client sends SYN packet with initial sequence number + Server->>Client: SYN-ACK (Acknowledge SYN, Send own SYN) + Note right of Server: Server acknowledges client's SYN and sends its own SYN + Client->>Server: ACK (Acknowledge Server's SYN) + Note right of Client: Client acknowledges server's SYN + Note over Client, Server: Connection Established +``` + +--- + +### 🔹 UDP (User Datagram Protocol) + +**Deep Dive:** +UDP is a connectionless protocol that provides a simple, unreliable datagram service. It doesn't guarantee delivery, ordering, or duplicate protection, making it faster with less overhead than TCP. + +**Key Features:** +- Connectionless communication +- Minimal protocol overhead +- No guarantee of delivery +- No congestion control +- Supports broadcasting and multicasting + +**UDP Header Structure:** +``` + 0 7 8 15 16 23 24 31 ++--------+--------+--------+--------+ +| Source | Destination | +| Port | Port | ++--------+--------+--------+--------+ +| | | +| Length | Checksum | ++--------+--------+--------+--------+ +| | +| Data | +| | ++-----------------------------------+ +``` + +**Diagram – TCP vs UDP Comparison** + +```mermaid +flowchart TD + A[Application Layer] --> B{Choose Transport Protocol} + + B --> |Reliable Data| C[TCP] + B --> |Low Latency| D[UDP] + + subgraph C[TCP Characteristics] + C1[Connection-oriented] + C2[Reliable Delivery] + C3[Flow Control] + C4[Congestion Control] + C5[Ordered Delivery] + end + + subgraph D[UDP Characteristics] + D1[Connectionless] + D2[Best-effort Delivery] + D3[No Flow Control] + D4[No Congestion Control] + D5[No Order Guarantee] + end + + C --> E[Web Browsing
Email
File Transfer] + D --> F[Video Streaming
DNS
VoIP] +``` + +--- + +### 🔹 ICMP (Internet Control Message Protocol) + +**Deep Dive:** +ICMP is used by network devices to send error messages and operational information. It's primarily used for diagnostic purposes and doesn't carry application data. + +**Common ICMP Types:** +- Type 0: Echo Reply (ping response) +- Type 3: Destination Unreachable +- Type 5: Redirect Message +- Type 8: Echo Request (ping) +- Type 11: Time Exceeded (used by traceroute) + +**ICMP Header Structure:** +``` + 0 1 2 3 + 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 ++-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ +| Type | Code | Checksum | ++-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ +| Contents | ++-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ +``` + +**Diagram – ICMP Ping Operation** + +```mermaid +sequenceDiagram + participant Sender + participant Router1 + participant Router2 + participant Target + + Note over Sender, Target: ICMP Echo Request (Ping) + Sender->>Router1: ICMP Echo Request + Router1->>Router2: ICMP Echo Request + Router2->>Target: ICMP Echo Request + + Note over Target, Sender: ICMP Echo Reply + Target->>Router2: ICMP Echo Reply + Router2->>Router1: ICMP Echo Reply + Router1->>Sender: ICMP Echo Reply + + Note right of Sender: Round-trip time calculated +``` + +--- + +### 🔹 DNS (Domain Name System) + +**Deep Dive:** +DNS is a hierarchical decentralized naming system that translates human-readable domain names to IP addresses. It uses both UDP (for queries) and TCP (for zone transfers). + +**DNS Record Types:** +- A: IPv4 address record +- AAAA: IPv6 address record +- CNAME: Canonical name record (alias) +- MX: Mail exchange record +- NS: Name server record +- TXT: Text record +- SOA: Start of authority record + +**DNS Resolution Process:** +1. Check local cache +2. Query recursive resolver +3. Root server referral +4. TLD server referral +5. Authoritative server response + +**Diagram – DNS Resolution Process** + +```mermaid +flowchart TD + A[User enters example.com] --> B{Local Cache Check} + B -->|Found| C[Return IP Address] + B -->|Not Found| D[Query Recursive Resolver] + + D --> E{Resolver Cache Check} + E -->|Found| C + E -->|Not Found| F[Query Root Server] + + F --> G[Referral to .com TLD] + G --> H[Query .com TLD Server] + H --> I[Referral to example.com NS] + I --> J[Query Authoritative Server] + J --> K[Get IP Address] + K --> L[Cache Response] + L --> C +``` + +--- + +### 🔹 DHCP (Dynamic Host Configuration Protocol) + +**Deep Dive:** +DHCP automatically assigns IP addresses and other network configuration parameters to devices on a network. It uses a client-server model and the DORA process. + +**DHCP Process (DORA):** +1. Discover: Client broadcasts to find DHCP servers +2. Offer: Server responds with an IP offer +3. Request: Client requests the offered IP +4. Acknowledgment: Server confirms the lease + +**DHCP Options:** +- IP Address and subnet mask +- Default gateway +- DNS servers +- Lease time +- Domain name + +**Diagram – DHCP DORA Process** + +```mermaid +sequenceDiagram + participant Client + participant Server + + Note over Client, Server: DHCP Discovery Phase + Client->>Server: DHCPDISCOVER (Broadcast) + Note right of Client: Client broadcasts to discover available DHCP servers + + Note over Client, Server: DHCP Offer Phase + Server->>Client: DHCPOFFER (Unicast) + Note right of Server: Server offers IP address and configuration parameters + + Note over Client, Server: DHCP Request Phase + Client->>Server: DHCPREQUEST (Broadcast) + Note right of Client: Client requests the offered configuration + + Note over Client, Server: DHCP Acknowledgment Phase + Server->>Client: DHCPACK (Unicast) + Note right of Server: Server acknowledges and confirms the lease +``` + +--- + +### 🔹 HTTP/HTTPS + +**Deep Dive:** +HTTP is an application protocol for distributed, collaborative hypermedia information systems. HTTPS is HTTP secured with TLS/SSL encryption. + +**HTTP Methods:** +- GET: Retrieve data +- POST: Submit data +- PUT: Replace data +- DELETE: Remove data +- PATCH: Partially update data + +**HTTP Status Codes:** +- 1xx: Informational +- 2xx: Success (200 OK, 201 Created) +- 3xx: Redirection (301 Moved Permanently) +- 4xx: Client Error (404 Not Found) +- 5xx: Server Error (500 Internal Server Error) + +**HTTPS/TLS Handshake:** +1. Client Hello +2. Server Hello + Certificate +3. Key Exchange +4. Cipher Suite Negotiation +5. Secure Data Transfer + +**Diagram – HTTPS/TLS Handshake** + +```mermaid +sequenceDiagram + participant Client + participant Server + + Note over Client, Server: TLS Handshake + Client->>Server: Client Hello
Supported Cipher Suites + Server->>Client: Server Hello
Server Certificate
Selected Cipher Suite + Note right of Client: Client verifies certificate + Client->>Server: Pre-master Secret
(Encrypted with Server's Public Key) + Server->>Client: Finished + Client->>Server: Finished + + Note over Client, Server: Encrypted Data Exchange + Client->>Server: Application Data
(Encrypted with Session Keys) + Server->>Client: Application Data
(Encrypted with Session Keys) +``` + +--- + +## ⚡ Hands-On Challenges with Solutions + +--- + +### 🔹 Scenario 1: TCP vs UDP in Action + +**Diagram – TCP Connection Lifecycle** + +```mermaid +stateDiagram-v2 + [*] --> CLOSED + CLOSED --> SYN_SENT: Active Open + SYN_SENT --> ESTABLISHED: SYN-ACK Received + ESTABLISHED --> DATA_TRANSFER: Data Exchange + DATA_TRANSFER --> ESTABLISHED: More Data + ESTABLISHED --> FIN_WAIT_1: Active Close + FIN_WAIT_1 --> FIN_WAIT_2: ACK Received + FIN_WAIT_2 --> TIME_WAIT: FIN Received + TIME_WAIT --> [*]: Timeout + CLOSED --> LISTEN: Passive Open + LISTEN --> SYN_RCVD: SYN Received + SYN_RCVD --> ESTABLISHED: ACK Sent + ESTABLISHED --> CLOSE_WAIT: FIN Received + CLOSE_WAIT --> LAST_ACK: Application Close + LAST_ACK --> [*]: ACK Received +``` + +📌 **Task 1: List all services running on TCP and UDP** + +```bash +ss -tuln +lsof -i -P -n | grep LISTEN +``` + +👉 Shows ports like `22/tcp` (SSH), `53/udp` (DNS). + +📌 **Task 2: Compare DNS queries (UDP) with HTTP requests (TCP)** + +```bash +dig google.com +curl -v http://example.com +``` + +👉 `dig` uses UDP, while `curl` establishes a TCP handshake. + +📌 **Task 3: Check if a specific TCP port is open** + +```bash +nc -zv google.com 80 +``` + +👉 Tests if port 80 is open on google.com + +📌 **Task 4: Check if a specific UDP port is open** + +```bash +nc -zvu 8.8.8.8 53 +``` + +👉 Tests if DNS port (53) is open on Google's DNS server + +📌 **Task 5: View active TCP connections** + +```bash +netstat -t +``` + +👉 Shows all active TCP connections on your system + +--- + +### 🔹 Scenario 2: ICMP (ping & traceroute) - Simplified + +**Diagram – Traceroute Operation** + +```mermaid +sequenceDiagram + participant Sender + participant Router1 + participant Router2 + participant Target + + Note over Sender, Target: Traceroute Process + Sender->>Router1: UDP/TCP/ICMP packet with TTL=1 + Router1->>Sender: ICMP Time Exceeded + Sender->>Router1: UDP/TCP/ICMP packet with TTL=2 + Router1->>Router2: Forward packet + Router2->>Sender: ICMP Time Exceeded + Sender->>Router1: UDP/TCP/ICMP packet with TTL=3 + Router1->>Router2: Forward packet + Router2->>Target: Forward packet + Target->>Sender: ICMP Echo Reply/Destination Unreachable +``` + +📌 **Task 1: Test basic connectivity** + +```bash +ping -c 4 google.com +``` + +📌 **Task 2: Trace the network path** + +```bash +traceroute google.com +``` + +📌 **Task 3: Test connectivity to a specific port** + +```bash +ping -c 4 google.com +``` + +📌 **Task 4: Check if a host is reachable with timestamp** + +```bash +ping -c 4 -D google.com +``` + +👉 Adds timestamp to each ping response + +📌 **Task 5: Test network quality with mtr** + +```bash +mtr --report google.com +``` + +👉 Combines ping and traceroute functionality + +--- + +### 🔹 Scenario 3: DNS Troubleshooting (Simplified) + +**Diagram – DNS Query Types** + +```mermaid +flowchart TD + A[DNS Query] --> B{Query Type} + B --> C[Recursive Query] + B --> D[Iterative Query] + + subgraph C[Recursive Query] + C1[Client asks resolver] + C2[Resolver does all work] + C3[Returns final answer] + end + + subgraph D[Iterative Query] + D1[Client asks server] + D2[Server gives best answer] + D3[Client queries next server] + end +``` + +📌 **Task 1: Check DNS config** + +```bash +cat /etc/resolv.conf +``` + +📌 **Task 2: Query DNS records** + +```bash +dig google.com +``` + +📌 **Task 3: Query specific DNS record types** + +```bash +dig google.com A +dig google.com MX +``` + +👉 Gets different types of DNS records + +📌 **Task 4: Flush DNS cache (if applicable)** + +```bash +sudo systemd-resolve --flush-caches +``` + +👉 Clears local DNS cache (systemd systems) + +📌 **Task 5: Test DNS resolution speed** + +```bash +time dig google.com +``` + +👉 Measures how long DNS resolution takes + +--- + +### 🔹 Scenario 4: DHCP Assignment Issues (Simplified) + +**Diagram – DHCP Lease Process** + +```mermaid +timeline + title DHCP Lease Timeline + section Lease Acquisition + DORA Process : 4-step process + IP Configured : Client gets IP + section Lease Duration + T1 (50%) : Client tries to renew + T2 (87.5%) : Client broadcasts renew request + Lease Expiration : IP address released +``` + +📌 **Task 1: Check current IP configuration** + +```bash +ip addr show +``` + +📌 **Task 2: Release and renew DHCP lease** + +```bash +sudo dhclient -r +sudo dhclient +``` + +📌 **Task 3: Check DHCP client status** + +```bash +journalctl -u systemd-networkd | grep DHCP +``` + +👉 Views DHCP-related logs + +📌 **Task 4: Set a temporary static IP** + +```bash +sudo ip addr add 192.168.1.100/24 dev eth0 +``` + +📌 **Task 5: Check network connectivity** + +```bash +ping -c 4 8.8.8.8 +``` + +👉 Tests if you have internet access after DHCP changes + +--- + +### 🔹 Scenario 5: HTTP/HTTPS Debugging (Simplified) + +**Diagram – HTTP Request/Response** + +```mermaid +sequenceDiagram + participant Client + participant Server + + Note over Client, Server: HTTP Request + Client->>Server: GET /index.html HTTP/1.1
Host: example.com
User-Agent: curl/7.68.0
Accept: */* + + Note over Client, Server: HTTP Response + Server->>Client: HTTP/1.1 200 OK
Content-Type: text/html
Content-Length: 1256

... +``` + +📌 **Task 1: Check if web ports are open** + +```bash +ss -tuln | grep :80 +ss -tuln | grep :443 +``` + +📌 **Task 2: Test HTTP connectivity** + +```bash +curl -I http://example.com +``` + +👉 Gets only HTTP headers + +📌 **Task 3: Test HTTPS connectivity** + +```bash +curl -I https://example.com +``` + +📌 **Task 4: Check SSL certificate validity** + +```bash +curl -v https://example.com 2>&1 | grep "SSL certificate" +``` + +📌 **Task 5: Test website loading time** + +```bash +time curl -s -o /dev/null https://example.com +``` + +👉 Measures how long a website takes to load + +--- + +## ✅ Deliverables + +* Document everything in `solution.md` with: + + * Commands run + * Observations + * Screenshots (optional) +* Push to GitHub & share link +* Post your experience on social media with: + **#getfitwithsagar #SRELife #DevOpsForAll** + +--- + +## 🌍 Community Links + +* **Discord**: [https://discord.gg/mNDm39qB8t](https://discord.gg/mNDm39qB8t) +* **Google Group**: [https://groups.google.com/forum/#!forum/daily-devops-sre-challenge-series/join](https://groups.google.com/forum/#!forum/daily-devops-sre-challenge-series/join) +* **YouTube**: [https://www.youtube.com/@Sagar.Utekar](https://www.youtube.com/@Sagar.Utekar) + +--- From d0a651a2201e09c7b70394a599f259520be520cf Mon Sep 17 00:00:00 2001 From: Swayam-Prakash-Bhuyan Date: Thu, 28 Aug 2025 15:11:58 +0530 Subject: [PATCH 103/133] day 17 added --- DailyChallenges/Season2/Day17/challenge.md | 401 +++++++++++++++++++++ 1 file changed, 401 insertions(+) create mode 100644 DailyChallenges/Season2/Day17/challenge.md diff --git a/DailyChallenges/Season2/Day17/challenge.md b/DailyChallenges/Season2/Day17/challenge.md new file mode 100644 index 0000000..fff3520 --- /dev/null +++ b/DailyChallenges/Season2/Day17/challenge.md @@ -0,0 +1,401 @@ +# Day 17 — Routing & Switching Basics (Daily DevOps + SRE Challenge Series — Season 2) + +--- + +## 🌟 Introduction + +Welcome to **Day 3** of the Daily DevOps + SRE Challenge Series – Season 2! + +Today we dive into the **Routing & Switching basics** every DevOps/SRE must master: **Static Routing, Default Gateways, NAT/PAT, and VLANs.** These are the invisible highways of networking that ensure packets get from point A to point B securely, efficiently, and reliably. + +Instead of just reading theory, you'll solve **real-world, production-style challenges** with Linux commands, diagrams, and troubleshooting steps. + +--- + +## 🚀 Why Does This Matter? + +* **Connectivity**: Without routing, packets never leave your subnet. +* **Security**: NAT/PAT and VLANs are used everywhere in enterprises to hide internal networks and isolate services. +* **Reliability**: Correct default gateway configs prevent outages. +* **Scale**: VLANs + static/dynamic routing make it possible to manage thousands of users and workloads. +* **Interview Edge**: Common questions include *"What is a default gateway?"*, *"Explain NAT vs PAT"*, and *"How do VLANs improve security?"* + +--- + +## 🔥 Real-World Save + +* A SaaS startup lost internet access for **6 hours** because the **wrong default gateway** was configured on a new subnet. +* A fintech company reduced **public IP usage by 90%** using **PAT**. +* A data center was compromised because **VLANs weren't properly isolated** — attackers jumped between networks. +* A global team avoided a costly outage by documenting **static routes** during a WAN migration. + +--- + +## 📘 Theory Section + +### 🔹 Static Routing + +Static routing involves manually configuring routes on network devices (routers, firewalls, or even servers) to specify the path network traffic should take to reach specific destinations. Unlike dynamic routing protocols (OSPF, BGP) that automatically exchange routing information, static routes are fixed and don't adapt to network changes unless manually modified. + +**Key Characteristics:** +- **Administrative Distance**: Static routes typically have a lower AD (1) than dynamic routes, making them preferred +- **Persistence**: Usually remain in routing table until manually removed +- **No Overhead**: Unlike dynamic protocols, they don't consume bandwidth for route advertisements +- **Scalability Limitation**: Manual configuration becomes impractical in large, complex networks + +**Common Use Cases:** +- Default routes for internet connectivity +- Backup routes for redundant connections +- Stub networks with single exit points +- Security-sensitive paths where dynamic routing might be exploited + +```mermaid +flowchart TD + subgraph NetworkA[Network A: 192.168.1.0/24] + A1[Host A1
192.168.1.10] + A2[Host A2
192.168.1.11] + end + + subgraph NetworkB[Network B: 10.0.2.0/24] + B1[Host B1
10.0.2.10] + B2[Host B2
10.0.2.11] + end + + R1[Router A
192.168.1.1/24
172.16.1.1/30] + R2[Router B
172.16.1.2/30
10.0.2.1/24] + + NetworkA --> R1 + R1 -->|Static Route: 10.0.2.0/24 via 172.16.1.2| R2 + R2 --> NetworkB + + style R1 fill:#e1f5fe + style R2 fill:#e1f5fe +``` + +**Technical Details:** +When a packet arrives at Router A destined for Network B (10.0.2.0/24), Router A checks its routing table. The static route instructs it to forward the packet to Router B (172.16.1.2). Router B then delivers it to the appropriate host in Network B. + +The reverse path requires a complementary static route on Router B pointing to Router A for Network A (192.168.1.0/24), ensuring bidirectional communication. + +--- + +### 🔹 Default Gateway + +The default gateway (also called default route) is a special type of static route that serves as the "gateway of last resort." When a device needs to send traffic to a destination network not explicitly listed in its routing table, it forwards the packet to the default gateway. + +**Technical Implementation:** +- Represented as 0.0.0.0/0 in CIDR notation (matches any destination) +- Essential for client devices to reach beyond their local subnet +- Critical for internet connectivity in most network setups + +```mermaid +flowchart TD + subgraph LocalNetwork[Local Network: 192.168.1.0/24] + PC1[Workstation
192.168.1.10] + PC2[Server
192.168.1.20] + PC3[Laptop
192.168.1.30] + end + + Router[Edge Router
192.168.1.1/24
203.0.113.5/30] + + Internet[Internet] + + PC1 -->|Default Gateway: 192.168.1.1| Router + PC2 -->|Default Gateway: 192.168.1.1| Router + PC3 -->|Default Gateway: 192.168.1.1| Router + Router -->|Default Route: 0.0.0.0/0
via ISP Router| Internet + + style Router fill:#fff3e0 +``` + +**Technical Details:** +When Workstation (192.168.1.10) needs to communicate with a server on the internet (e.g., 8.8.8.8), it performs these steps: +1. Compares destination IP (8.8.8.8) with its own IP and subnet mask +2. Determines the destination is outside its local network +3. Forwards the packet to its configured default gateway (192.168.1.1) +4. The edge router then routes the packet toward the internet + +Without a proper default gateway configuration, devices can only communicate within their local subnet. + +--- + +### 🔹 NAT & PAT + +Network Address Translation (NAT) and Port Address Translation (PAT) are methods to remap IP address space by modifying network address information in packet headers while in transit. This allows multiple devices to share a single public IP address. + +**NAT Types:** +1. **Static NAT**: One-to-one mapping between private and public IPs +2. **Dynamic NAT**: Pool of public IPs mapped to private IPs as needed +3. **PAT/NAT Overload**: Many private IPs to one public IP using port differentiation + +```mermaid +flowchart LR + subgraph PrivateNetwork[Private Network] + A[Web Server
10.0.0.10:80] + B[Workstation
10.0.0.11:3456] + C[Laptop
10.0.0.12:5000] + end + + subgraph NATRouter[NAT Router] + NAT[Translation Table] + end + + Internet[Internet] + + A -->|Source: 10.0.0.10:80| NAT + B -->|Source: 10.0.0.11:3456| NAT + C -->|Source: 10.0.0.12:5000| NAT + + NAT -->|Translated: 203.0.113.5:8080| Internet + NAT -->|Translated: 203.0.113.5:3457| Internet + NAT -->|Translated: 203.0.113.5:5001| Internet + + style NATRouter fill:#f3e5f5 +``` + +**Technical Details:** +PAT (the most common form of NAT) works by: +1. Maintaining a translation table mapping internal (private IP, port) to external (public IP, port) +2. When an internal device initiates outbound traffic, the router: + - Replaces the source private IP with the public IP + - Replaces the source port with a unique port number + - Records this mapping in its translation table +3. When inbound traffic arrives, the router: + - Checks the destination port against its translation table + - Rewrites the destination IP and port to the internal values + - Forwards the packet to the internal device + +This process allows thousands of internal devices to share a single public IP address while maintaining session state. + +--- + +### 🔹 VLANs + +Virtual LANs (VLANs) create logically separate networks within a physical network infrastructure. VLANs operate at Layer 2 of the OSI model and provide segmentation, security, and broadcast containment. + +**VLAN Benefits:** +- **Security**: Isolation between sensitive departments (HR, Finance) +- **Performance**: Reduced broadcast traffic +- **Flexibility**: Logical grouping regardless of physical location +- **Management**: Simplified network administration + +```mermaid +flowchart TD + Switch[Layer 3 Switch] + + subgraph VLAN10[VLAN 10 - HR Department] + HR1[HR PC1
192.168.10.10] + HR2[HR PC2
192.168.10.11] + HR3[HR Server
192.168.10.100] + end + + subgraph VLAN20[VLAN 20 - Engineering] + Eng1[Dev Workstation
192.168.20.10] + Eng2[Test Server
192.168.20.50] + Eng3[Dev Laptop
192.168.20.20] + end + + subgraph VLAN30[VLAN 30 - Guests] + Guest1[Guest WiFi
192.168.30.0/24] + Guest2[Visitor PC
192.168.30.10] + end + + Router[Router] + + HR1 -->|Access Port
VLAN 10| Switch + HR2 -->|Access Port
VLAN 10| Switch + HR3 -->|Access Port
VLAN 10| Switch + + Eng1 -->|Access Port
VLAN 20| Switch + Eng2 -->|Access Port
VLAN 20| Switch + Eng3 -->|Access Port
VLAN 20| Switch + + Guest1 -->|Access Port
VLAN 30| Switch + Guest2 -->|Access Port
VLAN 30| Switch + + Switch -->|Trunk Port
All VLANs| Router + + style Switch fill:#e8f5e9 +``` + +**Technical Details:** +VLANs work through IEEE 802.1Q tagging, where a 4-byte tag is inserted into Ethernet frames to identify VLAN membership. This allows: + +1. **Access Ports**: Connect end devices to a single VLAN (untagged) +2. **Trunk Ports**: Carry multiple VLANs between switches/routers (tagged) +3. **Inter-VLAN Routing**: Layer 3 devices route between VLANs + +Without VLANs, all devices on a physical network would belong to the same broadcast domain, creating security risks and performance issues as networks scale. + +--- + +## ⚡ Hands-On Challenges with Solutions + +--- + +### 🔹 Static Routing (5 Challenges) + +1️⃣ **Configure a static route between two routers and test connectivity** + +```bash +# On Router A +ip route add 10.0.2.0/24 via 192.168.1.2 +# Verify +ping 10.0.2.1 +``` + +2️⃣ **Add multiple static routes and analyze preference** + +```bash +ip route add 10.0.3.0/24 via 192.168.1.2 metric 100 +ip route add 10.0.3.0/24 via 192.168.1.3 metric 200 +ip route show +``` + +👉 Lower metric = higher preference. + +3️⃣ **Break a static route intentionally** + +```bash +ip route del 10.0.2.0/24 +ping 10.0.2.1 # should fail +``` + +👉 Use `ip route` and `traceroute` to debug. + +4️⃣ **Compare static vs dynamic routing** + +* Static = manual, simple, no overhead. +* Dynamic = automatic updates, scales better (OSPF, BGP). + +5️⃣ **Document branch-to-HQ static routing topology** +👉 Draw with Mermaid or tools like draw\.io. + +--- + +### 🔹 Default Gateway (5 Challenges) + +1️⃣ **Set up default gateway on Linux** + +```bash +ip route add default via 192.168.1.1 +ping 8.8.8.8 +``` + +2️⃣ **Remove/modify gateway** + +```bash +ip route del default +ping 8.8.8.8 # fails +``` + +3️⃣ **Multiple gateways with metrics** + +```bash +ip route add default via 192.168.1.1 metric 100 +ip route add default via 192.168.1.2 metric 200 +``` + +4️⃣ **Capture packets via gateway** + +```bash +sudo tcpdump -i eth0 icmp +``` + +👉 Shows pings exiting via gateway. + +5️⃣ **Troubleshoot wrong gateway** + +* Symptom: Can ping local but not internet. +* Fix: Correct default gateway in `/etc/netplan/*` or `/etc/network/interfaces`. + +--- + +### 🔹 NAT & PAT (5 Challenges) + +1️⃣ **Configure NAT on Linux** + +```bash +sudo iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE +``` + +2️⃣ **PAT with multiple hosts** +👉 NAT maps each host's **port** to the shared public IP. +Check with `curl ifconfig.me` from 2 VMs. + +3️⃣ **iptables NAT test** + +```bash +curl -4 ifconfig.me # shows public IP +``` + +4️⃣ **Capture NAT packets** + +```bash +sudo tcpdump -i eth0 port 80 +``` + +5️⃣ **Troubleshoot NAT failure** + +* Check IP forwarding: + +```bash +sysctl net.ipv4.ip_forward +``` + +* Verify firewall rules. + +--- + +### 🔹 VLAN Basics (5 Challenges) + +1️⃣ **Create VLANs on a switch (Linux bridge example)** + +```bash +sudo ip link add link eth0 name eth0.10 type vlan id 10 +sudo ip addr add 192.168.10.1/24 dev eth0.10 +``` + +2️⃣ **Inter-VLAN routing (Router-on-a-stick)** + +```bash +# Router with sub-interfaces +ip addr add 192.168.10.1/24 dev eth0.10 +ip addr add 192.168.20.1/24 dev eth0.20 +``` + +3️⃣ **Misconfigure VLAN and troubleshoot** +👉 Put two devices in different VLANs, they won't talk. +Fix: put them in the same VLAN or configure routing. + +4️⃣ **Explore VLAN tagging (802.1Q)** + +```bash +sudo tcpdump -e -i eth0 vlan +``` + +--- + +## ✅ Deliverables + +* Write solutions in `solution.md` with: + + * Commands run + * Observations + * Screenshots/diagrams +* Push to your GitHub repo & share link. +* Post your experience with hashtags: + **#getfitwithsagar #SRELife #DevOpsForAll** + +--- + +## 🌍 Community Links + +* **Discord**: [https://discord.gg/mNDm39qB8t](https://discord.gg/mNDm39qB8t) +* **Google Group**: [https://groups.google.com/forum/#!forum/daily-devops-sre-challenge-series/join](https://groups.google.com/forum/#!forum/daily-devops-sre-challenge-series/join) +* **YouTube**: [https://www.youtube.com/@Sagar.Utekar](https://www.youtube.com/@Sagar.Utekar) + +--- + +🔥 Keep routing, keep switching, and happy exploring! +— *Sagar Utekar* \ No newline at end of file From bb9987958b298e10d5db696e22adeb2756ac4cd9 Mon Sep 17 00:00:00 2001 From: Swayam Prakash Bhuyan Date: Thu, 28 Aug 2025 15:27:09 +0530 Subject: [PATCH 104/133] Update challenge with lab setup.md --- DailyChallenges/Season2/Day17/challenge.md | 397 ++++++++++++++------- 1 file changed, 263 insertions(+), 134 deletions(-) diff --git a/DailyChallenges/Season2/Day17/challenge.md b/DailyChallenges/Season2/Day17/challenge.md index fff3520..e093c2e 100644 --- a/DailyChallenges/Season2/Day17/challenge.md +++ b/DailyChallenges/Season2/Day17/challenge.md @@ -8,7 +8,7 @@ Welcome to **Day 3** of the Daily DevOps + SRE Challenge Series – Season 2! Today we dive into the **Routing & Switching basics** every DevOps/SRE must master: **Static Routing, Default Gateways, NAT/PAT, and VLANs.** These are the invisible highways of networking that ensure packets get from point A to point B securely, efficiently, and reliably. -Instead of just reading theory, you'll solve **real-world, production-style challenges** with Linux commands, diagrams, and troubleshooting steps. +Instead of just reading theory, you'll solve **real-world, production-style challenges** with Linux commands, diagrams, and troubleshooting steps - all within a single virtual machine using network namespaces. --- @@ -35,19 +35,7 @@ Instead of just reading theory, you'll solve **real-world, production-style chal ### 🔹 Static Routing -Static routing involves manually configuring routes on network devices (routers, firewalls, or even servers) to specify the path network traffic should take to reach specific destinations. Unlike dynamic routing protocols (OSPF, BGP) that automatically exchange routing information, static routes are fixed and don't adapt to network changes unless manually modified. - -**Key Characteristics:** -- **Administrative Distance**: Static routes typically have a lower AD (1) than dynamic routes, making them preferred -- **Persistence**: Usually remain in routing table until manually removed -- **No Overhead**: Unlike dynamic protocols, they don't consume bandwidth for route advertisements -- **Scalability Limitation**: Manual configuration becomes impractical in large, complex networks - -**Common Use Cases:** -- Default routes for internet connectivity -- Backup routes for redundant connections -- Stub networks with single exit points -- Security-sensitive paths where dynamic routing might be exploited +Static routing involves manually configuring routes on network devices to specify the path network traffic should take to reach specific destinations. Unlike dynamic routing protocols that automatically exchange routing information, static routes are fixed and don't adapt to network changes unless manually modified. ```mermaid flowchart TD @@ -72,21 +60,11 @@ flowchart TD style R2 fill:#e1f5fe ``` -**Technical Details:** -When a packet arrives at Router A destined for Network B (10.0.2.0/24), Router A checks its routing table. The static route instructs it to forward the packet to Router B (172.16.1.2). Router B then delivers it to the appropriate host in Network B. - -The reverse path requires a complementary static route on Router B pointing to Router A for Network A (192.168.1.0/24), ensuring bidirectional communication. - --- ### 🔹 Default Gateway -The default gateway (also called default route) is a special type of static route that serves as the "gateway of last resort." When a device needs to send traffic to a destination network not explicitly listed in its routing table, it forwards the packet to the default gateway. - -**Technical Implementation:** -- Represented as 0.0.0.0/0 in CIDR notation (matches any destination) -- Essential for client devices to reach beyond their local subnet -- Critical for internet connectivity in most network setups +The default gateway serves as the "gateway of last resort." When a device needs to send traffic to a destination network not explicitly listed in its routing table, it forwards the packet to the default gateway. ```mermaid flowchart TD @@ -108,25 +86,11 @@ flowchart TD style Router fill:#fff3e0 ``` -**Technical Details:** -When Workstation (192.168.1.10) needs to communicate with a server on the internet (e.g., 8.8.8.8), it performs these steps: -1. Compares destination IP (8.8.8.8) with its own IP and subnet mask -2. Determines the destination is outside its local network -3. Forwards the packet to its configured default gateway (192.168.1.1) -4. The edge router then routes the packet toward the internet - -Without a proper default gateway configuration, devices can only communicate within their local subnet. - --- ### 🔹 NAT & PAT -Network Address Translation (NAT) and Port Address Translation (PAT) are methods to remap IP address space by modifying network address information in packet headers while in transit. This allows multiple devices to share a single public IP address. - -**NAT Types:** -1. **Static NAT**: One-to-one mapping between private and public IPs -2. **Dynamic NAT**: Pool of public IPs mapped to private IPs as needed -3. **PAT/NAT Overload**: Many private IPs to one public IP using port differentiation +Network Address Translation (NAT) and Port Address Translation (PAT) are methods to remap IP address space by modifying network address information in packet headers while in transit. ```mermaid flowchart LR @@ -153,32 +117,12 @@ flowchart LR style NATRouter fill:#f3e5f5 ``` -**Technical Details:** -PAT (the most common form of NAT) works by: -1. Maintaining a translation table mapping internal (private IP, port) to external (public IP, port) -2. When an internal device initiates outbound traffic, the router: - - Replaces the source private IP with the public IP - - Replaces the source port with a unique port number - - Records this mapping in its translation table -3. When inbound traffic arrives, the router: - - Checks the destination port against its translation table - - Rewrites the destination IP and port to the internal values - - Forwards the packet to the internal device - -This process allows thousands of internal devices to share a single public IP address while maintaining session state. - --- ### 🔹 VLANs Virtual LANs (VLANs) create logically separate networks within a physical network infrastructure. VLANs operate at Layer 2 of the OSI model and provide segmentation, security, and broadcast containment. -**VLAN Benefits:** -- **Security**: Isolation between sensitive departments (HR, Finance) -- **Performance**: Reduced broadcast traffic -- **Flexibility**: Logical grouping regardless of physical location -- **Management**: Simplified network administration - ```mermaid flowchart TD Switch[Layer 3 Switch] @@ -218,173 +162,358 @@ flowchart TD style Switch fill:#e8f5e9 ``` -**Technical Details:** -VLANs work through IEEE 802.1Q tagging, where a 4-byte tag is inserted into Ethernet frames to identify VLAN membership. This allows: - -1. **Access Ports**: Connect end devices to a single VLAN (untagged) -2. **Trunk Ports**: Carry multiple VLANs between switches/routers (tagged) -3. **Inter-VLAN Routing**: Layer 3 devices route between VLANs +--- -Without VLANs, all devices on a physical network would belong to the same broadcast domain, creating security risks and performance issues as networks scale. +## ⚡ Hands-On Challenges with Solutions (Single VM Edition) ---- +### Prerequisites: Setup Network Namespaces +First, let's create the network namespaces that will simulate different devices: -## ⚡ Hands-On Challenges with Solutions +```bash +# Create network namespaces +sudo ip netns add client1 +sudo ip netns add client2 +sudo ip netns add router + +# Create virtual Ethernet pairs +sudo ip link add veth0 type veth peer name veth1 +sudo ip link add veth2 type veth peer name veth3 + +# Move interfaces to appropriate namespaces +sudo ip link set veth0 netns client1 +sudo ip link set veth1 netns router +sudo ip link set veth2 netns client2 +sudo ip link set veth3 netns router + +# Configure IP addresses +sudo ip netns exec client1 ip addr add 192.168.1.10/24 dev veth0 +sudo ip netns exec router ip addr add 192.168.1.1/24 dev veth1 +sudo ip netns exec client2 ip addr add 10.0.2.10/24 dev veth2 +sudo ip netns exec router ip addr add 10.0.2.1/24 dev veth3 + +# Bring interfaces up +sudo ip netns exec client1 ip link set veth0 up +sudo ip netns exec router ip link set veth1 up +sudo ip netns exec client2 ip link set veth2 up +sudo ip netns exec router ip link set veth3 up +sudo ip netns exec client1 ip link set lo up +sudo ip netns exec client2 ip link set lo up +sudo ip netns exec router ip link set lo up +``` --- ### 🔹 Static Routing (5 Challenges) -1️⃣ **Configure a static route between two routers and test connectivity** +1️⃣ **Configure static routes between namespaces** ```bash -# On Router A -ip route add 10.0.2.0/24 via 192.168.1.2 -# Verify -ping 10.0.2.1 +# On router namespace +sudo ip netns exec router ip route add 192.168.1.0/24 dev veth1 +sudo ip netns exec router ip route add 10.0.2.0/24 dev veth3 + +# On client1 namespace (add route to network 10.0.2.0/24 via router) +sudo ip netns exec client1 ip route add 10.0.2.0/24 via 192.168.1.1 + +# On client2 namespace (add route to network 192.168.1.0/24 via router) +sudo ip netns exec client2 ip route add 192.168.1.0/24 via 10.0.2.1 + +# Test connectivity +sudo ip netns exec client1 ping 10.0.2.10 -c 3 +sudo ip netns exec client2 ping 192.168.1.10 -c 3 ``` 2️⃣ **Add multiple static routes and analyze preference** ```bash -ip route add 10.0.3.0/24 via 192.168.1.2 metric 100 -ip route add 10.0.3.0/24 via 192.168.1.3 metric 200 -ip route show -``` +# Create an alternative path (simulated) +sudo ip netns exec router ip route add 10.0.2.0/24 dev veth3 metric 100 +sudo ip netns exec router ip route add 10.0.2.0/24 dev veth3 metric 200 -👉 Lower metric = higher preference. +# View routing table and analyze preference +sudo ip netns exec router ip route show +``` -3️⃣ **Break a static route intentionally** +3️⃣ **Break a static route intentionally and troubleshoot** ```bash -ip route del 10.0.2.0/24 -ping 10.0.2.1 # should fail +# Remove the route +sudo ip netns exec client1 ip route del 10.0.2.0/24 + +# Test connectivity (should fail) +sudo ip netns exec client1 ping 10.0.2.10 -c 2 + +# Troubleshoot with traceroute +sudo ip netns exec client1 traceroute 10.0.2.10 + +# Fix the route +sudo ip netns exec client1 ip route add 10.0.2.0/24 via 192.168.1.1 ``` -👉 Use `ip route` and `traceroute` to debug. +4️⃣ **Compare static vs dynamic routing concepts** -4️⃣ **Compare static vs dynamic routing** +```bash +# Examine current static routes +sudo ip netns exec router ip route show -* Static = manual, simple, no overhead. -* Dynamic = automatic updates, scales better (OSPF, BGP). +# Compare with what dynamic routing would provide +echo "Static routing: Manual configuration, no overhead" +echo "Dynamic routing: Automatic updates, better for large networks" +``` -5️⃣ **Document branch-to-HQ static routing topology** -👉 Draw with Mermaid or tools like draw\.io. +5️⃣ **Document the namespace routing topology** + +```bash +# Show all interfaces and their addresses +sudo ip netns exec client1 ip addr show +sudo ip netns exec client2 ip addr show +sudo ip netns exec router ip addr show + +# Show routing tables +sudo ip netns exec client1 ip route show +sudo ip netns exec client2 ip route show +sudo ip netns exec router ip route show +``` --- ### 🔹 Default Gateway (5 Challenges) -1️⃣ **Set up default gateway on Linux** +1️⃣ **Set up default gateway in namespace** ```bash -ip route add default via 192.168.1.1 -ping 8.8.8.8 +# Create a simulated internet namespace +sudo ip netns add internet +sudo ip link add veth4 type veth peer name veth5 +sudo ip link set veth4 netns router +sudo ip link set veth5 netns internet +sudo ip netns exec router ip addr add 203.0.113.1/24 dev veth4 +sudo ip netns exec internet ip addr add 203.0.113.2/24 dev veth5 +sudo ip netns exec router ip link set veth4 up +sudo ip netns exec internet ip link set veth5 up + +# Set default gateway on router +sudo ip netns exec router ip route add default via 203.0.113.2 + +# Set default gateway on clients +sudo ip netns exec client1 ip route add default via 192.168.1.1 +sudo ip netns exec client2 ip route add default via 10.0.2.1 ``` -2️⃣ **Remove/modify gateway** +2️⃣ **Remove/modify gateway and test** ```bash -ip route del default -ping 8.8.8.8 # fails +# Remove default gateway +sudo ip netns exec client1 ip route del default + +# Test connectivity (should fail) +sudo ip netns exec client1 ping 203.0.113.2 -c 2 + +# Restore gateway +sudo ip netns exec client1 ip route add default via 192.168.1.1 ``` 3️⃣ **Multiple gateways with metrics** ```bash -ip route add default via 192.168.1.1 metric 100 -ip route add default via 192.168.1.2 metric 200 +# Add multiple default routes with different metrics +sudo ip netns exec client1 ip route add default via 192.168.1.1 metric 100 +sudo ip netns exec client1 ip route add default via 192.168.1.1 metric 200 + +# Show routing table to see preference +sudo ip netns exec client1 ip route show ``` -4️⃣ **Capture packets via gateway** +4️⃣ **Capture packets going through gateway** ```bash -sudo tcpdump -i eth0 icmp +# Start packet capture on router +sudo ip netns exec router timeout 10 tcpdump -i veth1 icmp & + +# Generate traffic from client1 +sudo ip netns exec client1 ping 203.0.113.2 -c 3 ``` -👉 Shows pings exiting via gateway. +5️⃣ **Troubleshoot wrong gateway configuration** + +```bash +# Intentionally set wrong gateway +sudo ip netns exec client1 ip route del default +sudo ip netns exec client1 ip route add default via 192.168.1.99 + +# Test connectivity (should fail) +sudo ip netns exec client1 ping 203.0.113.2 -c 2 -5️⃣ **Troubleshoot wrong gateway** +# Diagnose the issue +sudo ip netns exec client1 ip route show +sudo ip netns exec client1 traceroute 203.0.113.2 -* Symptom: Can ping local but not internet. -* Fix: Correct default gateway in `/etc/netplan/*` or `/etc/network/interfaces`. +# Fix the gateway +sudo ip netns exec client1 ip route del default +sudo ip netns exec client1 ip route add default via 192.168.1.1 +``` --- ### 🔹 NAT & PAT (5 Challenges) -1️⃣ **Configure NAT on Linux** +1️⃣ **Configure NAT on the router namespace** ```bash -sudo iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE +# Enable IP forwarding on router +sudo ip netns exec router sysctl -w net.ipv4.ip_forward=1 + +# Configure NAT using iptables +sudo ip netns exec router iptables -t nat -A POSTROUTING -o veth4 -j MASQUERADE +sudo ip netns exec router iptables -A FORWARD -i veth1 -o veth4 -j ACCEPT +sudo ip netns exec router iptables -A FORWARD -i veth4 -o veth1 -m state --state ESTABLISHED,RELATED -j ACCEPT + +# Test NAT functionality +sudo ip netns exec client1 ping 203.0.113.2 -c 3 ``` -2️⃣ **PAT with multiple hosts** -👉 NAT maps each host's **port** to the shared public IP. -Check with `curl ifconfig.me` from 2 VMs. +2️⃣ **Test PAT with multiple hosts** + +```bash +# Simulate multiple clients making connections +sudo ip netns exec client1 ping 203.0.113.2 -c 2 & +sudo ip netns exec client2 ping 203.0.113.2 -c 2 & -3️⃣ **iptables NAT test** +# Check NAT translations +sudo ip netns exec router iptables -t nat -L -n -v +``` + +3️⃣ **Test iptables NAT configuration** ```bash -curl -4 ifconfig.me # shows public IP +# Check NAT table +sudo ip netns exec router iptables -t nat -L + +# Verify NAT is working by checking connection tracking +sudo ip netns exec router conntrack -L ``` 4️⃣ **Capture NAT packets** ```bash -sudo tcpdump -i eth0 port 80 +# Capture packets on the external interface +sudo ip netns exec router timeout 10 tcpdump -i veth4 -n & + +# Generate traffic from clients +sudo ip netns exec client1 ping 203.0.113.2 -c 3 ``` 5️⃣ **Troubleshoot NAT failure** -* Check IP forwarding: - ```bash -sysctl net.ipv4.ip_forward -``` +# Disable IP forwarding to simulate NAT failure +sudo ip netns exec router sysctl -w net.ipv4.ip_forward=0 + +# Test connectivity (should fail) +sudo ip netns exec client1 ping 203.0.113.2 -c 2 + +# Check sysctl settings +sudo ip netns exec router sysctl net.ipv4.ip_forward -* Verify firewall rules. +# Re-enable and verify +sudo ip netns exec router sysctl -w net.ipv4.ip_forward=1 +sudo ip netns exec client1 ping 203.0.113.2 -c 2 +``` --- ### 🔹 VLAN Basics (5 Challenges) -1️⃣ **Create VLANs on a switch (Linux bridge example)** +1️⃣ **Create VLAN interfaces in namespaces** ```bash -sudo ip link add link eth0 name eth0.10 type vlan id 10 -sudo ip addr add 192.168.10.1/24 dev eth0.10 +# Create VLAN interfaces on router +sudo ip netns exec router ip link add link veth1 name veth1.10 type vlan id 10 +sudo ip netns exec router ip link add link veth1 name veth1.20 type vlan id 20 +sudo ip netns exec router ip addr add 192.168.10.1/24 dev veth1.10 +sudo ip netns exec router ip addr add 192.168.20.1/24 dev veth1.20 +sudo ip netns exec router ip link set veth1.10 up +sudo ip netns exec router ip link set veth1.20 up + +# Create VLAN interfaces on client1 (VLAN 10) +sudo ip netns exec client1 ip link add link veth0 name veth0.10 type vlan id 10 +sudo ip netns exec client1 ip addr add 192.168.10.10/24 dev veth0.10 +sudo ip netns exec client1 ip link set veth0.10 up + +# Create VLAN interfaces on client2 (VLAN 20) +sudo ip netns exec client2 ip link add link veth2 name veth2.20 type vlan id 20 +sudo ip netns exec client2 ip addr add 192.168.20.10/24 dev veth2.20 +sudo ip netns exec client2 ip link set veth2.20 up ``` -2️⃣ **Inter-VLAN routing (Router-on-a-stick)** +2️⃣ **Configure inter-VLAN routing** ```bash -# Router with sub-interfaces -ip addr add 192.168.10.1/24 dev eth0.10 -ip addr add 192.168.20.1/24 dev eth0.20 +# Enable IP forwarding on router +sudo ip netns exec router sysctl -w net.ipv4.ip_forward=1 + +# Test inter-VLAN connectivity +sudo ip netns exec client1 ping 192.168.20.10 -c 3 +sudo ip netns exec client2 ping 192.168.10.10 -c 3 ``` 3️⃣ **Misconfigure VLAN and troubleshoot** -👉 Put two devices in different VLANs, they won't talk. -Fix: put them in the same VLAN or configure routing. -4️⃣ **Explore VLAN tagging (802.1Q)** +```bash +# Put client2 in wrong VLAN +sudo ip netns exec client2 ip link del veth2.20 +sudo ip netns exec client2 ip link add link veth2 name veth2.30 type vlan id 30 +sudo ip netns exec client2 ip addr add 192.168.30.10/24 dev veth2.30 +sudo ip netns exec client2 ip link set veth2.30 up + +# Test connectivity (should fail) +sudo ip netns exec client1 ping 192.168.30.10 -c 2 + +# Troubleshoot +sudo ip netns exec client1 ip addr show +sudo ip netns exec client2 ip addr show +sudo ip netns exec router ip addr show + +# Fix the configuration +sudo ip netns exec client2 ip link del veth2.30 +sudo ip netns exec client2 ip link add link veth2 name veth2.20 type vlan id 20 +sudo ip netns exec client2 ip addr add 192.168.20.10/24 dev veth2.20 +sudo ip netns exec client2 ip link set veth2.20 up +``` + +4️⃣ **Explore VLAN tagging** ```bash -sudo tcpdump -e -i eth0 vlan +# Capture VLAN tagged packets +sudo ip netns exec router timeout 10 tcpdump -i veth1 -e vlan & + +# Generate VLAN traffic +sudo ip netns exec client1 ping 192.168.20.10 -c 2 +``` + +5️⃣ **Document multi-VLAN setup** + +```bash +# Show VLAN configuration +echo "VLAN 10: 192.168.10.0/24 (Client1)" +echo "VLAN 20: 192.168.20.0/24 (Client2)" +echo "Router interfaces:" +sudo ip netns exec router ip addr show | grep -E "(veth1.10|veth1.20)" +echo "Client interfaces:" +sudo ip netns exec client1 ip addr show veth0.10 +sudo ip netns exec client2 ip addr show veth2.20 ``` --- ## ✅ Deliverables -* Write solutions in `solution.md` with: - - * Commands run - * Observations - * Screenshots/diagrams -* Push to your GitHub repo & share link. -* Post your experience with hashtags: +* Document your solutions in `solution.md` with: + * All commands executed + * Output observations and screenshots + * Network diagrams of your namespace setup +* Push to your GitHub repository +* Share your experience with hashtags: **#getfitwithsagar #SRELife #DevOpsForAll** --- @@ -398,4 +527,4 @@ sudo tcpdump -e -i eth0 vlan --- 🔥 Keep routing, keep switching, and happy exploring! -— *Sagar Utekar* \ No newline at end of file +— *Sagar Utekar* From 208502af03bbf85be5b6886c8bc365aa51426015 Mon Sep 17 00:00:00 2001 From: Swayam Prakash Bhuyan Date: Thu, 28 Aug 2025 15:30:51 +0530 Subject: [PATCH 105/133] spelling mistakes changed --- DailyChallenges/Season2/Day17/challenge.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/DailyChallenges/Season2/Day17/challenge.md b/DailyChallenges/Season2/Day17/challenge.md index e093c2e..9303106 100644 --- a/DailyChallenges/Season2/Day17/challenge.md +++ b/DailyChallenges/Season2/Day17/challenge.md @@ -4,7 +4,7 @@ ## 🌟 Introduction -Welcome to **Day 3** of the Daily DevOps + SRE Challenge Series – Season 2! +Welcome to **Day 17** of the Daily DevOps + SRE Challenge Series – Season 2! Today we dive into the **Routing & Switching basics** every DevOps/SRE must master: **Static Routing, Default Gateways, NAT/PAT, and VLANs.** These are the invisible highways of networking that ensure packets get from point A to point B securely, efficiently, and reliably. From 0ebc7ada760fbb2f10b147480a525cc1e7d736a5 Mon Sep 17 00:00:00 2001 From: Swayam-Prakash-Bhuyan Date: Sun, 31 Aug 2025 14:38:39 +0530 Subject: [PATCH 106/133] challenge 18 added --- DailyChallenges/Season2/Day18/challenge.md | 204 +++++++++++++++++++++ 1 file changed, 204 insertions(+) create mode 100644 DailyChallenges/Season2/Day18/challenge.md diff --git a/DailyChallenges/Season2/Day18/challenge.md b/DailyChallenges/Season2/Day18/challenge.md new file mode 100644 index 0000000..2e85d07 --- /dev/null +++ b/DailyChallenges/Season2/Day18/challenge.md @@ -0,0 +1,204 @@ +Of course. Here is the **Day 18 – Firewalls & Security** README, incorporating your requests precisely. I have added a detailed **Lab Setup** section and fixed the Mermaid diagram for the TLS handshake to be technically accurate, without altering any of the original content you provided. + +--- + +# **Day 18 – Firewalls & Security 🔥** + +--- + +## 📌 Introduction + +Hello Learners, + +Welcome to **Day 18 of the Daily DevOps + SRE Challenge Series – Season 2!** 🚀 + +Today, we’re diving into one of the **pillars of system and network security** — **firewalls, security groups, and TLS/SSL**. You’ll understand how Linux firewalls protect your servers, how cloud platforms enforce access policies, and how encryption safeguards your data in transit. + +This challenge will sharpen your **network defense skills** so you can prevent intrusions, lock down access, and troubleshoot connectivity issues like a battle-hardened SRE. + +--- + +## 🛠️ Lab Setup Instructions + +To complete the hands-on challenges, you will need an environment to work in. Choose one of the following setups: + +**Option A: Local Virtual Machine (Recommended)** +1. **Software:** Install VirtualBox or VMware. +2. **OS Image:** Download a Linux ISO (e.g., Ubuntu Server 22.04 LTS). +3. **Setup:** + * Create a new VM with at least 1 CPU, 2GB RAM, and 20GB disk space. + * Install the OS. During installation, ensure you install the `openssh-server` package. + * Once installed, boot the VM and note its IP address (`ip a`). + * You can now SSH into your VM from your host machine for practice. + +**Option B: Cloud Instance (e.g., AWS EC2, DigitalOcean Droplet)** +1. **Create an Account:** Sign up for a cloud provider that offers a free tier (AWS, GCP, Azure, DigitalOcean). +2. **Launch a Instance:** Create a new virtual machine instance (e.g., an AWS EC2 t2.micro instance with Ubuntu 22.04). +3. **Security Groups:** During setup, note the Security Group settings. For now, allow SSH (port 22) from your IP and All ICMP - IPv4 (for ping). +4. **Access:** Use the provided key pair to SSH into your instance. + +**Post-Setup (For both options):** +* Update your package lists: `sudo apt update && sudo apt upgrade -y` +* Install necessary tools: `sudo apt install -y iptables-persistent nginx tcpdump` (Confirm 'Yes' to save current IPv4 rules when prompted). + +--- + +## 🚀 Why it matters + +* **Security First:** Firewalls and TLS form the **first line of defense** against attacks. +* **Reliability:** A single wrong iptables rule can cut off production traffic — knowing how to **debug and fix quickly** is essential. +* **Cloud-native DevOps:** Security groups are a must-know for AWS, GCP, Azure engineers. +* **Compliance:** Most orgs require **encryption in transit (TLS/SSL)** and restricted access. +* **Interview Edge:** Expect “What’s the difference between iptables and security groups?” or “How does TLS handshake work?” in SRE/DevOps interviews. + +--- + +## 🔥 Real-world save stories + +* An SRE accidentally blocked SSH for the entire engineering team with a wrong `iptables -F` — recovery was only possible because a **cloud console out-of-band access** was enabled. +* A fintech startup avoided **man-in-the-middle attacks** by moving from HTTP to **TLS with Let’s Encrypt**, cutting down risks overnight. +* At scale, switching from **stateless packet filtering to stateful inspection** reduced false positives and prevented accidental drops of long-lived connections. + +--- + +## 📘 Theory with Mermaid diagrams + +### 🔹 Firewall Workflow + +```mermaid +flowchart TD + Client[Client Request] --> FW[Firewall Rules] + FW -->|Allowed| Server[Application Server] + FW -->|Blocked| Drop[Packet Dropped 🚫] +``` + +--- + +### 🔹 Stateful vs Stateless Filtering + +```mermaid +sequenceDiagram + participant Client + participant Firewall + participant Server + + Client->>Firewall: TCP SYN (New Connection) + Firewall->>Server: Allow (if rule matches) + Server->>Firewall: TCP SYN-ACK + Firewall->>Client: Allow (stateful remembers connection) +``` + +--- + +### 🔹 TLS Handshake Simplified (Corrected) +*Fixed the sequence to accurately reflect the key exchange and cipher setup.* + +```mermaid +sequenceDiagram + participant C as Client + participant S as Server + + Note over C, S: TLS Handshake + + C->>S: ClientHello (Supported Ciphers, Client Random) + S->>C: ServerHello (Selected Cipher, Server Random)
Server Certificate + Note right of C: Client validates certificate + C->>S: PreMaster Secret
(encrypted with Server's public key) + Note over S, C: Both sides generate
Session Keys from
ClientRandom, ServerRandom,
and PreMaster Secret + C->>S: ChangeCipherSpec (Finished) + S->>C: ChangeCipherSpec (Finished) + + Note over C, S: Secure Session Established 🔐
Encrypted Application Data +``` + +--- + +## ⚡ Hands-on challenges (with commands + troubleshooting) + +### **1. Packet Filtering vs Stateful Inspection** + +1. Explain difference between **stateless vs stateful firewalls** in your own words. +2. Block all ICMP traffic using `sudo iptables -A INPUT -p icmp -j DROP` and test with `ping`. +3. Allow only **established SSH connections** with: + ```bash + sudo iptables -A INPUT -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT + ``` +4. Use `sudo tcpdump -i any icmp` or `sudo tcpdump -i any port 22` to observe packets and analyze behavior. +5. Document **pros/cons** of stateless vs stateful filtering. + +--- + +### **2. iptables / nftables Basics** + +1. Flush existing rules (be careful!): `sudo iptables -F` +2. Set default policy to ACCEPT for now: `sudo iptables -P INPUT ACCEPT` +3. Now, set a default DROP policy and allow only SSH (22) and HTTP (80): + ```bash + sudo iptables -P INPUT DROP + sudo iptables -A INPUT -p tcp -m multiport --dports 22,80 -j ACCEPT + sudo iptables -A INPUT -j DROP + ``` +4. Save and persist rules with `sudo iptables-save | sudo tee /etc/iptables/rules.v4`. +5. Convert the same config into **nftables syntax** (research or use `iptables-translate`). +6. Flush all rules using `sudo iptables -F` → observe what breaks (troubleshooting!). +7. Compare performance and future-readiness of iptables vs nftables. + +--- + +### **3. Security Groups (Cloud)** + +1. Create a **Security Group** that allows SSH only from your IP. +2. Block all **outbound traffic** → test how apps fail when they need outbound internet. +3. Allow HTTP/HTTPS only → test with `curl http://` and `curl https://`. +4. Explain how **Security Groups differ from host firewalls**. +5. Design a **3-tier architecture Security Group model**: + * Web → App → DB, with only necessary ports open. + +--- + +### **4. TLS/SSL Basics** + +1. Generate a **self-signed certificate** with: + ```bash + sudo openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout /etc/ssl/private/nginx-selfsigned.key -out /etc/ssl/certs/nginx-selfsigned.crt + ``` +2. Configure Nginx to serve HTTPS using the cert. Edit `/etc/nginx/sites-available/default`: + ```nginx + server { + listen 443 ssl; + ssl_certificate /etc/ssl/certs/nginx-selfsigned.crt; + ssl_certificate_key /etc/ssl/private/nginx-selfsigned.key; + root /var/www/html; + index index.html index.htm; + } + ``` +3. Restart Nginx: `sudo systemctl restart nginx` +4. Verify cert details with: + ```bash + echo | openssl s_client -connect localhost:443 2>/dev/null | openssl x509 -noout -text + ``` +5. Draw the **TLS Handshake** sequence with a Mermaid diagram. +6. Compare **Let’s Encrypt vs Commercial Certs** – when would you choose each? + +--- + +## ✅ Deliverables + +* Create `solution.md` documenting: + * Your firewall configs (`iptables/nftables`). + * Security group screenshots or rules. + * TLS cert commands + verification outputs. + * Mermaid diagrams. +* Push the file to your GitHub repo. +* Share your learnings on social media with: + **#getfitwithsagar #SRELife #DevOpsForAll** + +--- + +## 🌍 Community links + +* Discord: [Join Here](https://discord.gg/mNDm39qB8t) +* Google Group: [Subscribe Here](https://groups.google.com/forum/#!forum/daily-devops-sre-challenge-series/join) +* YouTube: [Watch Tutorials](https://www.youtube.com/@Sagar.Utekar) + +--- \ No newline at end of file From a0f20aec9e71e6faef6c75def0c4666c5ba218d5 Mon Sep 17 00:00:00 2001 From: Swayam-Prakash-Bhuyan Date: Sun, 31 Aug 2025 14:44:22 +0530 Subject: [PATCH 107/133] theory added challenge 18 --- DailyChallenges/Season2/Day18/challenge.md | 39 ++++++++++++++++++++-- 1 file changed, 37 insertions(+), 2 deletions(-) diff --git a/DailyChallenges/Season2/Day18/challenge.md b/DailyChallenges/Season2/Day18/challenge.md index 2e85d07..1f0b112 100644 --- a/DailyChallenges/Season2/Day18/challenge.md +++ b/DailyChallenges/Season2/Day18/challenge.md @@ -1,4 +1,3 @@ -Of course. Here is the **Day 18 – Firewalls & Security** README, incorporating your requests precisely. I have added a detailed **Lab Setup** section and fixed the Mermaid diagram for the TLS handshake to be technically accurate, without altering any of the original content you provided. --- @@ -72,6 +71,16 @@ flowchart TD FW -->|Blocked| Drop[Packet Dropped 🚫] ``` +**Detailed Explanation:** +A firewall is a network security system that monitors and controls incoming and outgoing network traffic based on predetermined security rules. It acts as a barrier between a trusted internal network and untrusted external networks, like the Internet. + +* **Packet Filtering:** The core function. The firewall inspects each **packet** of data (looking at headers for source/destination IP, port, and protocol) and compares it against a set of **rules**. +* **Rule Evaluation:** Rules are processed in order. A packet is checked against the first rule; if it matches, the specified action (e.g., `ACCEPT`, `DROP`) is taken. If not, it moves to the next rule. If no rules match, a default policy (usually `DROP`) is applied. +* **Actions:** + * **ACCEPT:** Allows the packet to proceed. + * **DROP:** Silently discards the packet as if it never arrived. This is more secure than rejecting. + * **REJECT:** Discards the packet but sends an error message (e.g., a TCP reset packet) back to the sender. This is more polite but reveals the existence of the firewall. + --- ### 🔹 Stateful vs Stateless Filtering @@ -88,6 +97,17 @@ sequenceDiagram Firewall->>Client: Allow (stateful remembers connection) ``` +**Detailed Explanation:** +This diagram highlights the critical difference between stateful and stateless firewalls. + +* **Stateless Firewalls (Packet Filters):** + * **How they work:** They examine each packet in isolation, with no memory of previous packets. A rule like `ACCEPT TCP port 22` would allow *any* packet destined for port 22, whether it's a new connection attempt or a response to a previous one. + * **The Problem:** To allow legitimate responses, you'd have to open ports for all *potential* return traffic, creating a larger attack surface. They are simple and fast but lack context, making them vulnerable to spoofing attacks and less secure. + +* **Stateful Firewalls (The Modern Standard):** + * **How they work:** They maintain a **state table** that tracks the state of all active connections (e.g., `NEW`, `ESTABLISHED`, `RELATED`). This is the "memory" shown in the diagram. + * **The Advantage:** The firewall understands if a packet is starting a new connection or is part of an existing one. A simple rule like `ACCEPT ESTABLISHED,RELATED` traffic can allow return traffic for any outgoing connection, without needing specific rules for every return port. This is more secure and easier to manage. For example, if an internal client connects to a web server, the firewall automatically allows the return packets (SYN-ACK, etc.) back through, even if the default policy is to `DROP` incoming traffic. + --- ### 🔹 TLS Handshake Simplified (Corrected) @@ -111,6 +131,21 @@ sequenceDiagram Note over C, S: Secure Session Established 🔐
Encrypted Application Data ``` +**Detailed Explanation:** +The Transport Layer Security (TLS) handshake is a critical process that enables secure communication over a network by authenticating the server (and optionally the client) and establishing a shared secret key for symmetric encryption. + +1. **`ClientHello`:** The client initiates the handshake by sending a message containing the TLS versions it supports, a list of suggested cipher suites (cryptographic algorithms), and a random string of bytes (`Client Random`). + +2. **`ServerHello` & Certificate:** The server responds with its chosen TLS version and cipher suite from the client's list. It also sends its own `Server Random` and its **SSL Certificate**. This certificate contains the server's public key and is signed by a trusted Certificate Authority (CA), allowing the client to verify the server's identity. + +3. **Key Exchange (PreMaster Secret):** The client verifies the server's certificate. If trusted, it generates a `PreMaster Secret`, encrypts it with the server's public key (from the certificate), and sends it back. **Only the server possessing the corresponding private key can decrypt this secret.** + +4. **Session Keys Generated:** Both the client and server now independently generate the same **session keys** (symmetric keys for encryption/decryption) using the `Client Random`, `Server Random`, and `PreMaster Secret`. The PreMaster Secret is immediately discarded on both sides. This ensures **Forward Secrecy**. + +5. **`ChangeCipherSpec` & `Finished`:** Both parties send a `ChangeCipherSpec` message to signal that subsequent messages will be encrypted with the newly established session keys. A `Finished` message is sent encrypted to verify that the handshake was successful and the keys are working correctly. + +6. **Secure Communication:** The handshake is complete. All subsequent application data (HTTP, etc.) is encrypted and authenticated using the symmetric session keys, ensuring **confidentiality** and **integrity**. + --- ## ⚡ Hands-on challenges (with commands + troubleshooting) @@ -201,4 +236,4 @@ sequenceDiagram * Google Group: [Subscribe Here](https://groups.google.com/forum/#!forum/daily-devops-sre-challenge-series/join) * YouTube: [Watch Tutorials](https://www.youtube.com/@Sagar.Utekar) ---- \ No newline at end of file +--- From 7a387fb3478517a39e09301b9f00770e1aabbb58 Mon Sep 17 00:00:00 2001 From: Sagar Utekar Date: Fri, 5 Sep 2025 06:23:45 +0530 Subject: [PATCH 108/133] Update challenge.md --- DailyChallenges/Season2/Day15/challenge.md | 912 +++------------------ 1 file changed, 103 insertions(+), 809 deletions(-) diff --git a/DailyChallenges/Season2/Day15/challenge.md b/DailyChallenges/Season2/Day15/challenge.md index b7e3fb2..82ba764 100644 --- a/DailyChallenges/Season2/Day15/challenge.md +++ b/DailyChallenges/Season2/Day15/challenge.md @@ -1,860 +1,154 @@ -# Day 15 — Linux Networking Challenges (Daily DevOps + SRE Challenge Series — Season 2) +# Daily DevOps + SRE Challenge Series – Season 2 +## Day 15: OSI & TCP/IP Model Foundations for DevOps/SRE ---- - -## Introduction - -Welcome to Day 15 of the Daily DevOps + SRE Challenge Series – Season 2! 🎉 - -Today, you'll dive into essential **Linux networking skills** through practical, story-based scenarios. You'll configure IP addresses, set up namespaces, simulate routing, and debug traffic issues—all on your local machine without any complex setup. By the end, you'll: - -* Understand and troubleshoot the **OSI & TCP/IP layers** in real-world failures. -* Configure **IPv4/IPv6 addressing**, resolve subnetting conflicts, and simulate enterprise VPC designs. -* Master **ARP behavior**, from cache flushing to spoofing attacks. -* Gain hands-on skills to debug outages with tools like `iptables`, `ip`, and `tcpdump`. - ---- - -## Why This Matters? - -Networking is the backbone of everything—cloud apps, Kubernetes clusters, even CI/CD pipelines. If you can't debug connectivity, you'll get stuck in outages fast. - -* **Real-World Edge:** Most outages aren't due to servers crashing—they're due to misconfigured routes, DNS failures, or firewall issues. -* **SRE Superpower:** Knowing the **why ping works but app fails** scenario makes you 10x faster in production war rooms. -* **Cloud Readiness:** VPC, subnets, and dual-stack IPv4/IPv6 configs are daily tasks for AWS/GCP/Azure engineers. -* **Interview Gold:** "Why can two hosts ping but not load HTTP?" or "How do you debug duplicate IP conflicts?"—classic SRE questions. -* **Security Awareness:** ARP spoofing and wrong subnet masks are real threats in production, not just lab theory. - ---- - -## Real-World Save 🌍 - -At a fintech startup, payments randomly failed for EU customers. Engineers could **ping servers but HTTP calls kept timing out**. Debugging revealed a firewall rule blocking port 443 on just one subnet. Fixing that restored global transactions. Knowing **layer-by-layer troubleshooting** saved millions in lost revenue. - ---- - -# Theory (detailed) — **do not skip** - -> Below: full theory for the four topics you asked for. Read fully before doing labs. - ---- - -## OSI & TCP/IP Models — Introduction - -**What is it?** -OSI (Open Systems Interconnection) is a conceptual 7-layer model that describes how data moves through networked systems. TCP/IP model is a simpler practical model used in the Internet (4 layers). These help you think logically about where failures happen. - -**Why learn it?** -Because when something breaks (web app not loading, DNS failing, packets dropping), you must ask: *which layer broke?* — physical, link (MAC), IP (routing), TCP/UDP (ports), or the app (HTTP/DNS). This narrows down troubleshooting steps. - ---- - -### OSI layers (simple, one-line each) - -1. **Physical (L1)** — cables, radio waves, NIC hardware. -2. **Data Link (L2)** — frames, MAC addresses, switches, ARP. -3. **Network (L3)** — IP addressing and routing. -4. **Transport (L4)** — TCP/UDP, ports, connection reliability. -5. **Session (L5)** — sessions, dialogs (rarely directly debugged). -6. **Presentation (L6)** — encoding/encryption (TLS sits around here). -7. **Application (L7)** — HTTP, DNS, SSH—the services users interact with. - -**TCP/IP Stack mapping (practical):** - -* Link (L1–L2) — ARP, Ethernet -* Internet (L3) — IP, routing -* Transport (L4) — TCP, UDP -* Application (L7) — HTTP, DNS, SSH - ---- - -### Key concepts & simple analogies - -* **IP = house number** — tells you *where* to go. -* **MAC = person's face** — used to deliver inside the same building (LAN). -* **Ports (TCP/UDP) = apartment door numbers** — same house, many services. -* **Firewall = security guard at door** — can block specific ports/protocols. - ---- - -### Typical tools & what they tell you - -* `ping` (ICMP) → L3 reachability test. -* `arp` / `ip neigh` → L2 IP→MAC mapping. -* `ip addr`, `ip route` → addresses & routes (L3). -* `ss -tuln` / `nc` → ports and services (L4). -* `tcpdump` → raw packet capture (L2–L4). -* `iptables` or `nft` → firewall rules (L4-L3 policies). -* `traceroute` → shows path (L3/hops), but depends on ICMP/TCP/UDP responses. - ---- - -### Troubleshooting mental model (simple process) - -1. **Is the NIC up?** (`ip link`) — Physical/L1 -2. **Can I reach IP?** (`ping`) — Network/L3 -3. **Is the service listening?** (`ss -ltnp`, `nc`) — Transport/L4 & App/L7 -4. **Is DNS resolving?** (`dig`, `nslookup`) — App/L7 -5. **Capture packets** (`tcpdump`) — inspect headers (MAC, IP, TCP flags). -6. **Check firewall rules** (`iptables -L`, `nft list ruleset`). - ---- - -### Mermaid diagrams (OSI / TCP-IP overview) - -OSI stack: - -```mermaid -graph TB - A[Application HTTP/DNS/SSH] --> B[Transport TCP/UDP] - B --> C[Network IP/Routing] - C --> D[Data Link Ethernet/ARP] - D --> E[Physical Cable/Wireless] -``` - -Packet debugging flow: - -```mermaid -flowchart LR - UserAction[User opens browser] --> App[Application: HTTP request] - App --> Transport[TCP handshake/port] - Transport --> Network[IP routing/next-hop] - Network --> Link[ARP/MAC resolution] - Link --> Physical[Wire/WiFi] -``` - ---- - -## IP Addressing (IPv4 & IPv6) — Introduction - -**What is it?** -IP addresses uniquely identify hosts on a network. IPv4 is 32-bit (e.g., `192.168.1.10`), IPv6 is 128-bit (e.g., `2001:db8::1`). - -**Why important?** -If you don't know addressing basics, you'll misconfigure masks, gateways, or try to reach the wrong network. IP addressing also affects routing, ACLs, NAT — everything SREs care about. - ---- - -### IPv4 basics - -* **Format:** dotted decimal `a.b.c.d` (32 bits). -* **CIDR prefix:** `/24` means top 24 bits are network — addresses inside are on same subnet. -* **Private ranges:** - - * `10.0.0.0/8` - * `172.16.0.0/12` - * `192.168.0.0/16` -* **Special addresses:** `0.0.0.0` (this host), `127.0.0.1` (loopback), network & broadcast addresses. - -**How to compute host counts:** - -* `/24` → 256 addresses (254 hosts usable). -* `/26` → 64 addresses (62 hosts usable). - ---- - -### IPv6 basics - -* **Format:** hex groups `2001:db8::1` (128 bits). -* **Scopes:** Link-local (`fe80::/10`), global unicast (public), loopback (`::1`). -* **Advantages:** large address space, simpler auto-configuration (SLAAC), better routing aggregation. - ---- - -### Subnet mask & network boundary (simple) - -* Mask tells whether destination IP is on your local network or needs to be sent to a router. -* Example: `192.168.1.10/24` and `192.168.1.20/24` are local; `192.168.2.1` is remote. - ---- - -### Useful commands - -* Show addresses: `ip addr show` -* Show routes: `ip route show` -* Add address: `ip addr add 10.0.0.2/24 dev eth0` -* IPv6 show: `ip -6 addr show` -* Disable IPv4 (namespace): `sysctl -w net.ipv4.conf.all.disable_ipv4=1` - ---- - -### Routing decision (simple) - -```mermaid -flowchart LR - App[App creates packet] --> IPPkt[IP packet with dest IP] - IPPkt --> Router{Is dest in local subnet?} - Router -->|Yes| ARP[Do ARP resolve MAC] - ARP --> Send[Send frame on LAN] - Router -->|No| GW[Send to default gateway] -``` - ---- - -## Subnetting & CIDR — Introduction - -**What is it?** -CIDR (Classless Inter-Domain Routing) describes networks by prefix length (`/24`, `/26`) and enables flexible splitting of IP space. Subnetting divides a big network into smaller networks. - -**Why learn it?** -To allocate IPs to teams, limit broadcast domains, design VPCs, and avoid overlaps when merging networks. - ---- - -### How CIDR works (simple) - -* `10.0.0.0/24` → addresses `10.0.0.0`–`10.0.0.255`. -* Splitting `10.0.0.0/24` into `/26` yields: - - * `10.0.0.0/26` (0–63) - * `10.0.0.64/26` (64–127) - * `10.0.0.128/26` (128–191) - * `10.0.0.192/26` (192–255) - ---- - -### Common tasks - -* **Divide network** for teams → use /26, /27 etc. -* **Plan VPC** blocks to avoid future overlap. -* **Detect overlaps** before merging nets. - ---- - -### Overlap problems (simple) - -* If Net A thinks IP is local (on-link) and Net B thinks same IP is remote, packets can be dropped or blackholed. The fix: **renumber** or **NAT** at boundary. - ---- - -### Subnet split - -```mermaid -graph TD - A[10.0.0.0/24] --> B[10.0.0.0/26] - A --> C[10.0.0.64/26] - A --> D[10.0.0.128/26] - A --> E[10.0.0.192/26] -``` - ---- - -## ARP (Address Resolution Protocol) — Introduction - -**What is it?** -ARP maps IPv4 addresses to MAC addresses inside a LAN. It's how IP packets get delivered to the correct NIC inside the same broadcast domain. - -**Why ARP matters?** -If ARP is wrong or spoofed, your traffic can go to the wrong machine, be dropped, or be intercepted. - ---- - -### ARP normal flow (simple) - -1. Host A wants to send to IP X. -2. If it has no MAC for X in ARP cache, it sends **ARP Who-has X tell A** (broadcast). -3. The owner replies **ARP is-at MAC**. -4. Host A caches and sends frames to that MAC. - ---- - -### ARP cache and states - -* Entries expire — kernel keeps freshness flags (`REACHABLE`, `STALE`, etc.). -* Commands: `ip neigh show`, `arp -n`, `arping -I ` - ---- - -### ARP security problems - -* **Duplicate IP:** Two hosts claim same IP → flapping / inconsistent replies. -* **ARP spoofing/poisoning:** Attacker sends fake ARP replies linking victim/gateway IP to attacker's MAC → MITM. -* **Mitigations:** static ARP entries (for critical hosts), switch DAI, encryption (TLS). - ---- - -### ARP sequence - -```mermaid -sequenceDiagram - participant A as Host A - participant Net as Broadcast - participant B as Host B - - A->>Net: ARP Who-has 192.168.1.20? Tell 192.168.1.10 - B-->>Net: ARP is-at 11:22:33:44:55:66 - Net-->>A: ARP reply - A->>B: Frame delivered to MAC 11:22:33:44:55:66 -``` - ---- - -## Additional Theory Concepts - -### DNS (Domain Name System) — Introduction - -**What is it?** -DNS translates human-readable domain names (like google.com) to IP addresses. It uses a hierarchical, distributed database. - -**Why learn it?** -Because most network connections start with a DNS lookup. If DNS fails, your application fails—even if the target server is reachable by IP. - ---- - -### How DNS works (simple) - -1. Application requests resolution (e.g., browser asks for google.com). -2. The request is sent to a DNS resolver (usually provided by ISP or public ones like 8.8.8.8). -3. The resolver queries the root servers, then TLD servers, then authoritative servers for the domain. -4. The IP address is returned and cached. - ---- - -### Key concepts - -- **Record types:** A (IPv4), AAAA (IPv6), CNAME (alias), MX (mail), etc. -- **Port:** UDP 53 (queries), TCP 53 (large responses or zone transfers). -- **Tools:** `dig`, `nslookup`, `host`. - ---- - -### Common issues - -- Misconfigured `/etc/resolv.conf` -- Firewall blocking UDP 53 -- Slow DNS responses (high latency) - ---- - -### DNS lookup - -```mermaid -sequenceDiagram - participant C as Client - participant R as Resolver - participant A as Authoritative Server - - C->>R: Query for google.com - R->>A: Query - A-->>R: Response (IP) - R-->>C: Response (IP) -``` - ---- - -### MTU (Maximum Transmission Unit) — Introduction - -**What is it?** -MTU is the maximum size of a packet that can be transmitted without fragmentation over a network interface. - -**Why learn it?** -If packets are too large for a link, they get fragmented (which adds overhead) or dropped (which breaks connectivity). This is common in VPNs and tunnels. - ---- - -### How MTU works (simple) - -- Each link has an MTU (e.g., Ethernet is 1500 bytes). -- If a packet is larger than the MTU, it is fragmented into smaller packets (if allowed) or an ICMP "need to fragment" error is sent. -- **Path MTU Discovery (PMTUD)** is the process of determining the smallest MTU along the path to avoid fragmentation. - ---- - -### Common issues - -- MTU mismatch between endpoints or intermediate links causes black holes for large packets. -- VPNs reduce effective MTU due to encapsulation overhead. - ---- - -### PMTUD - -```mermaid -sequenceDiagram - participant S as Sender - participant R as Router - participant D as Dest - - S->>D: Large packet (1500 bytes) - R->>S: ICMP Frag Needed (MTU=1400) - S->>D: Packet split to 1400 - D-->>S: ACK -``` - ---- - -### Routing (Multiple NICs, Policy Routing) — Introduction - -**What is it?** -Routing decides which interface to use to send a packet. Policy routing allows routing decisions based on more than just the destination IP (e.g., source IP, protocol). - -**Why learn it?** -Multi-homed hosts (multiple NICs) may have complex routing requirements. Wrong routing can cause asymmetric paths or failures. - ---- - -### How routing works (simple) - -- The kernel uses the routing table to decide the next hop. -- Default route (0.0.0.0/0) is used when no specific route matches. -- **Metric** is used to choose between multiple routes to the same destination (lower is better). -- **Policy routing** uses multiple routing tables selected by rules (e.g., from a certain source IP, use a different table). - ---- - -### Common commands - -- `ip route show` -- `ip rule list` (for policy routing) -- `ip route add ... metric ...` -- `ip rule add from table
` - ---- - -### Routing decision - -```mermaid -flowchart LR - P[Packet] --> R[Routing Table] - R -->|match| I[Interface] - R -->|no match| D[Default Gateway] -``` - ---- ---- ---- ---- - -## Challenges - -🔹 **1. OSI & TCP/IP Models** - -**Ping works, but HTTP fails** -Run python3 -m http.server 8080 in one namespace/VM. -From another, ping works but block TCP port 8080 using iptables. -👉 Mimics: "Why I can ping the server but the website isn't loading?" - -**Solution:** - -```bash -# Setup -ip netns add srv; ip netns add cli -ip link add veth-s type veth peer name veth-c -ip link set veth-s netns srv; ip link set veth-c netns cli -ip netns exec srv ip addr add 10.0.0.1/24 dev veth-s -ip netns exec cli ip addr add 10.0.0.2/24 dev veth-c -ip netns exec srv ip link set veth-s up; ip netns exec cli ip link set veth-c up -ip netns exec srv python3 -m http.server 8080 & - -# Test -ip netns exec cli ping -c2 10.0.0.1 # ✅ works -ip netns exec cli curl -m3 10.0.0.1:8080 # ✅ works - -# Break (block TCP/8080) -ip netns exec srv iptables -A INPUT -p tcp --dport 8080 -j DROP -ip netns exec cli curl -m3 10.0.0.1:8080 # ❌ timeout - -# Interpretation: L3 OK; L4/L7 blocked by firewall. -# Fix -ip netns exec srv iptables -D INPUT -p tcp --dport 8080 -j DROP -``` - -**Flow diagram (packet journey → break → fix):** - -```mermaid -flowchart LR - C[Client] -->|ICMP Echo Request| S[Server] - S -->|ICMP Echo Reply| C - C -->|TCP SYN :8080| S - S -->|iptables DROP| X[Dropped] - S -->|Fix: iptables -D| S2[Server Fixed] - C -->|TCP SYN :8080| S2 - S2 -->|TCP SYN/ACK| C - C -->|HTTP GET| S2 - S2 -->|HTTP 200| C -``` - -*Note:* ping uses ICMP (L3). curl uses TCP (L4) + HTTP (L7). iptables blocked TCP/8080 — that's why ping succeeded but web failed. +### Introduction +Welcome to Day 15 of the Daily DevOps + SRE Challenge Series – Season 2! ---- - -**DNS Layer Check** -Disable /etc/resolv.conf or block UDP 53. -Test browsing (curl google.com fails, but curl 142.250.182.14 works). -👉 Mimics: "App fails because DNS is misconfigured." - -**Solution:** - -```bash -# Setup one namespace as client -ip netns add ns1; ip netns exec ns1 ip link set lo up - -# Break DNS safely (in namespace only) -ip netns exec ns1 iptables -A OUTPUT -p udp --dport 53 -j DROP - -# Test -ip netns exec ns1 curl -m3 http://google.com # ❌ fails (could not resolve) -ip netns exec ns1 curl -m3 http://142.250.182.14 # ✅ works (direct IP) - -# Interpretation: IP path fine; DNS resolution broken. -# Fix -ip netns exec ns1 iptables -D OUTPUT -p udp --dport 53 -j DROP -``` - -**Flow diagram:** +Today marks the start of your Networking for DevOps journey. Before diving into commands and configs, you'll build a strong mental model of how networking actually works—layer by layer. Understanding the OSI and TCP/IP models will make you a faster troubleshooter, a better cloud/network engineer, and help you communicate clearly with your team. -```mermaid -flowchart LR - C[Client] -->|UDP DNS Query :53| DNS[Resolver] - DNS -->|blocked| X[Dropped] - C -->|HTTP to numeric IP| S[Remote Server] - S -->|HTTP 200| C -``` +By the end of today, you'll be able to: +- Draw and describe both the OSI and TCP/IP models +- Map common protocols and tools (ping, SSH, HTTP, DNS, etc.) to the right layer +- Explain where most real-world DevOps/SRE problems occur +- Use this model to guide troubleshooting (“where is it broken?”) -*Note:* block DNS (UDP/53) prevents name resolution; direct IP works. +Why this matters: +- Troubleshooting: Pinpoint failures (e.g., “Is it a firewall or DNS issue?”) +- Communication: Clearly explain problems to network, security, and cloud teams +- Design: Choose the right tool or protocol for any scenario +- Interviews: “At what layer does X work?” is a common question --- -**MTU Mismatch** -Set MTU on one interface to 1400 and the other to 900. -Try large file transfer with scp. -👉 Mimics: "Packets drop due to MTU mismatch in VPN tunnels." - -**Solution:** - -```bash -# Setup -ip netns add a; ip netns add b -ip link add va type veth peer name vb -ip link set va netns a; ip link set vb netns b -ip netns exec a ip addr add 10.1.1.1/24 dev va -ip netns exec b ip addr add 10.1.1.2/24 dev vb -ip netns exec a ip link set va up mtu 1400 -ip netns exec b ip link set vb up mtu 900 - -# Test small vs large -ip netns exec a ping -c2 10.1.1.2 # ✅ often works -ip netns exec a ping -M do -s 1200 10.1.1.2 # ❌ fails due to MTU - -# (Optional TCP test) -ip netns exec b nc -l -p 9000 >/dev/null & -head -c 200000 |Large IP packet| Link[Link mtu=900] - Link -->|cannot forward| Router[drop or ICMP] - Router -->|ICMP frag needed| C - C -->|Fix: align MTU| Fixed[Link mtu=1400] -``` - ---- - -**Layer 2 vs Layer 3 Failure** -Assign two machines same subnet but no switch/bridge between them. -👉 Mimics: "Why hosts on same subnet can't reach each other?" - -**Solution:** - -```bash -# Setup (two separate links with no shared L2 domain) -ip netns add n1; ip netns add n2 -ip link add v1 type veth peer name v1p -ip link add v2 type veth peer name v2p -ip link set v1 netns n1; ip link set v2 netns n2 -ip netns exec n1 ip addr add 192.168.50.1/24 dev v1 -ip netns exec n2 ip addr add 192.168.50.2/24 dev v2 -ip netns exec n1 ip link set v1 up; ip netns exec n2 ip link set v2 up - -# Test -ip netns exec n1 ping -c2 192.168.50.2 # ❌ fails (no L2 connectivity) - -# Interpretation: Same subnet needs shared L2 (bridge/switch). -# Fix: create bridge and attach peers (host side v1p & v2p) -ip link add br0 type bridge; ip link set br0 up -ip link set v1p master br0; ip link set v2p master br0 -ip netns exec n1 ping -c2 192.168.50.2 # ✅ works now -``` +#### TCP/IP Model (4 Layers) +| Layer | OSI Layer(s) | Example Protocols/Tools | +|--------------|--------------------------|------------------------------| +| Application | 7, 6, 5 | HTTP, SSH, DNS, TLS, curl | +| Transport | 4 | TCP, UDP, nc, ss | +| Internet | 3 | IP, ICMP, ping, traceroute | +| Network Access| 2, 1 | Ethernet, ARP, ip link, arp | -**Flow diagram:** - -```mermaid -flowchart LR - A[Host A] -->|ARP Who-has| None[No shared L2 domain] - None -->|No path| X[Dropped] - B[Host B] - A -->|Fix: add bridge| BR[Bridge] - BR --> B -``` +**Quick mapping:** +- “Can’t SSH/ping/curl?” → Which layer fails? +- “Is DNS broken?” → Application +- “Is port blocked?” → Transport +- “Wrong IP/gateway?” → Network/Internet +- “Cable unplugged?” → Physical/Data Link --- -**Layer-wise Debugging** -Capture traffic with tcpdump. Try ICMP, TCP, UDP one by one. -👉 Helps visualize OSI/TCP-IP model in action. - -**Solution:** - -```bash -# Setup two namespaces -ip netns add c1; ip netns add c2 -ip link add t1 type veth peer name t2 -ip link set t1 netns c1; ip link set t2 netns c2 -ip netns exec c1 ip addr add 10.9.0.1/24 dev t1 -ip netns exec c2 ip addr add 10.9.0.2/24 dev t2 -ip netns exec c1 ip link set t1 up; ip netns exec c2 ip link set t2 up - -# Start capture on c2 -ip netns exec c2 tcpdump -i t2 -n & - -# Generate traffic from c1 -ip netns exec c1 ping -c2 10.9.0.2 # ICMP -ip netns exec c2 python3 -m http.server 8081 & # TCP server -ip netns exec c1 curl -m3 10.9.0.2:8081 # TCP -ip netns exec c1 sh -lc 'echo hi | nc -u 10.9.0.2 9001' # UDP - -# Interpretation: See L2 (eth), L3 (IP), L4 (ICMP/TCP/UDP) in capture. -``` - -**Flow diagram (example capture view):** - -```mermaid -flowchart LR - C[Client] -->|ICMP Echo| S[Server] - S -->|ICMP Reply| C - C -->|TCP SYN:8081| S - S -->|TCP SYN/ACK| C - C -->|UDP packet| S - S -->|no ACK required| C -``` - ---- - -🔹 **2. IP Addressing (IPv4, IPv6)** - -**Duplicate IP Conflict** -Assign 192.168.1.10 to two hosts. Ping from a third host → observe flapping replies. -👉 Mimics: "Two servers misconfigured with same IP." - -**Solution:** - -```bash -# Setup three namespaces on a bridge -ip netns add h1; ip netns add h2; ip netns add h3 -ip link add br0 type bridge; ip link set br0 up - -# Connect h1 -ip link add h1v type veth peer name h1p -ip link set h1v netns h1; ip link set h1p master br0 -ip netns exec h1 ip addr add 192.168.1.10/24 dev h1v; ip netns exec h1 ip link set h1v up - -# Connect h2 -ip link add h2v type veth peer name h2p -ip link set h2v netns h2; ip link set h2p master br0 -ip netns exec h2 ip addr add 192.168.1.10/24 dev h2v; ip netns exec h2 ip link set h2v up - -# Connect h3 -ip link add h3v type veth peer name h3p -ip link set h3v netns h3; ip link set h3p master br0 -ip netns exec h3 ip addr add 192.168.1.30/24 dev h3v; ip netns exec h3 ip link set h3v up - -# Test from h3 -ip netns exec h3 arping -I h3v -c 3 192.168.1.10 # 🔁 multiple MACs/replies (flap) -ip netns exec h3 ping -c4 192.168.1.10 # ❗ inconsistent replies - -# Interpretation: Duplicate IP causing ARP instability. -# Fix: give h2 a unique IP, e.g. -ip netns exec h2 ip addr flush dev h2v -ip netns exec h2 ip addr add 192.168.1.11/24 dev h2v -``` - -**Flow diagram:** +### 2) Visual Diagrams +#### OSI Model Overview ```mermaid -sequenceDiagram - participant H3 as Tester (h3) - participant BR as Bridge - participant H1 as Host1 (192.168.1.10 @ MAC A) - participant H2 as Host2 (192.168.1.10 @ MAC B) - - H3->>BR: ARP Who has 192.168.1.10? - BR->>H1: Who has? - H1-->>BR: I am 192.168.1.10 at MAC A - BR-->>H3: ARP reply MAC A - BR->>H2: Who has? - H2-->>BR: I am 192.168.1.10 at MAC B - BR-->>H3: ARP reply MAC B - Note over H3: Replies flap -> connectivity inconsistent -``` - ---- - -**IPv6 Only Network** -Assign IPv6 addresses only (`2001:db8::1/64`). Disable IPv4 → try reaching services. -👉 Mimics: "Transitioning to IPv6 in enterprise." - -**Solution:** - -```bash -# Setup -ip netns add v6a; ip netns add v6b -ip link add v6aif type veth peer name v6bif -ip link set v6aif netns v6a; ip link set v6bif netns v6b -ip netns exec v6a ip -6 addr add 2001:db8::1/64 dev v6aif -ip netns exec v6b ip -6 addr add 2001:db8::2/64 dev v6bif -ip netns exec v6a ip link set v6aif up; ip netns exec v6b ip link set v6bif up - -# Disable IPv4 on both interfaces -ip netns exec v6a sysctl -w net.ipv4.conf.all.disable_ipv4=1 -ip netns exec v6b sysctl -w net.ipv4.conf.all.disable_ipv4=1 - -# Test IPv6 reachability -ip netns exec v6a ping -6 -c2 2001:db8::2 # ✅ works -# Try reaching an IPv4-only IP from v6a (will fail) -ip netns exec v6a ping -6 -c2 142.250.182.14 # ❌ cannot (v4 only target) - -# Interpretation: IPv6-only cannot reach IPv4-only services. -# Fix: deploy dual-stack or NAT64/DNS64. +flowchart TD + L7[Application :HTTP, SSH, DNS] + L6[Presentation :TLS, SSL, Encoding] + L5[Session :TLS session, NetBIOS] + L4[Transport :TCP, UDP] + L3[Network :IP, ICMP] + L2[Data Link :Ethernet, ARP] + L1[Physical :Cable, Wi-Fi] + L7 --> L6 --> L5 --> L4 --> L3 --> L2 --> L1 ``` -**Flow diagram:** - +#### TCP/IP Model & Protocol Mapping ```mermaid -flowchart LR - A[IPv6-only Host] -->|IPv6 packet| B[IPv6 Server] - B -->|reply| A - A -->|tries IPv4 address| C[IPv4-only Server] - C -->|No route| X[Dropped] +flowchart TD + App[Application: HTTP, SSH, DNS, SMTP, curl] + Trans[Transport: TCP, UDP, nc, ss] + Net[Internet: IP, ICMP, ping, traceroute] + Link[Network Access: Ethernet, ARP, ip link] + App --> Trans --> Net --> Link ``` --- -**Wrong Subnet Mask** -Host A: 192.168.1.10/24, Host B: 192.168.1.20/16. Traffic works one way but not the other. -👉 Mimics: "Subnet mask mismatch during manual config." +### 3) Hands-On Challenge -**Solution:** +#### Scenario A: Draw & Map the Layers -```bash -# Setup -ip netns add A; ip netns add B -ip link add av type veth peer name bv -ip link set av netns A; ip link set bv netns B -ip netns exec A ip addr add 192.168.1.10/24 dev av -ip netns exec B ip addr add 192.168.1.20/16 dev bv -ip netns exec A ip link set av up; ip netns exec B ip link set bv up +- Draw both models (on paper or using a tool) +- List at least 2 protocols and 2 tools for each layer (see tables above) +- Save a photo/image/markdown table in your solution -# Test both ways -ip netns exec A ping -c2 192.168.1.20 # may ❌ (A expects router) -ip netns exec B ping -c2 192.168.1.10 # may ✅ (B thinks on-link) +#### Scenario B: Protocol Mapping Exercise -# Interpretation: Asymmetric routing due to mask mismatch. -# Fix: make both /24 or both /16 consistently. -``` +Given these tools/protocols, map each to its layer(s): +- ping +- ssh +- dig +- curl +- nc +- ip addr +- arp +- openssl +- traceroute +- tcpdump -**Flow diagram:** +Fill out a table like: +| Tool/Protocol | OSI Layer(s) | TCP/IP Layer | Description | +|---------------|--------------|--------------|-------------| +| ping | 3, 1 | Internet | Tests IP reachability using ICMP | +| ssh | 7, 4 | Application, Transport | Secure shell (needs network and transport) | +| ... | ... | ... | ... | -```mermaid -flowchart LR - A[Host A 192.168.1.10/24] -->|Sees 192.168.1.20 as remote| GW[Gateway] - B[Host B 192.168.1.20/16] -->|Sees 192.168.1.10 as local| A - A -->|send via gateway| X[not delivered] - B -->|ARP| A - A -->|reply| B -``` +#### Scenario C: Mini Incident Simulation ---- +A developer says: +> “I can’t reach the app at http://10.10.10.20:5000 from my laptop. Ping fails, but DNS resolves.” -**Multiple NICs Routing** -One host with 192.168.1.10 (LAN) and 10.0.0.10 (WAN). Configure routing → see which NIC is chosen. -👉 Mimics: "Multi-homed servers with wrong default gateway." - -**Solution:** - -```bash -# Setup one namespace with two NICs via two bridges -ip netns add mh -ip link add br1 type bridge; ip link set br1 up -ip link add br2 type bridge; ip link set br2 up - -ip link add mhv1 type veth peer name tap1 -ip link add mhv2 type veth peer name tap2 -ip link set mhv1 netns mh; ip link set mhv2 netns mh -ip link set tap1 master br1; ip link set tap2 master br2 -ip netns exec mh ip addr add 192.168.1.10/24 dev mhv1 -ip netns exec mh ip addr add 10.0.0.10/24 dev mhv2 -ip netns exec mh ip link set mhv1 up; ip netns exec mh ip link set mhv2 up - -# Default route selection -ip netns exec mh ip route add default via 10.0.0.1 dev mhv2 metric 100 -ip netns exec mh ip route add default via 192.168.1.1 dev mhv1 metric 200 -ip netns exec mh ip route show - -# Policy routing example (force source-based egress) -ip netns exec mh ip rule add from 192.168.1.10/32 table 100 -ip netns exec mh ip route add default via 192.168.1.1 dev mhv1 table 100 -ip netns exec mh ip rule list -``` +Your task: +1. Identify which layer(s) might be failing based on symptoms +2. List which commands/tools you would use in order to troubleshoot—from bottom to top -**Flow diagram:** - -```mermaid -flowchart TD - subgraph Multi-homed Host - NIC1[eth0: 192.168.1.10/24] - NIC2[eth1: 10.0.0.10/24] - end - - NIC1 -->|default route metric 200| GW1[192.168.1.1] - NIC2 -->|default route metric 100| GW2[10.0.0.1] - - GW1 --> Internet - GW2 --> Internet - - Note[Traffic prefers lower metric: uses GW2] -``` - -*Note:* The host will prefer the route with lower metric (100) for most traffic. Policy routing can override this based on source IP. +Document your answer in solution.md. --- -This completes the full networking challenge with comprehensive theory and practical exercises. The additional theory sections cover DNS, MTU, and routing concepts that are essential for complete networking understanding. - ---- - -Deliverables -- A markdown file solution.md with: - - Commands and outputs for Tasks - - 3–5 bullet notes on what you learned +### 4) What to Watch For (Common Gotchas) +- Not every protocol is at only one layer (e.g., DNS = Application, but depends on Transport/Network) +- Some tools work at multiple layers (e.g., tcpdump sees almost everything!) +- Real-world cloud infra may “hide” some layers (e.g., you don’t see cables, but you still check link status in VMs) --- ## Submission Guidelines -- Store your findings and execution steps in solution.md -- Submit it in your GitHub repository and share the link -- Post your experience on social media with #getfitwithsagar #SRELife #DevOpsForAll -- Tag us in your posts for visibility and networking +- Create `solution.md` including: + - Your diagrams/tables for OSI & TCP/IP models + - Completed protocol mapping table + - Your troubleshooting approach for the mini incident + - 3–5 short bullet points on what you learned + +- Push to your GitHub repo and share the link +- Post your experience with #getfitwithsagar #SRELife #DevOpsForAll and tag us --- ## Join Our Community - Discord – Ask questions and collaborate: https://discord.gg/mNDm39qB8t -- Google Group – Get updates and discussions: https://groups.google.com/forum/#!forum/daily-devops-sre-challenge-series/join -- YouTube – Watch solution videos and tutorials: https://www.youtube.com/@Sagar.Utekar +- Google Group – Updates and discussions: https://groups.google.com/forum/#!forum/daily-devops-sre-challenge-series/join +- YouTube – Tutorials: https://www.youtube.com/@Sagar.Utekar --- ## Stay Motivated! -Keep it simple. Master the basics. You’ll use these commands every week in real DevOps/SRE work. +A solid mental model makes every future network problem easier. Build your foundation today! Happy Learning! Best regards, -Sagar Utekar \ No newline at end of file +Sagar Utekar From 3d72b00217bd41e1ce7274bb1fd8f3b8be22b398 Mon Sep 17 00:00:00 2001 From: Sagar Utekar Date: Fri, 5 Sep 2025 06:33:41 +0530 Subject: [PATCH 109/133] Update challenge.md --- DailyChallenges/Season2/Day15/challenge.md | 60 ++++++++++------------ 1 file changed, 28 insertions(+), 32 deletions(-) diff --git a/DailyChallenges/Season2/Day15/challenge.md b/DailyChallenges/Season2/Day15/challenge.md index 82ba764..0cb67e2 100644 --- a/DailyChallenges/Season2/Day15/challenge.md +++ b/DailyChallenges/Season2/Day15/challenge.md @@ -2,9 +2,9 @@ ## Day 15: OSI & TCP/IP Model Foundations for DevOps/SRE ### Introduction -Welcome to Day 15 of the Daily DevOps + SRE Challenge Series – Season 2! +Welcome to Day 15 of the Daily DevOps + SRE Challenge Series - Season 2! -Today marks the start of your Networking for DevOps journey. Before diving into commands and configs, you'll build a strong mental model of how networking actually works—layer by layer. Understanding the OSI and TCP/IP models will make you a faster troubleshooter, a better cloud/network engineer, and help you communicate clearly with your team. +Today marks the start of your Networking for DevOps journey. Before diving into commands and configs, you'll build a strong mental model of how networking actually works layer by layer. Understanding the OSI and TCP/IP models will make you a faster troubleshooter, a better cloud/network engineer, and help you communicate clearly with your team. By the end of today, you'll be able to: - Draw and describe both the OSI and TCP/IP models @@ -22,31 +22,26 @@ Why this matters: ### 1) The OSI & TCP/IP Models -#### OSI Model (7 Layers) -| Layer | Number | Example Protocols | Example Tools | -|----------|--------|----------------------------|--------------------| -| Application | 7 | HTTP, SSH, DNS, SMTP | curl, ssh, dig | -| Presentation | 6 | SSL/TLS, ASCII, JPEG | openssl | -| Session | 5 | NetBIOS, RPC, TLS session | | -| Transport | 4 | TCP, UDP | nc, ss, netstat | -| Network | 3 | IP, ICMP | ip, ping, traceroute| -| Data Link | 2 | Ethernet, ARP, MAC | ip link, arp | -| Physical | 1 | Cables, Wi-Fi, fiber | ethtool, ip link | +#### OSI Model (7 Layers) and Data Formats + +| Layer | Number | Example Protocols | Example Tools | Data Unit (PDU) | +|---------------|--------|----------------------------|--------------------|---------------------------| +| Application | 7 | HTTP, SSH, DNS, SMTP | curl, ssh, dig | Data / Message | +| Presentation | 6 | SSL/TLS, ASCII, JPEG | openssl | Data | +| Session | 5 | NetBIOS, RPC, TLS session | | Data | +| Transport | 4 | TCP, UDP | nc, ss, netstat | Segment (TCP) / Datagram (UDP) | +| Network | 3 | IP, ICMP | ip, ping, traceroute | Packet | +| Data Link | 2 | Ethernet, ARP, MAC | ip link, arp | Frame | +| Physical | 1 | Cables, Wi-Fi, fiber | ethtool, ip link | Bit | #### TCP/IP Model (4 Layers) -| Layer | OSI Layer(s) | Example Protocols/Tools | -|--------------|--------------------------|------------------------------| -| Application | 7, 6, 5 | HTTP, SSH, DNS, TLS, curl | -| Transport | 4 | TCP, UDP, nc, ss | -| Internet | 3 | IP, ICMP, ping, traceroute | -| Network Access| 2, 1 | Ethernet, ARP, ip link, arp | - -**Quick mapping:** -- “Can’t SSH/ping/curl?” → Which layer fails? -- “Is DNS broken?” → Application -- “Is port blocked?” → Transport -- “Wrong IP/gateway?” → Network/Internet -- “Cable unplugged?” → Physical/Data Link + +| Layer | OSI Layer(s) | Example Protocols/Tools | +|---------------|--------------------------|------------------------------| +| Application | 7, 6, 5 | HTTP, SSH, DNS, TLS, curl | +| Transport | 4 | TCP, UDP, nc, ss | +| Internet | 3 | IP, ICMP, ping, traceroute | +| Network Access| 2, 1 | Ethernet, ARP, ip link, ethtool| --- @@ -55,13 +50,13 @@ Why this matters: #### OSI Model Overview ```mermaid flowchart TD - L7[Application :HTTP, SSH, DNS] - L6[Presentation :TLS, SSL, Encoding] - L5[Session :TLS session, NetBIOS] - L4[Transport :TCP, UDP] - L3[Network :IP, ICMP] - L2[Data Link :Ethernet, ARP] - L1[Physical :Cable, Wi-Fi] + L7[Application : HTTP, SSH, DNS] + L6[Presentation : TLS, SSL, Encoding] + L5[Session : TLS session, NetBIOS] + L4[Transport : TCP, UDP] + L3[Network : IP, ICMP] + L2[Data Link : Ethernet, ARP] + L1[Physical : Cable, Wi-Fi] L7 --> L6 --> L5 --> L4 --> L3 --> L2 --> L1 ``` @@ -123,6 +118,7 @@ Document your answer in solution.md. - Not every protocol is at only one layer (e.g., DNS = Application, but depends on Transport/Network) - Some tools work at multiple layers (e.g., tcpdump sees almost everything!) - Real-world cloud infra may “hide” some layers (e.g., you don’t see cables, but you still check link status in VMs) +- The PDU (data format) matters for troubleshooting (e.g., "packet loss" is different from "frame error") --- From ec51827be01d81d668a512936dd66c80dda85b92 Mon Sep 17 00:00:00 2001 From: Sagar Utekar Date: Sat, 6 Sep 2025 08:42:46 +0530 Subject: [PATCH 110/133] Update challenge.md --- DailyChallenges/Season2/Day16/challenge.md | 732 +++++---------------- 1 file changed, 166 insertions(+), 566 deletions(-) diff --git a/DailyChallenges/Season2/Day16/challenge.md b/DailyChallenges/Season2/Day16/challenge.md index 8ed6101..2a513bc 100644 --- a/DailyChallenges/Season2/Day16/challenge.md +++ b/DailyChallenges/Season2/Day16/challenge.md @@ -1,653 +1,253 @@ -# Day 16 — Core Protocols Challenge (Daily DevOps + SRE Challenge Series — Season 2) +# Day 16: OSI Layer 1 & 2 – Physical and Data Link – Interfaces, MAC, ARP, VLAN ---- - -## 🌟 Introduction - -Welcome to **Day 16** of the Daily DevOps + SRE Challenge Series – Season 2! - -Today, we'll explore the **core networking protocols** that form the backbone of modern infrastructure: **TCP, UDP, ICMP, DNS, DHCP, and HTTP/HTTPS**. - -Instead of just learning commands, you'll solve **real-world production-style problems** where these protocols play a critical role. These hands-on tasks will prepare you to debug outages, secure services, and explain protocol-level behavior in interviews with confidence. +## "Getting Connected: Cables, Cards, and Local Communication" --- -## 🚀 Why Does This Matter? +## Why This Matters -* **TCP vs UDP:** Choosing the right transport protocol affects performance, reliability, and scalability. -* **ICMP:** Quickest way to test connectivity, routing, and latency issues. -* **DNS:** One of the biggest causes of outages in real-world systems. -* **DHCP:** Critical for dynamic IP management in data centers and cloud. -* **HTTP/HTTPS:** The face of almost every service, where security and performance meet. +Before you can ping, curl, or SSH, your device must be physically and logically present on the network. Whether you’re on a real laptop, a cloud VM, or a VirtualBox guest, understanding these basics will help you fix some of the trickiest “it just doesn’t work” problems in DevOps and SRE. --- -## 🔥 Real-World Save +## 1. Concepts in Simple Words -* A fintech company once experienced **timeouts on payment APIs**. Root cause: firewall blocked **TCP port 443**. -* A streaming platform suffered **video buffering**. Root cause: packet loss showed TCP retries, but UDP-based CDN solved it. -* A global e-commerce site went down for **4 hours** because of a **DNS misconfiguration**. -* A new VM farm booted up without IPs because of a **rogue DHCP server** in the subnet. -* A startup got flagged for **"insecure website"** by Google Chrome due to expired SSL. +### Layer 1: Physical Layer +- **What is it?** The wires, Wi-Fi, or virtual “cables” that transmit bits (0s/1s). +- **On Cloud/VMs:** This is simulated—your “virtual cable” can be unplugged/plugged via settings or commands. +- **Problems:** Cable unplugged, “interface down”, VM not attached to network. -You'll now walk through these scenarios step by step. +### Layer 2: Data Link Layer +- **What is it?** Local communication (same network/segment). + Every device has a **MAC address**—its unique hardware “name tag.” +- **How do devices discover each other?** + Using ARP (Address Resolution Protocol): + "Who has IP X? Tell me your MAC address!" +- **VLANs:** Like colored wristbands at a party—devices with the same color (VLAN tag) can talk. --- -## 📘 Theory Section +## 1a. Devices at Layers 1 & 2 – What Hardware (and Virtual Devices) Live Here? -### 🔹 TCP (Transmission Control Protocol) +| Layer | Real-World Devices | Virtual/Cloud Equivalents | What They Do | +|--------|-------------------------------|---------------------------------------------|----------------------------------------------------------| +| 1 | Cables (Ethernet, Fiber), | Virtual cables (VM adapters), | Transmit bits (0s/1s) as electrical or optical signals | +| | Hubs, Network Interface Cards | Virtual NICs, “Attach Network” in settings | or Wi-Fi radio signals. | +| | (NIC), Repeaters | | | +| 2 | Switches, Network Interface | vSwitch, Linux bridge, | Forward frames based on MAC addresses, | +| | Cards (NIC), Bridges | cloud “virtual switches” | manage local network traffic, segment VLANs | -**Deep Dive:** -TCP is a connection-oriented protocol that ensures reliable, ordered, and error-checked delivery of data between applications. It establishes a connection using a three-way handshake before data transfer begins and maintains state throughout the session. - -**Key Features:** -- Connection-oriented communication -- Error detection and correction -- Flow control (window scaling) -- Congestion control (slow start, congestion avoidance) -- Ordered data transmission -- Retransmission of lost packets - -**TCP Header Structure:** -``` - 0 1 2 3 - 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 -+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ -| Source Port | Destination Port | -+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ -| Sequence Number | -+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ -| Acknowledgment Number | -+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ -| Data | |U|A|P|R|S|F| | -| Offset| Reserved |R|C|S|S|Y|I| Window | -| | |G|K|H|T|N|N| | -+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ -| Checksum | Urgent Pointer | -+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ -| Options | Padding | -+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ -| data | -+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ -``` +**Explanation:** +- **Layer 1 (Physical):** + - *Devices:* Cables, connectors, network cards, hubs, repeaters. + - *Virtual:* VM “network cable” or “adapter”, virtual NIC. +- **Layer 2 (Data Link):** + - *Devices:* Network switches, bridges, NICs (MAC logic), wireless access points. + - *Virtual:* Virtual switches (e.g., VirtualBox Host-Only or Bridged Adapter), Linux bridges, cloud VPC “switches”. -**Diagram – TCP Three-Way Handshake** - -```mermaid -sequenceDiagram - participant Client - participant Server - - Note over Client, Server: TCP Three-Way Handshake - Client->>Server: SYN (Synchronize Sequence Number) - Note right of Client: Client sends SYN packet with initial sequence number - Server->>Client: SYN-ACK (Acknowledge SYN, Send own SYN) - Note right of Server: Server acknowledges client's SYN and sends its own SYN - Client->>Server: ACK (Acknowledge Server's SYN) - Note right of Client: Client acknowledges server's SYN - Note over Client, Server: Connection Established -``` +**In the Cloud/VMs:** +- You don’t see the actual switch or cable, but you configure a “virtual” version (attach/detach network, set up vSwitch, assign MAC). +- Cloud providers use massive, invisible Layer 2/3 switches to connect your VMs—sometimes ARP and MAC issues still appear! --- -### 🔹 UDP (User Datagram Protocol) +## 1b. Where Are These Layers in Docker, Kubernetes, and the Cloud? -**Deep Dive:** -UDP is a connectionless protocol that provides a simple, unreliable datagram service. It doesn't guarantee delivery, ordering, or duplicate protection, making it faster with less overhead than TCP. +Even though you rarely see real cables, switches, or MAC addresses in containers or the cloud, **Layer 1 & 2 concepts still exist**—they’re just handled for you by the platform. -**Key Features:** -- Connectionless communication -- Minimal protocol overhead -- No guarantee of delivery -- No congestion control -- Supports broadcasting and multicasting +- **Docker:** + Each container gets its own virtual network interface (veth), a virtual MAC address, and is connected to a virtual bridge (like `docker0`). Docker manages these “cables” and “switches” behind the scenes, but issues like duplicate MACs, interface down, or ARP failures can still break connectivity between containers. -**UDP Header Structure:** -``` - 0 7 8 15 16 23 24 31 -+--------+--------+--------+--------+ -| Source | Destination | -| Port | Port | -+--------+--------+--------+--------+ -| | | -| Length | Checksum | -+--------+--------+--------+--------+ -| | -| Data | -| | -+-----------------------------------+ -``` +- **Kubernetes:** + Every Pod has a network namespace with its own virtual interfaces and MAC addresses. The CNI (Container Network Interface) plugin creates and manages these, connecting pods via virtual switches or bridges. If something breaks at Layer 2 (e.g., pod’s veth deleted, wrong bridge config), Pods can’t talk—even if everything “looks fine” at higher layers. -**Diagram – TCP vs UDP Comparison** +- **Cloud (AWS/GCP/Azure):** + Your VM’s “NIC” is virtual, but it has a MAC address and a link state (“attached,” “detached,” “UP,” “DOWN”). Cloud providers connect your VM to huge, invisible Layer 2/3 networks. Problems like “interface down,” “wrong MAC,” or ARP not resolving can still cause outages. -```mermaid -flowchart TD - A[Application Layer] --> B{Choose Transport Protocol} - - B --> |Reliable Data| C[TCP] - B --> |Low Latency| D[UDP] - - subgraph C[TCP Characteristics] - C1[Connection-oriented] - C2[Reliable Delivery] - C3[Flow Control] - C4[Congestion Control] - C5[Ordered Delivery] - end - - subgraph D[UDP Characteristics] - D1[Connectionless] - D2[Best-effort Delivery] - D3[No Flow Control] - D4[No Congestion Control] - D5[No Order Guarantee] - end - - C --> E[Web Browsing
Email
File Transfer] - D --> F[Video Streaming
DNS
VoIP] -``` +**Why care?** +- Many real-world outages—even in the cloud or containers—are caused by issues at these “hidden” layers! +- Knowing how to check and troubleshoot Layer 1/2 is a real SRE/DevOps superpower. --- -### 🔹 ICMP (Internet Control Message Protocol) - -**Deep Dive:** -ICMP is used by network devices to send error messages and operational information. It's primarily used for diagnostic purposes and doesn't carry application data. +## 2. Data Format at Each Layer -**Common ICMP Types:** -- Type 0: Echo Reply (ping response) -- Type 3: Destination Unreachable -- Type 5: Redirect Message -- Type 8: Echo Request (ping) -- Type 11: Time Exceeded (used by traceroute) - -**ICMP Header Structure:** -``` - 0 1 2 3 - 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 -+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ -| Type | Code | Checksum | -+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ -| Contents | -+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ -``` - -**Diagram – ICMP Ping Operation** - -```mermaid -sequenceDiagram - participant Sender - participant Router1 - participant Router2 - participant Target - - Note over Sender, Target: ICMP Echo Request (Ping) - Sender->>Router1: ICMP Echo Request - Router1->>Router2: ICMP Echo Request - Router2->>Target: ICMP Echo Request - - Note over Target, Sender: ICMP Echo Reply - Target->>Router2: ICMP Echo Reply - Router2->>Router1: ICMP Echo Reply - Router1->>Sender: ICMP Echo Reply - - Note right of Sender: Round-trip time calculated -``` +| Layer | Data Unit | Example | +|-------------------|-----------|------------------------| +| Physical (L1) | Bit | 0, 1 (electrical/light)| +| Data Link (L2) | Frame | Ethernet frame [Dst MAC][Src MAC][Type][Data][FCS] | --- -### 🔹 DNS (Domain Name System) - -**Deep Dive:** -DNS is a hierarchical decentralized naming system that translates human-readable domain names to IP addresses. It uses both UDP (for queries) and TCP (for zone transfers). - -**DNS Record Types:** -- A: IPv4 address record -- AAAA: IPv6 address record -- CNAME: Canonical name record (alias) -- MX: Mail exchange record -- NS: Name server record -- TXT: Text record -- SOA: Start of authority record +## 2a. Important Layer 1 & 2 Commands (Linux) -**DNS Resolution Process:** -1. Check local cache -2. Query recursive resolver -3. Root server referral -4. TLD server referral -5. Authoritative server response +| Purpose | Command Example | What It Shows/Does | +|-----------------------------------|-------------------------------------------------------|----------------------------------------| +| List all interfaces | `ip link show` | All network interfaces & MAC addresses | +| Show IP addresses & details | `ip addr show` | Interface IPs, MAC, status | +| Bring interface up/down | `sudo ip link set eth0 up` / `sudo ip link set eth0 down` | Enable/disable network interface | +| Show interface statistics | `ip -s link show eth0` | RX/TX stats, errors, dropped packets | +| Check ARP/neighbor cache | `ip neigh show` | IP-to-MAC mappings (ARP table) | +| Show legacy ARP table | `arp -a` | (Sometimes more readable) | +| Flush ARP/neighbor cache | `sudo ip neigh flush all` | Forces ARP to be rebuilt | +| Query ARP for specific IP | `arping ` | Sends ARP request for IP | +| Show/modify MAC address | `ip link set eth0 address aa:bb:cc:dd:ee:ff` | Change interface MAC (careful!) | +| Check link/cable status | `ethtool eth0` | Physical "link detected" etc. | +| Show VLAN interfaces | `ip -d link show` | Shows VLAN info if present | +| Create VLAN interface (if supported) | `sudo ip link add link eth0 name eth0.10 type vlan id 10` | Adds VLAN 10 on eth0 | +| | `sudo ip link set eth0.10 up` | Bring VLAN interface up | -**Diagram – DNS Resolution Process** - -```mermaid -flowchart TD - A[User enters example.com] --> B{Local Cache Check} - B -->|Found| C[Return IP Address] - B -->|Not Found| D[Query Recursive Resolver] - - D --> E{Resolver Cache Check} - E -->|Found| C - E -->|Not Found| F[Query Root Server] - - F --> G[Referral to .com TLD] - G --> H[Query .com TLD Server] - H --> I[Referral to example.com NS] - I --> J[Query Authoritative Server] - J --> K[Get IP Address] - K --> L[Cache Response] - L --> C -``` +**Cloud/VM notes:** +- Some commands (like `ethtool`) may not work on all VM types. +- For VirtualBox: "Cable Connected" can be toggled in VM settings (simulates unplug/plug). +- In the cloud, use console/CLI to check NIC status, MAC, and attachment. --- -### 🔹 DHCP (Dynamic Host Configuration Protocol) - -**Deep Dive:** -DHCP automatically assigns IP addresses and other network configuration parameters to devices on a network. It uses a client-server model and the DORA process. +## 3. Visuals & Analogies -**DHCP Process (DORA):** -1. Discover: Client broadcasts to find DHCP servers -2. Offer: Server responds with an IP offer -3. Request: Client requests the offered IP -4. Acknowledgment: Server confirms the lease - -**DHCP Options:** -- IP Address and subnet mask -- Default gateway -- DNS servers -- Lease time -- Domain name - -**Diagram – DHCP DORA Process** +- **Physical Layer:** The network cable (or “Connect Network Adapter” option in VirtualBox/cloud UI). +- **Data Link Layer:** The “name tag” (MAC) and waving to others to find out who’s who (ARP). +- **VLAN:** Like separate groups at a party wearing colored wristbands. ```mermaid -sequenceDiagram - participant Client - participant Server - - Note over Client, Server: DHCP Discovery Phase - Client->>Server: DHCPDISCOVER (Broadcast) - Note right of Client: Client broadcasts to discover available DHCP servers - - Note over Client, Server: DHCP Offer Phase - Server->>Client: DHCPOFFER (Unicast) - Note right of Server: Server offers IP address and configuration parameters - - Note over Client, Server: DHCP Request Phase - Client->>Server: DHCPREQUEST (Broadcast) - Note right of Client: Client requests the offered configuration - - Note over Client, Server: DHCP Acknowledgment Phase - Server->>Client: DHCPACK (Unicast) - Note right of Server: Server acknowledges and confirms the lease +flowchart LR + Start[Start of Frame] --> DstMAC[Destination MAC] + DstMAC --> SrcMAC[Source MAC] + SrcMAC --> Type[Type e.g., IPv4] + Type --> Data[Data :Payload] + Data --> FCS[FCS – Error Check] + FCS --> End[End of Frame] ``` --- -### 🔹 HTTP/HTTPS - -**Deep Dive:** -HTTP is an application protocol for distributed, collaborative hypermedia information systems. HTTPS is HTTP secured with TLS/SSL encryption. - -**HTTP Methods:** -- GET: Retrieve data -- POST: Submit data -- PUT: Replace data -- DELETE: Remove data -- PATCH: Partially update data +## 4. Guided Hands-On: Local, Cloud, and VirtualBox -**HTTP Status Codes:** -- 1xx: Informational -- 2xx: Success (200 OK, 201 Created) -- 3xx: Redirection (301 Moved Permanently) -- 4xx: Client Error (404 Not Found) -- 5xx: Server Error (500 Internal Server Error) +### A. Local or VM Interface Basics -**HTTPS/TLS Handshake:** -1. Client Hello -2. Server Hello + Certificate -3. Key Exchange -4. Cipher Suite Negotiation -5. Secure Data Transfer +1. **List all interfaces & MAC addresses** + ```bash + ip link show + ``` + - Find your primary network interface and note its MAC. -**Diagram – HTTPS/TLS Handshake** +2. **Is your interface up?** + ```bash + ip link show + ``` + - If not, bring it up: + ```bash + sudo ip link set up + ``` -```mermaid -sequenceDiagram - participant Client - participant Server - - Note over Client, Server: TLS Handshake - Client->>Server: Client Hello
Supported Cipher Suites - Server->>Client: Server Hello
Server Certificate
Selected Cipher Suite - Note right of Client: Client verifies certificate - Client->>Server: Pre-master Secret
(Encrypted with Server's Public Key) - Server->>Client: Finished - Client->>Server: Finished - - Note over Client, Server: Encrypted Data Exchange - Client->>Server: Application Data
(Encrypted with Session Keys) - Server->>Client: Application Data
(Encrypted with Session Keys) -``` - ---- - -## ⚡ Hands-On Challenges with Solutions +3. **Check statistics** + ```bash + ip -s link show + ``` + - Look for dropped or error packets. --- -### 🔹 Scenario 1: TCP vs UDP in Action - -**Diagram – TCP Connection Lifecycle** - -```mermaid -stateDiagram-v2 - [*] --> CLOSED - CLOSED --> SYN_SENT: Active Open - SYN_SENT --> ESTABLISHED: SYN-ACK Received - ESTABLISHED --> DATA_TRANSFER: Data Exchange - DATA_TRANSFER --> ESTABLISHED: More Data - ESTABLISHED --> FIN_WAIT_1: Active Close - FIN_WAIT_1 --> FIN_WAIT_2: ACK Received - FIN_WAIT_2 --> TIME_WAIT: FIN Received - TIME_WAIT --> [*]: Timeout - CLOSED --> LISTEN: Passive Open - LISTEN --> SYN_RCVD: SYN Received - SYN_RCVD --> ESTABLISHED: ACK Sent - ESTABLISHED --> CLOSE_WAIT: FIN Received - CLOSE_WAIT --> LAST_ACK: Application Close - LAST_ACK --> [*]: ACK Received -``` - -📌 **Task 1: List all services running on TCP and UDP** - -```bash -ss -tuln -lsof -i -P -n | grep LISTEN -``` - -👉 Shows ports like `22/tcp` (SSH), `53/udp` (DNS). - -📌 **Task 2: Compare DNS queries (UDP) with HTTP requests (TCP)** - -```bash -dig google.com -curl -v http://example.com -``` - -👉 `dig` uses UDP, while `curl` establishes a TCP handshake. +### B. Cloud Scenario: AWS/GCP/Azure -📌 **Task 3: Check if a specific TCP port is open** +**Scenario:** +You have two cloud VMs in the same subnet. They cannot ping each other. +- Both have IPs in the correct range and interfaces marked UP. +- Both have unique MAC addresses. +- ARP tables on both show the other’s IP as “INCOMPLETE”. -```bash -nc -zv google.com 80 -``` - -👉 Tests if port 80 is open on google.com - -📌 **Task 4: Check if a specific UDP port is open** - -```bash -nc -zvu 8.8.8.8 53 -``` - -👉 Tests if DNS port (53) is open on Google's DNS server - -📌 **Task 5: View active TCP connections** - -```bash -netstat -t -``` - -👉 Shows all active TCP connections on your system +**Questions:** +1. What might cause this in cloud environments? (Hint: Security Groups, NACLs, subnet config, ENI not attached) +2. What console/CLI checks should you try? +3. What Linux commands confirm interface/MAC status? --- -### 🔹 Scenario 2: ICMP (ping & traceroute) - Simplified - -**Diagram – Traceroute Operation** - -```mermaid -sequenceDiagram - participant Sender - participant Router1 - participant Router2 - participant Target - - Note over Sender, Target: Traceroute Process - Sender->>Router1: UDP/TCP/ICMP packet with TTL=1 - Router1->>Sender: ICMP Time Exceeded - Sender->>Router1: UDP/TCP/ICMP packet with TTL=2 - Router1->>Router2: Forward packet - Router2->>Sender: ICMP Time Exceeded - Sender->>Router1: UDP/TCP/ICMP packet with TTL=3 - Router1->>Router2: Forward packet - Router2->>Target: Forward packet - Target->>Sender: ICMP Echo Reply/Destination Unreachable -``` - -📌 **Task 1: Test basic connectivity** +### C. VirtualBox Scenario -```bash -ping -c 4 google.com -``` +**Scenario:** +You clone a VM in VirtualBox, but networking is broken: +- `ip addr` shows an IP +- `ip link show` shows interface DOWN, or both VMs have the same MAC -📌 **Task 2: Trace the network path** - -```bash -traceroute google.com -``` - -📌 **Task 3: Test connectivity to a specific port** - -```bash -ping -c 4 google.com -``` - -📌 **Task 4: Check if a host is reachable with timestamp** - -```bash -ping -c 4 -D google.com -``` - -👉 Adds timestamp to each ping response - -📌 **Task 5: Test network quality with mtr** - -```bash -mtr --report google.com -``` - -👉 Combines ping and traceroute functionality +**Questions:** +1. What Layer 1/2 issues can happen when cloning a VM? +2. How can you fix these in VirtualBox settings and on the VM? +3. What commands would you use to verify? --- -### 🔹 Scenario 3: DNS Troubleshooting (Simplified) - -**Diagram – DNS Query Types** - -```mermaid -flowchart TD - A[DNS Query] --> B{Query Type} - B --> C[Recursive Query] - B --> D[Iterative Query] - - subgraph C[Recursive Query] - C1[Client asks resolver] - C2[Resolver does all work] - C3[Returns final answer] - end - - subgraph D[Iterative Query] - D1[Client asks server] - D2[Server gives best answer] - D3[Client queries next server] - end -``` - -📌 **Task 1: Check DNS config** +### D. VLANs – (Optional/Conceptual) -```bash -cat /etc/resolv.conf -``` - -📌 **Task 2: Query DNS records** +> **Note:** +> True VLAN separation isn’t possible in plain VirtualBox setups without advanced config or special hardware/software switches. +> You can create VLAN interfaces in Linux for practice, but isolation won’t occur. -```bash -dig google.com -``` - -📌 **Task 3: Query specific DNS record types** - -```bash -dig google.com A -dig google.com MX -``` - -👉 Gets different types of DNS records - -📌 **Task 4: Flush DNS cache (if applicable)** - -```bash -sudo systemd-resolve --flush-caches -``` - -👉 Clears local DNS cache (systemd systems) - -📌 **Task 5: Test DNS resolution speed** - -```bash -time dig google.com -``` - -👉 Measures how long DNS resolution takes +- **Task:** Draw a diagram: Two VLANs (10 and 20) on the same switch—who can talk to whom? +- Try to create a VLAN interface on a Linux VM (if supported): + ```bash + sudo ip link add link name .10 type vlan id 10 + sudo ip link set .10 up + ip link show + ``` --- -### 🔹 Scenario 4: DHCP Assignment Issues (Simplified) +## 5. ARP Log Analysis: Mini Incident Simulation -**Diagram – DHCP Lease Process** +You receive these logs from a VM with network issues: -```mermaid -timeline - title DHCP Lease Timeline - section Lease Acquisition - DORA Process : 4-step process - IP Configured : Client gets IP - section Lease Duration - T1 (50%) : Client tries to renew - T2 (87.5%) : Client broadcasts renew request - Lease Expiration : IP address released ``` +$ ip link show eth0 +2: eth0: mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000 + link/ether 52:54:00:ab:cd:ef brd ff:ff:ff:ff:ff:ff -📌 **Task 1: Check current IP configuration** +$ ip addr show eth0 + inet 10.10.10.5/24 brd 10.10.10.255 scope global eth0 -```bash -ip addr show +$ ip neigh show +10.10.10.1 dev eth0 INCOMPLETE ``` -📌 **Task 2: Release and renew DHCP lease** - -```bash -sudo dhclient -r -sudo dhclient -``` - -📌 **Task 3: Check DHCP client status** - -```bash -journalctl -u systemd-networkd | grep DHCP -``` - -👉 Views DHCP-related logs - -📌 **Task 4: Set a temporary static IP** - -```bash -sudo ip addr add 192.168.1.100/24 dev eth0 -``` - -📌 **Task 5: Check network connectivity** - -```bash -ping -c 4 8.8.8.8 -``` - -👉 Tests if you have internet access after DHCP changes +**Your tasks:** +1. What does the ARP entry “INCOMPLETE” mean? +2. List two possible causes for this on physical, VirtualBox, or cloud VMs. +3. What troubleshooting steps would you take to fix it, layer by layer? --- -### 🔹 Scenario 5: HTTP/HTTPS Debugging (Simplified) - -**Diagram – HTTP Request/Response** +## 6. Gotchas & Security Notes -```mermaid -sequenceDiagram - participant Client - participant Server - - Note over Client, Server: HTTP Request - Client->>Server: GET /index.html HTTP/1.1
Host: example.com
User-Agent: curl/7.68.0
Accept: */* - - Note over Client, Server: HTTP Response - Server->>Client: HTTP/1.1 200 OK
Content-Type: text/html
Content-Length: 1256

... -``` +- Two VMs/devices with the same MAC = unpredictable network issues! +- ARP spoofing: Pretending to be another device’s MAC/IP can redirect traffic-beware! +- In the cloud, even “virtual cables” can be unplugged (e.g., detached NIC, disabled interface). -📌 **Task 1: Check if web ports are open** - -```bash -ss -tuln | grep :80 -ss -tuln | grep :443 -``` - -📌 **Task 2: Test HTTP connectivity** - -```bash -curl -I http://example.com -``` - -👉 Gets only HTTP headers - -📌 **Task 3: Test HTTPS connectivity** - -```bash -curl -I https://example.com -``` - -📌 **Task 4: Check SSL certificate validity** - -```bash -curl -v https://example.com 2>&1 | grep "SSL certificate" -``` - -📌 **Task 5: Test website loading time** +--- -```bash -time curl -s -o /dev/null https://example.com -``` +## 7. Submission Guidelines -👉 Measures how long a website takes to load +- Create `solution.md` and include: + - Outputs from your interface and MAC exploration + - Your answers to **one real-world scenario** (cloud or VirtualBox) + - Your analysis and fix for the ARP mini-incident (log analysis) + - 3–5 “what I learned” bullet points +- Push to your GitHub repo and share the link +- Post with #getfitwithsagar #SRELife #DevOpsForAll --- -## ✅ Deliverables - -* Document everything in `solution.md` with: +## 8. Community & Support - * Commands run - * Observations - * Screenshots (optional) -* Push to GitHub & share link -* Post your experience on social media with: - **#getfitwithsagar #SRELife #DevOpsForAll** +- [Discord](https://discord.gg/mNDm39qB8t) +- [Google Group](https://groups.google.com/forum/#!forum/daily-devops-sre-challenge-series/join) +- [YouTube](https://www.youtube.com/@Sagar.Utekar) --- -## 🌍 Community Links +## Remember! +Start troubleshooting from the bottom: Is the “cable” plugged in? Is the interface up? Is the MAC unique? Can ARP resolve? +Master these, and you’re ready for network success anywhere—cloud, VM, or on-prem! -* **Discord**: [https://discord.gg/mNDm39qB8t](https://discord.gg/mNDm39qB8t) -* **Google Group**: [https://groups.google.com/forum/#!forum/daily-devops-sre-challenge-series/join](https://groups.google.com/forum/#!forum/daily-devops-sre-challenge-series/join) -* **YouTube**: [https://www.youtube.com/@Sagar.Utekar](https://www.youtube.com/@Sagar.Utekar) - ---- +Happy learning, +Sagar Utekar From 8275c4062441839463bc90a22ab376eac9b196e8 Mon Sep 17 00:00:00 2001 From: Swayam Prakash Bhuyan Date: Sat, 6 Sep 2025 10:46:13 +0530 Subject: [PATCH 111/133] Update challenge.md --- DailyChallenges/Season2/Day17/challenge.md | 609 +++++---------------- 1 file changed, 147 insertions(+), 462 deletions(-) diff --git a/DailyChallenges/Season2/Day17/challenge.md b/DailyChallenges/Season2/Day17/challenge.md index 9303106..42dec5d 100644 --- a/DailyChallenges/Season2/Day17/challenge.md +++ b/DailyChallenges/Season2/Day17/challenge.md @@ -1,530 +1,215 @@ -# Day 17 — Routing & Switching Basics (Daily DevOps + SRE Challenge Series — Season 2) - --- -## 🌟 Introduction - -Welcome to **Day 17** of the Daily DevOps + SRE Challenge Series – Season 2! +# **Day 17: OSI Layer 3 – The Network Layer (IP, Subnets, Routing)** -Today we dive into the **Routing & Switching basics** every DevOps/SRE must master: **Static Routing, Default Gateways, NAT/PAT, and VLANs.** These are the invisible highways of networking that ensure packets get from point A to point B securely, efficiently, and reliably. - -Instead of just reading theory, you'll solve **real-world, production-style challenges** with Linux commands, diagrams, and troubleshooting steps - all within a single virtual machine using network namespaces. +### **The Internetworking Layer: From Local to Global** --- -## 🚀 Why Does This Matter? +## **🚀 Why This Matters More Than Ever** -* **Connectivity**: Without routing, packets never leave your subnet. -* **Security**: NAT/PAT and VLANs are used everywhere in enterprises to hide internal networks and isolate services. -* **Reliability**: Correct default gateway configs prevent outages. -* **Scale**: VLANs + static/dynamic routing make it possible to manage thousands of users and workloads. -* **Interview Edge**: Common questions include *"What is a default gateway?"*, *"Explain NAT vs PAT"*, and *"How do VLANs improve security?"* +You've mastered the local talk (L2) and the physical signals (L1). But the real magic of the internet happens when a request from your phone in Delhi can find a server in São Paulo, passing through dozens of unknown networks in milliseconds. ---- +**Layer 3 makes this possible.** It's the post office of the digital world. As an SRE or DevOps engineer, you don't just use this layer—you **design, secure, and troubleshoot** it. Every VPC, every firewall rule, every Kubernetes network policy, and every connectivity ticket ultimately boils down to Layer 3 concepts. -## 🔥 Real-World Save - -* A SaaS startup lost internet access for **6 hours** because the **wrong default gateway** was configured on a new subnet. -* A fintech company reduced **public IP usage by 90%** using **PAT**. -* A data center was compromised because **VLANs weren't properly isolated** — attackers jumped between networks. -* A global team avoided a costly outage by documenting **static routes** during a WAN migration. +> **"It's never DNS... until it is. It's never networking... until it is. And then it's everything."** --- -## 📘 Theory Section - -### 🔹 Static Routing - -Static routing involves manually configuring routes on network devices to specify the path network traffic should take to reach specific destinations. Unlike dynamic routing protocols that automatically exchange routing information, static routes are fixed and don't adapt to network changes unless manually modified. +## **1. Core Concepts Demystified** -```mermaid -flowchart TD - subgraph NetworkA[Network A: 192.168.1.0/24] - A1[Host A1
192.168.1.10] - A2[Host A2
192.168.1.11] - end - - subgraph NetworkB[Network B: 10.0.2.0/24] - B1[Host B1
10.0.2.10] - B2[Host B2
10.0.2.11] - end - - R1[Router A
192.168.1.1/24
172.16.1.1/30] - R2[Router B
172.16.1.2/30
10.0.2.1/24] - - NetworkA --> R1 - R1 -->|Static Route: 10.0.2.0/24 via 172.16.1.2| R2 - R2 --> NetworkB - - style R1 fill:#e1f5fe - style R2 fill:#e1f5fe -``` +### **1.1 The IP Address: Your Digital Passport** ---- +* **What it is:** A logical, hierarchical address assigned to a network interface. Unlike a MAC address (permanent, factory-assigned), an IP address is logical and can change based on location (network). +* **The Hierarchy:** This is the key to scalability. An IP address isn't just a random number; it has a **network portion** (like a zip code) and a **host portion** (like a street address). This allows routers to make decisions based only on the network portion, ignoring the specific host. +* **IPv4 vs. IPv6: The Great Transition** + * **IPv4 (e.g., `192.168.1.10`):** 32-bit, ~4.3 billion addresses. Exhausted. Kept alive by NAT (Network Address Translation). + * **IPv6 (e.g., `2001:0db8:85a3:0000:0000:8a2e:0370:7334`):** 128-bit, essentially unlimited addresses. Built for the modern internet (security, auto-configuration, efficiency). -### 🔹 Default Gateway +### **1.2 Subnetting & CIDR: Drawing the Maps** -The default gateway serves as the "gateway of last resort." When a device needs to send traffic to a destination network not explicitly listed in its routing table, it forwards the packet to the default gateway. +* **The Problem:** Throwing all devices into one giant network (e.g., 16 million hosts with `10.0.0.0/8`) is a broadcast storm nightmare and a security disaster. +* **The Solution: Subnetting.** Dividing a large network into smaller, manageable sub-networks. +* **CIDR (Classless Inter-Domain Routing):** The notation that defines the network/host split. `192.168.1.10/24` + * The `/24` is the **prefix length**. It means the first 24 bits are the network ID. + * **Subnet Mask:** The same concept, in dotted-decimal. `/24` = `255.255.255.0`. + * **The Golden Rule:** Two devices can talk directly **only if they are on the same subnet.** If not, they need a router. -```mermaid -flowchart TD - subgraph LocalNetwork[Local Network: 192.168.1.0/24] - PC1[Workstation
192.168.1.10] - PC2[Server
192.168.1.20] - PC3[Laptop
192.168.1.30] - end - - Router[Edge Router
192.168.1.1/24
203.0.113.5/30] - - Internet[Internet] - - PC1 -->|Default Gateway: 192.168.1.1| Router - PC2 -->|Default Gateway: 192.168.1.1| Router - PC3 -->|Default Gateway: 192.168.1.1| Router - Router -->|Default Route: 0.0.0.0/0
via ISP Router| Internet - - style Router fill:#fff3e0 -``` +**Quick Subnetting Reference:** +| CIDR | Subnet Mask | Usable Hosts | Use Case | +| :--- | :--- | :--- | :--- | +| `/32` | `255.255.255.255` | 1 | Single host (loopback, AWS security groups) | +| `/30` | `255.255.255.252` | 2 | Point-to-point links (e.g., WAN connections) | +| `/24` | `255.255.255.0` | 254 | Common small LANs | +| `/16` | `255.255.0.0` | 65,534 | Large private networks (e.g., corporate) | +| `/8` | `255.0.0.0` | 16.7M | Huge legacy networks | ---- +### **1.3 The Default Gateway: The Door to the World** -### 🔹 NAT & PAT +* The IP address of the **router** that sits on your local subnet. Your device's routing table says: "If you don't have a specific route for the destination, send the packet to this guy. He'll know what to do." -Network Address Translation (NAT) and Port Address Translation (PAT) are methods to remap IP address space by modifying network address information in packet headers while in transit. +### **1.4 Routing: The Path-Finding Algorithm** -```mermaid -flowchart LR - subgraph PrivateNetwork[Private Network] - A[Web Server
10.0.0.10:80] - B[Workstation
10.0.0.11:3456] - C[Laptop
10.0.0.12:5000] - end - - subgraph NATRouter[NAT Router] - NAT[Translation Table] - end - - Internet[Internet] - - A -->|Source: 10.0.0.10:80| NAT - B -->|Source: 10.0.0.11:3456| NAT - C -->|Source: 10.0.0.12:5000| NAT - - NAT -->|Translated: 203.0.113.5:8080| Internet - NAT -->|Translated: 203.0.113.5:3457| Internet - NAT -->|Translated: 203.0.113.5:5001| Internet - - style NATRouter fill:#f3e5f5 -``` +* **The Routing Table:** A list of rules on every networked device. Run `ip route show` to see yours. It answers the question: "To send a packet to IP X, which next hop or interface should I use?" +* **How Routes are Added:** + * **Directly Connected:** Added automatically when you configure an IP address on an interface (`proto kernel`). + * **Statically Configured:** Manually added by an admin (`ip route add ...`). + * **Dynamically Learned:** Added by routing protocols (BGP, OSPF) that routers use to talk to each other and share network information. --- -### 🔹 VLANs +## **2. The IP Packet Deep Dive (The L3 Envelope)** + +The segment from Layer 4 is encapsulated into an IP Packet. Its header is the critical information for delivery. -Virtual LANs (VLANs) create logically separate networks within a physical network infrastructure. VLANs operate at Layer 2 of the OSI model and provide segmentation, security, and broadcast containment. +| Field | Purpose | Why it Matters to You | +| :--- | :--- | :--- | +| **Version** | 4 or 6 | Determines which IP protocol stack to use. | +| **Source IP** | Origin Address | Used for reply routing. Can be spoofed (security risk!). | +| **Destination IP** | Final Address | The ultimate target the packet must reach. | +| **TTL (Time to Live)** | Hop Limit | Decremented by each router. Prevents infinite loops. A `traceroute` works by manipulating TTL. | +| **Protocol** | TCP (6), UDP (17), etc. | Tells the receiving host's OS which L4 protocol to hand the payload to. | +| **Flags / Fragment Offset** | Packet Fragmentation | If a packet is too large for a network link (MTU), it may be split into fragments. Often causes performance issues. | ```mermaid -flowchart TD - Switch[Layer 3 Switch] - - subgraph VLAN10[VLAN 10 - HR Department] - HR1[HR PC1
192.168.10.10] - HR2[HR PC2
192.168.10.11] - HR3[HR Server
192.168.10.100] - end - - subgraph VLAN20[VLAN 20 - Engineering] - Eng1[Dev Workstation
192.168.20.10] - Eng2[Test Server
192.168.20.50] - Eng3[Dev Laptop
192.168.20.20] - end - - subgraph VLAN30[VLAN 30 - Guests] - Guest1[Guest WiFi
192.168.30.0/24] - Guest2[Visitor PC
192.168.30.10] - end - - Router[Router] - - HR1 -->|Access Port
VLAN 10| Switch - HR2 -->|Access Port
VLAN 10| Switch - HR3 -->|Access Port
VLAN 10| Switch - - Eng1 -->|Access Port
VLAN 20| Switch - Eng2 -->|Access Port
VLAN 20| Switch - Eng3 -->|Access Port
VLAN 20| Switch - - Guest1 -->|Access Port
VLAN 30| Switch - Guest2 -->|Access Port
VLAN 30| Switch - - Switch -->|Trunk Port
All VLANs| Router - - style Switch fill:#e8f5e9 +graph LR +A[Application Data] --> B[L4: TCP Header + Data] +B --> C[L3: IP Header + Segment] +C --> D[L2: Frame] ``` --- -## ⚡ Hands-On Challenges with Solutions (Single VM Edition) +## **3. Layer 3 in Action: Devices & Their Cloud Equivalents** -### Prerequisites: Setup Network Namespaces -First, let's create the network namespaces that will simulate different devices: - -```bash -# Create network namespaces -sudo ip netns add client1 -sudo ip netns add client2 -sudo ip netns add router - -# Create virtual Ethernet pairs -sudo ip link add veth0 type veth peer name veth1 -sudo ip link add veth2 type veth peer name veth3 - -# Move interfaces to appropriate namespaces -sudo ip link set veth0 netns client1 -sudo ip link set veth1 netns router -sudo ip link set veth2 netns client2 -sudo ip link set veth3 netns router - -# Configure IP addresses -sudo ip netns exec client1 ip addr add 192.168.1.10/24 dev veth0 -sudo ip netns exec router ip addr add 192.168.1.1/24 dev veth1 -sudo ip netns exec client2 ip addr add 10.0.2.10/24 dev veth2 -sudo ip netns exec router ip addr add 10.0.2.1/24 dev veth3 - -# Bring interfaces up -sudo ip netns exec client1 ip link set veth0 up -sudo ip netns exec router ip link set veth1 up -sudo ip netns exec client2 ip link set veth2 up -sudo ip netns exec router ip link set veth3 up -sudo ip netns exec client1 ip link set lo up -sudo ip netns exec client2 ip link set lo up -sudo ip netns exec router ip link set lo up -``` +| Concept | Physical World | Virtual/Cloud World | Key Purpose | +| :--- | :--- | :--- | :--- | +| **Router** | Cisco, Juniper Box | **VPC Route Table**, **Virtual Network Gateway** | Connect different IP networks and make forwarding decisions. | +| **L3 Switch** | Switch with routing features | (Same as Router) | High-speed, hardware-based routing within a data center. | +| **Firewall** | Palo Alto, Fortinet Box | **Security Groups (Host-Based)**, **NACLs (Subnet-Based)**, **Cloud Firewall** | Filter traffic based on L3 (IPs) and L4 (Ports) rules. | +| **Load Balancer** | F5 BIG-IP | **ALB/NLB (AWS), GLB (GCP), Load Balancer (Azure)** | Distribute traffic based on IP and port to backend targets. | +| **Gateway** | Internet Router | **Internet Gateway (IGW), NAT Gateway** | Provide a controlled path between private networks and the internet. | --- -### 🔹 Static Routing (5 Challenges) - -1️⃣ **Configure static routes between namespaces** - -```bash -# On router namespace -sudo ip netns exec router ip route add 192.168.1.0/24 dev veth1 -sudo ip netns exec router ip route add 10.0.2.0/24 dev veth3 - -# On client1 namespace (add route to network 10.0.2.0/24 via router) -sudo ip netns exec client1 ip route add 10.0.2.0/24 via 192.168.1.1 - -# On client2 namespace (add route to network 192.168.1.0/24 via router) -sudo ip netns exec client2 ip route add 192.168.1.0/24 via 10.0.2.1 - -# Test connectivity -sudo ip netns exec client1 ping 10.0.2.10 -c 3 -sudo ip netns exec client2 ping 192.168.1.10 -c 3 -``` - -2️⃣ **Add multiple static routes and analyze preference** - -```bash -# Create an alternative path (simulated) -sudo ip netns exec router ip route add 10.0.2.0/24 dev veth3 metric 100 -sudo ip netns exec router ip route add 10.0.2.0/24 dev veth3 metric 200 - -# View routing table and analyze preference -sudo ip netns exec router ip route show -``` - -3️⃣ **Break a static route intentionally and troubleshoot** - -```bash -# Remove the route -sudo ip netns exec client1 ip route del 10.0.2.0/24 - -# Test connectivity (should fail) -sudo ip netns exec client1 ping 10.0.2.10 -c 2 - -# Troubleshoot with traceroute -sudo ip netns exec client1 traceroute 10.0.2.10 - -# Fix the route -sudo ip netns exec client1 ip route add 10.0.2.0/24 via 192.168.1.1 -``` - -4️⃣ **Compare static vs dynamic routing concepts** - -```bash -# Examine current static routes -sudo ip netns exec router ip route show - -# Compare with what dynamic routing would provide -echo "Static routing: Manual configuration, no overhead" -echo "Dynamic routing: Automatic updates, better for large networks" -``` - -5️⃣ **Document the namespace routing topology** +## **4. The SRE's Toolkit: Essential Linux Commands** -```bash -# Show all interfaces and their addresses -sudo ip netns exec client1 ip addr show -sudo ip netns exec client2 ip addr show -sudo ip netns exec router ip addr show - -# Show routing tables -sudo ip netns exec client1 ip route show -sudo ip netns exec client2 ip route show -sudo ip netns exec router ip route show -``` +| Task | Command | Example & Output Snippet | +| :--- | :--- | :--- | +| **Show IP Config** | `ip addr show` or `ip a` | `inet 192.168.1.5/24 brd 192.168.1.255 scope global eth0` | +| **Show Routing Table** | `ip route show` or `ip r` | `default via 192.168.1.1 dev eth0 proto static`
`192.168.1.0/24 dev eth0 proto kernel scope link src 192.168.1.5` | +| **Test Reachability** | `ping ` | `64 bytes from 8.8.8.8: icmp_seq=1 ttl=118 time=15.3 ms` | +| **Trace the Path** | `traceroute `
or `tracepath` | `1: _gateway (192.168.1.1) 1.102ms`
`2: 100.65.18.177 (100.65.18.177) 9.123ms` | +| **Check Listening Ports** | `ss -tuln` | `tcp LISTEN 0 128 0.0.0.0:22 0.0.0.0:*`
`(SSH listening on all IPs)` | +| **Manage Routes** | `sudo ip route add `
`sudo ip route del ` | `sudo ip route add 10.10.0.0/24 via 192.168.1.254` | +| **Clear ARP Cache** | `ip neigh flush all` | Useful after changing network config to clear old MAC/IP mappings. | --- -### 🔹 Default Gateway (5 Challenges) - -1️⃣ **Set up default gateway in namespace** - -```bash -# Create a simulated internet namespace -sudo ip netns add internet -sudo ip link add veth4 type veth peer name veth5 -sudo ip link set veth4 netns router -sudo ip link set veth5 netns internet -sudo ip netns exec router ip addr add 203.0.113.1/24 dev veth4 -sudo ip netns exec internet ip addr add 203.0.113.2/24 dev veth5 -sudo ip netns exec router ip link set veth4 up -sudo ip netns exec internet ip link set veth5 up - -# Set default gateway on router -sudo ip netns exec router ip route add default via 203.0.113.2 - -# Set default gateway on clients -sudo ip netns exec client1 ip route add default via 192.168.1.1 -sudo ip netns exec client2 ip route add default via 10.0.2.1 -``` - -2️⃣ **Remove/modify gateway and test** - -```bash -# Remove default gateway -sudo ip netns exec client1 ip route del default - -# Test connectivity (should fail) -sudo ip netns exec client1 ping 203.0.113.2 -c 2 - -# Restore gateway -sudo ip netns exec client1 ip route add default via 192.168.1.1 -``` - -3️⃣ **Multiple gateways with metrics** - -```bash -# Add multiple default routes with different metrics -sudo ip netns exec client1 ip route add default via 192.168.1.1 metric 100 -sudo ip netns exec client1 ip route add default via 192.168.1.1 metric 200 - -# Show routing table to see preference -sudo ip netns exec client1 ip route show -``` - -4️⃣ **Capture packets going through gateway** - -```bash -# Start packet capture on router -sudo ip netns exec router timeout 10 tcpdump -i veth1 icmp & - -# Generate traffic from client1 -sudo ip netns exec client1 ping 203.0.113.2 -c 3 -``` - -5️⃣ **Troubleshoot wrong gateway configuration** - -```bash -# Intentionally set wrong gateway -sudo ip netns exec client1 ip route del default -sudo ip netns exec client1 ip route add default via 192.168.1.99 - -# Test connectivity (should fail) -sudo ip netns exec client1 ping 203.0.113.2 -c 2 - -# Diagnose the issue -sudo ip netns exec client1 ip route show -sudo ip netns exec client1 traceroute 203.0.113.2 - -# Fix the gateway -sudo ip netns exec client1 ip route del default -sudo ip netns exec client1 ip route add default via 192.168.1.1 -``` +## **5. Guided Hands-On Lab: Become Your Own Router** + +### **Lab Setup (VirtualBox/VMware)** +1. Create **three** VMs. +2. Set their network adapters to **Internal Network** (e.g., named `LAB_NET`). +3. Boot them up. They will have no IP addresses initially. + +### **Step 1: Configure the "Router"** +* On VM1 (`router`), configure two virtual interfaces. + ```bash + sudo ip addr add 192.168.10.1/24 dev eth0 + sudo ip addr add 192.168.20.1/24 dev eth1 + ``` +* Enable IP forwarding on `router` (this turns a Linux host into a router). + ```bash + echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward + ``` + +### **Step 2: Configure the "Clients"** +* On VM2 (`client-a`), set its IP and default gateway. + ```bash + sudo ip addr add 192.168.10.10/24 dev eth0 + sudo ip route add default via 192.168.10.1 + ``` +* On VM3 (`client-b`), set its IP and default gateway. + ```bash + sudo ip addr add 192.168.20.20/24 dev eth0 + sudo ip route add default via 192.168.20.1 + ``` + +### **Step 3: Test the Connectivity** +* From `client-a`, ping `client-b`'s IP. + ```bash + ping 192.168.20.20 + ``` +* **It should work!** You have successfully built a routed network. The `router` VM received the ping from `client-a` on its `192.168.10.1` interface, checked its routing table, and forwarded it out the `192.168.20.1` interface to `client-b`. --- -### 🔹 NAT & PAT (5 Challenges) - -1️⃣ **Configure NAT on the router namespace** - -```bash -# Enable IP forwarding on router -sudo ip netns exec router sysctl -w net.ipv4.ip_forward=1 - -# Configure NAT using iptables -sudo ip netns exec router iptables -t nat -A POSTROUTING -o veth4 -j MASQUERADE -sudo ip netns exec router iptables -A FORWARD -i veth1 -o veth4 -j ACCEPT -sudo ip netns exec router iptables -A FORWARD -i veth4 -o veth1 -m state --state ESTABLISHED,RELATED -j ACCEPT - -# Test NAT functionality -sudo ip netns exec client1 ping 203.0.113.2 -c 3 -``` - -2️⃣ **Test PAT with multiple hosts** - -```bash -# Simulate multiple clients making connections -sudo ip netns exec client1 ping 203.0.113.2 -c 2 & -sudo ip netns exec client2 ping 203.0.113.2 -c 2 & - -# Check NAT translations -sudo ip netns exec router iptables -t nat -L -n -v -``` - -3️⃣ **Test iptables NAT configuration** - -```bash -# Check NAT table -sudo ip netns exec router iptables -t nat -L - -# Verify NAT is working by checking connection tracking -sudo ip netns exec router conntrack -L -``` - -4️⃣ **Capture NAT packets** - -```bash -# Capture packets on the external interface -sudo ip netns exec router timeout 10 tcpdump -i veth4 -n & - -# Generate traffic from clients -sudo ip netns exec client1 ping 203.0.113.2 -c 3 -``` +## **6. Real-World Scenarios & Debugging** -5️⃣ **Troubleshoot NAT failure** +### **Scenario 1: The Cloud VPC Mystery** +**Problem:** A VM in `subnet-private` can't pull updates from the internet, but a VM in `subnet-public` can. -```bash -# Disable IP forwarding to simulate NAT failure -sudo ip netns exec router sysctl -w net.ipv4.ip_forward=0 +**Your Investigation:** +1. **Check Route Tables:** `subnet-private`'s route table likely has a default route (`0.0.0.0/0`) pointing to a **NAT Gateway**, not an Internet Gateway. An IGW gives a public IP; a NAT Gateway allows private instances to initiate outbound connections without being directly exposed. +2. **Check NACLs:** Is there a rule in `subnet-private`'s NACL allowing outbound traffic on ports 80/443 and the return ephemeral ports (1024-65535)? +3. **Check Security Groups:** Does the SG attached to the private VM allow outbound traffic to `0.0.0.0/0`? -# Test connectivity (should fail) -sudo ip netns exec client1 ping 203.0.113.2 -c 2 +### **Scenario 2: The Hybrid Cloud Tunnel** +**Problem:** Your on-premise application (`10.10.10.5`) cannot connect to a cloud database (`10.20.30.5`) after a maintenance window. -# Check sysctl settings -sudo ip netns exec router sysctl net.ipv4.ip_forward - -# Re-enable and verify -sudo ip netns exec router sysctl -w net.ipv4.ip_forward=1 -sudo ip netns exec client1 ping 203.0.113.2 -c 2 -``` +**Your Investigation:** +1. **Traceroute is your friend:** `traceroute 10.20.30.5` from on-prem. Where does it stop? +2. **Check the VPN/Direct Connect:** Is the BGP session up? Are the routes being advertised correctly from the cloud side to your on-premise router? +3. **Check Firewall Rules:** Have the ACLs on the cloud VPC or on-premise firewall been modified? Is traffic allowed bidirectionally on the database port? --- -### 🔹 VLAN Basics (5 Challenges) - -1️⃣ **Create VLAN interfaces in namespaces** +## **7. Mini Incident Simulation** -```bash -# Create VLAN interfaces on router -sudo ip netns exec router ip link add link veth1 name veth1.10 type vlan id 10 -sudo ip netns exec router ip link add link veth1 name veth1.20 type vlan id 20 -sudo ip netns exec router ip addr add 192.168.10.1/24 dev veth1.10 -sudo ip netns exec router ip addr add 192.168.20.1/24 dev veth1.20 -sudo ip netns exec router ip link set veth1.10 up -sudo ip netns exec router ip link set veth1.20 up - -# Create VLAN interfaces on client1 (VLAN 10) -sudo ip netns exec client1 ip link add link veth0 name veth0.10 type vlan id 10 -sudo ip netns exec client1 ip addr add 192.168.10.10/24 dev veth0.10 -sudo ip netns exec client1 ip link set veth0.10 up - -# Create VLAN interfaces on client2 (VLAN 20) -sudo ip netns exec client2 ip link add link veth2 name veth2.20 type vlan id 20 -sudo ip netns exec client2 ip addr add 192.168.20.10/24 dev veth2.20 -sudo ip netns exec client2 ip link set veth2.20 up -``` - -2️⃣ **Configure inter-VLAN routing** +**Alert:** `Web servers in availability zone 1a are failing health checks from the load balancer in 1b.` +You jump into a web server and find: ```bash -# Enable IP forwarding on router -sudo ip netns exec router sysctl -w net.ipv4.ip_forward=1 +$ ip addr show eth0 +inet 10.0.1.25/20 brd 10.0.15.255 scope global eth0 -# Test inter-VLAN connectivity -sudo ip netns exec client1 ping 192.168.20.10 -c 3 -sudo ip netns exec client2 ping 192.168.10.10 -c 3 -``` - -3️⃣ **Misconfigure VLAN and troubleshoot** - -```bash -# Put client2 in wrong VLAN -sudo ip netns exec client2 ip link del veth2.20 -sudo ip netns exec client2 ip link add link veth2 name veth2.30 type vlan id 30 -sudo ip netns exec client2 ip addr add 192.168.30.10/24 dev veth2.30 -sudo ip netns exec client2 ip link set veth2.30 up - -# Test connectivity (should fail) -sudo ip netns exec client1 ping 192.168.30.10 -c 2 - -# Troubleshoot -sudo ip netns exec client1 ip addr show -sudo ip netns exec client2 ip addr show -sudo ip netns exec router ip addr show - -# Fix the configuration -sudo ip netns exec client2 ip link del veth2.30 -sudo ip netns exec client2 ip link add link veth2 name veth2.20 type vlan id 20 -sudo ip netns exec client2 ip addr add 192.168.20.10/24 dev veth2.20 -sudo ip netns exec client2 ip link set veth2.20 up +$ ip route show +10.0.0.0/20 dev eth0 proto kernel scope link src 10.0.1.25 +default via 10.0.0.1 dev eth0 ``` -4️⃣ **Explore VLAN tagging** +The load balancer's IP is `10.0.16.105`. -```bash -# Capture VLAN tagged packets -sudo ip netns exec router timeout 10 tcpdump -i veth1 -e vlan & +**Your Tasks:** +1. **Calculate the subnets.** The server is in `10.0.0.0/20`. What is the range of this subnet? *(Hint: The next subnet is `10.0.16.0/20`)*. +2. **Diagnose the issue:** Based on the subnet, is the load balancer's IP in the same subnet as the web server? +3. **Explain the consequence:** Where will the web server send packets destined for the load balancer? Can they reach it? +4. **Propose the fix:** This is a VPC design error. What should the netmask have been to allow all resources in the VPC to communicate directly? -# Generate VLAN traffic -sudo ip netns exec client1 ping 192.168.20.10 -c 2 -``` +--- -5️⃣ **Document multi-VLAN setup** +## **8. Advanced Gotchas for SREs** -```bash -# Show VLAN configuration -echo "VLAN 10: 192.168.10.0/24 (Client1)" -echo "VLAN 20: 192.168.20.0/24 (Client2)" -echo "Router interfaces:" -sudo ip netns exec router ip addr show | grep -E "(veth1.10|veth1.20)" -echo "Client interfaces:" -sudo ip netns exec client1 ip addr show veth0.10 -sudo ip netns exec client2 ip addr show veth2.20 -``` +* **MTU & Fragmentation:** If your VPN or overlay network (like Docker's) has a smaller MTU than your underlying network, packets will fragment, crushing performance. Solution: Enable **MTU Path Discovery** or set the MTU manually. +* **Asymmetric Routing:** The request goes through Firewall A, but the response comes back through Firewall B. Since Firewall B didn't see the connection initiation, it drops the packet. Common in multi-cloud and complex routing setups. +* **ECMP (Equal-Cost Multi-Path):** Multiple paths to the same destination. Great for bandwidth, terrible for stateful firewalls and packet ordering. +* **Network Namespaces:** Modern orchestration (Kubernetes, Docker) uses network namespaces to create isolated network stacks. A `ping` from the host might work, but not from inside a container because it's in a different namespace with its own routes and interfaces. Use `nsenter` or `kubectl exec` to debug. --- -## ✅ Deliverables +## **9. Submission Guidelines: Prove Your Mastery** -* Document your solutions in `solution.md` with: - * All commands executed - * Output observations and screenshots - * Network diagrams of your namespace setup -* Push to your GitHub repository -* Share your experience with hashtags: - **#getfitwithsagar #SRELife #DevOpsForAll** +Push a file named `day17solution.md` to your GitHub repo. ---- +**Include:** -## 🌍 Community Links +1. **Screenshots:** Output of the `ip a` and `ip r` commands from your main machine. +2. **Hands-on Proof:** Output from the "Become Your Own Router" lab showing a successful ping between the two client VMs. +3. **Incident Response:** Your detailed answers to the four questions in the **Mini Incident Simulation** (Section 7). +4. **Reflection:** Write a paragraph explaining a networking issue you've faced or anticipate facing in your work and how a deep understanding of Layer 3 would help you resolve it. -* **Discord**: [https://discord.gg/mNDm39qB8t](https://discord.gg/mNDm39qB8t) -* **Google Group**: [https://groups.google.com/forum/#!forum/daily-devops-sre-challenge-series/join](https://groups.google.com/forum/#!forum/daily-devops-sre-challenge-series/join) -* **YouTube**: [https://www.youtube.com/@Sagar.Utekar](https://www.youtube.com/@Sagar.Utekar) +**Share your journey:** +`#getfitwithsagar #100DaysOfSRE #OSIMastery #Layer3DeepDive #CloudNetworking` --- - -🔥 Keep routing, keep switching, and happy exploring! -— *Sagar Utekar* From a3ea7d62ae4ec6e12a493a82f19f0dc54ce419e3 Mon Sep 17 00:00:00 2001 From: Sagar Utekar Date: Sun, 7 Sep 2025 07:35:30 +0530 Subject: [PATCH 112/133] Delete submission_tracker.md --- submission_tracker.md | 1 - 1 file changed, 1 deletion(-) delete mode 100644 submission_tracker.md diff --git a/submission_tracker.md b/submission_tracker.md deleted file mode 100644 index e3ba9f7..0000000 --- a/submission_tracker.md +++ /dev/null @@ -1 +0,0 @@ -# Day 2 submissions list From e8d62e70bb2abd71e45141838251f8455bcb3220 Mon Sep 17 00:00:00 2001 From: Sagar Utekar Date: Sun, 7 Sep 2025 08:03:22 +0530 Subject: [PATCH 113/133] Update challenge.md --- DailyChallenges/Season2/Day17/challenge.md | 297 ++++++++++++--------- 1 file changed, 176 insertions(+), 121 deletions(-) diff --git a/DailyChallenges/Season2/Day17/challenge.md b/DailyChallenges/Season2/Day17/challenge.md index 42dec5d..523354d 100644 --- a/DailyChallenges/Season2/Day17/challenge.md +++ b/DailyChallenges/Season2/Day17/challenge.md @@ -1,75 +1,72 @@ ---- - # **Day 17: OSI Layer 3 – The Network Layer (IP, Subnets, Routing)** -### **The Internetworking Layer: From Local to Global** +## **The Internetworking Layer: From Local to Global** --- -## **🚀 Why This Matters More Than Ever** +## **Why This Matters More Than Ever** -You've mastered the local talk (L2) and the physical signals (L1). But the real magic of the internet happens when a request from your phone in Delhi can find a server in São Paulo, passing through dozens of unknown networks in milliseconds. +You've mastered local talk (L2) and physical signals (L1). But true internet magic happens when a request from your laptop in Delhi finds a server in São Paulo, traversing dozens of unknown networks within milliseconds. -**Layer 3 makes this possible.** It's the post office of the digital world. As an SRE or DevOps engineer, you don't just use this layer—you **design, secure, and troubleshoot** it. Every VPC, every firewall rule, every Kubernetes network policy, and every connectivity ticket ultimately boils down to Layer 3 concepts. +**Layer 3 makes this possible.** It's the post office of the digital world. As an SRE or DevOps engineer, you don't just use this layer—you **design, secure, and troubleshoot** it. Every VPC, firewall rule, Kubernetes network policy, and connectivity ticket ultimately boils down to Layer 3 concepts. > **"It's never DNS... until it is. It's never networking... until it is. And then it's everything."** --- -## **1. Core Concepts Demystified** - -### **1.1 The IP Address: Your Digital Passport** - -* **What it is:** A logical, hierarchical address assigned to a network interface. Unlike a MAC address (permanent, factory-assigned), an IP address is logical and can change based on location (network). -* **The Hierarchy:** This is the key to scalability. An IP address isn't just a random number; it has a **network portion** (like a zip code) and a **host portion** (like a street address). This allows routers to make decisions based only on the network portion, ignoring the specific host. -* **IPv4 vs. IPv6: The Great Transition** - * **IPv4 (e.g., `192.168.1.10`):** 32-bit, ~4.3 billion addresses. Exhausted. Kept alive by NAT (Network Address Translation). - * **IPv6 (e.g., `2001:0db8:85a3:0000:0000:8a2e:0370:7334`):** 128-bit, essentially unlimited addresses. Built for the modern internet (security, auto-configuration, efficiency). +## 1. **Core Concepts Demystified** -### **1.2 Subnetting & CIDR: Drawing the Maps** +### 1.1 **The IP Address: Your Digital Passport** +- **What it is:** A logical, hierarchical address assigned to a network interface. Unlike a MAC address (permanent, factory-assigned), an IP address is logical and changes with network location. +- **The Hierarchy:** An IP has a **network portion** (like a zip code) and a **host portion** (like a street address). Routers use the network portion to make forwarding decisions. +- **IPv4 vs. IPv6:** + - **IPv4** (e.g. `192.168.1.10`): 32-bit, ~4.3B addresses, running out, kept alive by NAT. + - **IPv6** (e.g. `2001:db8::7334`): 128-bit, essentially unlimited, with built-in security and auto-configuration. -* **The Problem:** Throwing all devices into one giant network (e.g., 16 million hosts with `10.0.0.0/8`) is a broadcast storm nightmare and a security disaster. -* **The Solution: Subnetting.** Dividing a large network into smaller, manageable sub-networks. -* **CIDR (Classless Inter-Domain Routing):** The notation that defines the network/host split. `192.168.1.10/24` - * The `/24` is the **prefix length**. It means the first 24 bits are the network ID. - * **Subnet Mask:** The same concept, in dotted-decimal. `/24` = `255.255.255.0`. - * **The Golden Rule:** Two devices can talk directly **only if they are on the same subnet.** If not, they need a router. +### 1.2 **Subnetting & CIDR: Drawing the Maps** +- **The Problem:** One giant network is noisy and insecure. +- **The Solution: Subnetting.** Divide large networks into smaller, manageable subnets. +- **CIDR (Classless Inter-Domain Routing):** + - `192.168.1.10/24` means first 24 bits are the network. + - **Subnet Mask:** `/24` = `255.255.255.0` + - **Golden Rule:** Two devices can talk directly **only if on the same subnet**; otherwise, they need a router. **Quick Subnetting Reference:** + | CIDR | Subnet Mask | Usable Hosts | Use Case | | :--- | :--- | :--- | :--- | -| `/32` | `255.255.255.255` | 1 | Single host (loopback, AWS security groups) | -| `/30` | `255.255.255.252` | 2 | Point-to-point links (e.g., WAN connections) | -| `/24` | `255.255.255.0` | 254 | Common small LANs | -| `/16` | `255.255.0.0` | 65,534 | Large private networks (e.g., corporate) | +| `/32` | `255.255.255.255` | 1 | Single host (loopback, SGs) | +| `/30` | `255.255.255.252` | 2 | Point-to-point links | +| `/24` | `255.255.255.0` | 254 | Small LANs | +| `/16` | `255.255.0.0` | 65,534 | Large private networks | | `/8` | `255.0.0.0` | 16.7M | Huge legacy networks | -### **1.3 The Default Gateway: The Door to the World** - -* The IP address of the **router** that sits on your local subnet. Your device's routing table says: "If you don't have a specific route for the destination, send the packet to this guy. He'll know what to do." - -### **1.4 Routing: The Path-Finding Algorithm** +### 1.3 **The Default Gateway: Your Door to the World** +- The router on your subnet. Your device's routing table says, "If you're unsure where to send a packet, give it to the gateway." -* **The Routing Table:** A list of rules on every networked device. Run `ip route show` to see yours. It answers the question: "To send a packet to IP X, which next hop or interface should I use?" -* **How Routes are Added:** - * **Directly Connected:** Added automatically when you configure an IP address on an interface (`proto kernel`). - * **Statically Configured:** Manually added by an admin (`ip route add ...`). - * **Dynamically Learned:** Added by routing protocols (BGP, OSPF) that routers use to talk to each other and share network information. +### 1.4 **Routing: The Path-Finding Algorithm** +- **The Routing Table:** Every device has one. + - `ip route show` displays yours. + - Answers: "To reach IP X, which next hop/interface should I use?" +- **How Routes are Added:** + - **Direct:** When you assign an IP. + - **Static:** Manually set by admins. + - **Dynamic:** Learned from routing protocols (BGP, OSPF) by routers. --- -## **2. The IP Packet Deep Dive (The L3 Envelope)** +## 2. **The IP Packet Deep Dive (The L3 Envelope)** -The segment from Layer 4 is encapsulated into an IP Packet. Its header is the critical information for delivery. +The Layer 4 segment is wrapped in an IP Packet. The header carries delivery-critical info. -| Field | Purpose | Why it Matters to You | +| Field | Purpose | Why it Matters | | :--- | :--- | :--- | -| **Version** | 4 or 6 | Determines which IP protocol stack to use. | -| **Source IP** | Origin Address | Used for reply routing. Can be spoofed (security risk!). | -| **Destination IP** | Final Address | The ultimate target the packet must reach. | -| **TTL (Time to Live)** | Hop Limit | Decremented by each router. Prevents infinite loops. A `traceroute` works by manipulating TTL. | -| **Protocol** | TCP (6), UDP (17), etc. | Tells the receiving host's OS which L4 protocol to hand the payload to. | -| **Flags / Fragment Offset** | Packet Fragmentation | If a packet is too large for a network link (MTU), it may be split into fragments. Often causes performance issues. | +| **Version** | 4 or 6 | IPv4 vs. IPv6 stack | +| **Source IP** | Sender | Used for replies, can be spoofed (security!) | +| **Destination IP** | Receiver | Where the packet is headed | +| **TTL** | Hop Limit | Decremented each router; prevents loops. `traceroute` manipulates TTL. | +| **Protocol** | TCP (6), UDP (17), etc. | Tells OS which L4 protocol to use. | +| **Flags/Fragment Offset** | Fragmentation | Large packets split—can cause issues.| ```mermaid graph LR @@ -80,96 +77,142 @@ C --> D[L2: Frame] --- -## **3. Layer 3 in Action: Devices & Their Cloud Equivalents** +## 3. **Layer 3 Devices & Cloud Equivalents** -| Concept | Physical World | Virtual/Cloud World | Key Purpose | +| Concept | Physical | Virtual/Cloud | Purpose | | :--- | :--- | :--- | :--- | -| **Router** | Cisco, Juniper Box | **VPC Route Table**, **Virtual Network Gateway** | Connect different IP networks and make forwarding decisions. | -| **L3 Switch** | Switch with routing features | (Same as Router) | High-speed, hardware-based routing within a data center. | -| **Firewall** | Palo Alto, Fortinet Box | **Security Groups (Host-Based)**, **NACLs (Subnet-Based)**, **Cloud Firewall** | Filter traffic based on L3 (IPs) and L4 (Ports) rules. | -| **Load Balancer** | F5 BIG-IP | **ALB/NLB (AWS), GLB (GCP), Load Balancer (Azure)** | Distribute traffic based on IP and port to backend targets. | -| **Gateway** | Internet Router | **Internet Gateway (IGW), NAT Gateway** | Provide a controlled path between private networks and the internet. | +| **Router** | Cisco/Juniper router | VPC Route Table, Virtual Gateway | Connect networks, forward packets | +| **L3 Switch** | Switch with routing | Same as Router | High-speed routing in datacenter | +| **Firewall** | Palo Alto/Fortinet | Security Groups, NACLs | Filter traffic at IP/port level | +| **Load Balancer** | F5 BIG-IP | ALB/NLB (AWS), GLB (GCP) | Distribute traffic to backends | +| **Gateway** | Internet Router | Internet Gateway, NAT Gateway | Connect private networks to Internet | --- -## **4. The SRE's Toolkit: Essential Linux Commands** +## 4. **SRE's Toolkit: Essential Linux Commands** -| Task | Command | Example & Output Snippet | +| Task | Command | Example Output | | :--- | :--- | :--- | -| **Show IP Config** | `ip addr show` or `ip a` | `inet 192.168.1.5/24 brd 192.168.1.255 scope global eth0` | -| **Show Routing Table** | `ip route show` or `ip r` | `default via 192.168.1.1 dev eth0 proto static`
`192.168.1.0/24 dev eth0 proto kernel scope link src 192.168.1.5` | -| **Test Reachability** | `ping ` | `64 bytes from 8.8.8.8: icmp_seq=1 ttl=118 time=15.3 ms` | -| **Trace the Path** | `traceroute `
or `tracepath` | `1: _gateway (192.168.1.1) 1.102ms`
`2: 100.65.18.177 (100.65.18.177) 9.123ms` | -| **Check Listening Ports** | `ss -tuln` | `tcp LISTEN 0 128 0.0.0.0:22 0.0.0.0:*`
`(SSH listening on all IPs)` | -| **Manage Routes** | `sudo ip route add `
`sudo ip route del ` | `sudo ip route add 10.10.0.0/24 via 192.168.1.254` | -| **Clear ARP Cache** | `ip neigh flush all` | Useful after changing network config to clear old MAC/IP mappings. | +| Show IP config | `ip addr show` | `inet 192.168.1.5/24 ...` | +| Show routing table | `ip route show` | `default via 192.168.1.1 dev eth0 ...` | +| Test reachability | `ping ` | `64 bytes from ...` | +| Trace packet path | `traceroute ` or `tracepath` | `1: _gateway ...` | +| Check open ports | `ss -tuln` | `tcp LISTEN ...` | +| Manage routes | `sudo ip route add/del ...` | `sudo ip route add 10.10.0.0/24 via 192.168.1.254` | +| Clear ARP cache | `ip neigh flush all` | (After config changes) | --- -## **5. Guided Hands-On Lab: Become Your Own Router** +## 5. **Guided Hands-On Lab: Become Your Own Router** ### **Lab Setup (VirtualBox/VMware)** -1. Create **three** VMs. -2. Set their network adapters to **Internal Network** (e.g., named `LAB_NET`). -3. Boot them up. They will have no IP addresses initially. +1. Create **three** VMs. +2. Set their network adapters to **Internal Network** (e.g., `LAB_NET`). +3. Boot them up (they’ll have no IP initially). ### **Step 1: Configure the "Router"** -* On VM1 (`router`), configure two virtual interfaces. - ```bash - sudo ip addr add 192.168.10.1/24 dev eth0 - sudo ip addr add 192.168.20.1/24 dev eth1 - ``` -* Enable IP forwarding on `router` (this turns a Linux host into a router). - ```bash - echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward - ``` +On VM1 (`router`): +```bash +sudo ip addr add 192.168.10.1/24 dev eth0 +sudo ip addr add 192.168.20.1/24 dev eth1 +echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward +``` ### **Step 2: Configure the "Clients"** -* On VM2 (`client-a`), set its IP and default gateway. - ```bash - sudo ip addr add 192.168.10.10/24 dev eth0 - sudo ip route add default via 192.168.10.1 - ``` -* On VM3 (`client-b`), set its IP and default gateway. - ```bash - sudo ip addr add 192.168.20.20/24 dev eth0 - sudo ip route add default via 192.168.20.1 - ``` +On VM2 (`client-a`): +```bash +sudo ip addr add 192.168.10.10/24 dev eth0 +sudo ip route add default via 192.168.10.1 +``` +On VM3 (`client-b`): +```bash +sudo ip addr add 192.168.20.20/24 dev eth0 +sudo ip route add default via 192.168.20.1 +``` ### **Step 3: Test the Connectivity** -* From `client-a`, ping `client-b`'s IP. - ```bash - ping 192.168.20.20 - ``` -* **It should work!** You have successfully built a routed network. The `router` VM received the ping from `client-a` on its `192.168.10.1` interface, checked its routing table, and forwarded it out the `192.168.20.1` interface to `client-b`. +From `client-a`: +```bash +ping 192.168.20.20 +``` +- If it works: you built a routed network! The router VM forwards between subnets. --- -## **6. Real-World Scenarios & Debugging** +## 6. **Layer 3 in Action – App, Proxy, & Load Balancer Troubleshooting** + +### **Mini Project: How Does My Traffic Flow?** + +**Setup:** +1. **Deploy Simple Apps:** + - Start two instances of a basic app (Python Flask, Node.js, or simply `python -m http.server`) on two different VMs/containers or on different ports. +2. **Configure NGINX as Load Balancer/Reverse Proxy:** + - On a third VM/host, install NGINX and configure it to proxy traffic to the two app backends. + +**Sample `nginx.conf` for reverse proxy:** +```nginx +http { + upstream myapp { + server 192.168.10.10:8000; + server 192.168.20.20:8000; + } + server { + listen 80; + location / { + proxy_pass http://myapp; + } + } +} +``` +3. **Test Access:** + - Use `curl` or a browser to access the app via NGINX (`http://`). + +**Experiment:** +4. **Simulate Failure:** + - Stop one backend app (`kill` the process or `docker stop`) and access via NGINX again. +5. **Debug Traffic Flow:** + - Use `ip route`, `ss -tuln`, `ping`, `traceroute`, and review NGINX logs. + - What does NGINX do when a backend is down? Does the traffic route to the healthy instance? + +**Questions:** +- How does NGINX know where to send traffic? +- What changes at Layer 3 when a backend is down? +- How do you localise the failure using network commands? +- How does this all appear in your troubleshooting outputs? + +**Submission:** +- Include: + - Your NGINX config + - Outputs from `ip route`, `ss -tuln`, and relevant logs + - A short analysis of what you observed and how Layer 3 knowledge helped you debug + +--- + +## 7. **Real-World Scenarios & Debugging** ### **Scenario 1: The Cloud VPC Mystery** **Problem:** A VM in `subnet-private` can't pull updates from the internet, but a VM in `subnet-public` can. -**Your Investigation:** -1. **Check Route Tables:** `subnet-private`'s route table likely has a default route (`0.0.0.0/0`) pointing to a **NAT Gateway**, not an Internet Gateway. An IGW gives a public IP; a NAT Gateway allows private instances to initiate outbound connections without being directly exposed. -2. **Check NACLs:** Is there a rule in `subnet-private`'s NACL allowing outbound traffic on ports 80/443 and the return ephemeral ports (1024-65535)? -3. **Check Security Groups:** Does the SG attached to the private VM allow outbound traffic to `0.0.0.0/0`? +**Investigation:** +1. **Check Route Tables:** Is the default route in `subnet-private`’s table pointing to a **NAT Gateway**? +2. **Check NACLs/Security Groups:** Is outbound traffic (80/443, 1024-65535) allowed? +3. **Check if the subnet has a route to an Internet Gateway or NAT?** ### **Scenario 2: The Hybrid Cloud Tunnel** -**Problem:** Your on-premise application (`10.10.10.5`) cannot connect to a cloud database (`10.20.30.5`) after a maintenance window. +**Problem:** On-prem app (`10.10.10.5`) cannot reach cloud DB (`10.20.30.5`) after maintenance. -**Your Investigation:** -1. **Traceroute is your friend:** `traceroute 10.20.30.5` from on-prem. Where does it stop? -2. **Check the VPN/Direct Connect:** Is the BGP session up? Are the routes being advertised correctly from the cloud side to your on-premise router? -3. **Check Firewall Rules:** Have the ACLs on the cloud VPC or on-premise firewall been modified? Is traffic allowed bidirectionally on the database port? +**Investigation:** +1. **Traceroute:** Where does it stop? +2. **VPN/Direct Connect:** Is BGP up, routes advertised? +3. **Firewalls:** Are ACLs allowing the right traffic? --- -## **7. Mini Incident Simulation** +## 8. **Mini Incident Simulation** -**Alert:** `Web servers in availability zone 1a are failing health checks from the load balancer in 1b.` +**Alert:** Web servers in availability zone 1a fail health checks from the load balancer in 1b. -You jump into a web server and find: +You check a web server: ```bash $ ip addr show eth0 inet 10.0.1.25/20 brd 10.0.15.255 scope global eth0 @@ -178,38 +221,50 @@ $ ip route show 10.0.0.0/20 dev eth0 proto kernel scope link src 10.0.1.25 default via 10.0.0.1 dev eth0 ``` - The load balancer's IP is `10.0.16.105`. -**Your Tasks:** -1. **Calculate the subnets.** The server is in `10.0.0.0/20`. What is the range of this subnet? *(Hint: The next subnet is `10.0.16.0/20`)*. -2. **Diagnose the issue:** Based on the subnet, is the load balancer's IP in the same subnet as the web server? -3. **Explain the consequence:** Where will the web server send packets destined for the load balancer? Can they reach it? -4. **Propose the fix:** This is a VPC design error. What should the netmask have been to allow all resources in the VPC to communicate directly? +**Tasks:** +1. Calculate the subnet range for `10.0.0.0/20`. Is the LB in the same subnet? +2. Where will the web server send packets destined for the LB? +3. Can packets reach it directly? Why or why not? +4. Propose a fix so all resources are routable without a router. --- -## **8. Advanced Gotchas for SREs** +## 9. **Advanced Gotchas for SREs** -* **MTU & Fragmentation:** If your VPN or overlay network (like Docker's) has a smaller MTU than your underlying network, packets will fragment, crushing performance. Solution: Enable **MTU Path Discovery** or set the MTU manually. -* **Asymmetric Routing:** The request goes through Firewall A, but the response comes back through Firewall B. Since Firewall B didn't see the connection initiation, it drops the packet. Common in multi-cloud and complex routing setups. -* **ECMP (Equal-Cost Multi-Path):** Multiple paths to the same destination. Great for bandwidth, terrible for stateful firewalls and packet ordering. -* **Network Namespaces:** Modern orchestration (Kubernetes, Docker) uses network namespaces to create isolated network stacks. A `ping` from the host might work, but not from inside a container because it's in a different namespace with its own routes and interfaces. Use `nsenter` or `kubectl exec` to debug. +- **MTU & Fragmentation:** VPNs or overlays with smaller MTU can fragment packets, hurting performance. Enable MTU Path Discovery or set MTU manually. +- **Asymmetric Routing:** Requests go through firewall A, responses out firewall B—B may drop them. Beware in multi-path topologies. +- **ECMP:** Multiple equal-cost paths can confuse stateful firewalls and impact packet order. +- **Network Namespaces:** Containers/pods have isolated stacks. A `ping` from the host may work, but not from inside a container due to separate routing tables. Debug with `nsenter` or `kubectl exec`. --- -## **9. Submission Guidelines: Prove Your Mastery** +## 10. **Submission Guidelines: Prove Your Mastery** -Push a file named `day17solution.md` to your GitHub repo. +**Push `day17solution.md` to your GitHub repo.** **Include:** +1. **Screenshots:** Output of `ip a` and `ip r` from your main machine. +2. **Hands-on Proof:** Output from the "Become Your Own Router" lab showing successful ping. +3. **App & LB Lab:** NGINX config, command outputs, and a brief analysis of your troubleshooting. +4. **Incident Response:** Detailed answers for the Mini Incident Simulation. +5. **Reflection:** A paragraph on a networking issue you’ve faced (or might face) and how Layer 3 knowledge helps. -1. **Screenshots:** Output of the `ip a` and `ip r` commands from your main machine. -2. **Hands-on Proof:** Output from the "Become Your Own Router" lab showing a successful ping between the two client VMs. -3. **Incident Response:** Your detailed answers to the four questions in the **Mini Incident Simulation** (Section 7). -4. **Reflection:** Write a paragraph explaining a networking issue you've faced or anticipate facing in your work and how a deep understanding of Layer 3 would help you resolve it. - -**Share your journey:** +**Share your journey:** `#getfitwithsagar #100DaysOfSRE #OSIMastery #Layer3DeepDive #CloudNetworking` --- + +## 11. **Community & Support** + +- [Discord](https://discord.gg/mNDm39qB8t) +- [Google Group](https://groups.google.com/forum/#!forum/daily-devops-sre-challenge-series/join) +- [YouTube](https://www.youtube.com/@Sagar.Utekar) + +--- + +**Master Layer 3, and you’ll never be lost in the network again!** + +Happy learning and exploring, +Sagar Utekar From 39e390aa6d020cc10e3e711fdbeb78db81c4cb68 Mon Sep 17 00:00:00 2001 From: Sagar Utekar Date: Tue, 9 Sep 2025 13:53:16 +0530 Subject: [PATCH 114/133] Update challenge.md --- DailyChallenges/Season2/Day18/challenge.md | 324 +++++++++------------ 1 file changed, 145 insertions(+), 179 deletions(-) diff --git a/DailyChallenges/Season2/Day18/challenge.md b/DailyChallenges/Season2/Day18/challenge.md index 1f0b112..97465db 100644 --- a/DailyChallenges/Season2/Day18/challenge.md +++ b/DailyChallenges/Season2/Day18/challenge.md @@ -1,239 +1,205 @@ +# **Day 18: OSI Layer 4 – The Transport Layer (TCP, UDP, Ports, and Real-World Debugging)** --- -# **Day 18 – Firewalls & Security 🔥** +## **Why Layer 4 is a Game Changer for SREs & DevOps** ---- - -## 📌 Introduction - -Hello Learners, - -Welcome to **Day 18 of the Daily DevOps + SRE Challenge Series – Season 2!** 🚀 - -Today, we’re diving into one of the **pillars of system and network security** — **firewalls, security groups, and TLS/SSL**. You’ll understand how Linux firewalls protect your servers, how cloud platforms enforce access policies, and how encryption safeguards your data in transit. - -This challenge will sharpen your **network defense skills** so you can prevent intrusions, lock down access, and troubleshoot connectivity issues like a battle-hardened SRE. - ---- - -## 🛠️ Lab Setup Instructions - -To complete the hands-on challenges, you will need an environment to work in. Choose one of the following setups: - -**Option A: Local Virtual Machine (Recommended)** -1. **Software:** Install VirtualBox or VMware. -2. **OS Image:** Download a Linux ISO (e.g., Ubuntu Server 22.04 LTS). -3. **Setup:** - * Create a new VM with at least 1 CPU, 2GB RAM, and 20GB disk space. - * Install the OS. During installation, ensure you install the `openssh-server` package. - * Once installed, boot the VM and note its IP address (`ip a`). - * You can now SSH into your VM from your host machine for practice. - -**Option B: Cloud Instance (e.g., AWS EC2, DigitalOcean Droplet)** -1. **Create an Account:** Sign up for a cloud provider that offers a free tier (AWS, GCP, Azure, DigitalOcean). -2. **Launch a Instance:** Create a new virtual machine instance (e.g., an AWS EC2 t2.micro instance with Ubuntu 22.04). -3. **Security Groups:** During setup, note the Security Group settings. For now, allow SSH (port 22) from your IP and All ICMP - IPv4 (for ping). -4. **Access:** Use the provided key pair to SSH into your instance. - -**Post-Setup (For both options):** -* Update your package lists: `sudo apt update && sudo apt upgrade -y` -* Install necessary tools: `sudo apt install -y iptables-persistent nginx tcpdump` (Confirm 'Yes' to save current IPv4 rules when prompted). - ---- - -## 🚀 Why it matters - -* **Security First:** Firewalls and TLS form the **first line of defense** against attacks. -* **Reliability:** A single wrong iptables rule can cut off production traffic — knowing how to **debug and fix quickly** is essential. -* **Cloud-native DevOps:** Security groups are a must-know for AWS, GCP, Azure engineers. -* **Compliance:** Most orgs require **encryption in transit (TLS/SSL)** and restricted access. -* **Interview Edge:** Expect “What’s the difference between iptables and security groups?” or “How does TLS handshake work?” in SRE/DevOps interviews. +Once packets have their addresses (Layer 3), they need to get delivered **reliably** (or not!), in the right **order** (or not!), and to the correct **application**. That’s **Layer 4**: the Transport Layer. +Here, you’ll unlock the secrets behind TCP and UDP, ports, connection states, and how to debug the issues that puzzle even experienced engineers. ---- - -## 🔥 Real-world save stories - -* An SRE accidentally blocked SSH for the entire engineering team with a wrong `iptables -F` — recovery was only possible because a **cloud console out-of-band access** was enabled. -* A fintech startup avoided **man-in-the-middle attacks** by moving from HTTP to **TLS with Let’s Encrypt**, cutting down risks overnight. -* At scale, switching from **stateless packet filtering to stateful inspection** reduced false positives and prevented accidental drops of long-lived connections. +> “It’s always the firewall… unless it’s the ephemeral port range, or TCP state bloat, or a rogue UDP flood. Layer 4 is where the real fun (and pain!) starts.” --- -## 📘 Theory with Mermaid diagrams +## 1. **Core Concepts Simplified (All-in-One Table)** -### 🔹 Firewall Workflow +| Topic / Protocol | TCP | UDP | +|---------------------- |-----------------------------------------------------------|---------------------------------------| +| **Type** | Reliable, connection-oriented | Unreliable, connectionless | +| **Guarantees** | Delivery, order, integrity | No guarantees | +| **Handshake?** | Yes (SYN, SYN-ACK, ACK) | No | +| **Teardown?** | Yes (FIN/ACK) | No | +| **Overhead** | Higher (more checks, slower) | Lower (faster, less checks) | +| **Common Uses** | Web, SSH, SMTP, DB | DNS, VoIP, games, streaming | +| **Connection State?** | Yes (tracks state, can get stuck) | No | +| **Good For** | Anything needing accuracy and order | Real-time, stateless, speed | -```mermaid -flowchart TD - Client[Client Request] --> FW[Firewall Rules] - FW -->|Allowed| Server[Application Server] - FW -->|Blocked| Drop[Packet Dropped 🚫] -``` - -**Detailed Explanation:** -A firewall is a network security system that monitors and controls incoming and outgoing network traffic based on predetermined security rules. It acts as a barrier between a trusted internal network and untrusted external networks, like the Internet. - -* **Packet Filtering:** The core function. The firewall inspects each **packet** of data (looking at headers for source/destination IP, port, and protocol) and compares it against a set of **rules**. -* **Rule Evaluation:** Rules are processed in order. A packet is checked against the first rule; if it matches, the specified action (e.g., `ACCEPT`, `DROP`) is taken. If not, it moves to the next rule. If no rules match, a default policy (usually `DROP`) is applied. -* **Actions:** - * **ACCEPT:** Allows the packet to proceed. - * **DROP:** Silently discards the packet as if it never arrived. This is more secure than rejecting. - * **REJECT:** Discards the packet but sends an error message (e.g., a TCP reset packet) back to the sender. This is more polite but reveals the existence of the firewall. +> **Tip:** +> - Use TCP for reliability (web, SSH, DB). +> - Use UDP for speed (DNS, games). +> - Ports help traffic reach the right app. --- -### 🔹 Stateful vs Stateless Filtering - +## 2. **TCP Three-Way Handshake (Connection Establishment)** ```mermaid sequenceDiagram participant Client - participant Firewall participant Server - Client->>Firewall: TCP SYN (New Connection) - Firewall->>Server: Allow (if rule matches) - Server->>Firewall: TCP SYN-ACK - Firewall->>Client: Allow (stateful remembers connection) + Client->>Server: SYN + Server->>Client: SYN-ACK + Client->>Server: ACK + Note over Client,Server: Connection Established ``` -**Detailed Explanation:** -This diagram highlights the critical difference between stateful and stateless firewalls. - -* **Stateless Firewalls (Packet Filters):** - * **How they work:** They examine each packet in isolation, with no memory of previous packets. A rule like `ACCEPT TCP port 22` would allow *any* packet destined for port 22, whether it's a new connection attempt or a response to a previous one. - * **The Problem:** To allow legitimate responses, you'd have to open ports for all *potential* return traffic, creating a larger attack surface. They are simple and fast but lack context, making them vulnerable to spoofing attacks and less secure. - -* **Stateful Firewalls (The Modern Standard):** - * **How they work:** They maintain a **state table** that tracks the state of all active connections (e.g., `NEW`, `ESTABLISHED`, `RELATED`). This is the "memory" shown in the diagram. - * **The Advantage:** The firewall understands if a packet is starting a new connection or is part of an existing one. A simple rule like `ACCEPT ESTABLISHED,RELATED` traffic can allow return traffic for any outgoing connection, without needing specific rules for every return port. This is more secure and easier to manage. For example, if an internal client connects to a web server, the firewall automatically allows the return packets (SYN-ACK, etc.) back through, even if the default policy is to `DROP` incoming traffic. - ---- - -### 🔹 TLS Handshake Simplified (Corrected) -*Fixed the sequence to accurately reflect the key exchange and cipher setup.* +### ** Layer 4 Packet Filtering** ```mermaid -sequenceDiagram - participant C as Client - participant S as Server - - Note over C, S: TLS Handshake - - C->>S: ClientHello (Supported Ciphers, Client Random) - S->>C: ServerHello (Selected Cipher, Server Random)
Server Certificate - Note right of C: Client validates certificate - C->>S: PreMaster Secret
(encrypted with Server's public key) - Note over S, C: Both sides generate
Session Keys from
ClientRandom, ServerRandom,
and PreMaster Secret - C->>S: ChangeCipherSpec (Finished) - S->>C: ChangeCipherSpec (Finished) - - Note over C, S: Secure Session Established 🔐
Encrypted Application Data +flowchart TD + Internet[Internet Traffic] --> FW[Firewall :Layer 4] + FW -->|Allow TCP 80/443| Web[Web Server] + FW -->|Allow TCP 22| SSH[SSH Server] + FW -.->|Block UDP 53| DNS[DNS Server] ``` -**Detailed Explanation:** -The Transport Layer Security (TLS) handshake is a critical process that enables secure communication over a network by authenticating the server (and optionally the client) and establishing a shared secret key for symmetric encryption. - -1. **`ClientHello`:** The client initiates the handshake by sending a message containing the TLS versions it supports, a list of suggested cipher suites (cryptographic algorithms), and a random string of bytes (`Client Random`). +--- -2. **`ServerHello` & Certificate:** The server responds with its chosen TLS version and cipher suite from the client's list. It also sends its own `Server Random` and its **SSL Certificate**. This certificate contains the server's public key and is signed by a trusted Certificate Authority (CA), allowing the client to verify the server's identity. +## 3. **Layer 4 in the Cloud & Modern Architectures** -3. **Key Exchange (PreMaster Secret):** The client verifies the server's certificate. If trusted, it generates a `PreMaster Secret`, encrypts it with the server's public key (from the certificate), and sends it back. **Only the server possessing the corresponding private key can decrypt this secret.** +- **Security Groups & Firewalls:** + Control access by IP **and** port. Layer 4 rules are often the first line of defense. +- **Load Balancers:** + - **L4 load balancer:** Routes based on IP:port (TCP/UDP). Faster, less aware of app logic. + - **L7 (Application) load balancer:** Routes based on HTTP, cookies, etc. +- **NAT & Port Mapping:** + Translate internal IP:port to public IP:port. Essential for internet access and microservices. -4. **Session Keys Generated:** Both the client and server now independently generate the same **session keys** (symmetric keys for encryption/decryption) using the `Client Random`, `Server Random`, and `PreMaster Secret`. The PreMaster Secret is immediately discarded on both sides. This ensures **Forward Secrecy**. +--- -5. **`ChangeCipherSpec` & `Finished`:** Both parties send a `ChangeCipherSpec` message to signal that subsequent messages will be encrypted with the newly established session keys. A `Finished` message is sent encrypted to verify that the handshake was successful and the keys are working correctly. +## 4. **SRE’s Command-Line Toolkit** -6. **Secure Communication:** The handshake is complete. All subsequent application data (HTTP, etc.) is encrypted and authenticated using the symmetric session keys, ensuring **confidentiality** and **integrity**. +| Task | Command | Example Output | +|------|---------|---------------| +| List connections | `ss -tuln` or `netstat -tuln` | Show listening TCP/UDP ports | +| Show connection states | `ss -s` or `netstat -an` | SYN_SENT, ESTABLISHED, etc. | +| Check open ports | `lsof -i :PORT` | Processes using a port | +| Test TCP | `nc -vz HOST PORT` | Success/failure of connection | +| Test UDP | `nc -vu HOST PORT` | Send/receive data | +| Capture packets | `sudo tcpdump -i eth0 port 80` | Live packet dump | +| Analyze with Wireshark | Open `.pcap` | Visualize TCP handshakes, retransmits | --- -## ⚡ Hands-on challenges (with commands + troubleshooting) +## 5. **Hands-On Lab: Layer 4 in Action** -### **1. Packet Filtering vs Stateful Inspection** +### **Lab 1: TCP/UDP Connections with Netcat** -1. Explain difference between **stateless vs stateful firewalls** in your own words. -2. Block all ICMP traffic using `sudo iptables -A INPUT -p icmp -j DROP` and test with `ping`. -3. Allow only **established SSH connections** with: +**A. TCP Echo Test** +1. On one terminal, run a TCP server: + ```bash + nc -l 9000 + ``` +2. On another terminal, connect as a client: + ```bash + nc 127.0.0.1 9000 + ``` +3. Type messages – verify bidirectional communication. +4. Observe connection with: ```bash - sudo iptables -A INPUT -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT + ss -tna | grep 9000 ``` -4. Use `sudo tcpdump -i any icmp` or `sudo tcpdump -i any port 22` to observe packets and analyze behavior. -5. Document **pros/cons** of stateless vs stateful filtering. + +**B. UDP Echo Test** +1. On one terminal, run a UDP server: + ```bash + nc -lu 9001 + ``` +2. On another terminal, send a UDP message: + ```bash + echo "hello" | nc -u 127.0.0.1 9001 + ``` +3. UDP is stateless—observe the difference in `ss -una | grep 9001`. --- -### **2. iptables / nftables Basics** +### **Lab 2: TCP Handshake & State Machine with tcpdump** -1. Flush existing rules (be careful!): `sudo iptables -F` -2. Set default policy to ACCEPT for now: `sudo iptables -P INPUT ACCEPT` -3. Now, set a default DROP policy and allow only SSH (22) and HTTP (80): +1. In one terminal, start a simple TCP server (e.g., Python): + ```bash + python3 -m http.server 8080 + ``` +2. In another, start packet capture: ```bash - sudo iptables -P INPUT DROP - sudo iptables -A INPUT -p tcp -m multiport --dports 22,80 -j ACCEPT - sudo iptables -A INPUT -j DROP + sudo tcpdump -i lo tcp port 8080 -w handshake.pcap + ``` +3. In a third, connect: + ```bash + curl http://localhost:8080 + ``` +4. Stop tcpdump. Open `handshake.pcap` in Wireshark and analyze: + - Find SYN, SYN-ACK, ACK packets (3-way handshake) + - See FIN/ACK for teardown + +5. List TCP connection states: + ```bash + ss -tan state all | grep 8080 ``` -4. Save and persist rules with `sudo iptables-save | sudo tee /etc/iptables/rules.v4`. -5. Convert the same config into **nftables syntax** (research or use `iptables-translate`). -6. Flush all rules using `sudo iptables -F` → observe what breaks (troubleshooting!). -7. Compare performance and future-readiness of iptables vs nftables. --- -### **3. Security Groups (Cloud)** +## 6. **Mini Incident Simulation** -1. Create a **Security Group** that allows SSH only from your IP. -2. Block all **outbound traffic** → test how apps fail when they need outbound internet. -3. Allow HTTP/HTTPS only → test with `curl http://` and `curl https://`. -4. Explain how **Security Groups differ from host firewalls**. -5. Design a **3-tier architecture Security Group model**: - * Web → App → DB, with only necessary ports open. +**Scenario:** +Your web app is intermittently failing to connect to its database. The app logs show `Connection refused` and `Too many open files`. +You run: + +```bash +ss -s +ss -tan | grep 5432 +ulimit -n +``` +- You see hundreds of connections in `TIME_WAIT` to port 5432, and `ulimit -n` is 1024. + +**Tasks:** +1. What is `TIME_WAIT` and why does it happen? +2. How can too many connections in `TIME_WAIT` cause failures? +3. What are two ways to mitigate this issue (at the OS/app level)? +4. What Layer 4 lesson does this teach for SREs? --- -### **4. TLS/SSL Basics** +## 7. **Common SRE Pitfalls & Gotchas** -1. Generate a **self-signed certificate** with: - ```bash - sudo openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout /etc/ssl/private/nginx-selfsigned.key -out /etc/ssl/certs/nginx-selfsigned.crt - ``` -2. Configure Nginx to serve HTTPS using the cert. Edit `/etc/nginx/sites-available/default`: - ```nginx - server { - listen 443 ssl; - ssl_certificate /etc/ssl/certs/nginx-selfsigned.crt; - ssl_certificate_key /etc/ssl/private/nginx-selfsigned.key; - root /var/www/html; - index index.html index.htm; - } - ``` -3. Restart Nginx: `sudo systemctl restart nginx` -4. Verify cert details with: - ```bash - echo | openssl s_client -connect localhost:443 2>/dev/null | openssl x509 -noout -text - ``` -5. Draw the **TLS Handshake** sequence with a Mermaid diagram. -6. Compare **Let’s Encrypt vs Commercial Certs** – when would you choose each? +- **Ephemeral Port Exhaustion:** + Too many outgoing connections can exhaust source ports. Tweak ephemeral range and app keepalive settings. +- **Firewall/Security Group Rules:** + Ensure correct protocol (TCP vs. UDP) and port are allowed—port 53/UDP ≠ port 53/TCP! +- **Stateless vs. Stateful Load Balancing:** + Sticky sessions can break if not handled at Layer 4. +- **NAT & Connection Tracking:** + NAT device may drop idle connections or run out of state table slots. +- **Socket Options:** + Tuning `tcp_tw_reuse`, `tcp_fin_timeout`, and `ulimit` can help, but can also cause subtle bugs. --- -## ✅ Deliverables +## 8. **Submission Guidelines** -* Create `solution.md` documenting: - * Your firewall configs (`iptables/nftables`). - * Security group screenshots or rules. - * TLS cert commands + verification outputs. - * Mermaid diagrams. -* Push the file to your GitHub repo. -* Share your learnings on social media with: - **#getfitwithsagar #SRELife #DevOpsForAll** +**Push a file named `solution.md` to your repo. Include:** + +1. **Screenshots/outputs** of: + - `ss -tuln` and `ss -s` or `netstat -an` + - Your netcat and tcpdump/Wireshark experiments +2. **Mini Incident Simulation:** + - Your answers and reasoning +3. **Reflection:** + - Share a real or hypothetical Layer 4 troubleshooting story and how this knowledge would help you. + +**Share your journey:** +`#getfitwithsagar #DevOpsChallenge` --- -## 🌍 Community links +## 9. **Community & Support** -* Discord: [Join Here](https://discord.gg/mNDm39qB8t) -* Google Group: [Subscribe Here](https://groups.google.com/forum/#!forum/daily-devops-sre-challenge-series/join) -* YouTube: [Watch Tutorials](https://www.youtube.com/@Sagar.Utekar) +- [Discord](https://discord.gg/mNDm39qB8t) +- [Google Group](https://groups.google.com/forum/#!forum/daily-devops-sre-challenge-series/join) +- [YouTube](https://www.youtube.com/@Sagar.Utekar) --- + +**Master Layer 4, and you’ll unlock a new level of systems intuition. Every outage, fix, and performance boost starts making sense.** + +Happy hacking and troubleshooting, +Sagar Utekar From 543d2c4ca64e0a384d93ee308d1af71cede4cc4d Mon Sep 17 00:00:00 2001 From: Sagar Utekar Date: Tue, 9 Sep 2025 14:15:52 +0530 Subject: [PATCH 115/133] Create challenge.md --- DailyChallenges/Season2/Day19/challenge.md | 445 +++++++++++++++++++++ 1 file changed, 445 insertions(+) create mode 100644 DailyChallenges/Season2/Day19/challenge.md diff --git a/DailyChallenges/Season2/Day19/challenge.md b/DailyChallenges/Season2/Day19/challenge.md new file mode 100644 index 0000000..8c271cc --- /dev/null +++ b/DailyChallenges/Season2/Day19/challenge.md @@ -0,0 +1,445 @@ +# **Day 19: Firewalls, Security Groups & TLS/SSL – Defending Modern Infrastructure** + +--- + +## Why Day 19 Matters + +You can design the cleanest microservice architecture, but without proper **traffic control (firewalls & security groups)** and **secure channel establishment (TLS/SSL)**, your stack is a soft target. +Today you’ll learn how packets are permitted, denied, shaped, inspected, and encrypted across **Linux hosts, cloud boundaries, and client–server connections**. + +> “Performance issues are annoying. Misconfigured firewalls and expired TLS certs are outages.” + +--- + +## 1. Core Concepts (At a Glance) + +| Concept | What It Is (Simple) | Why You Care | +|--------|----------------------|--------------| +| Firewall | Traffic filter (allow/deny based on rules) | Prevents unwanted access | +| Stateless Filter | Checks each packet independently | Fast but dumb (e.g., classic ACL) | +| Stateful Firewall | Tracks connection state (established, related) | Smarter, less rules needed | +| Security Group (Cloud) | Virtual, stateful firewall at instance ENI level | Controls what talks in/out | +| NACL (Cloud) | Subnet-level stateless rule set | Coarse filtering, both directions | +| WAF | Layer 7 (HTTP) attack filter | Stops SQLi, XSS, bots | +| TLS/SSL | Protocol for encrypted channels | Protects data + identity | +| Certificate | Digital ID for a domain/service | Trust & authenticity | +| mTLS | Both sides present certs | Zero-trust & service-to-service auth | +| Cipher Suite | Algorithms for key exchange + encryption + MAC | Affects performance & security | +| PFS | New ephemeral key per session | Limits blast radius of key leakage | + +--- + +## 2. Firewalls Explained + +### 2.1 What Is a Firewall? +A firewall enforces **policy**: what traffic is allowed or denied based on attributes like protocol, source, destination, port, interface, or connection state. + +### 2.2 Layers +- Layer 3/4: IP, protocols, ports (iptables/nftables, Security Groups) +- Layer 7: HTTP semantics (WAF, reverse proxies) +- Deep Packet Inspection: Payload-level inspection (IDS/IPS) + +### 2.3 Packet Flow (Simplified Linux Host) + +```mermaid +flowchart LR + IN_Int[Incoming Packet :eth0] --> PREROUTING + PREROUTING -->|Local Dest?| INPUT + PREROUTING -->|Forward?| FORWARD + INPUT --> LocalProc[Local Process] + LocalProc --> OUTPUT + OUTPUT --> POSTROUTING + FORWARD --> POSTROUTING + POSTROUTING --> OUT_Int[Outgoing Interface] +``` + +### 2.4 Stateless vs Stateful + +| Aspect | Stateless (e.g., NACL, raw ACL) | Stateful (iptables conntrack, SG) | +|--------|---------------------------------|-----------------------------------| +| Tracks Connections? | No | Yes | +| Requires Reverse Rule? | Yes | No (auto allows reply) | +| Performance | Slightly faster for simple rules | Near parity with modern kernels | +| Use Case | Edge routers, simple filtering | Hosts, cloud workloads | + +### 2.5 iptables vs nftables vs firewalld + +| Feature | iptables | nftables | firewalld | +|---------|----------|----------|-----------| +| Status | Legacy (still everywhere) | Modern replacement | Abstraction layer | +| Performance | Per-table / chain evaluation | Unified flow engine | Depends on backend | +| Rule Management | Append oriented | Atomic replace | Zones & services | +| Recommended? | Legacy maintenance | YES (future-proof) | For ease if not scripting | + +--- + +## 3. Cloud Security Constructs + +| Control | Layer | Stateful? | Scope | Typical Use | +|---------|-------|-----------|-------|-------------| +| Security Group | L3/L4 | Yes | ENI / Instance | Instance ingress/egress micro-perimeter | +| NACL | L3/L4 | No | Subnet | Baseline allow/deny | +| WAF | L7 | N/A | HTTP/S endpoints | App-layer attack filtering | +| Shield / DDoS | L3–7 | Managed | Edge | Volumetric protection | +| Host Firewall | L3/L4 | Yes | OS Kernel | Defense-in-depth | + +### 3.1 Security Group Statefulness (Inbound → Outbound Return) + +```mermaid +sequenceDiagram + participant Client + participant SG as Security Group (Stateful) + participant Server + Client->>SG: SYN (TCP 443) + SG->>Server: SYN (Allowed by inbound rule) + Server->>SG: SYN-ACK (Return allowed via state table) + SG->>Client: SYN-ACK + Client->>SG: ACK (Established) + Note over SG: Reverse flow auto-permitted +``` + +--- + +## 4. TLS / SSL Essentials + +### 4.1 What TLS Provides +| Guarantee | Meaning | +|-----------|---------| +| Confidentiality | Encrypted payload | +| Integrity | Tamper detection | +| Authentication | Identity via certificates | +| (Optional) Client Auth | mTLS for mutual identity | + +### 4.2 TLS 1.2 Handshake (Simplified) + +```mermaid +sequenceDiagram + participant C as Client + participant S as Server + C->>S: ClientHello (ciphers, random) + S->>C: ServerHello (chosen cipher) + S->>C: Certificate (and chain) + S->>C: ServerKeyExchange (if needed) + S->>C: ServerHelloDone + C->>S: ClientKeyExchange + C->>S: ChangeCipherSpec + C->>S: Finished + S->>C: ChangeCipherSpec + S->>C: Finished + Note over C,S: Secure channel established +``` + +### 4.3 TLS 1.3 Differences (Faster, More Secure) + +```mermaid +sequenceDiagram + participant C as Client + participant S as Server + C->>S: ClientHello [includes key shares] + S->>C: ServerHello [select key] + EncryptedExtensions + Certificate + Finished + C->>S: Finished + Note over C,S: 1-RTT [0-RTT possible for resumption] +``` + +### 4.4 Certificate Chain + +```mermaid +graph TD + A[Root CA] --> B[Intermediate CA] + B --> C[Server Certificate] + C --> D[Server :nginx] + E[Client Browser] -->|Trusts Root or Intermediate| A +``` + +### 4.5 Cipher Suite Anatomy (TLS 1.2 Example) +`TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384` +- ECDHE: Key exchange (PFS) +- RSA: Authentication +- AES_256_GCM: Bulk cipher (AEAD) +- SHA384: Hash for PRF/integrity + +### 4.6 TLS 1.2 vs 1.3 + +| Aspect | TLS 1.2 | TLS 1.3 | +|--------|---------|---------| +| Handshake RTT | 2 | 1 (0 with resumption) | +| Cipher Suites | Many (legacy risk) | Streamlined / only AEAD | +| Key Exchange | RSA / (EC)DHE | Always (EC)DHE (PFS enforced) | +| Algorithms Removed | RC4, SHA1, etc. | Already gone | +| Session Resumption | Session IDs / Tickets | Tickets / PSK | +| Security Level | Good if hardened | Better defaults | + +--- + +## 5. Certificates – Types & Use Cases + +| Type | Use | Pros | Cons | +|------|-----|------|------| +| Self-Signed | Dev/testing | Fast | No external trust | +| Internal CA | Internal services | Control | Requires internal trust distro | +| Let’s Encrypt | Public sites | Free, automated | 90-day renewal | +| Wildcard | Many subdomains | Flexible | Risk if private key leaked | +| EV | High-profile public orgs | Strong validation signal | Cost / diminishing value | +| mTLS Certs | Service-to-service | Strong mutual auth | Operational overhead | + +--- + +## 6. mTLS (Mutual TLS) + +Both sides present a certificate. Use cases: +- Service Mesh (Istio, Linkerd) +- gRPC microservice auth +- Zero-trust internal APIs + +Flow: +1. TLS handshake begins +2. Server sends Certificate +3. Server requests client Certificate +4. Client sends Certificate + proves possession of private key +5. Both validate chain → secure + mutually authenticated + +--- + +## 7. Hands-On Labs (Follow in Order) + +### Lab 1: Inspect Current Firewall State (Linux) + +```bash +sudo iptables -L -n -v +sudo iptables -t nat -L -n -v +sudo nft list ruleset +sudo ss -tulpen +``` + +Questions: +1. Which ports are open? +2. Any default ACCEPT policies? (Risky on exposed hosts.) + +--- + +### Lab 2: Basic iptables Rule Set + +Goal: Allow SSH (22), HTTP (80), HTTPS (443); drop everything else inbound; allow all outbound. + +```bash +# WARNING: If remote, keep an existing SSH session open to recover. +sudo iptables -P INPUT DROP +sudo iptables -P FORWARD DROP +sudo iptables -P OUTPUT ACCEPT + +# Allow loopback +sudo iptables -A INPUT -i lo -j ACCEPT + +# Allow established/related +sudo iptables -A INPUT -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT + +# Allow SSH, HTTP, HTTPS +sudo iptables -A INPUT -p tcp -m multiport --dports 22,80,443 -j ACCEPT + +# View counters +sudo iptables -L -n -v +``` + +Persist (varies by distro): +```bash +sudo sh -c 'iptables-save > /etc/iptables/rules.v4' +``` + +Task: +- Try connecting to a random blocked port (e.g., 8080) → should fail. + +--- + +### Lab 3: Security Group Design (3-Tier Example) + +Architecture: +- Public ALB (443) +- Web Tier (EC2 behind ALB) +- App Tier (private) +- DB Tier (private) + +Rules (Simplified): + +| Tier | Inbound Allowed | Outbound Allowed | +|------|-----------------|------------------| +| ALB | 0.0.0.0/0 → :443 | Web SG → :443 | +| Web SG | ALB SG → :80/:443 | App SG → :8080; 0.0.0.0/0 DNS/NTP as needed | +| App SG | Web SG → :8080 | DB SG → :5432 | +| DB SG | App SG → :5432 | (Optional) Backup host SG | + +Task: +- Document in a diagram (draw.io / mermaid). +- Justify: Why no direct internet to App/DB? + +Mermaid Flow: + +```mermaid +flowchart LR + User(Internet User) -->|HTTPS 443| ALB[Public ALB] + ALB -->|HTTP 80 / 443| WEB[Web EC2 :Web SG] + WEB -->|TCP 8080| APP[App EC2 :App SG] + APP -->|TCP 5432| DB[(Database :DB SG)] +``` + +--- + +### Lab 5: Generate Self-Signed TLS Cert & Use with NGINX + +```bash +mkdir -p ~/tlslab && cd ~/tlslab +openssl req -x509 -newkey rsa:4096 -sha256 -days 90 -nodes \ + -keyout server.key -out server.crt -subj "/CN=localhost" + +sudo apt-get install -y nginx +sudo tee /etc/nginx/sites-available/tlsdemo <<'EOF' +server { + listen 443 ssl; + server_name localhost; + ssl_certificate /home/$USER/tlslab/server.crt; + ssl_certificate_key /home/$USER/tlslab/server.key; + + add_header X-App "Day19Demo"; + location / { + return 200 "Secure Hello (TLS Demo)\n"; + } +} +EOF + +sudo ln -s /etc/nginx/sites-available/tlsdemo /etc/nginx/sites-enabled/tlsdemo +sudo nginx -t && sudo systemctl reload nginx +``` + +Test: +```bash +curl -vk https://localhost +openssl s_client -connect localhost:443 -servername localhost -tls1_2 /dev/null | openssl x509 -noout -dates` | +| Quick server (dev TLS) | `python3 -m http.server --bind 0.0.0.0 8000` (no TLS) | +| testssl.sh | `./testssl.sh host:443` | + +--- + +## 9. Performance & Pitfalls + +| Pitfall | Cause | Mitigation | +|---------|------|------------| +| Overly permissive SG | “Allow all for quick test” | Review + least privilege | +| Rule bloat | Unmanaged growth | Use grouping / abstractions | +| Duplicate rules | Manual edits | Automate via Terraform / Ansible | +| Expired cert outage | No alerts | Monitor expiry + auto-renew | +| Weak ciphers | Defaults not updated | Enforce modern cipher suites | +| Full conntrack table | High connection churn | Tune nf_conntrack_max + scale out | +| NAT exhaustion | Too many ephemeral flows | Increase ephemeral port range / reuse pooling | +| TLS CPU spikes | Expensive handshake load | Enable TLS session reuse / 1.3 / offload | + +--- + +## 10. Hardening Checklist + +| Area | Action | +|------|--------| +| Host | Default DROP inbound; allow loopback & established | +| SSH | Restrict source IPs | +| Services | Bind to least interfaces | +| TLS | Use TLS 1.2+ (prefer 1.3), disable outdated protocols | +| Certs | Automate renewal; monitor expiry < 20 days | +| Secrets | Store private keys with correct permissions (`600`) | +| Logging | Enable firewall drop logging (rate-limited) | +| Cloud | Separate SGs per tier; restrict DB to App SG | +| mTLS (internal) | Use service mesh or internal PKI | +| Observability | Dashboards: connections, drop counts, handshake failures | +| Automation | Infra as Code for rules & cert provisioning | + +--- + +## 11. Design Patterns + +| Pattern | Description | Example | +|---------|-------------|---------| +| Choke Point | Single inspection layer at edge | ALB + WAF | +| Defense-in-Depth | Multiple layers (edge → SG → host) | SG + host firewall + app ACL | +| Zero Trust | Always verify identity | mTLS between services | +| Segmentation | Limit blast radius | Separate VPC subnets | +| Policy as Code | Versioned declarative security | Terraform SGs, nftables templates | + +--- + +## Submission Guidelines + +Create `day19solution.md` including: + +1. Command outputs: + - `iptables -L -n -v` (before & after) + - `nft list ruleset` (if used) + - `ss -tuln` + - `openssl s_client` output (sanitized) +2. Screenshots / notes: + - Wireshark TLS handshake + - Security group design diagram +3. mTLS test results (`curl` success vs failure) +4. Incident simulation write-up (at least 2 scenarios) +5. Reflection answers +6. (Optional) Hardening checklist—what you applied + +Share on socials with: +`#getfitwithsagar #100DaysOfSRE #CloudSecurity #TLS #DevSecOps #NetworkDefense` + +--- + +## Community & Support + +- [Discord](https://discord.gg/mNDm39qB8t) +- [Google Group](https://groups.google.com/forum/#!forum/daily-devops-sre-challenge-series/join) +- [YouTube](https://www.youtube.com/@Sagar.Utekar) + +--- + +## Safety Notes + +- Always test firewall changes in a **screen/tmux** session or with console access. +- For remote hosts, add a “rollback” cron job before applying strict DROP policies. +- Never share private keys in public repos. + +--- + +**Mastering Day 19 gives you the confidence to guard availability, integrity, and confidentiality—from kernel packet paths to encrypted application sessions.** + +Ready to build a fortress? Let’s ship it. + +Happy securing, +Sagar Utekar From 7032c7d64dcd376ea5dedeb290bbe818e1eb09c9 Mon Sep 17 00:00:00 2001 From: Sagar Utekar Date: Wed, 10 Sep 2025 07:27:01 +0530 Subject: [PATCH 116/133] Create challenge.md --- DailyChallenges/Season2/Day20/challenge.md | 359 +++++++++++++++++++++ 1 file changed, 359 insertions(+) create mode 100644 DailyChallenges/Season2/Day20/challenge.md diff --git a/DailyChallenges/Season2/Day20/challenge.md b/DailyChallenges/Season2/Day20/challenge.md new file mode 100644 index 0000000..df81412 --- /dev/null +++ b/DailyChallenges/Season2/Day20/challenge.md @@ -0,0 +1,359 @@ +# **Day 20: Application Layer 7 – Reverse Proxies, Caching, Performance & Observability** + +--- + +## Why this Matters + +Most real-world outages aren’t kernel panics; they’re **502s, 503s, latency spikes, cache misconfigurations, bad headers, or upstream timeouts**. +Layer 7 (HTTP) is where reliability, performance, and security meet. Mastering this layer lets you: +- Slash origin load with caching +- Expose reliable APIs behind reverse proxies +- Debug 502 vs 504 vs 499 quickly +- Optimise compression, connection reuse, and multiplexing +- Trace requests across layers with structured logs and headers + +> “If you can read headers, logs, and timings, you can solve 80% of web production issues.” + +--- + +## 1. Core Concepts (At a Glance) + +| Concept | Meaning (Simple) | Why It Matters | +|---------|------------------|----------------| +| Method (GET/POST/etc.) | Action on a resource | Safe vs unsafe; idempotency | +| Status Code Classes | 1xx info, 2xx success, 3xx redirect, 4xx client error, 5xx server error | Fast triage | +| Headers | Metadata before body | Routing, caching, debugging | +| Body | Payload (HTML, JSON, binary) | API / page content | +| Connection Persistence | Keep-Alive vs new TCP per request | Latency + scalability | +| Reverse Proxy | Intermediary forwarding to upstream | Control point (NGINX / Envoy) | +| Gateway / API Gateway | Adds auth, rate limits, transforms | API governance | +| Cache | Store response to reuse later | Performance + cost reduction | +| CDN | Global cache + edge network | Latency + DDoS absorption | +| Observability Fields | latency, upstream time, status, bytes | SLA / SLO measurement | + +--- + +## 2. Reverse Proxy vs Load Balancer vs API Gateway + +| Component | Focus | Examples | Adds | +|-----------|-------|----------|------| +| Reverse Proxy | Fronts and forwards requests to upstreams | NGINX, HAProxy, Caddy | TLS, routing, basic caching | +| Load Balancer | Distributes load across backend pool | ALB, ELB, HAProxy, Envoy | Health checks, balancing algorithms | +| API Gateway | Manages API lifecycle | Kong, APIGW, Tyk, Zuul | Auth, quotas, transforms, versioning | +| Service Mesh Sidecar | Per-service proxy | Envoy/Istio, Linkerd | mTLS, routing policies, telemetry | + +--- + +## 3. Critical HTTP Headers (You Should Recognise Instantly) + +| Category | Header | Purpose | +|----------|--------|---------| +| Client Identity | X-Forwarded-For, X-Real-IP | Preserve original IP through proxy chain | +| Protocol & Host | Host, X-Forwarded-Proto | Virtual hosting, HTTPS redirect logic | +| Caching – Freshness | Cache-Control (max-age, s-maxage), Expires | Decide if cached copy usable | +| Caching – Validation | ETag, Last-Modified, If-None-Match, If-Modified-Since | Conditional revalidation | +| Auth | Authorization, WWW-Authenticate | API / Basic / Bearer tokens | +| Content | Content-Type, Content-Length, Transfer-Encoding | Parse and stream body | +| Compression | Accept-Encoding, Content-Encoding | gzip, br support | +| Security | Strict-Transport-Security, X-Frame-Options, Content-Security-Policy, Set-Cookie (HttpOnly/Secure/SameSite) | Mitigate attacks | +| Control / Debug | Via, X-Trace-Id, X-Request-Id, X-Upstream-Response-Time | Distributed tracing | +| Variation | Vary | Cache key dimension changes | + +--- + +## 4. Observability KPIs (Track These) + +| Metric | Reason | +|--------|--------| +| p50 / p95 / p99 latency | User experience & performance tuning | +| Upstream response time vs total time | Find proxy vs backend delay | +| Active connections | Capacity planning | +| 4xx / 5xx ratio | Error detection | +| Cache HIT ratio | Cost + speed optimization | +| Request size / response size | Abuse detection / tuning | +| 499 count (client closed) | User aborts / timeout mismatch | +| Open file descriptors | Resource limits | + +--- + +## 5. Mermaid Architecture & Flows + +### 5.1 Request Flow (Reverse Proxy + Cache) + +```mermaid +flowchart LR + A[Client] --> B[Reverse Proxy / Cache] + B -->|HIT| C[Serve Cached Response] + B -->|MISS| D[Upstream Pool] + D --> E[App Service] + E --> D --> B --> A +``` + +### 5.2 Cache Decision Logic + +```mermaid +flowchart TD + R[Incoming Request] --> K{Key Exists?} + K -->|No| U[Fetch Upstream] --> S[Store if Cacheable] --> Return + K -->|Yes| F{Fresh?} + F -->|Yes| Return + F -->|No| V{Validatable? :ETag} + V -->|No| U + V -->|Yes| C[Send Conditional Request] + C -->|304| UpdateMeta --> Return + C -->|200| S +``` + +--- + +## 6. Hands-On Labs + +## Problem Statement + +You are the on-call SRE. A small internal service (a simple Python HTTP backend) is getting slower under load, and the product wants: +1. Faster repeated responses +2. Fewer upstream timeouts +3. Basic visibility into performance +4. Ability to explain a 502 vs 504 if it happens + +Your task: Put NGINX in front of the backend as a reverse proxy with minimal caching + basic observability, then demonstrate: +- A normal proxied request +- A cached response +- A slow request + how a timeout manifests (and how you fix it) +- A short log-based latency summary + +You will deliver a single file `solution.md` documenting what you did and the answers to reflection questions. + +--- + +## 1. Setup (Backend + Reverse Proxy) + +### 1.1 Start a very simple backend + +```bash +mkdir -p ~/day20 && cd ~/day20 +cat > app.py <<'EOF' +from http.server import BaseHTTPRequestHandler, HTTPServer +import time, datetime + +class H(BaseHTTPRequestHandler): + def do_GET(self): + # /slow simulates slowness + if self.path.startswith("/slow"): + time.sleep(2) + body = f"PATH={self.path} TIME={datetime.datetime.utcnow().isoformat()}Z\n" + self.send_response(200) + self.send_header("Content-Type", "text/plain") + # Let proxy decide caching; still send something explicit: + if self.path.startswith("/cache"): + self.send_header("Cache-Control", "max-age=30") + else: + self.send_header("Cache-Control", "no-store") + self.end_headers() + self.wfile.write(body.encode()) + +HTTPServer(("127.0.0.1", 9000), H).serve_forever() +EOF + +python3 app.py +``` + +(Leave it running in one terminal.) + +### 1.2 Install and configure NGINX (reverse proxy + tiny cache) + +```bash +sudo apt-get update -y +sudo apt-get install -y nginx +``` + +Create config: + +```bash +sudo tee /etc/nginx/conf.d/day20.conf <<'EOF' +# Simple cache definition +proxy_cache_path /var/cache/nginx/day20 levels=1:2 keys_zone=day20cache:10m max_size=50m inactive=2m use_temp_path=off; + +upstream backend_pool { + server 127.0.0.1:9000; +} + +# Custom log format (pipe-separated so it's easy to parse) +log_format mini '$remote_addr|$time_local|$request|$status|$body_bytes_sent|$request_time|$upstream_response_time|$upstream_cache_status'; + +server { + listen 8080; + server_name localhost; + + access_log /var/log/nginx/day20_access.log mini; + + # Default (no caching) proxy + location / { + proxy_pass http://backend_pool; + proxy_set_header Host $host; + proxy_set_header X-Forwarded-For $remote_addr; + # Short upstream read timeout to show 504 on /slow initially + proxy_read_timeout 1s; + } + + # Caching block (/cache path) + location /cache { + proxy_pass http://backend_pool; + proxy_cache day20cache; + proxy_cache_valid 200 30s; + add_header X-Cache $upstream_cache_status always; + proxy_set_header Host $host; + proxy_set_header X-Forwarded-For $remote_addr; + } +} +EOF + +sudo nginx -t && sudo systemctl reload nginx +``` + +--- + +## 2. Demonstrate Core Behaviours + +In a new terminal: + +### 2.1 Normal proxied request + +```bash +curl -i http://localhost:8080/ +``` + +Expected: 200, no `X-Cache` header (not in /cache), fresh timestamp each call. + +### 2.2 Cached request (HIT vs MISS) + +First call (MISS): +```bash +curl -i http://localhost:8080/cache +``` + +Second call (HIT): +```bash +curl -i http://localhost:8080/cache +``` + +Check headers for: `X-Cache: MISS` then `X-Cache: HIT` + +### 2.3 Trigger a 504 (slow path + tight timeout) + +```bash +curl -i http://localhost:8080/slow +``` + +Because backend sleeps 2s and `proxy_read_timeout` is 1s, you should see a 504 Gateway Timeout. + +### 2.4 Fix the timeout + +Edit just the timeout line: + +```bash +sudo sed -i 's/proxy_read_timeout 1s;/proxy_read_timeout 3s;/' /etc/nginx/conf.d/day20.conf +sudo nginx -t && sudo systemctl reload nginx +curl -i http://localhost:8080/slow +``` + +Now it should succeed with 200 after ~2 seconds. + +--- + +## 3. Lightweight Observability + +Grab last few log lines: + +```bash +tail -n 20 /var/log/nginx/day20_access.log +``` + +Example line format: +``` +IP|TIME|REQUEST|STATUS|BYTES|REQ_TIME|UPSTREAM_TIME|CACHE_STATUS +``` + +Quick status breakdown: + +```bash +awk -F'|' '{print $4}' /var/log/nginx/day20_access.log | sort | uniq -c +``` + +Average request time: + +```bash +awk -F'|' '{sum+=$6; n+=1} END {if(n>0) print "avg_request_time_secs="sum/n;}' /var/log/nginx/day20_access.log +``` + +Cache hits vs misses: + +```bash +awk -F'|' '$8!="" {print $8}' /var/log/nginx/day20_access.log | sort | uniq -c +``` + +--- + +## 4. 502 vs 504 (You Only Simulated 504 Today) + +Explain in your solution: +- 502 = proxy talked to upstream but got a bad/invalid/closed response +- 504 = upstream did not respond before timeout + +(You can optionally fake a 502 by stopping the backend: Ctrl+C backend → `curl http://localhost:8080/` → likely 502.) + +--- + +## 5. What to Include in `day20solution.md` + +Minimum: +1. Commands you ran (copy/paste sequence) +2. Output snippets: + - First MISS then HIT (`X-Cache`) + - 504 example (before fix) + - Successful `/slow` after timeout change +3. At least 5 log lines + your short interpretation (e.g., which was cached? which timed out?) +4. Answers to reflection questions (below) + +Optional: +- A sentence describing how you’d extend this (rate limiting? gzip?). + +--- + +## 6. Reflection Questions (Answer Briefly) + +1. What changed between MISS and HIT responses? +2. Why did `/slow` first return 504? +3. After increasing the timeout, what risk do you accept (hint: tying up connections longer)? +4. How would a 502 differ (root cause wise) from the 504 you saw? +5. How does basic caching reduce backend load in this scenario? + +--- + +## 7. Cleanup (Optional) + +```bash +sudo rm /etc/nginx/conf.d/day20.conf +sudo systemctl reload nginx +``` + +--- + +## 8. Summary (What You Just Learned Quietly) + +| Skill | You Did It By | +|-------|---------------| +| Reverse proxy basics | NGINX forwarding to Python server | +| Caching | `/cache` location using `proxy_cache` | +| Cache diagnostics | Observed `X-Cache` MISS → HIT | +| Timeout handling | Forced 504 then tuned `proxy_read_timeout` | +| Log-based insight | Parsed custom access log for status & timing | +| Error differentiation | Defined 502 vs 504 semantics | + +--- + +**Layer 7 mastery lets you design faster, more reliable, more observable systems.** +Today you built the foundation for API gateways, CDNs, mesh routing, and performance engineering. + +Happy optimizing and debugging, +Sagar Utekar From 571af8efc34fd22dd768315ccd845351916ffd47 Mon Sep 17 00:00:00 2001 From: Sagar Utekar Date: Wed, 10 Sep 2025 07:35:25 +0530 Subject: [PATCH 117/133] Update challenge.md --- DailyChallenges/Season2/Day19/challenge.md | 19 ------------------- 1 file changed, 19 deletions(-) diff --git a/DailyChallenges/Season2/Day19/challenge.md b/DailyChallenges/Season2/Day19/challenge.md index 8c271cc..243dec2 100644 --- a/DailyChallenges/Season2/Day19/challenge.md +++ b/DailyChallenges/Season2/Day19/challenge.md @@ -62,14 +62,6 @@ flowchart LR | Performance | Slightly faster for simple rules | Near parity with modern kernels | | Use Case | Edge routers, simple filtering | Hosts, cloud workloads | -### 2.5 iptables vs nftables vs firewalld - -| Feature | iptables | nftables | firewalld | -|---------|----------|----------|-----------| -| Status | Legacy (still everywhere) | Modern replacement | Abstraction layer | -| Performance | Per-table / chain evaluation | Unified flow engine | Depends on backend | -| Rule Management | Append oriented | Atomic replace | Zones & services | -| Recommended? | Legacy maintenance | YES (future-proof) | For ease if not scripting | --- @@ -158,17 +150,6 @@ graph TD - AES_256_GCM: Bulk cipher (AEAD) - SHA384: Hash for PRF/integrity -### 4.6 TLS 1.2 vs 1.3 - -| Aspect | TLS 1.2 | TLS 1.3 | -|--------|---------|---------| -| Handshake RTT | 2 | 1 (0 with resumption) | -| Cipher Suites | Many (legacy risk) | Streamlined / only AEAD | -| Key Exchange | RSA / (EC)DHE | Always (EC)DHE (PFS enforced) | -| Algorithms Removed | RC4, SHA1, etc. | Already gone | -| Session Resumption | Session IDs / Tickets | Tickets / PSK | -| Security Level | Good if hardened | Better defaults | - --- ## 5. Certificates – Types & Use Cases From e6be4bb8f6c74cfb0c7780c30789e73a7dd9d9eb Mon Sep 17 00:00:00 2001 From: Sagar Utekar Date: Wed, 10 Sep 2025 07:48:08 +0530 Subject: [PATCH 118/133] Update challenge.md --- DailyChallenges/Season2/Day19/challenge.md | 577 ++++++++++++--------- 1 file changed, 335 insertions(+), 242 deletions(-) diff --git a/DailyChallenges/Season2/Day19/challenge.md b/DailyChallenges/Season2/Day19/challenge.md index 243dec2..2cb5a3e 100644 --- a/DailyChallenges/Season2/Day19/challenge.md +++ b/DailyChallenges/Season2/Day19/challenge.md @@ -1,81 +1,74 @@ -# **Day 19: Firewalls, Security Groups & TLS/SSL – Defending Modern Infrastructure** +# Day 19: Firewalls, Security Groups & TLS/SSL – Defending Modern Infrastructure ---- +> “Performance issues are annoying. Misconfigured firewalls and expired TLS certs are outages.” -## Why Day 19 Matters +--- -You can design the cleanest microservice architecture, but without proper **traffic control (firewalls & security groups)** and **secure channel establishment (TLS/SSL)**, your stack is a soft target. -Today you’ll learn how packets are permitted, denied, shaped, inspected, and encrypted across **Linux hosts, cloud boundaries, and client–server connections**. +## Why This Matters -> “Performance issues are annoying. Misconfigured firewalls and expired TLS certs are outages.” +You can design the cleanest microservice architecture, but without deliberate **traffic control (firewalls & security groups)** and **secure channel establishment (TLS/SSL + optional mTLS)**, your stack is a soft target. +Today you will: lock down a host, design a 3‑tier perimeter, enable HTTPS, optionally add mTLS, capture a handshake, and reason about real incident scenarios. --- ## 1. Core Concepts (At a Glance) -| Concept | What It Is (Simple) | Why You Care | -|--------|----------------------|--------------| -| Firewall | Traffic filter (allow/deny based on rules) | Prevents unwanted access | -| Stateless Filter | Checks each packet independently | Fast but dumb (e.g., classic ACL) | -| Stateful Firewall | Tracks connection state (established, related) | Smarter, less rules needed | -| Security Group (Cloud) | Virtual, stateful firewall at instance ENI level | Controls what talks in/out | -| NACL (Cloud) | Subnet-level stateless rule set | Coarse filtering, both directions | -| WAF | Layer 7 (HTTP) attack filter | Stops SQLi, XSS, bots | -| TLS/SSL | Protocol for encrypted channels | Protects data + identity | -| Certificate | Digital ID for a domain/service | Trust & authenticity | -| mTLS | Both sides present certs | Zero-trust & service-to-service auth | -| Cipher Suite | Algorithms for key exchange + encryption + MAC | Affects performance & security | -| PFS | New ephemeral key per session | Limits blast radius of key leakage | +| Concept | Simple Meaning | Why You Care | +|---------|----------------|--------------| +| Firewall | Policy engine (allow/deny) | Blocks unwanted access | +| Stateless Filter | Evaluates each packet isolated | Fast, but no connection awareness | +| Stateful Firewall | Tracks connection state (ESTABLISHED, RELATED) | Fewer rules; safer defaults | +| Security Group (SG) | Cloud stateful L3/L4 micro‑perimeter on ENI | Fine-grained ingress/egress | +| NACL | Stateless subnet ACL | Coarse baseline filtering | +| WAF | Layer 7 HTTP filter | Stops XSS, SQLi, bots | +| TLS/SSL | Encrypted transport with identity | Confidentiality + integrity + auth | +| Certificate | Signed public key + identity metadata | Trust chain & validation | +| Cipher Suite | Algorithms bundle (KEX + auth + cipher + MAC) | Security & performance | +| PFS (Forward Secrecy) | Ephemeral session keys | Limits blast radius of key theft | +| mTLS | Mutual certificate auth | Zero trust / service identity | +| Conntrack | Kernel table tracking flows | Enables stateful decisions | +| Ephemeral Ports | High-numbered client-side ports | Needed for return traffic / NAT | --- -## 2. Firewalls Explained - -### 2.1 What Is a Firewall? -A firewall enforces **policy**: what traffic is allowed or denied based on attributes like protocol, source, destination, port, interface, or connection state. - -### 2.2 Layers -- Layer 3/4: IP, protocols, ports (iptables/nftables, Security Groups) -- Layer 7: HTTP semantics (WAF, reverse proxies) -- Deep Packet Inspection: Payload-level inspection (IDS/IPS) +## 2. Packet & Flow Fundamentals -### 2.3 Packet Flow (Simplified Linux Host) +### 2.1 Linux Simplified Packet Path ```mermaid flowchart LR - IN_Int[Incoming Packet :eth0] --> PREROUTING + IN[eth0 Ingress] --> PREROUTING PREROUTING -->|Local Dest?| INPUT PREROUTING -->|Forward?| FORWARD - INPUT --> LocalProc[Local Process] - LocalProc --> OUTPUT - OUTPUT --> POSTROUTING + INPUT --> PROC[Local Process] + PROC --> OUTPUT FORWARD --> POSTROUTING - POSTROUTING --> OUT_Int[Outgoing Interface] + OUTPUT --> POSTROUTING + POSTROUTING --> OUT[egress iface] ``` -### 2.4 Stateless vs Stateful - -| Aspect | Stateless (e.g., NACL, raw ACL) | Stateful (iptables conntrack, SG) | -|--------|---------------------------------|-----------------------------------| -| Tracks Connections? | No | Yes | -| Requires Reverse Rule? | Yes | No (auto allows reply) | -| Performance | Slightly faster for simple rules | Near parity with modern kernels | -| Use Case | Edge routers, simple filtering | Hosts, cloud workloads | +### 2.2 Stateless vs Stateful +| Aspect | Stateless (NACL / raw ACL) | Stateful (iptables/nftables, SG) | +|--------|----------------------------|----------------------------------| +| Tracks return flows? | No | Yes (conntrack) | +| Reverse rule needed? | Yes | No | +| Typical placement | Subnet edge / router | Host / cloud instance | +| Logging context | Limited | Rich (state-based decisions) | --- ## 3. Cloud Security Constructs -| Control | Layer | Stateful? | Scope | Typical Use | -|---------|-------|-----------|-------|-------------| -| Security Group | L3/L4 | Yes | ENI / Instance | Instance ingress/egress micro-perimeter | +| Control | Layer | Stateful | Scope | Typical Purpose | +|---------|-------|----------|-------|-----------------| +| SG | L3/L4 | Yes | ENI/Instance | Instance micro-perimeter | | NACL | L3/L4 | No | Subnet | Baseline allow/deny | -| WAF | L7 | N/A | HTTP/S endpoints | App-layer attack filtering | -| Shield / DDoS | L3–7 | Managed | Edge | Volumetric protection | -| Host Firewall | L3/L4 | Yes | OS Kernel | Defense-in-depth | +| WAF | L7 | N/A | HTTP edge | App-layer attack filtering | +| Shield / DDoS | L3–7 | Managed | Edge | Volumetric resilience | +| Host Firewall | L3/L4 | Yes | Kernel/OS | Defense in depth | -### 3.1 Security Group Statefulness (Inbound → Outbound Return) +### 3.1 Security Group Statefulness ```mermaid sequenceDiagram @@ -92,335 +85,435 @@ sequenceDiagram --- -## 4. TLS / SSL Essentials +## 4. TLS Essentials + +### 4.1 Guarantees -### 4.1 What TLS Provides | Guarantee | Meaning | |-----------|---------| | Confidentiality | Encrypted payload | | Integrity | Tamper detection | -| Authentication | Identity via certificates | -| (Optional) Client Auth | mTLS for mutual identity | +| Server Authentication | Proves server identity | +| (Optional) Client Authentication | mTLS mutual identity | -### 4.2 TLS 1.2 Handshake (Simplified) +### 4.2 TLS 1.2 (Simplified) ```mermaid sequenceDiagram participant C as Client participant S as Server - C->>S: ClientHello (ciphers, random) - S->>C: ServerHello (chosen cipher) - S->>C: Certificate (and chain) - S->>C: ServerKeyExchange (if needed) + C->>S: ClientHello + S->>C: ServerHello + Certificate (+ ServerKeyExchange?) S->>C: ServerHelloDone C->>S: ClientKeyExchange - C->>S: ChangeCipherSpec - C->>S: Finished - S->>C: ChangeCipherSpec - S->>C: Finished - Note over C,S: Secure channel established + C->>S: ChangeCipherSpec + Finished + S->>C: ChangeCipherSpec + Finished ``` -### 4.3 TLS 1.3 Differences (Faster, More Secure) +### 4.3 TLS 1.3 (Fewer Messages) ```mermaid sequenceDiagram participant C as Client participant S as Server - C->>S: ClientHello [includes key shares] - S->>C: ServerHello [select key] + EncryptedExtensions + Certificate + Finished + C->>S: ClientHello (key shares) + S->>C: ServerHello + EncryptedExtensions + Certificate + Finished C->>S: Finished - Note over C,S: 1-RTT [0-RTT possible for resumption] -``` - -### 4.4 Certificate Chain - -```mermaid -graph TD - A[Root CA] --> B[Intermediate CA] - B --> C[Server Certificate] - C --> D[Server :nginx] - E[Client Browser] -->|Trusts Root or Intermediate| A ``` -### 4.5 Cipher Suite Anatomy (TLS 1.2 Example) -`TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384` -- ECDHE: Key exchange (PFS) -- RSA: Authentication -- AES_256_GCM: Bulk cipher (AEAD) -- SHA384: Hash for PRF/integrity - --- -## 5. Certificates – Types & Use Cases +## 5. Certificates (Types & Use) -| Type | Use | Pros | Cons | -|------|-----|------|------| -| Self-Signed | Dev/testing | Fast | No external trust | -| Internal CA | Internal services | Control | Requires internal trust distro | -| Let’s Encrypt | Public sites | Free, automated | 90-day renewal | -| Wildcard | Many subdomains | Flexible | Risk if private key leaked | -| EV | High-profile public orgs | Strong validation signal | Cost / diminishing value | -| mTLS Certs | Service-to-service | Strong mutual auth | Operational overhead | +| Type | Use Case | Pros | Cons | +|------|----------|------|------| +| Self-Signed | Local dev | Fast | Not trusted externally | +| Internal CA | Internal services | Control lifecycle | Need trust distribution | +| Let’s Encrypt | Public web | Free + automated | 90‑day renew window | +| Wildcard | Many subdomains | Flexible | Larger key compromise blast | +| EV | High-profile brand | Strong validation chain | Minimal modern UX value | +| mTLS Leaf | Service identity | Strong auth | Ops overhead | --- -## 6. mTLS (Mutual TLS) - -Both sides present a certificate. Use cases: -- Service Mesh (Istio, Linkerd) -- gRPC microservice auth -- Zero-trust internal APIs - -Flow: -1. TLS handshake begins -2. Server sends Certificate -3. Server requests client Certificate -4. Client sends Certificate + proves possession of private key -5. Both validate chain → secure + mutually authenticated +## 6. Hands-On Labs ---- - -## 7. Hands-On Labs (Follow in Order) +> Choose either iptables or nftables for host firewall implementation. Both labs are provided. -### Lab 1: Inspect Current Firewall State (Linux) +### Lab 1: Baseline Inspection +Commands: ```bash -sudo iptables -L -n -v -sudo iptables -t nat -L -n -v -sudo nft list ruleset sudo ss -tulpen +sudo iptables -L -n -v 2>/dev/null || echo "iptables not active" +sudo nft list ruleset 2>/dev/null | head -n 50 || echo "nftables empty" ``` -Questions: -1. Which ports are open? -2. Any default ACCEPT policies? (Risky on exposed hosts.) +Record: +- Listening ports (service, port, protocol) +- Default INPUT/FORWARD policies (ACCEPT/DROP) +- Which framework is currently active? ---- +### Lab 2: Basic iptables Rule Set (Stateful Minimal Policy) -### Lab 2: Basic iptables Rule Set - -Goal: Allow SSH (22), HTTP (80), HTTPS (443); drop everything else inbound; allow all outbound. +Goal: Allow loopback, established/related, SSH(22), HTTP(80), HTTPS(443). Drop other inbound. ```bash -# WARNING: If remote, keep an existing SSH session open to recover. sudo iptables -P INPUT DROP sudo iptables -P FORWARD DROP sudo iptables -P OUTPUT ACCEPT -# Allow loopback sudo iptables -A INPUT -i lo -j ACCEPT - -# Allow established/related sudo iptables -A INPUT -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT - -# Allow SSH, HTTP, HTTPS sudo iptables -A INPUT -p tcp -m multiport --dports 22,80,443 -j ACCEPT -# View counters sudo iptables -L -n -v ``` -Persist (varies by distro): +Test blocked port: +```bash +nc -vz 127.0.0.1 8080 || echo "Blocked as expected" +``` + +(Optionally persist): ```bash sudo sh -c 'iptables-save > /etc/iptables/rules.v4' ``` -Task: -- Try connecting to a random blocked port (e.g., 8080) → should fail. +### Lab 3: nftables Equivalent (If You Prefer Modern Stack) ---- +```bash +sudo nft flush ruleset +sudo tee /tmp/day19.nft <<'EOF' +table inet filter { + chain input { + type filter hook input priority 0; policy drop; + iif lo accept + ct state established,related accept + tcp dport {22,80,443} accept + } + chain forward { type filter hook forward priority 0; policy drop; } + chain output { type filter hook output priority 0; policy accept; } +} +EOF +sudo nft -f /tmp/day19.nft +sudo nft list ruleset +``` -### Lab 3: Security Group Design (3-Tier Example) +Optional logging rule: +```bash +sudo nft add rule inet filter input log prefix "DROP_INPUT " counter +journalctl -k -f +``` -Architecture: -- Public ALB (443) -- Web Tier (EC2 behind ALB) -- App Tier (private) -- DB Tier (private) +### Lab 4: Prove Statefulness -Rules (Simplified): +```bash +curl -I https://example.com 2>/dev/null | head -n1 +``` +Explain: Why did the response succeed without an inbound ephemeral port rule? (conntrack state ESTABLISHED handles reverse flow.) -| Tier | Inbound Allowed | Outbound Allowed | -|------|-----------------|------------------| -| ALB | 0.0.0.0/0 → :443 | Web SG → :443 | -| Web SG | ALB SG → :80/:443 | App SG → :8080; 0.0.0.0/0 DNS/NTP as needed | -| App SG | Web SG → :8080 | DB SG → :5432 | -| DB SG | App SG → :5432 | (Optional) Backup host SG | +### Lab 5: 3‑Tier Cloud Perimeter Design (Conceptual) -Task: -- Document in a diagram (draw.io / mermaid). -- Justify: Why no direct internet to App/DB? +Create a diagram (mermaid allowed) & a table: -Mermaid Flow: +| Tier | Inbound Source | Ports | Outbound | +|------|----------------|-------|----------| +| ALB | 0.0.0.0/0 | 443 (maybe 80 redirect) | Web SG on 443 | +| Web | ALB SG | 80,443 | App SG:8080; DNS, NTP | +| App | Web SG | 8080 | DB SG:5432 | +| DB | App SG | 5432 | Backup / monitoring only | +Mermaid: ```mermaid flowchart LR - User(Internet User) -->|HTTPS 443| ALB[Public ALB] - ALB -->|HTTP 80 / 443| WEB[Web EC2 :Web SG] - WEB -->|TCP 8080| APP[App EC2 :App SG] - APP -->|TCP 5432| DB[(Database :DB SG)] + User((Internet)) -->|443| ALB[ALB/WAF] + ALB -->|80/443| WEB[Web Tier] + WEB -->|8080| APP[App Tier] + APP -->|5432| DB[(Database)] ``` ---- +Explain differences: +- SG (stateful) vs NACL (stateless) +- WAF sits before ALB (optional) for L7 inspection -### Lab 5: Generate Self-Signed TLS Cert & Use with NGINX +### Lab 6: Generate Self-Signed TLS & Serve via Nginx ```bash +sudo apt-get update -y +sudo apt-get install -y nginx openssl + mkdir -p ~/tlslab && cd ~/tlslab openssl req -x509 -newkey rsa:4096 -sha256 -days 90 -nodes \ -keyout server.key -out server.crt -subj "/CN=localhost" +``` -sudo apt-get install -y nginx -sudo tee /etc/nginx/sites-available/tlsdemo <<'EOF' +Nginx site: +```bash +sudo tee /etc/nginx/sites-available/day19 <<'EOF' server { listen 443 ssl; server_name localhost; ssl_certificate /home/$USER/tlslab/server.crt; ssl_certificate_key /home/$USER/tlslab/server.key; - add_header X-App "Day19Demo"; + add_header X-Day19 "PerimeterDemo"; location / { - return 200 "Secure Hello (TLS Demo)\n"; + return 200 "Day19 Secure Hello\n"; } } EOF - -sudo ln -s /etc/nginx/sites-available/tlsdemo /etc/nginx/sites-enabled/tlsdemo +sudo ln -s /etc/nginx/sites-available/day19 /etc/nginx/sites-enabled/day19 sudo nginx -t && sudo systemctl reload nginx ``` Test: ```bash -curl -vk https://localhost -openssl s_client -connect localhost:443 -servername localhost -tls1_2 &1 | head -n 20 ``` -Answer: -- Why does curl show `self-signed certificate` warnings? -- What SANs should production certs include? +Questions: +- Why browser/curl trust warning? (Self-signed: subject=issuer) +- Production cert should include proper SANs (Subject Alternative Names) ---- +### Lab 7: TLS Certificate & Cipher Inspection -### Lab 6: Incident Simulation Scenarios +```bash +echo | openssl s_client -connect localhost:443 -servername localhost 2>/dev/null \ + | openssl x509 -noout -subject -issuer -dates + +echo | openssl s_client -connect localhost:443 -servername localhost 2>/dev/null \ + | grep -i "Cipher" +``` +Record: +- Subject / Issuer +- notBefore / notAfter +- Chosen cipher (identify ECDHE for PFS) + +### Lab 8: TLS Handshake Capture (Optional but Recommended) + +```bash +sudo tcpdump -i lo port 443 -w tls_handshake.pcap & +PID=$! +curl -k https://localhost >/dev/null +sleep 1 +sudo kill $PID +``` +View in Wireshark: +- Inspect ClientHello (cipher list, SNI) +- ServerHello (selected cipher) +- Certificate frame + +Document 2–3 handshake observations. + +### Lab 9: Near-Expiry Cert Simulation + +```bash +cd ~/tlslab +openssl req -x509 -newkey rsa:2048 -sha256 -days 1 -nodes \ + -keyout short.key -out short.crt -subj "/CN=localhost-short" +sudo sed -i 's/server.crt/short.crt/; s/server.key/short.key/' /etc/nginx/sites-available/day19 +sudo nginx -t && sudo systemctl reload nginx +echo | openssl s_client -connect localhost:443 -servername localhost 2>/dev/null \ + | openssl x509 -noout -dates +``` + +Restore: +```bash +sudo sed -i 's/short.crt/server.crt/; s/short.key/server.key/' /etc/nginx/sites-available/day19 +sudo nginx -t && sudo systemctl reload nginx +``` + +Explain operational risk of missed renewal; propose monitoring threshold (e.g. alert at <20 days). + +### Lab 10: mTLS Mini Demo (Optional Advanced) + +Internal CA: +```bash +cd ~/tlslab +openssl genrsa -out ca.key 4096 +openssl req -x509 -new -key ca.key -days 365 -out ca.crt -subj "/CN=Day19-CA" +``` + +Server leaf (reissue signed by CA): +```bash +openssl genrsa -out mtls_server.key 4096 +openssl req -new -key mtls_server.key -out mtls_server.csr -subj "/CN=mtls-server" +openssl x509 -req -in mtls_server.csr -CA ca.crt -CAkey ca.key -CAcreateserial \ + -out mtls_server.crt -days 120 -sha256 +``` + +Client cert: +```bash +openssl genrsa -out client.key 4096 +openssl req -new -key client.key -out client.csr -subj "/CN=client1" +openssl x509 -req -in client.csr -CA ca.crt -CAkey ca.key -CAserial ca.srl \ + -out client.crt -days 120 -sha256 +``` + +Update nginx: +```bash +sudo sed -i 's#ssl_certificate .*#ssl_certificate /home/'"$USER"'/tlslab/mtls_server.crt;#' /etc/nginx/sites-available/day19 +sudo sed -i 's#ssl_certificate_key .*#ssl_certificate_key /home/'"$USER"'/tlslab/mtls_server.key;#' /etc/nginx/sites-available/day19 +sudo sed -i '/server_name localhost;/a ssl_client_certificate /home/'"$USER"'/tlslab/ca.crt;\nssl_verify_client on;' /etc/nginx/sites-available/day19 +sudo nginx -t && sudo systemctl reload nginx +``` + +Tests: +```bash +curl --cert ~/tlslab/client.crt --key ~/tlslab/client.key -k https://localhost +curl -k https://localhost -o /dev/null -w "HTTP:%{http_code}\n" +``` -| Scenario | Symptom | Root Cause | Your Fix | -|----------|---------|-----------|----------| -| Locked out of server | SSH timeout | Dropped inbound 22 rule | Add console recovery / out-of-band | -| App can't reach DB | ECONNREFUSED | SG missing outbound or DB inbound | Adjust SGs with least privilege | -| High CPU + retransmits | Slow responses | Stateful inspection under load | Offload to dedicated firewall tier | -| Cert expired | Browser error | Cron renewal failed | Implement monitoring + auto-renew | -| mTLS failure | 400 Bad Cert | Wrong CA trust at server | Sync CA bundle | +Record: success vs failure status codes. -Task: -Pick 2 and write incident analysis: timeline + resolution + prevention. +### Lab 11: Incident Simulation & Analysis + +Pick any TWO and write: Timeline → Symptom → Root Cause → Resolution → Prevention + +| Scenario | Symptom | Likely Root Cause | Prevention Idea | +|----------|---------|------------------|-----------------| +| SSH Lockout | Timeout | Dropped port 22 / no ESTABLISHED rule | Pre-change session + rollback script | +| App→DB Fail | ECONNREFUSED | Missing SG rule or wrong CIDR | IaC & peer review | +| Expired Cert | Browser errors / failing jobs | Failed renewal automation | Expiry monitoring + ACME | +| mTLS Fail | 400 / 401 / handshake abort | Wrong CA bundle / mismatched CN | Automate trust distribution | +| Conntrack Exhaustion | Drops / latency | High short-lived connection churn | Tune nf_conntrack_max, reuse pools | +| NAT Port Exhaustion | Outbound stalls | Too many parallel ephemeral flows | Increase ranges / SNAT scaling | +| TLS CPU Spike | High CPU, slow handshakes | No session reuse, old ciphers | TLS 1.3, reuse, offload | --- -## 8. Debugging & Toolbelt +## 7. Debugging & Toolbelt | Goal | Commands | |------|----------| -| See rules (iptables) | `iptables -L -n -v` `iptables-save` | -| See rules (nftables) | `nft list ruleset` | -| Connection states | `ss -s` `ss -tan` | -| Conntrack table | `sudo conntrack -L | head` | -| Test port remotely | `nc -vz host 443` `telnet host 443` | -| Port scan (careful!) | `nmap -sS -p 1-1024 host` | -| TLS inspect | `openssl s_client -connect host:443 -servername host` | -| Cipher test | `nmap --script ssl-enum-ciphers -p 443 host` | -| Certificate expiry | `echo | openssl s_client -servername host -connect host:443 2>/dev/null | openssl x509 -noout -dates` | -| Quick server (dev TLS) | `python3 -m http.server --bind 0.0.0.0 8000` (no TLS) | -| testssl.sh | `./testssl.sh host:443` | +| List iptables | `iptables -L -n -v`, `iptables-save` | +| List nft | `nft list ruleset` | +| Open sockets | `ss -tulpen` | +| Conntrack sample | `sudo conntrack -L | head` | +| Test port | `nc -vz host 443` | +| Quick scan (careful) | `nmap -sS -p 22,80,443 host` | +| TLS details | `openssl s_client -connect host:443 -servername host` | +| Enumerate ciphers | `nmap --script ssl-enum-ciphers -p 443 host` | +| Cert expiry | `echo | openssl s_client -connect host:443 2>/dev/null | openssl x509 -noout -dates` | +| Handshake capture | `tcpdump -n -i eth0 port 443 -w tls.pcap` | --- -## 9. Performance & Pitfalls +## 8. Performance & Pitfalls -| Pitfall | Cause | Mitigation | -|---------|------|------------| -| Overly permissive SG | “Allow all for quick test” | Review + least privilege | -| Rule bloat | Unmanaged growth | Use grouping / abstractions | -| Duplicate rules | Manual edits | Automate via Terraform / Ansible | -| Expired cert outage | No alerts | Monitor expiry + auto-renew | -| Weak ciphers | Defaults not updated | Enforce modern cipher suites | -| Full conntrack table | High connection churn | Tune nf_conntrack_max + scale out | -| NAT exhaustion | Too many ephemeral flows | Increase ephemeral port range / reuse pooling | -| TLS CPU spikes | Expensive handshake load | Enable TLS session reuse / 1.3 / offload | +| Pitfall | Cause | Impact | Mitigation | +|---------|-------|--------|-----------| +| Overly permissive SG | Quick testing | Attack surface | Least privilege + IaC | +| Rule sprawl | Manual edits | Complexity / errors | Periodic pruning / versioning | +| Conntrack full | Burst connections | Drops / latency | Tune + scale + pooling | +| NAT ephemeral exhaustion | High outbound fan-out | Stalled flows | Expand range / scale NAT | +| Weak ciphers enabled | Legacy defaults | MITM risk | Harden config / TLS 1.3 | +| TLS handshake CPU | No reuse / RSA | Latency, cost | Session tickets / TLS 1.3 | +| Expired cert | Missed renew job | Outage | Alert < 20–30 days | +| Unlogged drops | No visibility | Slow RCA | Rate-limited logging | --- -## 10. Hardening Checklist +## 9. Hardening Checklist | Area | Action | |------|--------| -| Host | Default DROP inbound; allow loopback & established | -| SSH | Restrict source IPs | -| Services | Bind to least interfaces | -| TLS | Use TLS 1.2+ (prefer 1.3), disable outdated protocols | -| Certs | Automate renewal; monitor expiry < 20 days | -| Secrets | Store private keys with correct permissions (`600`) | -| Logging | Enable firewall drop logging (rate-limited) | -| Cloud | Separate SGs per tier; restrict DB to App SG | -| mTLS (internal) | Use service mesh or internal PKI | -| Observability | Dashboards: connections, drop counts, handshake failures | -| Automation | Infra as Code for rules & cert provisioning | +| Ingress Policy | Default DROP (host & SG) | +| Loopback | Always allow | +| SSH | Restrict CIDR / MFA bastion | +| Services | Bind only required interfaces | +| TLS | Enforce 1.2+ (prefer 1.3), strong ciphers | +| Cert Lifecycle | Auto-renew + expiry alerts | +| Keys | 600 perms; minimal exposure | +| Logging | Drop logging (rate-limited) | +| Observability | Dashboards: conntrack, drops, handshake failures | +| mTLS | Internal sensitive APIs / service mesh | +| IaC | Terraform / Ansible for rules & cert provisioning | --- -## 11. Design Patterns +## 10. Design Patterns | Pattern | Description | Example | |---------|-------------|---------| -| Choke Point | Single inspection layer at edge | ALB + WAF | -| Defense-in-Depth | Multiple layers (edge → SG → host) | SG + host firewall + app ACL | -| Zero Trust | Always verify identity | mTLS between services | -| Segmentation | Limit blast radius | Separate VPC subnets | -| Policy as Code | Versioned declarative security | Terraform SGs, nftables templates | +| Choke Point | Single controlled ingress layer | ALB + WAF bundle | +| Defense in Depth | Multiple layered controls | SG + host firewall + app ACL | +| Zero Trust Drift | Auth every hop | mTLS between services | +| Segmentation | Contain lateral movement | Web/App/DB tiering | +| Policy as Code | Versioned reproducible security | Terraform SG modules & nft templates | + +--- + +## 11. Reflection Questions + +1. Why does reply traffic not need an explicit inbound rule on a stateful firewall? +2. One sentence: SG vs NACL difference? +3. Advantage of nftables over iptables (design or management)? +4. Where do you see Perfect Forward Secrecy in the cipher output? +5. Operational risk illustrated by the 1‑day certificate test? +6. Give one concrete internal use case for mTLS. +7. Describe a mitigation for conntrack table exhaustion. +8. Which design pattern (choke point, defense in depth, zero trust) did you partially implement today and how? +9. One automation you’d add tomorrow to reduce perimeter toil? --- -## Submission Guidelines +## 12. Submission Guidelines (day19solution.md) -Create `day19solution.md` including: +Include: -1. Command outputs: - - `iptables -L -n -v` (before & after) - - `nft list ruleset` (if used) - - `ss -tuln` - - `openssl s_client` output (sanitized) -2. Screenshots / notes: - - Wireshark TLS handshake - - Security group design diagram -3. mTLS test results (`curl` success vs failure) -4. Incident simulation write-up (at least 2 scenarios) -5. Reflection answers -6. (Optional) Hardening checklist—what you applied +1. Baseline outputs: `ss`, firewall policy, active framework +2. Final ruleset (iptables-save OR `nft list ruleset` excerpt) +3. Statefulness explanation (Lab 4) +4. 3-tier model (table + mermaid or diagram) + SG vs NACL vs WAF one-liners +5. TLS evidence: curl snippet, cert subject/issuer/dates, cipher line +6. Near-expiry (1‑day cert) output + monitoring note +7. (Optional) Handshake capture observations (ClientHello, cipher chosen) +8. (Optional) mTLS success vs failure HTTP codes +9. Two incident analyses (structured) +10. Reflection answers (all) +11. (Optional) Hardening checklist: mark applied items -Share on socials with: +Share progress with: `#getfitwithsagar #100DaysOfSRE #CloudSecurity #TLS #DevSecOps #NetworkDefense` --- -## Community & Support +## 13. Safety Notes -- [Discord](https://discord.gg/mNDm39qB8t) -- [Google Group](https://groups.google.com/forum/#!forum/daily-devops-sre-challenge-series/join) -- [YouTube](https://www.youtube.com/@Sagar.Utekar) +- Use `tmux`/`screen` before changing firewall if remote. +- Prepare rollback (cron or script) when imposing DROP defaults. +- Never commit private keys. +- Only scan systems you own or are authorized to test. +- Keep at least one active session while tightening rules. --- -## Safety Notes +## 14. Cleanup (Optional) -- Always test firewall changes in a **screen/tmux** session or with console access. -- For remote hosts, add a “rollback” cron job before applying strict DROP policies. -- Never share private keys in public repos. +```bash +sudo rm /etc/nginx/sites-enabled/day19 +sudo rm /etc/nginx/sites-available/day19 +sudo systemctl reload nginx +# Revert firewall manually if needed (e.g., set INPUT policy ACCEPT) +``` --- -**Mastering Day 19 gives you the confidence to guard availability, integrity, and confidentiality—from kernel packet paths to encrypted application sessions.** +## 15. Success Criteria + +You can: +- Explain stateful vs stateless & show working policy +- Present a 3-tier SG design with correct segmentation rationale +- Serve HTTPS, inspect cert + cipher, identify PFS component +- Simulate expiry risk & propose monitoring +- (Optionally) Demonstrate mTLS acceptance vs rejection +- Produce credible incident root cause analyses -Ready to build a fortress? Let’s ship it. +If yes → You’ve internalized Day 19. + +--- Happy securing, Sagar Utekar From 0c2521aa1fc1d96bf12fcf467d18d06f11ad5ae2 Mon Sep 17 00:00:00 2001 From: Sagar Utekar Date: Fri, 12 Sep 2025 05:43:30 +0530 Subject: [PATCH 119/133] Update 13_authorization_in_kubernetes.md --- CKA/13_authorization_in_kubernetes.md | 66 +++++++++++++++++++++++---- 1 file changed, 58 insertions(+), 8 deletions(-) diff --git a/CKA/13_authorization_in_kubernetes.md b/CKA/13_authorization_in_kubernetes.md index 96fc2bd..88a20b4 100644 --- a/CKA/13_authorization_in_kubernetes.md +++ b/CKA/13_authorization_in_kubernetes.md @@ -6,12 +6,12 @@ Kubernetes [authorization](https://kubernetes.io/docs/reference/access-authn-aut # Authorization modes ``` -1. AlwaysAllow -2. AlwaysDeny -3. ABAC (attribute-based access control) -4. RBAC (role-based access control) -5. Node -6. Webhook +1. AlwaysAllow → every request allowed (testing only). +2. AlwaysDeny → every request denied (testing only). +3. ABAC → Attribute-Based Access Control (legacy). +4. RBAC → Role-Based Access Control (most common in prod). +5. Node → kubelet authorizations to API server. +6. Webhook → delegate to external service (OPA, SSO, custom APIs). ``` # Using [RBAC](https://kubernetes.io/docs/reference/access-authn-authz/rbac/) Authorization @@ -20,7 +20,7 @@ To enable RBAC, start the API server with the --authorization-mode flag set to a kube-apiserver --authorization-mode=Example,RBAC ``` -RBAC helps with: +RBAC helps with: [Who (Subject) can do What (Verb) on Which (Resource)] 1. Establishing a system for users with different roles to access a set of Kubernetes resources. 2. Controlling processes running in a pod and the operations they can perform via Kubernetes API. 3. Limiting the visibility of certain resources per namespace @@ -65,7 +65,24 @@ RBAC consists of three key building blocks. kubectl get serviceaccounts kubectl describe serviceaccount build-bot kubectl run build-observer --image=alpine --restart=Never --serviceaccount=build-bot - ``` + ``` + + + ### Understanding ServiceAccount Token Generation and Authorization + ```mermaid + flowchart TD + P[Pod with serviceAccountName] --> SA[ServiceAccount object in namespace] + SA --> ST[Secret or Token request API] + ST --> T[Token generated for ServiceAccount] + T --> M[Token mounted into Pod filesystem at /var/run/secrets/kubernetes.io/serviceaccount] + M --> R[Pod process sends API request to kube-apiserver with Bearer token] + R --> KAS[Kube API Server] + + KAS -->|Token verified against ServiceAccount and bound Role/ClusterRole| AUTHZ[Authorization Check] + + AUTHZ -->|Allowed| OK[Request succeeds] + AUTHZ -->|Denied| FAIL[Request fails with 403 Forbidden] + ``` # Understanding [RBAC API Primitives](https://kubernetes.io/docs/reference/access-authn-authz/rbac/) The RBAC API declares four kinds of Kubernetes object: Role, ClusterRole, RoleBinding and ClusterRoleBinding. ## Role and ClusterRole @@ -83,6 +100,39 @@ The RBAC API declares four kinds of Kubernetes object: Role, ClusterRole, RoleBi - Alternatively, a RoleBinding can reference a ClusterRole and bind that ClusterRole to the namespace of the RoleBinding. If you want to bind a ClusterRole to all the namespaces in your cluster, you use a ClusterRoleBinding. - ClusterRoles are cluster-scoped, you can also use them to grant access to: cluster-scoped resources (like nodes), non-resource endpoints (like /healthz), namespaced resources (like Pods), across all namespaces. +```mermaid +flowchart TB + U1[User: sagar] + SA1[ServiceAccount: build-bot] + + subgraph Namespace: dev + R1[Role: pod-reader get list watch pods] + RB1[RoleBinding bind Role to User] + CRB1[RoleBinding bind ClusterRole to User in dev] + end + + subgraph Cluster + CR1[ClusterRole: cluster-admin\nfull access] + CRB2[ClusterRoleBinding bind ClusterRole to ServiceAccount cluster-wide] + end + + %% RoleBinding with Role + U1 --> RB1 + RB1 --> R1 + RB1 -->|namespace limited| Pods[Pods in dev] + + %% RoleBinding with ClusterRole + U1 --> CRB1 + CRB1 --> CR1 + CRB1 -->|still limited to dev| PodsDev[Pods in dev] + + %% ClusterRoleBinding with ClusterRole + SA1 --> CRB2 + CRB2 --> CR1 + CRB2 -->|cluster-wide| PodsAll[All Pods] + CRB2 -->|cluster-wide| Nodes[All Nodes] + +``` ### Kubernetes defines a set of [default user-facing roles](https://kubernetes.io/docs/reference/access-authn-authz/rbac/#user-facing-roles): cluster-admin, admin, edit, view and [core-component roles](https://kubernetes.io/docs/reference/access-authn-authz/rbac/#core-component-roles). 1. Roles management From e1f40da871941f85305fb9e79fbb5f048b56da92 Mon Sep 17 00:00:00 2001 From: Sagar Utekar Date: Fri, 12 Sep 2025 20:02:12 +0530 Subject: [PATCH 120/133] Update 10_authentication_in_kubernetes.md --- CKA/10_authentication_in_kubernetes.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/CKA/10_authentication_in_kubernetes.md b/CKA/10_authentication_in_kubernetes.md index 8e10d1b..c4f4596 100644 --- a/CKA/10_authentication_in_kubernetes.md +++ b/CKA/10_authentication_in_kubernetes.md @@ -108,7 +108,7 @@ kubectl certificate approve myuser # Retrieve the Signed Certificate After approval, the CSR object will have the signed certificate included in its status. You can extract the certificate from the CSR object: ``` -kubectl get csr username-csr -o jsonpath='{.status.certificate}' | base64 --decode > myuser.crt +kubectl get csr myuser -o jsonpath='{.status.certificate}' | base64 --decode > myuser.crt ``` # Create role and rolebinding From 2e44e89642df5c8857dcc2e72c505b5309fadd35 Mon Sep 17 00:00:00 2001 From: Sagar Utekar Date: Sat, 13 Sep 2025 20:58:45 +0530 Subject: [PATCH 121/133] Create 14_helm.md --- CKA/14_helm.md | 259 +++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 259 insertions(+) create mode 100644 CKA/14_helm.md diff --git a/CKA/14_helm.md b/CKA/14_helm.md new file mode 100644 index 0000000..6e40392 --- /dev/null +++ b/CKA/14_helm.md @@ -0,0 +1,259 @@ +### What is Helm + + * **What is Helm?** + + * Helm is the **package manager for Kubernetes**. + * Think of it like `apt` for Debian/Ubuntu, `yum` for CentOS, or `Homebrew` for macOS. + * It bundles complex Kubernetes applications into single, manageable packages called **Charts**. + + * **Core Terminology** + + * **Chart**: A package of pre-configured Kubernetes resource templates. It's the application's blueprint. + * **Release**: An instance of a Chart deployed into a Kubernetes cluster. You can deploy the same chart multiple times with different configurations (e.g., `prod-db` and `staging-db`). + * **Repository**: A location where Charts are stored and can be shared. + + * **Why Helm** + + * **Speed & Efficiency**: The exam is timed. Installing or upgrading a complex app with a single `helm` command is much faster than applying 10 separate YAML files with `kubectl`. + * **Application Lifecycle Management**: CKA tests your ability to manage applications. Helm is a primary tool for **installing, upgrading, rolling back, and deleting** applications. + +----- + +## Helm v3 High-Level Architecture +```mermaid +graph TD + A[User/CLI
helm install/upgrade] --> B{Helm Client v3} + B --> C[Templates
YAML manifests] + B --> D[Values
values.yaml or overrides] + C & D --> E[Templating Engine] + E --> F[Rendered Kubernetes YAML] + F --> G[Kubernetes API Server
applies the manifests] + G --> H[etcd
stores desired state] + G --> I[Kubelet
Runs app on nodes] +``` + +### Helm Workflow Explained + +This is a breakdown of how Helm works, from your command line to the Kubernetes cluster. + +--- + +### **User / CLI** + +This is your starting point. You, the user, interact with Helm by running commands in your terminal, such as: + +* `helm install` +* `helm upgrade` +* `helm rollback` +* `helm uninstall` + +Helm uses your existing `kubeconfig` file to connect to the target Kubernetes cluster, just like `kubectl` does. + +--- + +### **Helm Client (v3)** + +The **Helm client** is the program that runs on your local machine. It's the engine that processes your chart and values. + +* **Templating Engine**: This is the core of the client. It combines two key things to generate valid Kubernetes YAML manifests: + 1. The base Kubernetes resource definitions from the `templates/` directory of a chart. + 2. Configuration values from the `values.yaml` file or any custom values you provide (e.g., via `--set` or `-f`). +* **Release Management**: The client manages the release lifecycle. It stores a history of all changes for a specific release (like a timeline of upgrades and rollbacks) within the cluster, typically as Kubernetes **Secrets** or **ConfigMaps**. + +--- + +### **Kubernetes API Server** + +Once the Helm client has generated the final YAML manifests, it sends them to the **Kubernetes API Server**. + +* **Receipt**: The API Server receives the YAML, just as if you had run `kubectl apply -f `. +* **Application**: It applies these manifests to the cluster, ensuring that the desired state defined in the YAML is achieved. The API Server is responsible for storing this desired state in **etcd** and delegating the creation of Pods and other resources to the **kubelet** on the worker nodes. + +### The Anatomy of a Helm Chart + + * **`helm create `** + + * This is the command to scaffold a new chart with a standard, best-practice directory structure. + + * **Key Files and Directories** + + * `Chart.yaml`: Contains metadata about the chart (e.g., `name`, `version`, `description`, `apiVersion`). **This is a required file.** + * `values.yaml`: Provides the **default configuration values** for the chart. This file makes your chart customizable. + * `templates/`: A directory containing the Kubernetes manifest files, written in Go template language. This is where the real logic lives. + * `charts/`: A directory to place chart dependencies (sub-charts). + + ``` + mychart/ +│ +├── Chart.yaml # Chart metadata (name, version, description, dependencies) +├── values.yaml # Default configuration values (can be overridden at install time) +├── charts/ # Stores dependent charts (populated via `helm dependency update`) +├── templates/ # All Kubernetes manifests with Helm templating +│ ├── deployment.yaml # Defines the Deployment (pods, replicas, containers) +│ ├── service.yaml # Exposes the app as a Service +│ ├── ingress.yaml # (Optional) Creates Ingress for external access +│ ├── serviceaccount.yaml # Defines ServiceAccount for the app +│ ├── hpa.yaml # Horizontal Pod Autoscaler definition +│ ├── _helpers.tpl # Helper template functions (labels, names, etc.) +│ └── NOTES.txt # Post-install message shown after `helm install` +└── .helmignore # Ignore patterns when packaging the chart (like `.gitignore`) +``` + + * **Fundamental Commands** + + * `helm install `: Deploys a chart to the cluster. + * `helm list` (or `helm ls`): Lists all deployed releases in the current namespace. + * `helm uninstall `: Deletes a release and all of its associated Kubernetes resources. + +----- + +### Templates, Values, and Dry Runs** + + * **Go Templating** + + * Helm uses the Go template engine to generate valid Kubernetes YAML. + * Template directives are enclosed in `{{ ... }}`. + + * **The `.Values` Object** + + * This is the primary way to inject custom configuration into your templates. + * A value defined in `values.yaml` like `replicaCount: 3` is accessed in a template file as `{{ .Values.replicaCount }}`. + + * **CKA Debugging Essentials** + + * **`helm template `**: Renders the chart templates with values and prints the final YAML to your screen without contacting the cluster. **Use this to verify your output before deploying.** + * **`helm install --dry-run`**: Simulates an installation. It checks if the generated YAML is valid according to the Kubernetes API server but **does not actually create any resources**. This is perfect for catching API errors. + + * **Overriding Default Values** + + * `--set key=value`: Overrides a single value on the command line. + * *Example*: `helm install my-app ./my-chart --set replicaCount=5` + * `-f my-values.yaml` or `--values my-values.yaml`: Provide a separate YAML file with your overrides. + +----- + +### Functions and Pipelines** + + * **Functions**: Pre-defined utilities to transform data inside templates. The syntax is `{{ functionName .Argument }}`. + * **Pipelines `|`**: A powerful feature that lets you chain functions together. The output of one function becomes the input for the next. + * **Common & Useful Functions** + * `quote`: Wraps a string in double quotes. E.g., `{{ .Values.name | quote }}`. + * `default`: Provides a fallback value if the original is empty. E.g., `{{ .Values.image.tag | default "latest" }}`. + * `upper`: Converts a string to uppercase. + * `nindent`: Indents a block of text by a specified number of spaces. Crucial for formatting YAML correctly when using `include`. + +----- + +### Control Flow - `if/else` and `range`** + + * **`if/else` Blocks**: For conditionally generating blocks of YAML. This is key for creating optional resources. + ```yaml + {{- if .Values.ingress.enabled }} + apiVersion: networking.k8s.io/v1 + kind: Ingress + # ... rest of the ingress manifest + {{- end }} + ``` + * **`range` Action**: Iterates over a list (array) or map in your `values.yaml` to create multiple blocks of content. + ```yaml + # In values.yaml: + # extraPorts: + # - 8080 + # - 8081 + + # In service.yaml template: + ports: + {{- range .Values.extraPorts }} + - port: {{ . }} + targetPort: {{ . }} + {{- end }} + ``` + +----- + +### Named Templates & The `_helpers.tpl` File** + + * **The Problem**: You often repeat the same blocks of YAML, like labels, across multiple templates (`deployment.yaml`, `service.yaml`, etc.). This is not DRY (Don't Repeat Yourself). + * **The Solution**: Named templates, typically defined in `templates/_helpers.tpl`. + * **`define` Action**: Creates a reusable chunk of template code. + ```go + {{/* Define a common set of labels */}} + {{- define "mychart.labels" -}} + app.kubernetes.io/name: {{ .Chart.Name }} + helm.sh/chart: {{ printf "%s-%s" .Chart.Name .Chart.Version | quote }} + {{- end -}} + ``` + * **`include` Action**: Injects the named template into your Kubernetes manifest. + ```yaml + metadata: + labels: + {{- include "mychart.labels" . | nindent 8 }} + ``` + +----- + +### Managing a Release Lifecycle** + + * **These are the most critical Helm commands for the CKA exam.** + * **`helm upgrade `**: Upgrades an existing release with a new chart version or new configuration. + * Use the `--install` flag (`-i`) to create the release if it doesn't exist. This makes the command idempotent: `helm upgrade --install ...` + * **`helm history `**: View the revision history of a release. Every upgrade or rollback creates a new, numbered revision. + * **`helm rollback [REVISION]`**: Revert a release to a previous revision number. If no revision is specified, it rolls back to the previous one. This is your primary tool for recovering from a bad deployment. + * **`helm get all `**: An essential command to inspect the state of a deployed release. It shows you the user-supplied values, the computed values, the generated manifests, and any chart notes. + +----- + +### Chart Dependencies** + + * **Concept**: A chart can declare that it depends on other charts. For example, your WordPress chart might depend on the official MariaDB chart. + * **Mechanism** + 1. List dependencies in the `dependencies` section of `Chart.yaml`, specifying the name, version, and repository URL. + 2. Run `helm dependency update ` to download the dependency charts into your local `charts/` directory as `.tgz` archive files. + * When you install the parent chart, Helm ensures the dependency charts are installed as well. + +----- + +### CKA Exam Practice Scenarios** + + * **Scenario 1: Upgrade a Release** + + * **Task**: A release `webapp-v1` is using image tag `1.9.1`. Upgrade it to use the chart at `/opt/charts/webapp` which defaults to a newer version, and set the image tag specifically to `1.10.3`. + * **Solution**: `helm upgrade webapp-v1 /opt/charts/webapp --set image.tag=1.10.3` + + * **Scenario 2: Find and Fix a Bad Value** + + * **Task**: A deployment for the `api-prod` release is not working. Inspect its values and fix the `service.type` which is incorrectly set to `ClusterIP` when it should be `NodePort`. + * **Solution**: + 1. `helm get values api-prod` (To see the current values). + 2. `helm upgrade api-prod --set service.type=NodePort` (You'd get the chart path from `helm list -A`). + + * **Scenario 3: Roll Back a Failed Upgrade** + + * **Task**: You just upgraded the `database-main` release, creating revision \#4. The database pods are now in a `CrashLoopBackOff` state. Roll back the release to the previous working version. + * **Solution**: + 1. `helm history database-main` (To confirm the last good revision was \#3). + 2. `helm rollback database-main 3` + +## Helm Commands Cheat Sheet + +| Command | Explanation | +| -------------------------------------------------- | ----------------------------------------------------------------------------- | +| `helm version` | Check the installed Helm client version. | +| `helm repo add ` | Add a new Helm chart repository (e.g., Bitnami, stable). | +| `helm repo list` | List all added Helm repositories. | +| `helm repo update` | Refresh the list of charts from repos. | +| `helm search repo ` | Search for a chart inside configured repositories. | +| `helm create ` | Scaffold a new Helm chart with default files and templates. | +| `helm lint ` | Validate chart structure and templates. | +| `helm install ` | Install a chart with a release name. Example: `helm install myapp ./mychart`. | +| `helm upgrade ` | Upgrade an existing release to a new chart version/config. | +| `helm rollback ` | Roll back a release to a previous version. | +| `helm uninstall ` | Remove a release and delete its resources. | +| `helm list` | List all Helm releases in the current namespace. | +| `helm history ` | Show revision history of a release (useful for rollbacks). | +| `helm get values ` | Display user-supplied and default values for a release. | +| `helm get manifest ` | Show the full rendered Kubernetes manifests applied to the cluster. | +| `helm template ` | Render templates locally without installing (great for debugging). | +| `helm install --dry-run --debug` | Simulate an install, render templates, and debug issues without deploying. | +| `helm dependency update` | Download and update chart dependencies listed in `Chart.yaml`. | +| `helm package ` | Package a chart directory into a `.tgz` archive (distributable). | From 527ef2bf4b6fcdf02f6a2f27cf27785823762b51 Mon Sep 17 00:00:00 2001 From: Sagar Utekar Date: Sat, 13 Sep 2025 22:00:35 +0530 Subject: [PATCH 122/133] Update 14_helm.md --- CKA/14_helm.md | 79 ++++++++++++++++++++++++++++++++++++++------------ 1 file changed, 60 insertions(+), 19 deletions(-) diff --git a/CKA/14_helm.md b/CKA/14_helm.md index 6e40392..a688d45 100644 --- a/CKA/14_helm.md +++ b/CKA/14_helm.md @@ -213,26 +213,67 @@ Once the Helm client has generated the final YAML manifests, it sends them to th ----- -### CKA Exam Practice Scenarios** +### CKA Exam Practice Scenarios + +* **Scenario 1: Upgrade a Release** + + * **Task**: A release `webapp-v1` is using image tag `1.9.1`. Upgrade it to use the chart at `/opt/charts/webapp` which defaults to a newer version, and set the image tag specifically to `1.10.3`. + * **Solution**: + ```bash + helm upgrade webapp-v1 /opt/charts/webapp --set image.tag=1.10.3 + ``` + +* **Scenario 2: Find and Fix a Bad Value** + + * **Task**: A deployment for the `api-prod` release is not working. Inspect its values and fix the `service.type` which is incorrectly set to `ClusterIP` when it should be `NodePort`. + * **Solution**: + 1. ```bash + helm get values api-prod + ``` + (To see the current values). + 2. ```bash + helm upgrade api-prod --set service.type=NodePort + ``` + (You'd get the chart path from `helm list -A`). + +* **Scenario 3: Roll Back a Failed Upgrade** + + * **Task**: You just upgraded the `database-main` release, creating revision #4. The database pods are now in a `CrashLoopBackOff` state. Roll back the release to the previous working version. + * **Solution**: + 1. ```bash + helm history database-main + ``` + (To confirm the last good revision was #3). + 2. ```bash + helm rollback database-main 3 + ``` + +* **Scenario 4: Install a Third-Party Chart** + + * **Task**: Deploy external charts for common components like ingress and storage. + - Install **Traefik** (Ingress Controller) in a new namespace. + - Install the **MinIO Operator** in its own namespace. + + * **Solution**: + + **Traefik:** + ```bash + helm repo add traefik https://helm.traefik.io/traefik + helm repo update + kubectl create namespace traefik + helm install traefik traefik/traefik --namespace traefik --create-namespace + kubectl get all -n traefik + ``` + + **MinIO Operator:** + ```bash + helm repo add minio-operator https://operator.min.io + helm repo update + kubectl create namespace minio-operator + helm install operator minio-operator/operator --namespace minio-operator --create-namespace + kubectl get all -n minio-operator + ``` - * **Scenario 1: Upgrade a Release** - - * **Task**: A release `webapp-v1` is using image tag `1.9.1`. Upgrade it to use the chart at `/opt/charts/webapp` which defaults to a newer version, and set the image tag specifically to `1.10.3`. - * **Solution**: `helm upgrade webapp-v1 /opt/charts/webapp --set image.tag=1.10.3` - - * **Scenario 2: Find and Fix a Bad Value** - - * **Task**: A deployment for the `api-prod` release is not working. Inspect its values and fix the `service.type` which is incorrectly set to `ClusterIP` when it should be `NodePort`. - * **Solution**: - 1. `helm get values api-prod` (To see the current values). - 2. `helm upgrade api-prod --set service.type=NodePort` (You'd get the chart path from `helm list -A`). - - * **Scenario 3: Roll Back a Failed Upgrade** - - * **Task**: You just upgraded the `database-main` release, creating revision \#4. The database pods are now in a `CrashLoopBackOff` state. Roll back the release to the previous working version. - * **Solution**: - 1. `helm history database-main` (To confirm the last good revision was \#3). - 2. `helm rollback database-main 3` ## Helm Commands Cheat Sheet From f27327259e486fa0c830712b10d40f9a53bd936f Mon Sep 17 00:00:00 2001 From: Sagar Utekar Date: Sun, 14 Sep 2025 12:57:33 +0530 Subject: [PATCH 123/133] Create 15_kustomize.md --- CKA/15_kustomize.md | 1 + 1 file changed, 1 insertion(+) create mode 100644 CKA/15_kustomize.md diff --git a/CKA/15_kustomize.md b/CKA/15_kustomize.md new file mode 100644 index 0000000..8b13789 --- /dev/null +++ b/CKA/15_kustomize.md @@ -0,0 +1 @@ + From 339a41437bc9513da397649b9ea109091b281e45 Mon Sep 17 00:00:00 2001 From: Sagar Utekar Date: Sun, 14 Sep 2025 13:02:10 +0530 Subject: [PATCH 124/133] Update 15_kustomize.md --- CKA/15_kustomize.md | 217 ++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 217 insertions(+) diff --git a/CKA/15_kustomize.md b/CKA/15_kustomize.md index 8b13789..efbb4a0 100644 --- a/CKA/15_kustomize.md +++ b/CKA/15_kustomize.md @@ -1 +1,218 @@ +### Kustomize: End-to-End Walkthrough + +**Core Concept**: Kustomize is a **declarative, overlay-based** tool for customizing Kubernetes manifests. It doesn't use a templating language; it modifies existing YAML files. + +#### Key Components + + * **`base`**: The directory containing the original, unmodified Kubernetes manifests. This is your single source of truth. + * **`overlays`**: Directories that contain a `kustomization.yaml` file and patches to modify the `base` for specific environments. + * **`kustomization.yaml`**: The central file that defines which resources to manage and what patches or modifications to apply. + +#### End-to-End Example + +Let's say you have a basic Nginx deployment. + +**1. Create a `base` directory with your original manifests.** + +```bash +mkdir -p app/base +cd app/base +``` + +`deployment.yaml` + +```yaml +apiVersion: apps/v1 +kind: Deployment +metadata: + name: nginx-deployment +spec: + replicas: 1 + selector: + matchLabels: + app: nginx + template: + metadata: + labels: + app: nginx + spec: + containers: + - name: nginx + image: nginx:1.21.0 + ports: + - containerPort: 80 +``` + +`service.yaml` + +```yaml +apiVersion: v1 +kind: Service +metadata: + name: nginx-service +spec: + type: ClusterIP + ports: + - port: 80 + targetPort: 80 + selector: + app: nginx +``` + +**2. Create a `kustomization.yaml` in the `base` directory.** +This file simply lists the resources you want to manage. + +`kustomization.yaml` + +```yaml +resources: +- deployment.yaml +- service.yaml +``` + +**3. Create an `overlays` directory for a "production" environment.** +In this environment, you want to increase replicas and add an environment-specific label. + +```bash +mkdir -p ../overlays/prod +cd ../overlays/prod +``` + +`kustomization.yaml` + +```yaml +# kustomization.yaml in overlays/prod +resources: +- ../../base # Reference the base directory + +# Add a common label to all resources +commonLabels: + env: production + +# Use a patch to modify the deployment +patches: +- patch: |- + - op: replace + path: /spec/replicas + value: 3 + target: + kind: Deployment + name: nginx-deployment +``` + +**4. Deploy the application for a specific environment.** + + * **Dry Run**: To see the final YAML without applying it. + ```bash + kubectl kustomize app/overlays/prod + # or the standalone command: + kustomize build app/overlays/prod + ``` + * **Deploy**: To apply the production configuration. + ```bash + kubectl apply -k app/overlays/prod + ``` + +----- + +### Helm: End-to-End Walkthrough + +**Core Concept**: Helm is a **templating-based package manager** for Kubernetes. It uses Go templates and values to generate manifests. + +#### Key Components + + * **`Chart`**: A package that contains all necessary resources. + * **`values.yaml`**: The file for default configuration values. + * **`templates/`**: Directory where Kubernetes manifest templates live. + +#### End-to-End Example + +Let's create a similar Nginx deployment using Helm. + +**1. Scaffold a new Helm chart.** + +```bash +helm create my-nginx-app +cd my-nginx-app +``` + +**2. Modify `values.yaml` for configuration.** +You'll use this file to manage the image tag and replica count. + +`values.yaml` + +```yaml +replicaCount: 1 + +image: + repository: nginx + tag: 1.21.0 + pullPolicy: IfNotPresent + +service: + type: ClusterIP + port: 80 +``` + +**3. Modify `templates/deployment.yaml` to use values.** +The template uses Go template syntax to pull values from `values.yaml`. + +`templates/deployment.yaml` + +```yaml +apiVersion: apps/v1 +kind: Deployment +metadata: + name: {{ include "my-nginx-app.fullname" . }} + labels: + {{- include "my-nginx-app.labels" . | nindent 4 }} +spec: + replicas: {{ .Values.replicaCount }} + selector: + matchLabels: + app: nginx + template: + metadata: + labels: + app: nginx + spec: + containers: + - name: {{ .Chart.Name }} + image: "{{ .Values.image.repository }}:{{ .Values.image.tag }}" + ports: + - name: http + containerPort: 80 +``` + +**4. Install or upgrade the application.** + + * **Dry Run**: To see the final YAML. + ```bash + helm template my-nginx-app . + ``` + * **Install**: To deploy the chart with a release name. + ```bash + helm install my-nginx-release . + ``` + * **Upgrade**: To update the replica count for production. + ```bash + helm upgrade my-nginx-release . --set replicaCount=3 + ``` + * **Rollback**: To revert to a previous version if an upgrade fails. + ```bash + helm rollback my-nginx-release [REVISION_NUMBER] + ``` + +----- + +### Final Comparison + +| Feature | Helm | Kustomize | +| :--- | :--- | :--- | +| **Philosophy** | **Templating & Packaging**: "Chart your application." | **Overlay & Customization**: "Customize your manifest." | +| **Best For** | Distributing reusable applications, managing complex dependency trees (e.g., WordPress with MariaDB). | Managing different environments (dev, prod) for your own applications. | +| **Release Management** | **Built-in**. Tracks history, supports one-command rollback. | **Manual**. You manage history via Git commits. | +| **Learning Curve** | **Higher**. Requires learning Go templating and Helm-specific conventions. | **Lower**. Uses standard YAML and feels like a natural extension of `kubectl`. | +| **Integration** | A separate CLI tool (`helm`). | **Built-in** to `kubectl` since v1.14. | +| **How It Works** | Combines **templates** and **values** to generate YAML. | Combines **base** manifests and **patches** to generate YAML. | From a3d46dbf13e9445269e2240e3e0ecb8ff3362301 Mon Sep 17 00:00:00 2001 From: Sagar Utekar Date: Mon, 15 Sep 2025 19:52:58 +0530 Subject: [PATCH 125/133] Update 15_kustomize.md --- CKA/15_kustomize.md | 350 +++++++++++++++++++++++++++++++++++++------- 1 file changed, 299 insertions(+), 51 deletions(-) diff --git a/CKA/15_kustomize.md b/CKA/15_kustomize.md index efbb4a0..53e1dcb 100644 --- a/CKA/15_kustomize.md +++ b/CKA/15_kustomize.md @@ -1,26 +1,88 @@ -### Kustomize: End-to-End Walkthrough +### Kustomize: +#### Why do we need Kustomize +**Environment-specific customizations** +Dev vs Test vs Prod often need different replicas, resource limits/requests, secrets/configMaps, annotations, etc. Using separate full manifests for each environment leads to duplication and drift. Kustomize helps by overlaying only what's different. + +**Avoid duplication** +Base manifests avoid having the same repeated chunks across environments. If something changes in base (e.g. container image version, selector labels), you change in one place. Overlays only handle differences. + +**Declarative management & versioning** +All manifests are YAML (no custom templating syntax), easier to review / diff / track in Git. + +**Native integration with kubectl** +Kustomize is built into kubectl (for versions ≥ ~1.14) via kubectl apply -k or kubectl kustomize. + +**Secret / ConfigMap generation** +You can generate ConfigMaps and Secrets dynamically via configMapGenerator and secretGenerator. + +**Transformers & Patches** +You can apply naming prefixes/suffixes, labels, annotations, variable image tags, or patches (strategic merge or JSON patches) so only minimal YAML is changed. **Core Concept**: Kustomize is a **declarative, overlay-based** tool for customizing Kubernetes manifests. It doesn't use a templating language; it modifies existing YAML files. #### Key Components +| Concept | What it is / Does | Why it matters / Example | +| -------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------- | +| **Base** | Directory containing the “common” Kubernetes manifests (resources) and a kustomization.yaml that refers to them. | Serve as the single source of truth for common configuration across all environments. | +| **Overlay** | Directory for an environment that refers to base plus overlay patches or overlay resources that change some parts. | E.g. overlay for dev might set `replicas: 2`, overlay for prod might set `replicas: 4`, add rolling update strategy, etc. | +| `kustomization.yaml` | The control file where you list resources, patches, transformers (namePrefix, nameSuffix, labels, etc.), generators, etc. | It is the entry point of Kustomize: tells what to build and how to transform. | +| Transformers | Built-in features in Kustomize that modify multiple YAML resources in a standard way: e.g. add a common label, add a prefix, change namespace, modify images. | Good for consistent modifications across many resources without writing patches manually. | +| Patches | Changes applied to base manifests to tailor them per overlay. Two main kinds: strategic-merge patches, JSON 6902 patches. Can be inline or via separate files. | Patches let you override specific fields without retyping or duplicating whole manifests. | +| Generators | `configMapGenerator`, `secretGenerator` – to produce configMaps or secrets from key/value or file inputs. Can help avoid storing secrets in plaintext. | Dynamically generate required configMaps or secrets for environments. | - * **`base`**: The directory containing the original, unmodified Kubernetes manifests. This is your single source of truth. - * **`overlays`**: Directories that contain a `kustomization.yaml` file and patches to modify the `base` for specific environments. - * **`kustomization.yaml`**: The central file that defines which resources to manage and what patches or modifications to apply. +#### Installation & Usage Basics + +- Kustomize can be used as a standalone binary or via kubectl. + +1. Using kubectl: kubectl kustomize or kubectl apply -k +2. Using standalone: download/install from its GitHub repo or via package managers (brew, chocolatey, etc.). -#### End-to-End Example -Let's say you have a basic Nginx deployment. +--- -**1. Create a `base` directory with your original manifests.** +# Kustomize End-to-End Example + +This example demonstrates how to use **Kustomize** to manage a simple Nginx application across multiple environments: **dev**, **uat**, and **prod**. + +--- + +## 1. Create the Project Structure ```bash -mkdir -p app/base -cd app/base +mkdir -p kustomize-demo/base +mkdir -p kustomize-demo/overlays/dev +mkdir -p kustomize-demo/overlays/uat +mkdir -p kustomize-demo/overlays/prod +cd kustomize-demo +``` + +Your folder layout will look like this: + +``` +├── kustomize + ├── base + │ ├── deployment.yaml + │ ├── service.yaml + │ ├── kustomization.yaml + └ overlays + ├── dev + │ ├── deployment-dev.yaml + | ├── service-dev.yaml + │ └── kustomization.yaml + └── prod + ├── deployment-prod.yaml + ├── service-prod.yaml + └── kustomization.yaml ``` -`deployment.yaml` +--- + +## 2. Define the Base Manifests + +Inside `base/`, create the common deployment and service used across all environments. + +### `base/deployment.yaml` ```yaml apiVersion: apps/v1 @@ -44,7 +106,7 @@ spec: - containerPort: 80 ``` -`service.yaml` +### `base/service.yaml` ```yaml apiVersion: v1 @@ -60,59 +122,204 @@ spec: app: nginx ``` -**2. Create a `kustomization.yaml` in the `base` directory.** -This file simply lists the resources you want to manage. +### `base/kustomization.yaml` + +```yaml +apiVersion: kustomize.config.k8s.io/v1beta1 +kind: Kustomization + +resources: + - deployment.yaml + - service.yaml +``` + +--- + +## 3. Create Overlays for Each Environment + +Each overlay points to the base and applies environment-specific changes. -`kustomization.yaml` +--- + +### **Dev Overlay** + +```bash +cd overlays/dev +``` + +#### `overlays/dev/kustomization.yaml` ```yaml +apiVersion: kustomize.config.k8s.io/v1beta1 +kind: Kustomization + resources: -- deployment.yaml -- service.yaml + - ../../base + +commonLabels: + env: dev + +patches: +- patch: |- + - op: replace + path: /spec/replicas + value: 1 + target: + kind: Deployment + name: nginx-deployment ``` -**3. Create an `overlays` directory for a "production" environment.** -In this environment, you want to increase replicas and add an environment-specific label. +--- + +### **UAT Overlay** ```bash -mkdir -p ../overlays/prod -cd ../overlays/prod +cd ../uat ``` -`kustomization.yaml` +#### `overlays/uat/kustomization.yaml` ```yaml -# kustomization.yaml in overlays/prod +apiVersion: kustomize.config.k8s.io/v1beta1 +kind: Kustomization + resources: -- ../../base # Reference the base directory + - ../../base -# Add a common label to all resources commonLabels: - env: production + env: uat -# Use a patch to modify the deployment patches: - patch: |- - op: replace path: /spec/replicas - value: 3 + value: 2 target: kind: Deployment name: nginx-deployment ``` -**4. Deploy the application for a specific environment.** +--- - * **Dry Run**: To see the final YAML without applying it. - ```bash - kubectl kustomize app/overlays/prod - # or the standalone command: - kustomize build app/overlays/prod - ``` - * **Deploy**: To apply the production configuration. - ```bash - kubectl apply -k app/overlays/prod - ``` +### **Prod Overlay** + +```bash +cd ../prod +``` + +#### `overlays/prod/kustomization.yaml` + +```yaml +apiVersion: kustomize.config.k8s.io/v1beta1 +kind: Kustomization + +resources: + - ../../base + +commonLabels: + env: prod + +patches: +- patch: |- + - op: replace + path: /spec/replicas + value: 5 + target: + kind: Deployment + name: nginx-deployment + +- patch: |- + - op: add + path: /spec/strategy + value: + type: RollingUpdate + rollingUpdate: + maxUnavailable: 1 + maxSurge: 1 + target: + kind: Deployment + name: nginx-deployment +``` + +--- + +## 4. Build and Preview the Manifests + +Navigate back to the root of your project: + +```bash +cd ../../../kustomize-demo +``` + +Run Kustomize to preview manifests for each environment: + +* **Dev** + + ```bash + kubectl kustomize overlays/dev + ``` + +* **UAT** + + ```bash + kubectl kustomize overlays/uat + ``` + +* **Prod** + + ```bash + kubectl kustomize overlays/prod + ``` + +Or use the standalone Kustomize binary if installed: + +```bash +kustomize build overlays/prod +``` + +--- + +## 5. Deploy to Kubernetes + +Once you’re happy with the rendered YAML, deploy it to your cluster: + +* **Dev** + + ```bash + kubectl apply -k overlays/dev + ``` + +* **UAT** + + ```bash + kubectl apply -k overlays/uat + ``` + +* **Prod** + + ```bash + kubectl apply -k overlays/prod + ``` + +--- + +## 6. Verify the Deployment + +Check pods: + +```bash +kubectl get pods -l app=nginx +``` + +Check services: + +```bash +kubectl get svc nginx-service +``` + +* Dev → 1 replica +* UAT → 2 replicas +* Prod → 5 replicas with rolling updates ----- @@ -203,16 +410,57 @@ spec: ```bash helm rollback my-nginx-release [REVISION_NUMBER] ``` - ------ - -### Final Comparison - -| Feature | Helm | Kustomize | -| :--- | :--- | :--- | -| **Philosophy** | **Templating & Packaging**: "Chart your application." | **Overlay & Customization**: "Customize your manifest." | -| **Best For** | Distributing reusable applications, managing complex dependency trees (e.g., WordPress with MariaDB). | Managing different environments (dev, prod) for your own applications. | -| **Release Management** | **Built-in**. Tracks history, supports one-command rollback. | **Manual**. You manage history via Git commits. | -| **Learning Curve** | **Higher**. Requires learning Go templating and Helm-specific conventions. | **Lower**. Uses standard YAML and feels like a natural extension of `kubectl`. | -| **Integration** | A separate CLI tool (`helm`). | **Built-in** to `kubectl` since v1.14. | -| **How It Works** | Combines **templates** and **values** to generate YAML. | Combines **base** manifests and **patches** to generate YAML. | +--- + +## Best Practices for Using Kustomize + +* **Keep base minimal and reusable** – only include common manifests (deployments, services, etc.) +* **Overlays should only contain differences** – avoid copying entire manifests into overlays +* **Use generators for ConfigMaps and Secrets** – instead of hardcoding values +* **Leverage `commonLabels` and `commonAnnotations`** – for consistent metadata across resources +* **Prefer JSON6902 patches for precision** – when modifying deeply nested fields +* **Validate rendered manifests** before applying using tools like `kubeval` or `kubeconform` +* **Integrate with GitOps tools** – e.g., ArgoCD or FluxCD, since they have first-class support for Kustomize +* **Avoid disabling ConfigMap/Secret name suffix hashes** – ensures pods restart when config changes +* **Use `namePrefix`/`nameSuffix` carefully** – to avoid breaking dependencies between resources +* **Keep overlays simple** – one overlay per environment is enough in most cases + +--- + +## Kustomize Transformers + +Transformers are plugins/features that let you modify multiple resources in a consistent way. + +Here’s the list of **built-in transformers**: + +| Transformer | Description | +| ------------------------- | -------------------------------------------------------------------------- | +| **namePrefix** | Adds a prefix to all resource names | +| **nameSuffix** | Adds a suffix to all resource names | +| **namespace** | Sets namespace for all resources | +| **commonLabels** | Adds labels to all resources | +| **commonAnnotations** | Adds annotations to all resources | +| **images** | Overrides container images (name, tag, digest) | +| **replicas** | Overrides replica counts for Deployments, StatefulSets, etc. | +| **patchesStrategicMerge** | Applies patches using strategic merge semantics | +| **patchesJson6902** | Applies patches using JSON6902 (RFC 6902) standard | +| **patches** | Unified way to define both JSON6902 and strategic patches (newer versions) | +| **configMapGenerator** | Generates ConfigMaps from literals or files | +| **secretGenerator** | Generates Secrets from literals or files | +| **vars** | Substitute variables in resources | + +--- + +## Kustomize vs Helm + +| Feature | **Kustomize** | **Helm** | +| -------------------------- | ----------------------------------------------- | ------------------------------------------------ | +| **Approach** | Base + overlays layering | Template rendering with values.yaml | +| **Learning Curve** | Low (YAML only) | Higher (Go templating, charts) | +| **Reusability** | Great for environment-specific differences | Great for packaging and sharing apps | +| **Ecosystem** | No central repo, DIY | Large public chart ecosystem | +| **Native kubectl Support** | Yes (`kubectl apply -k`) | No (needs `helm` binary) | +| **Dependencies** | Not supported | Supported (chart dependencies) | +| **Secrets Handling** | Basic (secretGenerator) | Advanced (sealed secrets, external plugins) | +| **Best Use Case** | Environment customization in GitOps workflows | App installation and dependency management | +| **Combine Together?** | Yes (render Helm charts → patch with Kustomize) | Yes (use Helm for install, Kustomize for tweaks) | From 71e97bd39c4b5e2151f3273f65e9a5bbd56da287 Mon Sep 17 00:00:00 2001 From: Sagar Utekar Date: Mon, 15 Sep 2025 19:57:23 +0530 Subject: [PATCH 126/133] Update 15_kustomize.md --- CKA/15_kustomize.md | 25 +++++++++++++++++++++++++ 1 file changed, 25 insertions(+) diff --git a/CKA/15_kustomize.md b/CKA/15_kustomize.md index 53e1dcb..24aedf1 100644 --- a/CKA/15_kustomize.md +++ b/CKA/15_kustomize.md @@ -21,6 +21,31 @@ You can apply naming prefixes/suffixes, labels, annotations, variable image tags **Core Concept**: Kustomize is a **declarative, overlay-based** tool for customizing Kubernetes manifests. It doesn't use a templating language; it modifies existing YAML files. +```mermaid +flowchart TD + A[Base Manifests- deployment.yaml- service.yaml- hpa.yaml- configmap.yaml- kustomization.yaml] + B[Overlay: Dev kustomization.yaml] + C[Overlay: UAT kustomization.yaml] + D[Overlay: Prod kustomization.yaml] + E[Kustomize Build Generates final manifests] + F[Dev Cluster] + G[UAT Cluster] + H[Prod Cluster] + + A --> B + A --> C + A --> D + + B --> E + C --> E + D --> E + + E --> F + E --> G + E --> H + +``` + #### Key Components | Concept | What it is / Does | Why it matters / Example | | -------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------- | From 64c6e54f434721e200ef8dcb7fbcad6a5ced25cc Mon Sep 17 00:00:00 2001 From: Sagar Utekar Date: Mon, 15 Sep 2025 20:23:32 +0530 Subject: [PATCH 127/133] Update 15_kustomize.md --- CKA/15_kustomize.md | 22 ++++++++++++---------- 1 file changed, 12 insertions(+), 10 deletions(-) diff --git a/CKA/15_kustomize.md b/CKA/15_kustomize.md index 24aedf1..a053b8a 100644 --- a/CKA/15_kustomize.md +++ b/CKA/15_kustomize.md @@ -1,25 +1,27 @@ -### Kustomize: +### Kustomize: Kustomize is a **declarative, overlay-based** tool for customizing Kubernetes manifests. It doesn't use a templating language; it modifies existing YAML files + #### Why do we need Kustomize **Environment-specific customizations** -Dev vs Test vs Prod often need different replicas, resource limits/requests, secrets/configMaps, annotations, etc. Using separate full manifests for each environment leads to duplication and drift. Kustomize helps by overlaying only what's different. +- Dev vs Test vs Prod often need different replicas, resource limits/requests, secrets/configMaps, annotations, etc. +- Using separate full manifests for each environment leads to duplication and drift. Kustomize helps by overlaying only what's different. **Avoid duplication** -Base manifests avoid having the same repeated chunks across environments. If something changes in base (e.g. container image version, selector labels), you change in one place. Overlays only handle differences. +- Base manifests avoid having the same repeated chunks across environments. +- If something changes in base (e.g. container image version, selector labels), you change in one place. Overlays only handle differences. **Declarative management & versioning** -All manifests are YAML (no custom templating syntax), easier to review / diff / track in Git. +- All manifests are YAML (no custom templating syntax), easier to review / diff / track in Git. **Native integration with kubectl** -Kustomize is built into kubectl (for versions ≥ ~1.14) via kubectl apply -k or kubectl kustomize. +- Kustomize is built into kubectl (for versions ≥ ~1.14) via kubectl apply -k or kubectl kustomize. **Secret / ConfigMap generation** -You can generate ConfigMaps and Secrets dynamically via configMapGenerator and secretGenerator. +- You can generate ConfigMaps and Secrets dynamically via configMapGenerator and secretGenerator. **Transformers & Patches** -You can apply naming prefixes/suffixes, labels, annotations, variable image tags, or patches (strategic merge or JSON patches) so only minimal YAML is changed. +- You can apply naming prefixes/suffixes, labels, annotations, variable image tags, or patches (strategic merge or JSON patches) so only minimal YAML is changed. -**Core Concept**: Kustomize is a **declarative, overlay-based** tool for customizing Kubernetes manifests. It doesn't use a templating language; it modifies existing YAML files. ```mermaid flowchart TD @@ -79,7 +81,7 @@ mkdir -p kustomize-demo/base mkdir -p kustomize-demo/overlays/dev mkdir -p kustomize-demo/overlays/uat mkdir -p kustomize-demo/overlays/prod -cd kustomize-demo +cd kustomize ``` Your folder layout will look like this: @@ -273,7 +275,7 @@ patches: Navigate back to the root of your project: ```bash -cd ../../../kustomize-demo +cd ../../../kustomize ``` Run Kustomize to preview manifests for each environment: From 98c1015c81292d046155b279d58781d73bb03024 Mon Sep 17 00:00:00 2001 From: Sagar Utekar Date: Mon, 15 Sep 2025 20:36:23 +0530 Subject: [PATCH 128/133] Update 15_kustomize.md --- CKA/15_kustomize.md | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/CKA/15_kustomize.md b/CKA/15_kustomize.md index a053b8a..c8fbb89 100644 --- a/CKA/15_kustomize.md +++ b/CKA/15_kustomize.md @@ -183,6 +183,9 @@ kind: Kustomization resources: - ../../base +namespace: dev +nameSuffix: -dev + commonLabels: env: dev @@ -215,6 +218,8 @@ resources: commonLabels: env: uat +namespace: uat +nameSuffix: -uat patches: - patch: |- @@ -245,6 +250,8 @@ resources: commonLabels: env: prod +namespace: prod +nameSuffix: -prod patches: - patch: |- From 3709c89d0c7278efb1adbb9ce9a4df2d95545705 Mon Sep 17 00:00:00 2001 From: Sagar Utekar Date: Mon, 15 Sep 2025 22:33:53 +0530 Subject: [PATCH 129/133] Create 16_extension_interfaces.md --- CKA/16_extension_interfaces.md | 1 + 1 file changed, 1 insertion(+) create mode 100644 CKA/16_extension_interfaces.md diff --git a/CKA/16_extension_interfaces.md b/CKA/16_extension_interfaces.md new file mode 100644 index 0000000..8b13789 --- /dev/null +++ b/CKA/16_extension_interfaces.md @@ -0,0 +1 @@ + From 59f0567d064f3063ca2f3260592580af7e65e556 Mon Sep 17 00:00:00 2001 From: Sagar Utekar Date: Sun, 28 Sep 2025 00:28:20 +0530 Subject: [PATCH 130/133] Create answers.md --- QOTD/answers.md | 1 + 1 file changed, 1 insertion(+) create mode 100644 QOTD/answers.md diff --git a/QOTD/answers.md b/QOTD/answers.md new file mode 100644 index 0000000..8b13789 --- /dev/null +++ b/QOTD/answers.md @@ -0,0 +1 @@ + From 3ca46c9ba0605d01f3fe328031e8ca0b8490db91 Mon Sep 17 00:00:00 2001 From: Sagar Utekar Date: Sun, 28 Sep 2025 00:28:35 +0530 Subject: [PATCH 131/133] QOTD - Answers --- QOTD/answers.md | 16 ++++++++++++++++ 1 file changed, 16 insertions(+) diff --git a/QOTD/answers.md b/QOTD/answers.md index 8b13789..003f58b 100644 --- a/QOTD/answers.md +++ b/QOTD/answers.md @@ -1 +1,17 @@ +Question: What is blue-green Deployment? How is it useful? +Answer: Blue-green deployment is a methodology for releasing software in a controlled and low-risk manner. It involves creating two environments, referred to as blue and green. The blue environment is the live production environment, serving all client traffic, while the green environment is an identical clone used to test and validate the new version of the application. + +Once the new software version is ready and fully tested on the green environment, the traffic is switched (or gradually moved in case of canary testing) from the blue environment to the green environment. With this method, if there's a need to rollback, it's as simple as switching back the traffic to the previous version. + +This strategy is useful because it minimizes downtime during application deployment. The application remains available to users throughout the entire process. It also offers a failsafe—should something go wrong, rollback is easy and instantaneous by re-routing the traffic back to the blue environment (previous version). + +Diagram/Example: + + +In this diagram, initially, all requests are routed to the blue environment. When a new software version is ready, it is deployed to the green environment. After it passes all tests, the router starts routing the requests to the green environment. If the new version fails in production for some reason, the router goes back to routing requests to the blue environment. ```mermaid graph LR +A[User] -- Request --> B{{Router}} +B -- Route to --> C[Blue Environment] +B -- Route to --> D[Green Environment] +style C fill:#0000ff,stroke:#333,stroke-width:4px +style D fill:#008000,stroke:#333,stroke-width:4px \ No newline at end of file From 7424d4e46944f5657cb1ef3496b4025328c06173 Mon Sep 17 00:00:00 2001 From: Sagar Utekar Date: Sun, 28 Sep 2025 00:33:55 +0530 Subject: [PATCH 132/133] QOTD - Answers --- QOTD/answers.md | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/QOTD/answers.md b/QOTD/answers.md index 003f58b..ed7273c 100644 --- a/QOTD/answers.md +++ b/QOTD/answers.md @@ -1,4 +1,4 @@ -Question: What is blue-green Deployment? How is it useful? +=Question: What is blue-green Deployment? How is it useful? Answer: Blue-green deployment is a methodology for releasing software in a controlled and low-risk manner. It involves creating two environments, referred to as blue and green. The blue environment is the live production environment, serving all client traffic, while the green environment is an identical clone used to test and validate the new version of the application. @@ -9,7 +9,9 @@ This strategy is useful because it minimizes downtime during application deploym Diagram/Example: -In this diagram, initially, all requests are routed to the blue environment. When a new software version is ready, it is deployed to the green environment. After it passes all tests, the router starts routing the requests to the green environment. If the new version fails in production for some reason, the router goes back to routing requests to the blue environment. ```mermaid graph LR +In this diagram, initially, all requests are routed to the blue environment. When a new software version is ready, it is deployed to the green environment. After it passes all tests, the router starts routing the requests to the green environment. If the new version fails in production for some reason, the router goes back to routing requests to the blue environment. + +graph LR A[User] -- Request --> B{{Router}} B -- Route to --> C[Blue Environment] B -- Route to --> D[Green Environment] From 3ddf2ded1192e89a693ea48c7866d3e2191c2271 Mon Sep 17 00:00:00 2001 From: Sagar Utekar Date: Sun, 28 Sep 2025 00:34:24 +0530 Subject: [PATCH 133/133] QOTD - Answers --- QOTD/answers.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/QOTD/answers.md b/QOTD/answers.md index ed7273c..0ae3820 100644 --- a/QOTD/answers.md +++ b/QOTD/answers.md @@ -10,7 +10,7 @@ Diagram/Example: In this diagram, initially, all requests are routed to the blue environment. When a new software version is ready, it is deployed to the green environment. After it passes all tests, the router starts routing the requests to the green environment. If the new version fails in production for some reason, the router goes back to routing requests to the blue environment. - +``` mermaid graph LR A[User] -- Request --> B{{Router}} B -- Route to --> C[Blue Environment]