diff --git a/.github/workflows/day7-challenge.yaml b/.github/workflows/day7-challenge.yaml new file mode 100644 index 0000000..c4399e0 --- /dev/null +++ b/.github/workflows/day7-challenge.yaml @@ -0,0 +1,42 @@ +name: Deploy Snake Game to EC2 + +on: + push: + branches: + - main + +jobs: + deploy: + runs-on: ubuntu-latest + + steps: + - name: Checkout repository + uses: actions/checkout@v4 + + - name: Deploy to EC2 + uses: appleboy/ssh-action@v0.1.10 + with: + host: ${{ secrets.EC2_HOST }} + username: ${{ secrets.EC2_USER }} + key: ${{ secrets.EC2_SSH_KEY }} + script: | + # Update system packages + sudo apt update -y + sudo apt install -y python3 python3-pip python3-venv git + + # Remove old directory if it exists + sudo rm -rf season2-snake_game + + # Clone fresh + git clone https://github.com/Sagar2366/season2-snake_game.git + cd season2-snake_game + ls -al + + # Set up a virtual environment + python3 -m venv venv + source venv/bin/activate + pip install --upgrade pip + pip install -r requirements.txt + sudo nohup python3 app.py > snake_game.log 2>&1 & + + echo "✅ Snake Game deployed successfully!" diff --git a/CKA/10_authentication_in_kubernetes.md b/CKA/10_authentication_in_kubernetes.md index 8e10d1b..c4f4596 100644 --- a/CKA/10_authentication_in_kubernetes.md +++ b/CKA/10_authentication_in_kubernetes.md @@ -108,7 +108,7 @@ kubectl certificate approve myuser # Retrieve the Signed Certificate After approval, the CSR object will have the signed certificate included in its status. You can extract the certificate from the CSR object: ``` -kubectl get csr username-csr -o jsonpath='{.status.certificate}' | base64 --decode > myuser.crt +kubectl get csr myuser -o jsonpath='{.status.certificate}' | base64 --decode > myuser.crt ``` # Create role and rolebinding diff --git a/CKA/13_authorization_in_kubernetes.md b/CKA/13_authorization_in_kubernetes.md index 96fc2bd..88a20b4 100644 --- a/CKA/13_authorization_in_kubernetes.md +++ b/CKA/13_authorization_in_kubernetes.md @@ -6,12 +6,12 @@ Kubernetes [authorization](https://kubernetes.io/docs/reference/access-authn-aut # Authorization modes ``` -1. AlwaysAllow -2. AlwaysDeny -3. ABAC (attribute-based access control) -4. RBAC (role-based access control) -5. Node -6. Webhook +1. AlwaysAllow → every request allowed (testing only). +2. AlwaysDeny → every request denied (testing only). +3. ABAC → Attribute-Based Access Control (legacy). +4. RBAC → Role-Based Access Control (most common in prod). +5. Node → kubelet authorizations to API server. +6. Webhook → delegate to external service (OPA, SSO, custom APIs). ``` # Using [RBAC](https://kubernetes.io/docs/reference/access-authn-authz/rbac/) Authorization @@ -20,7 +20,7 @@ To enable RBAC, start the API server with the --authorization-mode flag set to a kube-apiserver --authorization-mode=Example,RBAC ``` -RBAC helps with: +RBAC helps with: [Who (Subject) can do What (Verb) on Which (Resource)] 1. Establishing a system for users with different roles to access a set of Kubernetes resources. 2. Controlling processes running in a pod and the operations they can perform via Kubernetes API. 3. Limiting the visibility of certain resources per namespace @@ -65,7 +65,24 @@ RBAC consists of three key building blocks. kubectl get serviceaccounts kubectl describe serviceaccount build-bot kubectl run build-observer --image=alpine --restart=Never --serviceaccount=build-bot - ``` + ``` + + + ### Understanding ServiceAccount Token Generation and Authorization + ```mermaid + flowchart TD + P[Pod with serviceAccountName] --> SA[ServiceAccount object in namespace] + SA --> ST[Secret or Token request API] + ST --> T[Token generated for ServiceAccount] + T --> M[Token mounted into Pod filesystem at /var/run/secrets/kubernetes.io/serviceaccount] + M --> R[Pod process sends API request to kube-apiserver with Bearer token] + R --> KAS[Kube API Server] + + KAS -->|Token verified against ServiceAccount and bound Role/ClusterRole| AUTHZ[Authorization Check] + + AUTHZ -->|Allowed| OK[Request succeeds] + AUTHZ -->|Denied| FAIL[Request fails with 403 Forbidden] + ``` # Understanding [RBAC API Primitives](https://kubernetes.io/docs/reference/access-authn-authz/rbac/) The RBAC API declares four kinds of Kubernetes object: Role, ClusterRole, RoleBinding and ClusterRoleBinding. ## Role and ClusterRole @@ -83,6 +100,39 @@ The RBAC API declares four kinds of Kubernetes object: Role, ClusterRole, RoleBi - Alternatively, a RoleBinding can reference a ClusterRole and bind that ClusterRole to the namespace of the RoleBinding. If you want to bind a ClusterRole to all the namespaces in your cluster, you use a ClusterRoleBinding. - ClusterRoles are cluster-scoped, you can also use them to grant access to: cluster-scoped resources (like nodes), non-resource endpoints (like /healthz), namespaced resources (like Pods), across all namespaces. +```mermaid +flowchart TB + U1[User: sagar] + SA1[ServiceAccount: build-bot] + + subgraph Namespace: dev + R1[Role: pod-reader get list watch pods] + RB1[RoleBinding bind Role to User] + CRB1[RoleBinding bind ClusterRole to User in dev] + end + + subgraph Cluster + CR1[ClusterRole: cluster-admin\nfull access] + CRB2[ClusterRoleBinding bind ClusterRole to ServiceAccount cluster-wide] + end + + %% RoleBinding with Role + U1 --> RB1 + RB1 --> R1 + RB1 -->|namespace limited| Pods[Pods in dev] + + %% RoleBinding with ClusterRole + U1 --> CRB1 + CRB1 --> CR1 + CRB1 -->|still limited to dev| PodsDev[Pods in dev] + + %% ClusterRoleBinding with ClusterRole + SA1 --> CRB2 + CRB2 --> CR1 + CRB2 -->|cluster-wide| PodsAll[All Pods] + CRB2 -->|cluster-wide| Nodes[All Nodes] + +``` ### Kubernetes defines a set of [default user-facing roles](https://kubernetes.io/docs/reference/access-authn-authz/rbac/#user-facing-roles): cluster-admin, admin, edit, view and [core-component roles](https://kubernetes.io/docs/reference/access-authn-authz/rbac/#core-component-roles). 1. Roles management diff --git a/CKA/14_helm.md b/CKA/14_helm.md new file mode 100644 index 0000000..a688d45 --- /dev/null +++ b/CKA/14_helm.md @@ -0,0 +1,300 @@ +### What is Helm + + * **What is Helm?** + + * Helm is the **package manager for Kubernetes**. + * Think of it like `apt` for Debian/Ubuntu, `yum` for CentOS, or `Homebrew` for macOS. + * It bundles complex Kubernetes applications into single, manageable packages called **Charts**. + + * **Core Terminology** + + * **Chart**: A package of pre-configured Kubernetes resource templates. It's the application's blueprint. + * **Release**: An instance of a Chart deployed into a Kubernetes cluster. You can deploy the same chart multiple times with different configurations (e.g., `prod-db` and `staging-db`). + * **Repository**: A location where Charts are stored and can be shared. + + * **Why Helm** + + * **Speed & Efficiency**: The exam is timed. Installing or upgrading a complex app with a single `helm` command is much faster than applying 10 separate YAML files with `kubectl`. + * **Application Lifecycle Management**: CKA tests your ability to manage applications. Helm is a primary tool for **installing, upgrading, rolling back, and deleting** applications. + +----- + +## Helm v3 High-Level Architecture +```mermaid +graph TD + A[User/CLI
helm install/upgrade] --> B{Helm Client v3} + B --> C[Templates
YAML manifests] + B --> D[Values
values.yaml or overrides] + C & D --> E[Templating Engine] + E --> F[Rendered Kubernetes YAML] + F --> G[Kubernetes API Server
applies the manifests] + G --> H[etcd
stores desired state] + G --> I[Kubelet
Runs app on nodes] +``` + +### Helm Workflow Explained + +This is a breakdown of how Helm works, from your command line to the Kubernetes cluster. + +--- + +### **User / CLI** + +This is your starting point. You, the user, interact with Helm by running commands in your terminal, such as: + +* `helm install` +* `helm upgrade` +* `helm rollback` +* `helm uninstall` + +Helm uses your existing `kubeconfig` file to connect to the target Kubernetes cluster, just like `kubectl` does. + +--- + +### **Helm Client (v3)** + +The **Helm client** is the program that runs on your local machine. It's the engine that processes your chart and values. + +* **Templating Engine**: This is the core of the client. It combines two key things to generate valid Kubernetes YAML manifests: + 1. The base Kubernetes resource definitions from the `templates/` directory of a chart. + 2. Configuration values from the `values.yaml` file or any custom values you provide (e.g., via `--set` or `-f`). +* **Release Management**: The client manages the release lifecycle. It stores a history of all changes for a specific release (like a timeline of upgrades and rollbacks) within the cluster, typically as Kubernetes **Secrets** or **ConfigMaps**. + +--- + +### **Kubernetes API Server** + +Once the Helm client has generated the final YAML manifests, it sends them to the **Kubernetes API Server**. + +* **Receipt**: The API Server receives the YAML, just as if you had run `kubectl apply -f `. +* **Application**: It applies these manifests to the cluster, ensuring that the desired state defined in the YAML is achieved. The API Server is responsible for storing this desired state in **etcd** and delegating the creation of Pods and other resources to the **kubelet** on the worker nodes. + +### The Anatomy of a Helm Chart + + * **`helm create `** + + * This is the command to scaffold a new chart with a standard, best-practice directory structure. + + * **Key Files and Directories** + + * `Chart.yaml`: Contains metadata about the chart (e.g., `name`, `version`, `description`, `apiVersion`). **This is a required file.** + * `values.yaml`: Provides the **default configuration values** for the chart. This file makes your chart customizable. + * `templates/`: A directory containing the Kubernetes manifest files, written in Go template language. This is where the real logic lives. + * `charts/`: A directory to place chart dependencies (sub-charts). + + ``` + mychart/ +│ +├── Chart.yaml # Chart metadata (name, version, description, dependencies) +├── values.yaml # Default configuration values (can be overridden at install time) +├── charts/ # Stores dependent charts (populated via `helm dependency update`) +├── templates/ # All Kubernetes manifests with Helm templating +│ ├── deployment.yaml # Defines the Deployment (pods, replicas, containers) +│ ├── service.yaml # Exposes the app as a Service +│ ├── ingress.yaml # (Optional) Creates Ingress for external access +│ ├── serviceaccount.yaml # Defines ServiceAccount for the app +│ ├── hpa.yaml # Horizontal Pod Autoscaler definition +│ ├── _helpers.tpl # Helper template functions (labels, names, etc.) +│ └── NOTES.txt # Post-install message shown after `helm install` +└── .helmignore # Ignore patterns when packaging the chart (like `.gitignore`) +``` + + * **Fundamental Commands** + + * `helm install `: Deploys a chart to the cluster. + * `helm list` (or `helm ls`): Lists all deployed releases in the current namespace. + * `helm uninstall `: Deletes a release and all of its associated Kubernetes resources. + +----- + +### Templates, Values, and Dry Runs** + + * **Go Templating** + + * Helm uses the Go template engine to generate valid Kubernetes YAML. + * Template directives are enclosed in `{{ ... }}`. + + * **The `.Values` Object** + + * This is the primary way to inject custom configuration into your templates. + * A value defined in `values.yaml` like `replicaCount: 3` is accessed in a template file as `{{ .Values.replicaCount }}`. + + * **CKA Debugging Essentials** + + * **`helm template `**: Renders the chart templates with values and prints the final YAML to your screen without contacting the cluster. **Use this to verify your output before deploying.** + * **`helm install --dry-run`**: Simulates an installation. It checks if the generated YAML is valid according to the Kubernetes API server but **does not actually create any resources**. This is perfect for catching API errors. + + * **Overriding Default Values** + + * `--set key=value`: Overrides a single value on the command line. + * *Example*: `helm install my-app ./my-chart --set replicaCount=5` + * `-f my-values.yaml` or `--values my-values.yaml`: Provide a separate YAML file with your overrides. + +----- + +### Functions and Pipelines** + + * **Functions**: Pre-defined utilities to transform data inside templates. The syntax is `{{ functionName .Argument }}`. + * **Pipelines `|`**: A powerful feature that lets you chain functions together. The output of one function becomes the input for the next. + * **Common & Useful Functions** + * `quote`: Wraps a string in double quotes. E.g., `{{ .Values.name | quote }}`. + * `default`: Provides a fallback value if the original is empty. E.g., `{{ .Values.image.tag | default "latest" }}`. + * `upper`: Converts a string to uppercase. + * `nindent`: Indents a block of text by a specified number of spaces. Crucial for formatting YAML correctly when using `include`. + +----- + +### Control Flow - `if/else` and `range`** + + * **`if/else` Blocks**: For conditionally generating blocks of YAML. This is key for creating optional resources. + ```yaml + {{- if .Values.ingress.enabled }} + apiVersion: networking.k8s.io/v1 + kind: Ingress + # ... rest of the ingress manifest + {{- end }} + ``` + * **`range` Action**: Iterates over a list (array) or map in your `values.yaml` to create multiple blocks of content. + ```yaml + # In values.yaml: + # extraPorts: + # - 8080 + # - 8081 + + # In service.yaml template: + ports: + {{- range .Values.extraPorts }} + - port: {{ . }} + targetPort: {{ . }} + {{- end }} + ``` + +----- + +### Named Templates & The `_helpers.tpl` File** + + * **The Problem**: You often repeat the same blocks of YAML, like labels, across multiple templates (`deployment.yaml`, `service.yaml`, etc.). This is not DRY (Don't Repeat Yourself). + * **The Solution**: Named templates, typically defined in `templates/_helpers.tpl`. + * **`define` Action**: Creates a reusable chunk of template code. + ```go + {{/* Define a common set of labels */}} + {{- define "mychart.labels" -}} + app.kubernetes.io/name: {{ .Chart.Name }} + helm.sh/chart: {{ printf "%s-%s" .Chart.Name .Chart.Version | quote }} + {{- end -}} + ``` + * **`include` Action**: Injects the named template into your Kubernetes manifest. + ```yaml + metadata: + labels: + {{- include "mychart.labels" . | nindent 8 }} + ``` + +----- + +### Managing a Release Lifecycle** + + * **These are the most critical Helm commands for the CKA exam.** + * **`helm upgrade `**: Upgrades an existing release with a new chart version or new configuration. + * Use the `--install` flag (`-i`) to create the release if it doesn't exist. This makes the command idempotent: `helm upgrade --install ...` + * **`helm history `**: View the revision history of a release. Every upgrade or rollback creates a new, numbered revision. + * **`helm rollback [REVISION]`**: Revert a release to a previous revision number. If no revision is specified, it rolls back to the previous one. This is your primary tool for recovering from a bad deployment. + * **`helm get all `**: An essential command to inspect the state of a deployed release. It shows you the user-supplied values, the computed values, the generated manifests, and any chart notes. + +----- + +### Chart Dependencies** + + * **Concept**: A chart can declare that it depends on other charts. For example, your WordPress chart might depend on the official MariaDB chart. + * **Mechanism** + 1. List dependencies in the `dependencies` section of `Chart.yaml`, specifying the name, version, and repository URL. + 2. Run `helm dependency update ` to download the dependency charts into your local `charts/` directory as `.tgz` archive files. + * When you install the parent chart, Helm ensures the dependency charts are installed as well. + +----- + +### CKA Exam Practice Scenarios + +* **Scenario 1: Upgrade a Release** + + * **Task**: A release `webapp-v1` is using image tag `1.9.1`. Upgrade it to use the chart at `/opt/charts/webapp` which defaults to a newer version, and set the image tag specifically to `1.10.3`. + * **Solution**: + ```bash + helm upgrade webapp-v1 /opt/charts/webapp --set image.tag=1.10.3 + ``` + +* **Scenario 2: Find and Fix a Bad Value** + + * **Task**: A deployment for the `api-prod` release is not working. Inspect its values and fix the `service.type` which is incorrectly set to `ClusterIP` when it should be `NodePort`. + * **Solution**: + 1. ```bash + helm get values api-prod + ``` + (To see the current values). + 2. ```bash + helm upgrade api-prod --set service.type=NodePort + ``` + (You'd get the chart path from `helm list -A`). + +* **Scenario 3: Roll Back a Failed Upgrade** + + * **Task**: You just upgraded the `database-main` release, creating revision #4. The database pods are now in a `CrashLoopBackOff` state. Roll back the release to the previous working version. + * **Solution**: + 1. ```bash + helm history database-main + ``` + (To confirm the last good revision was #3). + 2. ```bash + helm rollback database-main 3 + ``` + +* **Scenario 4: Install a Third-Party Chart** + + * **Task**: Deploy external charts for common components like ingress and storage. + - Install **Traefik** (Ingress Controller) in a new namespace. + - Install the **MinIO Operator** in its own namespace. + + * **Solution**: + + **Traefik:** + ```bash + helm repo add traefik https://helm.traefik.io/traefik + helm repo update + kubectl create namespace traefik + helm install traefik traefik/traefik --namespace traefik --create-namespace + kubectl get all -n traefik + ``` + + **MinIO Operator:** + ```bash + helm repo add minio-operator https://operator.min.io + helm repo update + kubectl create namespace minio-operator + helm install operator minio-operator/operator --namespace minio-operator --create-namespace + kubectl get all -n minio-operator + ``` + + +## Helm Commands Cheat Sheet + +| Command | Explanation | +| -------------------------------------------------- | ----------------------------------------------------------------------------- | +| `helm version` | Check the installed Helm client version. | +| `helm repo add ` | Add a new Helm chart repository (e.g., Bitnami, stable). | +| `helm repo list` | List all added Helm repositories. | +| `helm repo update` | Refresh the list of charts from repos. | +| `helm search repo ` | Search for a chart inside configured repositories. | +| `helm create ` | Scaffold a new Helm chart with default files and templates. | +| `helm lint ` | Validate chart structure and templates. | +| `helm install ` | Install a chart with a release name. Example: `helm install myapp ./mychart`. | +| `helm upgrade ` | Upgrade an existing release to a new chart version/config. | +| `helm rollback ` | Roll back a release to a previous version. | +| `helm uninstall ` | Remove a release and delete its resources. | +| `helm list` | List all Helm releases in the current namespace. | +| `helm history ` | Show revision history of a release (useful for rollbacks). | +| `helm get values ` | Display user-supplied and default values for a release. | +| `helm get manifest ` | Show the full rendered Kubernetes manifests applied to the cluster. | +| `helm template ` | Render templates locally without installing (great for debugging). | +| `helm install --dry-run --debug` | Simulate an install, render templates, and debug issues without deploying. | +| `helm dependency update` | Download and update chart dependencies listed in `Chart.yaml`. | +| `helm package ` | Package a chart directory into a `.tgz` archive (distributable). | diff --git a/CKA/15_kustomize.md b/CKA/15_kustomize.md new file mode 100644 index 0000000..c8fbb89 --- /dev/null +++ b/CKA/15_kustomize.md @@ -0,0 +1,500 @@ + +### Kustomize: Kustomize is a **declarative, overlay-based** tool for customizing Kubernetes manifests. It doesn't use a templating language; it modifies existing YAML files + +#### Why do we need Kustomize +**Environment-specific customizations** +- Dev vs Test vs Prod often need different replicas, resource limits/requests, secrets/configMaps, annotations, etc. +- Using separate full manifests for each environment leads to duplication and drift. Kustomize helps by overlaying only what's different. + +**Avoid duplication** +- Base manifests avoid having the same repeated chunks across environments. +- If something changes in base (e.g. container image version, selector labels), you change in one place. Overlays only handle differences. + +**Declarative management & versioning** +- All manifests are YAML (no custom templating syntax), easier to review / diff / track in Git. + +**Native integration with kubectl** +- Kustomize is built into kubectl (for versions ≥ ~1.14) via kubectl apply -k or kubectl kustomize. + +**Secret / ConfigMap generation** +- You can generate ConfigMaps and Secrets dynamically via configMapGenerator and secretGenerator. + +**Transformers & Patches** +- You can apply naming prefixes/suffixes, labels, annotations, variable image tags, or patches (strategic merge or JSON patches) so only minimal YAML is changed. + + +```mermaid +flowchart TD + A[Base Manifests- deployment.yaml- service.yaml- hpa.yaml- configmap.yaml- kustomization.yaml] + B[Overlay: Dev kustomization.yaml] + C[Overlay: UAT kustomization.yaml] + D[Overlay: Prod kustomization.yaml] + E[Kustomize Build Generates final manifests] + F[Dev Cluster] + G[UAT Cluster] + H[Prod Cluster] + + A --> B + A --> C + A --> D + + B --> E + C --> E + D --> E + + E --> F + E --> G + E --> H + +``` + +#### Key Components +| Concept | What it is / Does | Why it matters / Example | +| -------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------- | +| **Base** | Directory containing the “common” Kubernetes manifests (resources) and a kustomization.yaml that refers to them. | Serve as the single source of truth for common configuration across all environments. | +| **Overlay** | Directory for an environment that refers to base plus overlay patches or overlay resources that change some parts. | E.g. overlay for dev might set `replicas: 2`, overlay for prod might set `replicas: 4`, add rolling update strategy, etc. | +| `kustomization.yaml` | The control file where you list resources, patches, transformers (namePrefix, nameSuffix, labels, etc.), generators, etc. | It is the entry point of Kustomize: tells what to build and how to transform. | +| Transformers | Built-in features in Kustomize that modify multiple YAML resources in a standard way: e.g. add a common label, add a prefix, change namespace, modify images. | Good for consistent modifications across many resources without writing patches manually. | +| Patches | Changes applied to base manifests to tailor them per overlay. Two main kinds: strategic-merge patches, JSON 6902 patches. Can be inline or via separate files. | Patches let you override specific fields without retyping or duplicating whole manifests. | +| Generators | `configMapGenerator`, `secretGenerator` – to produce configMaps or secrets from key/value or file inputs. Can help avoid storing secrets in plaintext. | Dynamically generate required configMaps or secrets for environments. | + +#### Installation & Usage Basics + +- Kustomize can be used as a standalone binary or via kubectl. + +1. Using kubectl: kubectl kustomize or kubectl apply -k +2. Using standalone: download/install from its GitHub repo or via package managers (brew, chocolatey, etc.). + + +--- + +# Kustomize End-to-End Example + +This example demonstrates how to use **Kustomize** to manage a simple Nginx application across multiple environments: **dev**, **uat**, and **prod**. + +--- + +## 1. Create the Project Structure + +```bash +mkdir -p kustomize-demo/base +mkdir -p kustomize-demo/overlays/dev +mkdir -p kustomize-demo/overlays/uat +mkdir -p kustomize-demo/overlays/prod +cd kustomize +``` + +Your folder layout will look like this: + +``` +├── kustomize + ├── base + │ ├── deployment.yaml + │ ├── service.yaml + │ ├── kustomization.yaml + └ overlays + ├── dev + │ ├── deployment-dev.yaml + | ├── service-dev.yaml + │ └── kustomization.yaml + └── prod + ├── deployment-prod.yaml + ├── service-prod.yaml + └── kustomization.yaml +``` + +--- + +## 2. Define the Base Manifests + +Inside `base/`, create the common deployment and service used across all environments. + +### `base/deployment.yaml` + +```yaml +apiVersion: apps/v1 +kind: Deployment +metadata: + name: nginx-deployment +spec: + replicas: 1 + selector: + matchLabels: + app: nginx + template: + metadata: + labels: + app: nginx + spec: + containers: + - name: nginx + image: nginx:1.21.0 + ports: + - containerPort: 80 +``` + +### `base/service.yaml` + +```yaml +apiVersion: v1 +kind: Service +metadata: + name: nginx-service +spec: + type: ClusterIP + ports: + - port: 80 + targetPort: 80 + selector: + app: nginx +``` + +### `base/kustomization.yaml` + +```yaml +apiVersion: kustomize.config.k8s.io/v1beta1 +kind: Kustomization + +resources: + - deployment.yaml + - service.yaml +``` + +--- + +## 3. Create Overlays for Each Environment + +Each overlay points to the base and applies environment-specific changes. + +--- + +### **Dev Overlay** + +```bash +cd overlays/dev +``` + +#### `overlays/dev/kustomization.yaml` + +```yaml +apiVersion: kustomize.config.k8s.io/v1beta1 +kind: Kustomization + +resources: + - ../../base + +namespace: dev +nameSuffix: -dev + +commonLabels: + env: dev + +patches: +- patch: |- + - op: replace + path: /spec/replicas + value: 1 + target: + kind: Deployment + name: nginx-deployment +``` + +--- + +### **UAT Overlay** + +```bash +cd ../uat +``` + +#### `overlays/uat/kustomization.yaml` + +```yaml +apiVersion: kustomize.config.k8s.io/v1beta1 +kind: Kustomization + +resources: + - ../../base + +commonLabels: + env: uat +namespace: uat +nameSuffix: -uat + +patches: +- patch: |- + - op: replace + path: /spec/replicas + value: 2 + target: + kind: Deployment + name: nginx-deployment +``` + +--- + +### **Prod Overlay** + +```bash +cd ../prod +``` + +#### `overlays/prod/kustomization.yaml` + +```yaml +apiVersion: kustomize.config.k8s.io/v1beta1 +kind: Kustomization + +resources: + - ../../base + +commonLabels: + env: prod +namespace: prod +nameSuffix: -prod + +patches: +- patch: |- + - op: replace + path: /spec/replicas + value: 5 + target: + kind: Deployment + name: nginx-deployment + +- patch: |- + - op: add + path: /spec/strategy + value: + type: RollingUpdate + rollingUpdate: + maxUnavailable: 1 + maxSurge: 1 + target: + kind: Deployment + name: nginx-deployment +``` + +--- + +## 4. Build and Preview the Manifests + +Navigate back to the root of your project: + +```bash +cd ../../../kustomize +``` + +Run Kustomize to preview manifests for each environment: + +* **Dev** + + ```bash + kubectl kustomize overlays/dev + ``` + +* **UAT** + + ```bash + kubectl kustomize overlays/uat + ``` + +* **Prod** + + ```bash + kubectl kustomize overlays/prod + ``` + +Or use the standalone Kustomize binary if installed: + +```bash +kustomize build overlays/prod +``` + +--- + +## 5. Deploy to Kubernetes + +Once you’re happy with the rendered YAML, deploy it to your cluster: + +* **Dev** + + ```bash + kubectl apply -k overlays/dev + ``` + +* **UAT** + + ```bash + kubectl apply -k overlays/uat + ``` + +* **Prod** + + ```bash + kubectl apply -k overlays/prod + ``` + +--- + +## 6. Verify the Deployment + +Check pods: + +```bash +kubectl get pods -l app=nginx +``` + +Check services: + +```bash +kubectl get svc nginx-service +``` + +* Dev → 1 replica +* UAT → 2 replicas +* Prod → 5 replicas with rolling updates + +----- + +### Helm: End-to-End Walkthrough + +**Core Concept**: Helm is a **templating-based package manager** for Kubernetes. It uses Go templates and values to generate manifests. + +#### Key Components + + * **`Chart`**: A package that contains all necessary resources. + * **`values.yaml`**: The file for default configuration values. + * **`templates/`**: Directory where Kubernetes manifest templates live. + +#### End-to-End Example + +Let's create a similar Nginx deployment using Helm. + +**1. Scaffold a new Helm chart.** + +```bash +helm create my-nginx-app +cd my-nginx-app +``` + +**2. Modify `values.yaml` for configuration.** +You'll use this file to manage the image tag and replica count. + +`values.yaml` + +```yaml +replicaCount: 1 + +image: + repository: nginx + tag: 1.21.0 + pullPolicy: IfNotPresent + +service: + type: ClusterIP + port: 80 +``` + +**3. Modify `templates/deployment.yaml` to use values.** +The template uses Go template syntax to pull values from `values.yaml`. + +`templates/deployment.yaml` + +```yaml +apiVersion: apps/v1 +kind: Deployment +metadata: + name: {{ include "my-nginx-app.fullname" . }} + labels: + {{- include "my-nginx-app.labels" . | nindent 4 }} +spec: + replicas: {{ .Values.replicaCount }} + selector: + matchLabels: + app: nginx + template: + metadata: + labels: + app: nginx + spec: + containers: + - name: {{ .Chart.Name }} + image: "{{ .Values.image.repository }}:{{ .Values.image.tag }}" + ports: + - name: http + containerPort: 80 +``` + +**4. Install or upgrade the application.** + + * **Dry Run**: To see the final YAML. + ```bash + helm template my-nginx-app . + ``` + * **Install**: To deploy the chart with a release name. + ```bash + helm install my-nginx-release . + ``` + * **Upgrade**: To update the replica count for production. + ```bash + helm upgrade my-nginx-release . --set replicaCount=3 + ``` + * **Rollback**: To revert to a previous version if an upgrade fails. + ```bash + helm rollback my-nginx-release [REVISION_NUMBER] + ``` +--- + +## Best Practices for Using Kustomize + +* **Keep base minimal and reusable** – only include common manifests (deployments, services, etc.) +* **Overlays should only contain differences** – avoid copying entire manifests into overlays +* **Use generators for ConfigMaps and Secrets** – instead of hardcoding values +* **Leverage `commonLabels` and `commonAnnotations`** – for consistent metadata across resources +* **Prefer JSON6902 patches for precision** – when modifying deeply nested fields +* **Validate rendered manifests** before applying using tools like `kubeval` or `kubeconform` +* **Integrate with GitOps tools** – e.g., ArgoCD or FluxCD, since they have first-class support for Kustomize +* **Avoid disabling ConfigMap/Secret name suffix hashes** – ensures pods restart when config changes +* **Use `namePrefix`/`nameSuffix` carefully** – to avoid breaking dependencies between resources +* **Keep overlays simple** – one overlay per environment is enough in most cases + +--- + +## Kustomize Transformers + +Transformers are plugins/features that let you modify multiple resources in a consistent way. + +Here’s the list of **built-in transformers**: + +| Transformer | Description | +| ------------------------- | -------------------------------------------------------------------------- | +| **namePrefix** | Adds a prefix to all resource names | +| **nameSuffix** | Adds a suffix to all resource names | +| **namespace** | Sets namespace for all resources | +| **commonLabels** | Adds labels to all resources | +| **commonAnnotations** | Adds annotations to all resources | +| **images** | Overrides container images (name, tag, digest) | +| **replicas** | Overrides replica counts for Deployments, StatefulSets, etc. | +| **patchesStrategicMerge** | Applies patches using strategic merge semantics | +| **patchesJson6902** | Applies patches using JSON6902 (RFC 6902) standard | +| **patches** | Unified way to define both JSON6902 and strategic patches (newer versions) | +| **configMapGenerator** | Generates ConfigMaps from literals or files | +| **secretGenerator** | Generates Secrets from literals or files | +| **vars** | Substitute variables in resources | + +--- + +## Kustomize vs Helm + +| Feature | **Kustomize** | **Helm** | +| -------------------------- | ----------------------------------------------- | ------------------------------------------------ | +| **Approach** | Base + overlays layering | Template rendering with values.yaml | +| **Learning Curve** | Low (YAML only) | Higher (Go templating, charts) | +| **Reusability** | Great for environment-specific differences | Great for packaging and sharing apps | +| **Ecosystem** | No central repo, DIY | Large public chart ecosystem | +| **Native kubectl Support** | Yes (`kubectl apply -k`) | No (needs `helm` binary) | +| **Dependencies** | Not supported | Supported (chart dependencies) | +| **Secrets Handling** | Basic (secretGenerator) | Advanced (sealed secrets, external plugins) | +| **Best Use Case** | Environment customization in GitOps workflows | App installation and dependency management | +| **Combine Together?** | Yes (render Helm charts → patch with Kustomize) | Yes (use Helm for install, Kustomize for tweaks) | diff --git a/DailyChallenges/Season2/Linux/syllabus.md b/CKA/16_extension_interfaces.md similarity index 100% rename from DailyChallenges/Season2/Linux/syllabus.md rename to CKA/16_extension_interfaces.md diff --git a/DailyChallenges/DAY10/solution.md b/DailyChallenges/DAY10/solution.md new file mode 100644 index 0000000..00abc22 --- /dev/null +++ b/DailyChallenges/DAY10/solution.md @@ -0,0 +1,248 @@ +## 3 Node rabbitMQ cluster setup and monitoring using Prometheus and Grafana + +### Steps to implement: + +- Create 3 EC2 instance with the following details: + + - **AMI:** Ubuntu + - **Instance type:** t2.medium + - **Volume:** 15GB + - **Security group ports**: + + - 5672 (AMQP) for RabbitMQ messaging + - 15672 (RabbitMQ management UI) + - 15692 (Prometheus metrics endpoint) + - 9090 (Prometheus) + - 3000 (Grafana) +--- + +- Update system (All nodes): +```bash +sudo su +apt update +``` +--- + +- create a shell script to install erlang and rabbitmq-server (All nodes): +```bash +#!/bin/sh + +sudo apt-get install curl gnupg apt-transport-https -y + +## Team RabbitMQ's main signing key +curl -1sLf "https://keys.openpgp.org/vks/v1/by-fingerprint/0A9AF2115F4687BD29803A206B73A36E6026DFCA" | sudo gpg --dearmor | sudo tee /usr/share/keyrings/com.rabbitmq.team.gpg > /dev/null +## Community mirror of Cloudsmith: modern Erlang repository +curl -1sLf https://github.com/rabbitmq/signing-keys/releases/download/3.0/cloudsmith.rabbitmq-erlang.E495BB49CC4BBE5B.key | sudo gpg --dearmor | sudo tee /usr/share/keyrings/rabbitmq.E495BB49CC4BBE5B.gpg > /dev/null +## Community mirror of Cloudsmith: RabbitMQ repository +curl -1sLf https://github.com/rabbitmq/signing-keys/releases/download/3.0/cloudsmith.rabbitmq-server.9F4587F226208342.key | sudo gpg --dearmor | sudo tee /usr/share/keyrings/rabbitmq.9F4587F226208342.gpg > /dev/null + +## Add apt repositories maintained by Team RabbitMQ +sudo tee /etc/apt/sources.list.d/rabbitmq.list < Node-2: vim /var/lib/rabbitmq/.erlang.cookie (PASTE)
+ Node-3: vim /var/lib/rabbitmq/.erlang.cookie (PASTE) + +--- + +- Restart rabbitmq-server (Node1, Node2 and Node3): +```bash +systemctl restart rabbitmq-server +``` +--- + +- Stop application to join cluster (Node2 and Node3): +```bash +rabbitmqctl stop_app +``` +--- + +- Join cluster (Node2 and Node3): +```bash +rabbitmqctl join_cluster rabbit@ip- +``` +--- + +- Start application (Node2 and Node3): +```bash +rabbitmqctl start_app +``` +--- + +- Go to Node-1, and check cluster status: +```bash +rabbitmqctl cluster_status +``` +--- + +- Enable the RabbitMQ Management Plugin to access UI: +```bash +rabbitmq-plugins enable rabbitmq_management +systemctl restart rabbitmq-server +``` + +`Congratulations, you have successfully setupped RabbitMQ 3 node cluster.` + +--- + +- Now go to the Node-1, and setup prometheus server: +```bash +wget https://github.com/prometheus/prometheus/releases/download/v2.47.0/prometheus-2.47.0.linux-amd64.tar.gz +tar zxvf prometheus-2.47.0.linux-amd64.tar.gz +``` +--- + +- Create a prometheus.service inside the /etc/systemd/system directory and paste the below content: `vim /etc/systemd/system/prometheus.service` +```bash +[Unit] + +Description=Prometheus Server + +Documentation=https://prometheus.io/docs/introduction/overview/ + +After=network-online.target + +[Service] + +User=root + +Restart=on-failure + +ExecStart=/home/ubuntu/prometheus-2.47.0.linux-amd64/prometheus --config.file=/home/ubuntu/prometheus-2.47.0.linux-amd64/prometheus.yml + +[Install] + +WantedBy=multi-user.target +``` +--- + +- Reload system daemon: +```bash +systemctl daemon-reload +``` +--- + +- Restart prometheus server: +```bash +systemctl restart prometheus +``` + +> [!Important] +> node exporter runs on 9090 port no. + +--- + +- Now, enable prometheus plugin (Node2 and Node3): +```bash +rabbitmq-plugins enable rabbitmq_prometheus +systemctl restart rabbitmq-server +``` +--- + +- Go to Node1, go to `prometheus-2.47.0.linux-amd64` directory and edit `prometheus.yml` file to scrape the metrics which are exposed from other cluster nodes +```bash +cd prometheus-2.47.0.linux-amd64 +vim prometheus.yml +``` +Add the below code to `prometheus.yml` +```bash + - job_name: "Node2" + + # metrics_path defaults to '/metrics' + # scheme defaults to 'http'. + + static_configs: + - targets: [":15692"] + + - job_name: "Node3" + + # metrics_path defaults to '/metrics' + # scheme defaults to 'http'. + + static_configs: + - targets: [":15692"] +``` +--- + +- Restart your prometheus server: +```bash +systemctl restart prometheus +``` +--- + +- Now, go to web browser, access your prometheus server and click on `Status` and then `Targets` and you will see all nodes metrics: + +![image](https://github.com/user-attachments/assets/98631c4e-3a3b-4009-954a-602c7af0e6bc) + +--- + +- Now, setup grafana on Node1: +```bash +sudo apt-get update +sudo apt-get install -y software-properties-common +sudo add-apt-repository "deb https://packages.grafana.com/oss/deb stable main" +sudo apt-get update +sudo apt-get install -y grafana +sudo systemctl start grafana-server +sudo systemctl enable grafana-server +``` + +> Grafana UI: http://\:3000 + +--- + +- Go to your grafana server, add prometheus as a datasource and go to dashboard and click on import Dashboard + - **Dashboard ID**: 10991 + +You will see beautiful RabbitMQ cluster Dashboard like this + +![image](https://github.com/user-attachments/assets/295083ed-7a31-4c56-b890-09cbbc59ed82) diff --git a/DailyChallenges/Day3/solution.md b/DailyChallenges/Day3/solution.md index a2f5bdf..0551b3f 100644 --- a/DailyChallenges/Day3/solution.md +++ b/DailyChallenges/Day3/solution.md @@ -1,3 +1,26 @@ +``` +1. Core Concepts +Client: Requests a service (e.g., browser accessing a website). + +Server: Responds to requests (e.g., Google’s servers). + +Proxy: Acts as an intermediary. + +Forward Proxy: Sits on the client side (e.g., corporate VPNs to monitor employee internet access). + +Reverse Proxy: Sits on the server side (e.g., Nginx routing traffic to backend services like Grafana/Jenkins). + +Load Balancer: Distributes traffic across servers (e.g., AWS ALB/NLB). + +2. Why Nginx? +Reverse Proxy + Load Balancing: Routes requests to multiple backend services. + +Security: SSL/TLS termination, IP whitelisting, rate limiting. + +Performance: Caching static content, reducing server load. +``` + + ## Launch EC2 Instances - Create Two EC2 Instances - One for Grafana, One for Jenkins. - Open required ports in the EC2 security group: @@ -24,26 +47,42 @@ sudo systemctl start grafana-server sudo systemctl enable grafana-server sudo systemctl status grafana-server ``` +Official Doc for Grafana Installation: https://grafana.com/grafana/download?edition=oss -## Install Java -``` -sudo apt update -sudo apt install fontconfig openjdk-17-jre -java -version -openjdk version "17.0.8" 2023-07-18 +## Install Java & Jenkins ``` +Downloading and installing Jenkins +Completing the previous steps enables you to download and install Jenkins on AWS. To download and install Jenkins: + +Ensure that your software packages are up to date on your instance by using the following command to perform a quick software update: + +[ec2-user ~]$ sudo yum update –y +Add the Jenkins repo using the following command: + +[ec2-user ~]$ sudo wget -O /etc/yum.repos.d/jenkins.repo \ + https://pkg.jenkins.io/redhat-stable/jenkins.repo +Import a key file from Jenkins-CI to enable installation from the package: + +[ec2-user ~]$ sudo rpm --import https://pkg.jenkins.io/redhat-stable/jenkins.io-2023.key +[ec2-user ~]$ sudo yum upgrade +Install Java (Amazon Linux 2023): + +[ec2-user ~]$ sudo dnf install java-17-amazon-corretto -y +Install Jenkins: + +[ec2-user ~]$ sudo yum install jenkins -y +Enable the Jenkins service to start at boot: + +[ec2-user ~]$ sudo systemctl enable jenkins +Start Jenkins as a service: + +[ec2-user ~]$ sudo systemctl start jenkins +You can check the status of the Jenkins service using the command: + +[ec2-user ~]$ sudo systemctl status jenkins -### Install Jenkins on the Jenkins EC2 instance: -``` -sudo wget -O /usr/share/keyrings/jenkins-keyring.asc \ - https://pkg.jenkins.io/debian-stable/jenkins.io-2023.key -echo "deb [signed-by=/usr/share/keyrings/jenkins-keyring.asc]" \ - https://pkg.jenkins.io/debian-stable binary/ | sudo tee \ - /etc/apt/sources.list.d/jenkins.list > /dev/null -sudo apt-get update -sudo apt-get install jenkins -sudo status jenkins ``` +Official Doc for Jenkins Installation: https://www.jenkins.io/doc/tutorials/tutorial-for-installing-jenkins-on-AWS/#prerequisites ### Install and Configure Nginx as a Reverse Proxy - Install Nginx on one EC2 instance (e.g., Grafana EC2): ```sudo apt-get install -y nginx``` diff --git a/DailyChallenges/Day3/solution_by_madhup/solution.md b/DailyChallenges/Day3/solution_by_madhup/solution.md new file mode 100644 index 0000000..a7b3667 --- /dev/null +++ b/DailyChallenges/Day3/solution_by_madhup/solution.md @@ -0,0 +1,111 @@ +## Launch EC2 Instances +- Create Two EC2 Instances - One for Grafana, One for Jenkins. +- Open required ports in the EC2 security group: +- Grafana: Port 3000. +- Jenkins: Port 8080. +- Nginx: Ports 80 and 443. + +## Update the system and install necessary packages on both instances: +``` +sudo apt update && sudo apt upgrade -y +``` + +### Install Grafana on the Grafana EC2 instance: +``` +sudo apt-get install -y apt-transport-https software-properties-common wget +sudo mkdir -p /etc/apt/keyrings/ +wget -q -O - https://apt.grafana.com/gpg.key | gpg --dearmor | sudo tee /etc/apt/keyrings/grafana.gpg > /dev/null +echo "deb [signed-by=/etc/apt/keyrings/grafana.gpg] https://apt.grafana.com stable main" | sudo tee -a /etc/apt/sources.list.d/grafana.list +echo "deb [signed-by=/etc/apt/keyrings/grafana.gpg] https://apt.grafana.com beta main" | sudo tee -a /etc/apt/sources.list.d/grafana.list +# Updates the list of available packages +sudo apt-get update +sudo apt-get install -y grafana +sudo systemctl start grafana-server +sudo systemctl enable grafana-server +sudo systemctl status grafana-server +``` + +## Install Java +``` +sudo apt update +sudo apt install fontconfig openjdk-17-jre +java -version +``` + +### Install Jenkins on the Jenkins EC2 instance: +``` +sudo wget -O /usr/share/keyrings/jenkins-keyring.asc \ + https://pkg.jenkins.io/debian-stable/jenkins.io-2023.key +echo "deb [signed-by=/usr/share/keyrings/jenkins-keyring.asc]" \ + https://pkg.jenkins.io/debian-stable binary/ | sudo tee \ + /etc/apt/sources.list.d/jenkins.list > /dev/null +sudo apt-get update +sudo apt-get install jenkins +sudo systemctl status jenkins +``` + +### Install and Configure Nginx as a Reverse Proxy +- Install Nginx on one EC2 instance (e.g., Grafana EC2): + +```bash +sudo apt-get install -y nginx +``` +- Configure Nginx: Edit the Nginx configuration file: + +```bash +sudo vim /etc/nginx/sites-available/default +``` + +Replace the file contents with: +``` +server { + listen 80; + server_name ; + + location / { + proxy_pass http://127.0.0.1:8080; + proxy_set_header Host $host; + proxy_set_header X-Real-IP $remote_addr; + proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; + proxy_set_header X-Forwarded-Proto $scheme; + } +} + +server { + listen 80; + server_name ; + + auth_basic "Restricted Access"; + auth_basic_user_file /etc/nginx/.htpasswd; + + location / { + proxy_pass http://127.0.0.1:3000; + proxy_set_header Host $host; + proxy_set_header X-Real-IP $remote_addr; + proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; + proxy_set_header X-Forwarded-Proto $scheme; + } +} +``` + +- Create a .htpasswd file for Grafana basic authentication: +``` +sudo apt-get install -y apache2-utils +sudo htpasswd -c /etc/nginx/.htpasswd admin +``` + +- Test and restart Nginx: +``` +sudo nginx -t +sudo systemctl restart nginx +``` + +- Generate a self-signed SSL certificate using certbot: + - Install python-certbot + ```bash + sudo apt install certbot python3-certbot-nginx + ``` + - Obtain an SSL Certificate + ```bash + sudo certbot --nginx -d -d + ``` diff --git a/DailyChallenges/Day7/solution.md b/DailyChallenges/Day7/solution.md new file mode 100644 index 0000000..2be0d81 --- /dev/null +++ b/DailyChallenges/Day7/solution.md @@ -0,0 +1,160 @@ +## Flask app deployment + +### Steps to deploy using python virtual environment: + +- Create an EC2 instance with the following details: + + - **AMI:** Ubuntu + - **Instance type:** t2.micro + - **Volume:** 10GB +--- +- Update the system +```bash +sudo apt update +``` +--- +- Install python virtual environment: +```bash +sudo apt install python3-venv +```` +--- +- Create virtual environment: +```bash +python3 -m venv venv +source venv/bin/activate +``` +--- +- Clone your source code: +```bash +git clone https://github.com/DevMadhup/flask.git +``` +--- +- Move to source code directory +```bash +cd flask/examples/tutorial +``` +--- +- Install the required dependencies: +```bash +pip install -e . +``` +--- +- Run tests: +```bash +pip install '.[test]' +pytest +``` +--- +- Initialize database: +```bash +flask --app flaskr init-db +``` +--- +- Run the application: +```bash +flask --app flaskr run --host=0.0.0.0 --debug +``` +> Note: Access the application on http://\:5000 + +--- +### Steps to deploy using Jenkins & Docker: +- Install docker: +```bash +sudo apt update +sudo apt install docker.io -y +sudo usermod -aG docker $USER && newgrp docker +``` +--- +- Create a Dockerfile and paste the below content: +```bash +vim Dockerfile +``` +```bash +# Use Alpine-based Python image +FROM python:3.12-alpine + +# Working directory +WORKDIR /app + +# Copy code +COPY . . + +# Run Tests +RUN pip install --upgrade flask +RUN pip install '.[test]' +RUN pytest + +# Install Dependencies +RUN pip install -e . + +# Expose port +EXPOSE 5000 + +# Run the app +RUN flask --app flaskr init-db +ENTRYPOINT ["flask", "--app", "flaskr", "run", "--host=0.0.0.0", "--debug"] +``` +--- +- Install Jenkins: +```bash +sudo apt update -y +sudo apt install fontconfig openjdk-17-jre -y + +sudo wget -O /usr/share/keyrings/jenkins-keyring.asc \ + https://pkg.jenkins.io/debian-stable/jenkins.io-2023.key + +echo "deb [signed-by=/usr/share/keyrings/jenkins-keyring.asc]" \ + https://pkg.jenkins.io/debian-stable binary/ | sudo tee \ + /etc/apt/sources.list.d/jenkins.list > /dev/null + +sudo apt-get update -y +sudo apt-get install jenkins -y +``` +--- +- Access Jenkins from the browser and create a declarative pipeline and paste the below declarative pipeline: +```bash +pipeline{ + agent any + stages{ + stage("Cleanup Workspace"){ + steps{ + cleanWs() + } + } + + stage("Clone code"){ + steps{ + git branch: "main", url: "https://github.com/DevMadhup/flask.git" + } + } + + stage("Build Docker Image"){ + steps{ + sh "ls -lrt" + dir("examples/tutorial/"){ + sh "ls -lrt" + sh "docker build -t flaskapp ." + } + } + } + + stage("Push Image to DockerHub"){ + steps{ + withCredentials([usernamePassword(credentialsId: 'Docker-cred', passwordVariable: 'DOCKERHUB_PASS', usernameVariable: 'DOCKERHUB_USER')]) { + sh "docker login -u $DOCKERHUB_USER -p $DOCKERHUB_PASS" + sh "docker tag flaskapp madhupdevops/flaskapp-getfitwithsagar" + sh "docker push madhupdevops/flaskapp-getfitwithsagar" + } + } + } + + stage("Deploy docker container"){ + steps{ + sh "docker run -itd --name flaskapp -p 5000:5000 madhupdevops/flaskapp-getfitwithsagar" + } + } + } +} +``` +--- +- Now, run the pipeline and after it get successfull access it on public ip and port 5000 diff --git a/DailyChallenges/Day8/solution.md b/DailyChallenges/Day8/solution.md new file mode 100644 index 0000000..315cac1 --- /dev/null +++ b/DailyChallenges/Day8/solution.md @@ -0,0 +1,69 @@ +## Build a Distributed Logging System (Part1) + +### Steps to implement (with RabbitMQ): + +- Create an EC2 instance with the following details: + + - **AMI:** Ubuntu + - **Instance type:** t2.micro + - **Volume:** 15GB + - **Security group ports**: 22, 443, 15672 +--- + +- Install RabbitMQ +```bash +sudo apt update && sudo apt install rabbitmq-server -y +``` + +- Enable management plugin +```bash +sudo rabbitmq-plugins enable rabbitmq_management +sudo systemctl restart rabbitmq-server +``` +--- + +- Set Up a User and Queue +```bash +sudo rabbitmqctl add_user admin admin +sudo rabbitmqctl set_permissions -p / admin ".*" ".*" ".*" +``` +--- + +- Restart RabbitMQ +```bash +sudo systemctl restart rabbitmq-server +``` +--- + +- Change created user to admin user +```bash +sudo rabbitmqctl set_user_tags admin administrator +sudo systemctl restart rabbitmq-server +``` +--- + +- Access the RabbitMQ UI at `http://:15672`. Default credentials: `admin/admin`. +--- + +- Add logs queue to rabbitMQ +```bash +sudo apt install -y curl +curl -o rabbitmqadmin http://localhost:15672/cli/rabbitmqadmin +chmod +x rabbitmqadmin +sudo mv rabbitmqadmin /usr/local/bin/ +``` +--- + +- Create queue +```bash +rabbitmqadmin declare queue name=logs_queue +``` +--- + +- Check created queues +```bash +rabbitmqadmin list queues +``` +`Example output:` + +![image](https://github.com/user-attachments/assets/487533b7-7c04-4a5e-a977-988a15193c42) diff --git a/DailyChallenges/ProjectChallengeSeries/Day1/readme.md b/DailyChallenges/ProjectChallengeSeries/Day1/readme.md new file mode 100644 index 0000000..8b13789 --- /dev/null +++ b/DailyChallenges/ProjectChallengeSeries/Day1/readme.md @@ -0,0 +1 @@ + diff --git a/DailyChallenges/ProjectChallengeSeries/project1/Day1.md b/DailyChallenges/ProjectChallengeSeries/project1/Day1.md new file mode 100644 index 0000000..f4e76d9 --- /dev/null +++ b/DailyChallenges/ProjectChallengeSeries/project1/Day1.md @@ -0,0 +1,159 @@ +# Enterprise-Grade AKS Platform Deployment Series: Day 1 Kickoff + +## Welcome to the Mission! + +Attention, platform engineers! You’re diving into a 7-day DevOps heist to build an **enterprise-grade Azure Kubernetes Service (AKS) platform** that’s secure, scalable, and production-ready. Your mission: deploy the OpenTelemetry Demo microservices using Terraform and other cutting-edge tools. Day 1 drops you into a high-stakes scenario where you’ll master **Terraform best practices** to create a rock-solid infrastructure foundation. Each day will unveil a new challenge, building toward a world-class platform so let’s execute this heist with precision! + +## Project Overview + +Your goal is to create a **multi-environment (dev, stage, prod) AKS platform** that’s automated, resilient, and enterprise-ready. Over the week, you’ll tackle automation, security, observability, and more, with surprises at every turn. Today, you’ll handle a real-world brownfield scenario using Terraform to provision and manage a private AKS infrastructure, mastering best practices like `terraform import`, `terraform refresh`, and modular design. + +## Day 1: Milestone 1 – Infrastructure Deployment with Terraform + +**Scenario**: +CloudHeist Inc. acquired a startup that manually deployed an Azure Virtual Network (VNet) and subnet for their AKS dev environment. Your task is to bring this existing VNet under Terraform management, then provision private AKS clusters, Azure Container Registry (ACR), and Azure Key Vault for dev and prod environments. You’ll use `terraform import` to manage the existing VNet, `terraform refresh` to align state, and Terraform workspaces to manage environments, ensuring a secure, private, and compliant infrastructure. + +**Objective**: +Provision a secure, highly available, private AKS infrastructure using Terraform, incorporating a pre-existing VNet, and master best practices like `terraform import`, `terraform refresh`, modules, and workspace management. + +**Tasks**: +- **Import Existing Resources**: Import the pre-existing VNet (`cloudheist-vnet`) and subnet (`aks-subnet`) in the `cloudheist-rg` resource group into Terraform state using `terraform import`. +- **State Refresh**: Run `terraform refresh` to align Terraform state with actual Azure resources, ensuring no discrepancies. +- **Modular Terraform**: Create reusable Terraform modules for AKS, networking, ACR, and Key Vault in a `modules/` directory. +- **Workspaces**: Use Terraform workspaces (`dev`, `staging`, `prod`) to manage environment-specific configurations with a single codebase. +- **Private AKS Cluster**: + - Deploy private AKS clusters with no public IP for the API server, using a private endpoint for communication between the control plane (in AKS-managed resource group) and node pools (in `cloudheist-rg`). + - Configure Azure CNI and availability zones for high availability. +- **Managed Identity**: + - Enable system-assigned managed identity on AKS clusters. + - Grant the managed identity permissions to create Azure Load Balancers, Managed Disks, and other dependent resources. +- **Authentication & Authorization**: + - Enable Azure AD workload identity federation on AKS, configuring `kubelogin` for authentication. + - Set up Azure RBAC for Kubernetes authorization, binding roles to Azure AD users or groups. +- **State Management**: Configure a secure remote backend in Azure Blob Storage (`tfstate12345`, container `tfstate`) with encryption and state locking via Azure Table Storage. +- **Provider Configuration**: Define the `azurerm` provider with version constraints in a `terraform` block in `provider.tf`. +- **Key Vault Integration**: Deploy Azure Key Vault with private endpoints, storing sensitive data (e.g., AKS admin credentials) and referencing them via Terraform `data` sources. +- **Provisioners**: Use `local-exec` provisioners to automate post-deployment tasks, like storing AKS kubeconfig in Key Vault. +- **Compliance and Governance**: Apply Azure Policy for AKS compliance (CIS benchmarks) and add Azure tags (`Environment`, `Project`) for cost tracking. + +**Terraform Best Practices**: +- **Modularity**: Create reusable Terraform modules for AKS, networking, ACR, and Key Vault to promote DRY principles. +- **Workspaces**: Use `terraform workspace` to manage dev and prod environments, isolating state files. +- **State Security**: Store Terraform state in Azure Blob Storage with encryption and locking to prevent conflicts. +- **Provider Versioning**: Pin `azurerm` provider versions in `provider.tf` to ensure consistent builds. +- **Sensitive Data**: Store secrets in Key Vault, avoiding hardcoding in `.tf` files. +- **Idempotency**: Ensure configurations are idempotent and reusable across environments. +- **State Management**: Use `terraform import` for brownfield resources and `terraform refresh` to align state with reality. +- **Documentation**: Document module inputs/outputs, workspace setup, import process, and state management in `README.md`. + +**Deliverables**: +- `infrastructure/` directory with: + - `main.tf`: Main configuration calling modules. + - `provider.tf`: Combined backend and provider configuration with a single `terraform` block. + - `variables.tf`: Variable declarations with types and defaults. + - `outputs.tf`: Outputs like AKS cluster IDs and Key Vault URIs. + - `terraform.tfvars`: Global variables. + - `modules/aks/`, `modules/networking/`, `modules/keyvault/`, `modules/acr/`: Reusable modules with `main.tf`, `variables.tf`, and `outputs.tf`. + - `environments/dev/`, `environments/prod/`: Environment-specific `terraform.tfvars`. + - `scripts/import-vnet.sh`: Shell script to import VNet and subnet using `terraform import`. + - `scripts/refresh-state.sh`: Shell script to run `terraform refresh`. +- Architecture diagram in Mermaid format showing infrastructure components. +- `./README.md` documenting: + - Terraform setup, module structure, and workspace usage. + - Detailed steps for `terraform import` and `terraform refresh`. + - State management, Key Vault integration, workload identity setup, and compliance details. + +**Success Criteria**: +- Existing VNet (`cloudheist-vnet`) and subnet (`aks-subnet`) in `cloudheist-rg` are imported into Terraform state using `terraform import`. +- `terraform refresh` aligns state with Azure resources without errors. +- AKS clusters are private (no public API server IP), use private endpoints for control plane communication, and have node pools in `cloudheist-rg` and control plane in AKS-managed resource group. +- System-assigned managed identity is enabled and used for creating Azure Load Balancers, Managed Disks, and other resources. +- Azure AD workload identity is configured with `kubelogin` for authentication, and Kubernetes RBAC is set up with Azure AD role bindings. +- Terraform state is encrypted, locked, and stored in Azure Blob Storage. +- Workspaces manage dev and prod environments with consistent configurations. +- Key Vault stores sensitive data, accessible via private endpoints. +- Provider versions are pinned in `provider.tf`, and infrastructure is tagged for cost tracking with Azure Policy applied for CIS compliance. + +**Starter Template**: +```bash +infrastructure/ +├── main.tf +├── provider.tf +├── variables.tf +├── outputs.tf +├── terraform.tfvars +├── modules/ +│ ├── aks/ +│ │ ├── main.tf +│ │ ├── variables.tf +│ │ └── outputs.tf +│ ├── networking/ +│ │ ├── main.tf +│ │ ├── variables.tf +│ │ └── outputs.tf +│ ├── keyvault/ +│ │ ├── main.tf +│ │ ├── variables.tf +│ │ └── outputs.tf +│ ├── acr/ +│ │ ├── main.tf +│ │ ├── variables.tf +│ │ └── outputs.tf +├── environments/ +│ ├── dev/ +│ │ └── terraform.tfvars +│ ├── prod/ +│ │ └── terraform.tfvars +├── scripts/ +│ ├── import-vnet.sh +│ └── refresh-state.sh +└── README.md +``` +## Architecture Sneak Peek + +Here’s the infrastructure you’ll build and manage today, setting the stage for an epic platform: + +```mermaid +flowchart TD + subgraph Azure + ACR[Azure Container Registry] + KV[Azure Key Vault] + subgraph AKS-Clusters + DEV[AKS Dev] + PROD[AKS Prod] + end + LA[Azure Monitor] + Storage[Azure Blob Storage - Terraform State] + Cost[Azure Cost Management] + end + + Terraform[Terraform]--import/refresh-->DEV + Terraform--modules-->DEV + Terraform--modules-->PROD + Terraform--modules-->ACR + Terraform--modules-->KV + Terraform--state-->Storage + Terraform--tags-->Cost + DEV--secrets-->KV + PROD--secrets-->KV + DEV--images-->ACR + PROD--images-->ACR + DEV--logs-->LA + PROD--logs-->LA +``` + +## What’s Next? + +Tomorrow, you’ll tackle the next challenge, diving into automation, security, and resilience. Each day will unlock new tools and techniques, building toward a fully automated, enterprise-grade AKS platform. Stay sharp—the heist is just getting started! + +## Getting Started + +1. Fork the repository: [LearnWithSagar/DailyChallenges](https://github.com/Sagar2366/LearnWithSagar). +2. Navigate to the `DailyChallenges/ProjectChallengeSeries/project1` directory to understand the problem statement. +3. Create VNet and Subnet Manually, Write Terraform plan and Initialize it: `terraform init` in `infrastructure/`. +4. Set up Terraform workspaces: `terraform workspace new dev`, `terraform workspace new staging` and `terraform workspace new prod`. +5. Import the existing VNet and subnet. +6. Deploy infrastructure with `terraform apply -var-file=environments/dev/terraform.tfvars`. +7. Document your setup in `README.md`, covering module structure, workspace usage, import process, state management, workload identity setup, and compliance. + +Good luck, and let’s execute this infrastructure heist flawlessly! diff --git a/DailyChallenges/Season2/DAY6/Dockerfile b/DailyChallenges/Season2/DAY6/Dockerfile new file mode 100644 index 0000000..13b31b0 --- /dev/null +++ b/DailyChallenges/Season2/DAY6/Dockerfile @@ -0,0 +1,16 @@ +FROM python:3.9-slim + +# Set working directory +WORKDIR /app + +# Copy files +COPY app.py /app + +# Install Flask +RUN pip install flask + +# Expose port 5000 +EXPOSE 5000 + +# Run Flask app +CMD ["python", "app.py"] diff --git a/DailyChallenges/Season2/DAY6/app.py b/DailyChallenges/Season2/DAY6/app.py new file mode 100644 index 0000000..f9a9aff --- /dev/null +++ b/DailyChallenges/Season2/DAY6/app.py @@ -0,0 +1,10 @@ +from flask import Flask + +app = Flask(__name__) + +@app.route('/') +def home(): + return "Hello, Welcome to Season 2! You are learning GitHub Actions." + +if __name__ == '__main__': + app.run(host='0.0.0.0', port=5000) diff --git a/DailyChallenges/Season2/DAY6/challenge.md b/DailyChallenges/Season2/DAY6/challenge.md index 7a693d4..11ea3d7 100644 --- a/DailyChallenges/Season2/DAY6/challenge.md +++ b/DailyChallenges/Season2/DAY6/challenge.md @@ -1,164 +1,188 @@ -Challenge Overview -Theory: Explain the key differences between GitHub and GitLab, the migration process, and the importance of migrating all objects. +# Day 6 : GitHub Actions Theory & Practical Challenge + +![Untitled design (1)](https://github.com/user-attachments/assets/085eac7f-92a3-4af4-a406-e94fced1e84a) + +## **1. What is GitHub Actions?** +GitHub Actions is a CI/CD automation tool that enables developers to automate workflows within their GitHub repositories. It allows you to automate software development workflows, such as building, testing, and deploying applications. + +## **2. When Should GitHub Actions Be Used? When Should It Not Be Used?** +### **When to Use GitHub Actions:** +- Automating CI/CD pipelines. +- Running tests automatically on code commits. +- Deploying applications to cloud services (AWS, Azure, GCP, etc.). +- Managing infrastructure as code (Terraform, Ansible, etc.). +- Automating security scans and linting. +- Scheduling periodic jobs (e.g., nightly builds, cron jobs). + +### **When Not to Use GitHub Actions:** +- When you need on-premises-only solutions. +- If strict security compliance requires hosting the CI/CD pipeline on internal infrastructure. +- If you need advanced enterprise features available only in other CI/CD tools like Jenkins. + +| Feature | GitHub Actions | Jenkins | +|------------------|---------------|---------| +| Hosting | Cloud-based (GitHub-hosted runners) | Self-hosted (or cloud-based with effort) | +| Ease of Use | Simple YAML-based workflows | Requires setup & configuration | +| Integration | Native GitHub integration | Supports multiple VCS (Git, SVN, etc.) | +| Cost | Free for public repos, limited free usage for private repos | Requires dedicated infrastructure | +| Plugins | Built-in Marketplace | Extensive plugin ecosystem | +| Scalability | Managed by GitHub | Requires manual scaling | + +## **3. Steps to Create a GitHub Action** +1. Navigate to your GitHub repository. +2. Click on the `Actions` tab. +3. Choose a predefined template or create a `.github/workflows/main.yml` file. +4. Define your workflow steps using YAML. +5. Commit and push the file to trigger the workflow. + +## **4. GitHub Actions Workflow Structure** +A typical GitHub Actions workflow consists of: +- **name**: Name of the workflow. +- **on**: Defines when the workflow should run (e.g., push, pull_request, schedule, workflow_dispatch). +- **jobs**: Contains the tasks to be executed. +- **steps**: Defines individual steps in a job. +- **uses**: Calls an external GitHub Action. +- **run**: Executes custom commands or scripts. + +Example: +```yaml +name: Example Workflow +on: push +jobs: + build: + runs-on: ubuntu-latest + steps: + - name: Checkout code + uses: actions/checkout@v4 + - name: Run a script + run: echo "Hello, GitHub Actions!" +``` + +## **5. What is a Job? What is an Action?** +- **Job**: A job is a set of steps that execute in the same runner. + ```yaml + jobs: + build: + runs-on: ubuntu-latest + steps: + - run: echo "This is a job" + ``` + +- **Action**: A reusable unit of work within GitHub Actions. + ```yaml + steps: + - name: Use a Docker-based Action + uses: actions/hello-world-docker-action@v1 + ``` + +## **6. What Are Inputs and Outputs in GitHub Actions?** +- **Inputs** allow workflows to accept dynamic values. +- **Outputs** store results from one step/job and pass them to another. + +Example: +```yaml +jobs: + example: + runs-on: ubuntu-latest + steps: + - id: step1 + run: echo "::set-output name=myoutput::Hello" + - run: echo "Output from step1: ${{ steps.step1.outputs.myoutput }}" +``` + +## **7. General Steps in a GitHub Action** +1. **Checkout Code** → Use `actions/checkout` to fetch repository code. +2. **Authenticate & Authorize** → Configure AWS, Azure, or any required credentials. +3. **Perform Necessary Operations** → Run scripts, deploy applications, execute tests, etc. + +## **8. Theory Challenge** +### **Answer the following questions:** +1. What are the key differences between GitHub Actions and Jenkins? +2. Describe the general structure of a GitHub Actions workflow. +3. How to manage variables and secrets in GitHub Actions + +## **9. Practical Challenge** +### **Challenge Scenario:** +You are required to create a GitHub Actions workflow that: +1. **Builds and pushes a Docker image** to Docker Hub. +2. **Deploys the application on GitHub Runners** +3. **Validates that the application is running correctly**. +4. **Sends an email notification** with the deployment status. + +### **Provided Code:** +#### **app.py** +```python +from flask import Flask + +app = Flask(__name__) + +@app.route('/') +def home(): + return "Hello, Welcome to Season 2! You are learning GitHub Actions." + +if __name__ == '__main__': + app.run(host='0.0.0.0', port=5000) +``` + +#### **Dockerfile** +```dockerfile +# Use official Python image +FROM python:3.9-slim + +# Set working directory +WORKDIR /app + +# Copy files +COPY app.py /app + +# Install Flask +RUN pip install flask + +# Expose port 5000 +EXPOSE 5000 + +# Run Flask app +CMD ["python", "app.py"] +``` + +### **Submission Guidelines:** +- Make a **GitHub repository** for the challenge. +- Take **screenshots** of significant steps. +- Document the **solution** in a `solution.md` file, explaining each step clearly. +- Mention **challenges faced** and how you overcame them. +- Ensure that your repository is well-structured and follows best practices. +- Post your experience on social media with #getfitwithsagar #SRELife #DevOpsForAll. +- Tag us in your posts for visibility and networking. + +### **Additional Notes:** +- Make sure to use proper authentication methods while pushing Docker images. +- Structure your workflow efficiently by separating build, deployment, and notification steps. +- Follow security best practices, such as setting permissions and using secrets. +- Ensure logging and error handling are in place to debug issues easily. + +--- + +## **Join Our Community** -Subtasks: +To make the most of this journey, connect with fellow learners: +- **Discord** – Ask questions and collaborate: [Join here](https://discord.gg/mNDm39qB8t) +- **Google Group** – Get updates and discussions: [Join here](https://groups.google.com/forum/#!forum/daily-devops-sre-challenge-series/join) +- **YouTube** – Watch solution videos and tutorials: [Subscribe here](https://www.youtube.com/@Sagar.Utekar) -Migrate repositories and their content. +--- -Migrate users, permissions, and teams. +--- -Migrate issues, pull requests, and projects. +## **Stay Motivated!** -Migrate GitHub Actions workflows to GitLab CI/CD. +Every challenge you complete brings you closer to mastering Git, DevOps, and SRE practices. Keep pushing your limits and collaborating with the community. -Migrate repository settings, webhooks, and integrations. +Happy Learning! -Process: Provide a step-by-step guide for the candidate to follow. +----------- -Evaluation: Assess the candidate's understanding, execution, and problem-solving skills. +**Best regards,** +Sagar Utekar & Raihan -Detailed Steps and Plan -1. Theory (Knowledge Assessment) -Ask the candidate to answer the following questions: -What are the key differences between GitHub and GitLab? - -What are the challenges of migrating all GitHub objects (repositories, users, workflows, etc.) to GitLab? - -How would you ensure data integrity during the migration? - -What are the best practices for migrating CI/CD pipelines and workflows? - -2. Subtasks -Subtask 1: Migrate Repositories and Content -The candidate should: - -Use GitLab's built-in GitHub importer (recommended in the GitLab blog) to migrate repositories. - -Ensure all branches, tags, and commit history are preserved. - -Verify that the repository content (code, files, etc.) is intact. - -Subtask 2: Migrate Users, Permissions, and Teams -The candidate should: - -Create corresponding users in GitLab. - -Map GitHub users to GitLab users. - -Migrate teams and permissions (e.g., admin, maintainer, developer) using GitLab groups and access controls. - -Subtask 3: Migrate Issues, Pull Requests, and Projects -The candidate should: - -Use GitLab's import tools or third-party tools (e.g., github-gitlab-importer) to migrate issues, pull requests, and projects. - -Ensure all metadata (e.g., labels, milestones, comments) is preserved. - -Subtask 4: Migrate GitHub Actions Workflows to GitLab CI/CD -The candidate should: - -Convert GitHub Actions workflows (.github/workflows/*.yml) to GitLab CI/CD pipelines (.gitlab-ci.yml). - -Replicate all steps (e.g., build, test, deploy) in GitLab CI/CD. - -Use GitLab CI/CD features like caching, artifacts, and environments to optimize the pipeline. - -Subtask 5: Migrate Repository Settings and Webhooks -The candidate should: - -Migrate repository settings (e.g., branch protection rules, merge request settings). - -Recreate webhooks in GitLab for integrations (e.g., Slack, Jira). - -3. Process -Provide the candidate with the following step-by-step instructions: - -Prepare for Migration: - -Audit the GitHub organization to identify all objects to be migrated (repositories, users, projects, workflows, etc.). - -Create a GitLab group or namespace to mirror the GitHub organization structure. - -Migrate Repositories: - -Use GitLab's GitHub importer (as described in the GitLab blog) to migrate repositories. - -Verify that all branches, tags, and commit history are preserved. - -Migrate Users, Permissions, and Teams: - -Create users in GitLab and map them to GitHub users. - -Assign appropriate roles and permissions in GitLab using groups and access controls. - -Migrate Issues, Pull Requests, and Projects: - -Use GitLab's import tools or third-party tools to migrate issues, pull requests, and projects. - -Verify that all metadata (labels, milestones, comments) is preserved. - -Migrate GitHub Actions Workflows: - -Convert GitHub Actions workflows to GitLab CI/CD pipelines. - -Test the pipelines to ensure they work as expected. - -Migrate Repository Settings and Webhooks: - -Recreate repository settings (e.g., branch protection rules) in GitLab. - -Recreate webhooks for integrations. - -Validate the Migration: - -Run the GitLab CI/CD pipelines and verify that all steps pass. - -Deploy the application (if applicable) and ensure it works as expected. - -Verify that all issues, pull requests, and projects are intact. - -Document the Process: - -Write a detailed report explaining the migration steps, challenges faced, and solutions implemented. - -4. Evaluation Criteria -Evaluate the candidate based on the following: - -Knowledge: Understanding of GitHub, GitLab, and migration concepts. - -Execution: Ability to migrate all objects and ensure data integrity. - -Problem-Solving: Handling errors and optimizing the migration process. - -Documentation: Clarity and completeness of the migration report. - -Tools and Resources -GitHub Organization: Provide access to a sample GitHub organization with repositories, users, projects, and workflows. - -GitLab Group: Provide access to a GitLab group for migration. - -Documentation: - -GitLab's GitHub Importer - -GitLab CI/CD Documentation - -Third-Party Migration Tools - -Timeline -Preparation: 1 hour (audit GitHub organization and plan migration). - -Migration: 4 hours (migrate repositories, users, projects, workflows, etc.). - -Validation and Documentation: 2 hours. - -Bonus Tasks (Optional) -Migrate GitHub Wikis and Pages to GitLab. - -Implement advanced GitLab CI/CD features (e.g., environments, artifacts). - -Set up monitoring and alerting for the migrated pipelines. - -This challenge will test the candidate's ability to perform a complete migration while ensuring data integrity, minimal downtime, and adherence to best practices. It also provides insight into their problem-solving and documentation skills. The referenced guides will serve as excellent resources for the candidate to follow during the challenge. diff --git a/DailyChallenges/Season2/Day10/0_filesystem.md b/DailyChallenges/Season2/Day10/0_filesystem.md new file mode 100644 index 0000000..4c87bc9 --- /dev/null +++ b/DailyChallenges/Season2/Day10/0_filesystem.md @@ -0,0 +1,33 @@ +### Linux Filesystem Overview + +The Linux filesystem is a hierarchical structure used to store all information on a computer. It is based on the UNIX system, where nearly everything, including data, commands, symbolic links, devices, and directories, is represented within the filesystem. + +### Structure and Navigation +Screenshot 2025-04-09 at 5 57 43 AM + +1. **Hierarchy**: The filesystem is organized like an upside-down tree with the root directory (`/`) at the top. Below the root are common directories such as `/bin`, `/dev`, `/home`, `/lib`, and `/tmp`. These directories can contain subdirectories, creating a hierarchical structure. +2. **Paths**: + * **Full Path**: Specifies the complete route from the root directory to a file or directory (e.g., `/home/joe/Documents/memos/memo1.doc`). + * **Relative Path**: Specifies the path relative to the current directory (e.g., `memo1.doc` if the current directory is `/home/joe/Documents/memos`). + +### Common Linux Directories + +#### Key Directories + +- **`/bin`**: Contains common Linux user commands like `ls`, `sort`, `date`, and `chmod`. +- **`/boot`**: Contains the bootable Linux kernel and boot loader configuration files (GRUB). +- **`/dev`**: Contains files representing access points to devices (e.g., `tty*`, `fd*`, `hd*`, `sd*`, `ram*`, `cd*`). +- **`/etc`**: Contains administrative configuration files, which are typically plaintext files that can be edited with a text editor. +- **`/home`**: Contains directories for each regular user account (except root, which uses `/root`). +- **`/media`**: Standard location for automatically mounting devices, particularly removable media. +- **`/lib`**: Contains shared libraries required by applications in `/bin` and `/sbin` to boot the system. +- **`/mnt`**: A common mount point for devices, often used for temporarily mounting local or remote filesystems. +- **`/misc`**: Sometimes used to automatically mount filesystems upon request. +- **`/opt`**: Used to store add-on application software. +- **`/proc`**: Contains information about system resources. +- **`/root`**: The home directory for the root user, located outside `/home` for security reasons. +- **`/sbin`**: Contains administrative commands and daemon processes. +- **`/tmp`**: Contains temporary files used by applications. +- **`/usr`**: Contains user documentation, games, graphical files (X11), libraries, and other commands and files not needed during the boot process. +- **`/var`**: Contains directories of data used by various applications, including FTP server files (`/var/ftp`), web server files (`/var/www`), system log files (`/var/log`), and spool files (`/var/spool`). + diff --git a/DailyChallenges/Season2/Day10/1_core_filesystem_commands.md b/DailyChallenges/Season2/Day10/1_core_filesystem_commands.md new file mode 100644 index 0000000..941050b --- /dev/null +++ b/DailyChallenges/Season2/Day10/1_core_filesystem_commands.md @@ -0,0 +1,319 @@ +As a Linux user or administrator, you frequently interact with the filesystem to navigate, create, and manage files and directories. When you log into a Linux system and open a shell, you start in your **home directory** (e.g., `/home/joe`), which serves as your personal workspace for files and subdirectories. Mastering basic commands and file management techniques is essential for organizing data, securing access, and performing administrative tasks. + +#### Home Directory + +The home directory is represented by `~`, which is your default base (e.g., `/home/$USER`). You can use `~` to navigate to directories relative to your home directory. + +### Core Filesystem Commands + +These commands are the foundation for navigating and managing the filesystem. + +#### 2.1 `cd` - Change Directory + +- **Purpose**: Moves you to another directory. +- **Usage**: + - `cd`: Returns to home (e.g., `/home/joe`). + - **Absolute Path**: `cd /usr/share → /usr/share`. (Starts from the root directory `/`) + - **Relative Path**: From `/usr/share`, `cd doc → /usr/share/doc`. (Relative to the current directory) + - `cd ~`: Home directory (e.g., `/home/joe`). + - `cd ~/Music`: Subdirectory of home (e.g., `/home/joe/Music`). + - `cd ..`: Up one level (e.g., from `/usr/share/doc` to `/usr/share`). + - `cd -`: Previous directory (`$OLDPWD`). +- **Key Concept**: `..` (parent) and `~` (home) simplify navigation. +- **Example**: + +```bash +$ cd /usr/share/ +$ pwd +/usr/share +$ cd doc +$ pwd +/usr/share/doc +$ cd +$ pwd +/home/chris +``` + + +#### 2.2 `pwd` - Print Working Directory + +- **Purpose**: Displays your current absolute path. +- **Usage**: `pwd → /home/joe`. +- **Use Case**: Confirms your location. + + +#### 2.3 `mkdir` - Make Directory + +- **Purpose**: Creates directories. +- **Usage**: + - `mkdir files`: Creates `files` in the current directory (e.g., `/home/joe/files`). + - `mkdir -p /tmp/test/docs`: Creates nested directories if parents don’t exist. +- **Use Case**: Organizing files. + + +#### 2.4 `chmod` - Change Permissions + +- **Purpose**: Sets file/directory permissions. +- **Usage**: `chmod 700 test → rwx------` (owner full access, others none). +- **Permissions**: `r` (4), `w` (2), `x` (1); `700 = 7` (owner), `0` (group), `0` (others). +- **Use Case**: Securing directories. + + +#### 2.5 `ls` - List Directory Contents + +- **Purpose**: Shows files and directories. +- **Usage**: + - `ls`: Non-hidden items. + - `ls -l`: Long listing (e.g., `drwxr-xr-x 2 joe sales 1024 Jan 24 12:17 test`). + - `ls -a`: Includes hidden files (e.g., `.hidden`). + - `ls -lrt`: Sorts by time, reversed (recent last). + - `ls -R`: Recursive listing. + - `ls -d`: Directory info only (e.g., `ls -ld test`). +- **Note**: Colors may vary (e.g., blue for dirs, green for executables). + + +### File Management Tasks + +Administrators perform key tasks to manage files effectively. + +#### 3.1 Working with Wildcards + +- **Purpose**: Match multiple files efficiently. +- **Wildcards**: + + +| Wildcard | Use | +| :-- | :-- | +| `*` | Refers to an unlimited number of any characters. `ls *` shows all files in the current directory (except those that have a name starting with a dot). | +| `?` | Used to refer to one specific character that can be any character. `ls c?t` would match `cat` as well as `cut`. | +| `[auo]` | Refers to one character that may be selected from the range that is specified between square brackets. `ls c[auo]t` would match `cat`, `cut`, and `cot`. | + +- **Use Case**: `touch apple banana grape → ls *e* → apple, grape`. + + +#### 3.2 Absolute and Relative Pathnames + +- **Absolute Pathname**: Complete path from the root directory `/` (e.g., `/home/lisa/file1`). + - Always works, regardless of the current directory. +- **Relative Pathname**: Path relative to the current directory (e.g., from `/home`, `lisa/file1 → /home/lisa/file1`). +- When working with relative filenames, `..` moves up one level (e.g., from `/home/lisa`, `cp file1 ../lara` copies `/home/lisa/file1` to `/home/lara`). + + +#### 3.3 Listing Files and Directories + +##### Detailed Explanation of `ls -la` + +The command `ls -la` is a powerful tool for listing files and directories in Linux with detailed information. It combines two options: + +- `-l`: Long listing format. +- `-a`: Show all files, including hidden files. + +When you run `ls -la`, you get a detailed view of all files and directories, including those that are hidden (i.e., those whose names start with a dot `.`). This is incredibly useful for examining your system's configuration files and hidden directories. + +##### Sample Output + +```bash +$ ls -la /home/joe +total 158 +drwxrwxrwx 2 joe sales 4096 May 12 13:55 . +drwxr-xr-x 3 root root 4096 May 10 01:49 .. +-rw------- 1 joe sales 2204 May 18 21:30 .bash_history +-rw-r--r-- 1 joe sales 24 May 10 01:50 .bash_logout +-rw-r--r-- 1 joe sales 230 May 10 01:50 .bash_profile +-rw-r--r-- 1 joe sales 124 May 10 01:50 .bashrc +drw-r--r-- 1 joe sales 4096 May 10 01:50 .kde +-rw-rw-r-- 1 joe sales 149872 May 11 22:49 letter +``` + + +##### Explanation of Columns + +| Column | Description | +| :-- | :-- | +| 1 | **File type and permissions**: The first character indicates the file type (e.g., `d` for directory, `-` for file, `l` for symbolic link). The next nine characters represent permissions for the owner, group, and others. | +| 2 | **Number of hard links**: The number of hard links to the file or directory. | +| 3 | **Owner**: The owner of the file or directory. | +| 4 | **Group**: The group associated with the file or directory. | +| 5 | **Size**: The size of the file in bytes. For directories, it is typically 4096 bytes, representing the size of the directory metadata. | +| 6 | **Last modified**: The date and time when the file was last modified. | +| 7 | **Name**: The name of the file or directory. | + +##### Key Details + +* **. (Current Directory)**: Represents the current directory (`/home/joe` in this example). +* **.. (Parent Directory)**: Represents the directory above the current directory (`/home` in this example). +* **Hidden Files**: Files starting with a dot (`.`) are hidden files and are typically configuration files or directories used by applications. + + +##### Special Characters in Permissions + +* **s (SetUID or SetGID)**: If you see an `s` instead of `x` in the permissions, it indicates a SetUID or SetGID program. + * SetUID: Allows any user to run the application with the owner's permissions. + * SetGID: Allows any user to run the application with the group's permissions. +* **t (Sticky Bit)**: If a `t` appears at the end of the permissions for a directory (e.g., `drwxrwxr-t`), it indicates the sticky bit is set. + * Sticky Bit: Allows users to add files to the directory but prevents them from deleting files owned by others. +* **+ (Extended Attributes)**: If a `+` appears at the end of the permission bits (e.g., `-rw-rw-r--+`), it means that extended attributes, such as Access Control Lists (ACLs) or SELinux, are set on the file. + + +##### Examples and Best Practices + +1. **Basic Usage** + +```bash +$ ls -la ~ +``` + +This command lists all files and directories in your home directory, including hidden ones, with detailed information. + +2. **Checking Permissions** + +```bash +$ ls -la /etc/passwd +``` + +Use this to check the permissions, owner, and group of the `/etc/passwd` file. + +3. **Troubleshooting** +If you're missing configuration files, running `ls -la` will show you if they exist and what their permissions are. + + +#### 3.4 Copying Files and Directories + +- **Command**: `cp source dest;`. +- **Purpose**: The `cp` command allows you to copy files and directories from one location to another. +- **Basic Usage**: + - To copy a single file: + +```bash +cp /path/to/source/file /path/to/destination/ +``` + + - Example: + +```bash +cp /etc/hosts /tmp/ +``` + +This copies the file `/etc/hosts` to the `/tmp/` directory. +- **Options**: + + +| Option | Description | +| :-- | :-- | +| `-R` | Recursive: Copies directories and their contents recursively. | +| `-a` | Archive: Preserves file attributes (permissions, ownership, timestamps). It is equivalent to `-dpR`. | +| `-i` | Interactive: Prompts before overwriting existing files. | +| `-u` | Update: Copies only when the source file is newer than the destination file or when the destination file is missing. | +| `-v` | Verbose: Shows the files being copied. | + +- **Important Considerations**: + - **Target Directory**: Always ensure the target directory exists. If you copy a file to a non-existent directory without a trailing `/`, it will create a file named as the directory instead. + +```bash +cp /etc/hosts /tmp # If /tmp doesn't exist, it creates a file named 'tmp' +cp /etc/hosts /tmp/ # Correct way, ensures /tmp is treated as a directory +``` + + - **Copying Directories Recursively**: +To copy an entire directory, use the `-R` option. + +```bash +cp -R /etc /tmp +``` + +This copies the `/etc` directory and all its contents to the `/tmp` directory. + - **Preserving Attributes**: +To keep the original file permissions and other attributes, use the `-a` option (archive mode). + +```bash +cp -a ~ /tmp +``` + +This copies your entire home directory to the `/tmp` directory while preserving all file attributes. + - **Hidden Files**: +Hidden files (files starting with `.`) are not copied by default. Here's how to handle them: + +1. **Using `.*`**: + +```bash +cp /somedir/.* /tmp +``` + +This copies hidden files from `/somedir/` to `/tmp`. Note that this will produce an error for the `.` and `..` directories unless you use the `-R` option. +2. **Copying the entire directory**: + +```bash +cp -a /somedir/ . +``` + +This creates a `somedir` subdirectory in the current directory and copies all contents, including hidden files. +3. **Copying all files, including hidden files, into the current directory**: + +```bash +cp -a /somedir/. . +``` + +Make sure there is a space between the two dots. This copies all files and directories (including hidden ones) from /somedir into the current directory. + + +### Best Practices and Tips + +- Ensure the destination directory exists to avoid creating a file instead of copying into a directory. +- Use the -a option when you want to preserve file attributes like permissions and timestamps. +- Be cautious when using wildcards, and always double-check your command before executing to avoid unintended consequences. + + + +#### 3.5 Moving Files and Directories + +- **Command**: `mv source dest`. +- **Usage**: + - `mv file1 /tmp`: Moves to `/tmp`. + - `mv file1 file2`: Renames. + - `mv dir /tmp`: Moves directory (works with contents). +- **Note**: Combines copy and delete. + + +#### 3.6 Deleting Files and Directories + +- **Command**: `rm target;`. +- **Usage**: + - `rm file1`: Deletes a file. + - `rm -r dir`: Deletes directory recursively. + - `rm -f`: Forces without prompting. +- **Note**: On RHEL, `rm` may alias to `rm -i` (prompts for confirmation); use `-f` to override. + + +### Metacharacters and Operators + +- **Brace Expansion**: `touch memo{1,2,3} → memo1 memo2 memo3`. +- **Redirection**: + - `>`: Overwrites file (e.g., `echo "test" > file`). + - `>>`: Appends (e.g., `echo "more" >> file`). + - `<`: Input from file (e.g., `mail root < file`). + - `<<WORD`: Here document (e.g., `cat <<END`). + + +### Practical Context + +- **Workflow**: `cd ~; mkdir test; chmod 700 test; cp -a test /tmp; rm -r /tmp/test`. +- **Permissions**: `ls -l` shows owner, group, and access (e.g., `rwxr-xr-x`). +- **Hidden Files**: Start with `.` (e.g., `.hidden`), visible with `ls -a`. + + +### Why This Matters + +- **Foundation**: Essential for Linux administration and scripting. +- **Efficiency**: Wildcards and operators save time. +- **DevOps/SRE**: Critical for server management and automation. +- **RHCSA**: Tests these skills explicitly. + + +### Key Takeaways for Learners + +- Navigate with `cd`, `pwd`. +- Create with `mkdir`, `touch`. +- Manage files with `cp`, `mv`, `rm`; secure with `chmod`. +- Use wildcards (`*`, `?`, `[]`) and paths (absolute/relative) for flexibility. +- List with `ls` variations (e.g., `-l`, `-a`, `-R`, `-d`). +- Understand permissions and file types via `ls -l`. diff --git a/DailyChallenges/Season2/Day10/2_mounts_in_linux.md b/DailyChallenges/Season2/Day10/2_mounts_in_linux.md new file mode 100644 index 0000000..8fbc2fe --- /dev/null +++ b/DailyChallenges/Season2/Day10/2_mounts_in_linux.md @@ -0,0 +1,121 @@ +### Understanding Mounts in the Linux Filesystem + +#### 1. What is a Mount? + +A **mount** is the process of connecting a storage device (e.g., a disk partition, logical volume, or network share) to a specific directory in the Linux filesystem. Once mounted, that directory—called the **mount point**—serves as the access point to the device’s contents. + +**Core Concept**: Linux presents a single, unified filesystem hierarchy starting at the root directory (`/`). Mounts integrate various devices into this hierarchy, making them appear as part of the same tree, even if they reside on physically separate hardware or remote systems. + +**How It Works**: When a device is mounted to a directory (e.g., `/mnt/data`), accessing that directory allows you to interact with the device’s filesystem as if it were part of the main structure. + +**Example**: Mounting `/dev/sda1` to `/boot` makes the partition’s boot files accessible under `/boot`. + +#### 2. Why Use Mounts? + +Mounting devices provides flexibility and addresses limitations of storing all files on a single filesystem. Using multiple mounts avoids several pitfalls: + +- **Preventing Overflows**: High activity in one area (e.g., log files in `/var/log`) could fill a single filesystem, disrupting services. Separate mounts isolate space usage. +- **Enhancing Security**: A single filesystem limits security customization. By mounting a device to a specific directory, you can apply tailored mount options (e.g., read-only or no execution) to meet security needs. +- **Improving Scalability**: If a single filesystem runs out of space, expanding it is complex. Separate mounts make it easier to add or replace storage devices without affecting the entire system. +- **Organizational Clarity**: Mounts allow admins to dedicate devices to specific purposes (e.g., a partition for `/home` vs. `/var`), improving management and maintenance. + + +#### 3. Common Mounted Directories + +Certain directories are frequently mounted on dedicated devices to optimize performance, security, or system reliability. The choice depends on the system’s needs and the administrator’s discretion, but common examples include: + +- **`/boot`**: + - **Purpose**: Stores essential boot files (e.g., the Linux kernel). + - **Why Separate?**: Often placed on a dedicated partition because the root filesystem (`/`) may use Logical Volume Manager (LVM), which isn’t bootable by default. A separate `/boot` ensures the system can start. +- **`/boot/EFI`**: + - **Purpose**: Holds files for systems using Extensible Firmware Interface (EFI) for booting. + - **Why Separate?**: EFI requires a dedicated filesystem (e.g., FAT32) for early boot processes, distinct from other filesystems. +- **`/var`**: + - **Purpose**: Contains dynamic data like logs (`/var/log`), mail, or runtime files. + - **Why Separate?**: `/var` grows unpredictably (e.g., due to log writes). A dedicated device prevents it from filling the root filesystem and crashing services. +- **`/home`**: + - **Purpose**: Stores user home directories. + - **Why Separate?**: Enhances security (e.g., mount with `noexec` to block executable files or `nodev` to prevent device access). Also preserves user data during OS reinstalls, as `/home` can remain untouched. +- **`/usr`**: + - **Purpose**: Contains operating system files and programs, typically read-only for users. + - **Why Separate?**: Can be mounted as read-only to protect system files from unauthorized changes. +- **Custom Mounts**: Administrators may mount other directories (e.g., `/data` or `/backup`) based on specific requirements, such as isolating application data or backups. + + +#### 4. Mount Options + +When mounting a device, administrators can specify **mount options** to control how the filesystem behaves. These options enhance security, performance, or functionality. Examples include: + +- **`ro`**: Mounts the filesystem as read-only, preventing modifications (e.g., for `/usr`). +- **`noexec`**: Prevents execution of binaries, reducing security risks (e.g., for `/home`). +- **`nodev`**: Blocks device files, limiting potential exploits (e.g., for `/home`). +- **`rw`**: Allows read-write access (default for most mounts). + +**Use Case**: Mounting `/home` with `noexec,nodev` ensures users can’t run malicious scripts or access raw devices. + +#### 5. Exploring Mounts with Commands + +Several commands help administrators view and understand the current mount configuration: + +- **`mount`**: + - **Purpose**: Displays all currently mounted devices by reading `/proc/mounts`, a kernel-maintained file listing active mounts. + - **Output**: Includes device, mount point, filesystem type, and options (e.g., `/dev/nvme0n1p1` on `/boot` type `xfs` (`rw,relatime,seclabel`)). + - **Note**: Output can be lengthy, as it includes kernel interfaces (e.g., `sysfs`, `proc`) alongside physical devices. + - **Use Case**: Quick check of what’s mounted and where. +- **`df -Th`**: + - **Purpose**: Shows disk space usage, mount points, and filesystem types for mounted devices. + - **Options**: + - **`-T`**: Displays the filesystem type (e.g., `xfs`, `ext4`). + - **`-h`**: Presents sizes in human-readable units (e.g., GiB instead of kibibytes). + - **Output Columns**: + - **Filesystem**: Device name (e.g., `/dev/sda1`) or virtual device (e.g., `tmpfs`). + - **Type**: Filesystem type. + - **Size**: Total capacity. + - **Used**: Space in use. + - **Avail**: Free space. + - **Use%**: Percentage used. + - **Mounted on**: Mount point (e.g., `/boot`). + - **Example**: + +```plaintext +Filesystem Type Size Used Avail Use% Mounted on +/dev/sda1 xfs 197M 131M 67M 67% /boot +``` + + - **Use Case**: Checking available space or confirming mount points. +- **`findmnt`**: + - **Purpose**: Presents mounts in a tree-like structure, highlighting relationships between mount points. + - **Advantage**: Cleaner and more readable than `mount`, focusing on physical and key virtual filesystems. + - **Use Case**: Visualizing how mounts fit into the hierarchy (e.g., `/` vs. `/boot` vs. `/var`). + + +#### 6. Practical Context + +Mounts in Administration: Mounts are configured in `/etc/fstab` to persist across reboots, ensuring devices are automatically attached to their mount points. + +**Real-World Example**: A server might have: + +- `/` on an LVM volume for flexibility. +- `/boot` on a separate partition for booting. +- `/var` on a dedicated disk to handle log growth. +- `/home` on a network share for centralized user data. + +**Dynamic Management**: Admins can manually mount devices (e.g., `mount /dev/sdb1 /mnt`) for temporary access or troubleshooting. + +#### 7. Why Mounts Matter + +- **System Reliability**: Separate mounts prevent one area (e.g., `/var`) from crashing the entire system by filling up. +- **Security**: Mount options like `noexec` or `ro` protect critical directories. +- **Flexibility**: Mounts allow admins to mix local disks, network shares, and virtual filesystems seamlessly. +- **RHCSA Relevance**: Understanding mounts is critical for tasks like configuring `/etc/fstab`, troubleshooting disk issues, or setting up secure filesystems. +- **DevOps/SRE Impact**: Mounts underpin storage management in cloud VMs, containers, and clusters, ensuring data availability and performance. + +**Key Takeaways for Learners**: + +- A mount connects a device to a directory, integrating it into the `/` hierarchy. +- Multiple mounts avoid space, security, and scalability issues of a single filesystem. +- Common mount points like `/boot`, `/var`, and `/home` serve specific purposes. +- Mount options (e.g., `ro`, `noexec`) customize behavior for security or functionality. +- Use `mount`, `df -Th`, and `findmnt` to inspect the mount structure. + + diff --git a/DailyChallenges/Season2/Day10/3_links.md b/DailyChallenges/Season2/Day10/3_links.md new file mode 100644 index 0000000..1de3fde --- /dev/null +++ b/DailyChallenges/Season2/Day10/3_links.md @@ -0,0 +1,101 @@ +# Using Links + +Links on Linux are like aliases assigned to a file. There are symbolic links and hard links. To understand a link, you need to know how the Linux filesystem uses inodes for filesystem administration. + +### Understanding Hard Links + +Linux stores administrative data about files in inodes. The inode stores all administrative data about files. Every file on Linux has an inode, and important information is stored in it: + +* The data block where the file contents are stored +* The creation, access, and modification date +* Permissions +* File owners + +The filename is not stored in the inode; names are stored in the directory. Each filename knows which inode it has to address to access further file information. An inode knows how many names are associated with it; these names are referred to as **hard links**. Every file always has one hard link to start with, which is the name of the file. When you create a file, you are creating a hard link. On a Linux filesystem, multiple hard links can be created to a file, which is useful if a file with the same contents needs to be available at multiple locations. If a change is applied to any one of the hard links, it will show in all other hard links because all hard links point to the same data blocks. + +Restrictions for hard links: + +* Hard links must exist all on the same device (partition, logical volume, etc.). +* You cannot create hard links to directories. + +When the last hard link to a file is removed, access to the file’s data is also removed. There is no difference between the first hard link and the second hard link; they are both just hard links. + +#### Inode Diagram + +Here's a simple representation of how hard links and inodes work: + + +----------+ +---------+ + | Directory| --; | Inode | --; Data Blocks (File Content) + +----------+ +---------+ + ^ + | + +----------+ + |Hard Link | + +----------+ + +### Understanding Symbolic Links + +A **symbolic link** (also referred to as a soft link) does not link directly to the inode but to the name of the file. This makes symbolic links more flexible but has some disadvantages. The advantage of symbolic links is that they can link to files on other devices and directories. The major disadvantage is that when the original file is removed, the symbolic link becomes invalid. + +#### Symbolic Link Diagram + + +----------+ +--------------+ +----------+ +---------+ + |Directory | --; | Symbolic Link| --; |Directory | --; | Inode | --; Data Blocks + +----------+ +--------------+ +----------+ +---------+ + (Points to name) + +### Creating Links + +Use the `ln` command to create links. It uses the same order of parameters as `cp` and `mv`; first, you mention the source name, followed by the destination name. If you want to create a symbolic link, use the option `-s`, and then specify the source and target file or directory. One important restriction applies: to be able to create hard links, you must be the owner of the item that you want to link to. + + +| Command | Explanation | +| :-- | :-- | +| `ln /etc/hosts .` | Creates a hard link to the file `/etc/hosts` in the current directory | +| `ln -s /etc/hosts .` | Creates a symbolic link to the file `/etc/hosts` in the current directory | +| `ln -s /home /tmp` | Creates a symbolic link to the directory `/home` in the directory `/tmp` | + +The `ls` command will reveal whether a file is a link: + +* In the output of the `ls -l` command, the first character is an `l` if the file is a symbolic link. +* If a file is a symbolic link, the output of `ls -l` shows the name of the item it links to after the filename. +* If a file is a hard link, `ls -l` shows the hard link counter. + +[root@localhost tmp]\# \ls -l +total 3 +lrwxrwxrwx. 1 root root 5 Jan 19 04:38 home -> /home +-rw-r--r--. 3 root root 158 Jun 7 2013 hosts + +In the example above, `home` is a symbolic link pointing to `/home`, and `hosts` is a file with 3 hard links. + +### Removing Links + +Removing links can be dangerous. Let’s consider the following procedure: + +1. Make a directory named `test` in your home directory: `mkdir ~/test` +2. Copy all files that have a name starting with a, b, c, d, or e from `/etc` to this directory: `cp /etc/[a-e]* ~/test` +3. Type `ls -l ~/test/` to verify the contents of the `test` directory. +4. Make sure that you are in your home directory, by using `cd` without arguments. +5. Type `ln -s test link` +6. Type `rm link`. This removes the symbolic link. (Do not use `-r` or `-f` to remove symbolic links, even if they are subdirectories.) +7. Type `ls -l`. You’ll see that the symbolic link has been removed. +8. Let’s do it again. Type `ln -s test link` to create the link again. +9. Type `rm -rf link/` (which is what you would get by using Bash command-line completion). +10. Type `ls`. You’ll see that the directory `link` still exists. +11. Type `ls test/`. You’ll see the directory `test` is now empty. + +### 1Exercise: Working with Symbolic Links and Hard Links + +1. Open a shell as the `student` user. +2. From your home directory, type `ln /etc/passwd .` (Make sure that the command ends with a dot that has a space before it!). This command gives you an “operation not permitted” error because you are not the owner of `/etc/passwd`. +3. Type `ln -s /etc/passwd .` (Again, make sure that the command ends with a space and a dot!). This works; you do not have to be the owner to create a symbolic link. +4. Type `ln -s /etc/hosts` (this time with no dot at the end of the command). You’ll notice this command also works. If the target is not specified, the link is created in the current directory. +5. Type `touch newfile` and create a hard link to this file by using `ln newfile linkedfile`. +6. Type `ls -l` and notice the link counter for `newfile` and `linkedfile`, which is currently set to `2`. +7. Type `ln -s newfile symlinkfile` to create a symbolic link to `newfile`. +8. Type `rm newfile` +9. Type `cat symlinkfile`. You will get a “no such file or directory” error message because the original file could not be found. +10. Type `cat linkedfile`. This gives no problem. +11. Type `ls -l` and look at the way the `symlinkfile` is displayed. Also, look at `linkedfile`, which now has the link counter set to `1`. +12. Type `ln linkedfile newfile` +13. Type `ls -l` again. You’ll see that the original situation has been restored. diff --git a/DailyChallenges/Season2/Day10/4_compression_archive.md b/DailyChallenges/Season2/Day10/4_compression_archive.md new file mode 100644 index 0000000..9c827e0 --- /dev/null +++ b/DailyChallenges/Season2/Day10/4_compression_archive.md @@ -0,0 +1,71 @@ +# Working with Archives and Compressed Files + +This document provides a guide on how to manage archives and compressed files on a Linux system using the `tar` command, `gzip`, `bzip2`, and `xz` utilities. + +## Managing Archives with tar + +The `tar` (Tape Archiver) utility is used to create, list, extract, compress, and uncompress archives. + +### Creating Archives + +- Syntax: `tar -cf archivename.tar /files-you-want-to-archive` +- `-c`: Creates an archive. +- `-v`: Shows verbose output. +- Example: `tar -cvf /root/homes.tar /home` (archives the /home directory to /root/homes.tar) +- `-r`: Adds a file to an existing archive. + - Example: `tar -rvf /root/homes.tar /etc/hosts` +- `-u`: Updates an existing archive with newer files. + - Example: `tar -uvf /root/homes.tar /home` + +### Monitoring and Extracting tar Files + +- `-t`: Lists the contents of an archive. + - Example: `tar -tvf /root/homes.tar` +- `file` command: Analyzes a file and shows its type. + - Example: `file somefile` +- `-x`: Extracts an archive. + - Syntax: `tar -xvf /archivename.tar` (extracts to the current directory) +- `-C /targetdir`: Specifies the target directory for extraction. + - Example: `tar -xvf homes.tar -C /tmp` (extracts to /tmp) +- Extracting a single file: `tar -xvf /archivename.tar file-you-want-to-extract` + - Example: `tar -xvf /root/etc.tar etc/hosts` + +## Using Compression + +- `gzip`: Compresses files. + - Example: `gzip home.tar` (creates home.tar.gz) +- `bzip2`: An alternative compression utility. +- `xz`: Another alternative for compression. +- `gunzip`: Decompresses files compressed with gzip. +- `bunzip2`: Decompresses files compressed with bzip2. + +### tar Options for Compression + +- `-z`: Compresses/decompresses using gzip. +- `-j`: Compresses/decompresses using bzip2. +- `-J`: Compresses/decompresses using xz. + +## Overview of tar Options + +| Option | Use | +| :----- | :----------------------------------------------------------------- | +| c | Creates an archive. | +| v | Shows verbose output while tar is working. | +| t | Shows the contents of an archive. | +| z | Compresses/decompresses the archive using gzip. | +| j | Compresses/decompresses the archive using bzip2. | +| J | Compresses/decompresses the archive using xz. | +| x | Extracts an archive. | +| u | Updates an archive; only newer files will be written to the archive. | +| C | Changes the working directory before performing the command. | +| r | Appends files to an archive. | + +## Exercise + +1. Archive `/etc` directory: `tar cvf etc.tar /etc` +2. Compress the tar file: `gzip etc.tar` +3. Extract a file: `tar xvf etc.tar.gz etc/hosts` +4. Decompress the file: `gunzip etc.tar.gz` +5. Extract to a specific directory: `tar xvf etc.tar -C /tmp etc/passwd` +6. Create a compressed archive of the home directory: `tar cjvf homes.tar /home` +7. Remove temporary files: `rm -f *gz *tar` diff --git a/DailyChallenges/Season2/Day10/readme.md b/DailyChallenges/Season2/Day10/readme.md new file mode 100644 index 0000000..91c0726 --- /dev/null +++ b/DailyChallenges/Season2/Day10/readme.md @@ -0,0 +1,200 @@ +# Daily DevOps + SRE Challenge Series – Season 2 +## Day 10: The Filesystem Heist – Crack the Vault of Linux Mastery +![ChatGPT Image Apr 9, 2025, 07_30_45 AM](https://github.com/user-attachments/assets/738656ce-b015-4ddd-8b42-fb310654cf39) + +### Introduction +Welcome to Day 10 of the Daily DevOps + SRE Challenge Series – Season 2! 🎉 + +Today, you’re stepping into the core of Linux: the filesystem. This isn’t just a tutorial—it’s a high-stakes mission to navigate, secure, and conquer the server’s deepest layers. You’ll wield essential tools to traverse directories, manipulate files, manage storage, and lock down secrets, building skills that power DevOps pipelines and SRE heroics. By the end, you’ll: +- Master filesystem navigation with precision and speed. +- Command file and directory operations like a seasoned operative. +- Secure storage mounts, links, and archives under pressure. + +#### Why This Matters? +The filesystem is the backbone of every Linux system—your key to deploying apps, debugging outages, and scaling infrastructure. Here’s why these skills are mission-critical: +- **Everywhere You Go**: From AWS EC2 to Kubernetes pods, `cd /etc` and `cp -r /data /backup` are universal moves. +- **Automation Fuel**: Commands like `mv logs/*.log /archive/$(date +%F)` keep CI/CD pipelines humming. +- **Outage Lifeline**: When chaos strikes, `df -h` or `tar -czf backup.tar.gz /var` can save the day—fast. +- **Interview Edge**: “How do you recover a deleted config?” or “How do you manage disk space?”—nail these with filesystem fluency. +- **Real-World Win**: An SRE at a streaming service used `mkdir -p /mnt/cache && mount /dev/nvme1n1 /mnt/cache` to shift live streams to a new volume during a disk failure, keeping millions watching without a glitch. + +Gear up, crack your terminal—it’s time to breach the vault! + +--- + +## Challenge Breakdown +Today, you’re not just learning—you’re on a mission! Imagine you’re an SRE tasked with securing a critical server after a chaotic deployment. You’ll navigate treacherous filesystem paths, manage secret files, attach and secure new storage, and even collaborate with rogue agents (users). By the end, you’ll: +- Navigate and manipulate the filesystem like a stealth operative. +- Secure files, directories, and mounts under pressure. +- Execute advanced ops—linking, archiving, and user management—to outsmart the chaos. + +--- + +## Challenge Description +You’re securing a server after a botched update. Your mission: navigate the filesystem, manage critical files, secure a new storage vault, collaborate with agents, and archive secrets—all while leaving no trace. Ready? + +## Theory Questions +Before you dive into the heist, test your knowledge with these questions. They’re based on the filesystem concepts you’ll need to succeed. Write your answers in a file `~/heist_vault/theory_answers.txt` and check them against the solutions later. + +1. **Filesystem Hierarchy**: What is the purpose of the `/var` directory, and why might it be mounted on a separate device? +2. **Paths**: Explain the difference between an absolute path and a relative path. Give an example of each that reaches `/home/user/docs` from `/usr/share`. +3. **Wildcards**: If you run `ls f[io]le*` in a directory containing `file1`, `fole2`, `file3`, and `flea`, what files will be listed? +4. **Permissions**: What does `chmod 640 file.txt` do to the permissions of `file.txt`? Describe the resulting access for owner, group, and others. +5. **Mounts**: Why might an administrator mount `/home` with the `noexec` option? What does this prevent? +6. **Copying Files**: How does `cp -a /dir1 /dir2` differ from `cp -r /dir1 /dir2` when copying a directory? +7. **Links**: What’s the difference between a hard link and a symbolic link? Why might a hard link to `/etc/passwd` fail when created in your home directory? +8. **Redirection**: What does `echo "Log entry" >> /var/log/mylog` do? How is it different from using `>`? +9. **Mount Commands**: Compare the output of `mount`, `df -h`, and `findmnt`. Which would you use to check available disk space? +10. **SGID**: What happens to the group ownership of files created in a directory with the SGID bit set (e.g., `chmod g+s`)? + +--- + +### II. Practical + +### Task 1: Infiltrate the Filesystem (as `ec2-user`) +- Confirm your entry point with `pwd` (expect `/home/ec2-user`). +- Infiltrate `/var/log` using an absolute path. +- Retreat to `/var` with a relative move. +- Slip back to base (`~`) using a shortcut. +- In one command, hit `/tmp` then bounce back to base, verifying with `pwd`. + +### Task 2: Set Up the Hideout (as `ec2-user`) +- Establish `heist_vault` in `~`. +- Create subdirs `newfiles` and `oldfiles` in one command. +- In `newfiles`, plant a hidden file `.secret` and a decoy `decoy.txt` using `touch`. +- Secure `newfiles` with `chmod 700` and verify with `ls -ld`. + +### Task 3: Secure the New Vault - Storage Management (Root) +- **Agent Brief**: A new 1GB EBS volume arrives (or simulate with a loop device). + 1. As root: If on AWS, attach a 1GB EBS volume via the console; note the device (e.g., `/dev/xvdf`). If not, create a 1GB loop device: `dd if=/dev/zero of=/root/disk.img bs=1M count=1000; losetup /dev/loop0 /root/disk.img`. + 2. Partition it: `fdisk /dev/xvdf` (or `/dev/loop0`) → `n` (new), `p` (primary), `1` (partition 1), defaults, `w` (write). + 3. Format as ext4: `mkfs.ext4 /dev/xvdf1` (or `/dev/loop0p1`). + 4. Mount at `/data`: `mkdir /data; mount /dev/xvdf1 /data` (or `/dev/loop0p1`). + 5. Verify: `df -h` (shows `/data` usage) and `lsblk` (lists device tree). +- **Tip**: If `lsblk` shows no partitions, recheck `fdisk` steps. + +### Task 4: Advanced File Ops Under Pressure (as `ec2-user`) +- From `heist_vault/oldfiles`: + 1. Copy `newfiles` (including `.secret`) into `oldfiles` with `cp -a ../newfiles/ .`. + 2. Remove the nested `newfiles` with `rm -rf newfiles`. + 3. Copy only visible files from `../newfiles` using `cp -a ../newfiles/* .`. + 4. Copy hidden files separately with `cp -a ../newfiles/. .` and verify with `ls -a`. +- In `~`: + 5. Create `projects` with `house1` to `house9` using brace expansion. + 6. List only `house*` files (exclude others like `heist_vault`). + 7. Build `$HOME/projects/houses/doors/` and plant: + - `$HOME/projects/houses/bungalow.txt` + - `$HOME/projects/houses/doors/bifold.txt` + - `$HOME/projects/outdoors/vegetation/landscape.txt` + 8. Copy `house1` and `house5` to `houses/`. + 9. Recursively copy `/usr/share/doc/initscripts*` to `projects/` with `-a`. + 10. List `projects/` recursively, paging with `less`. + 11. Wipe `house6` to `house8` non-interactively. + 12. Move `house3` and `house4` to `doors/`. + 13. Obliterate `doors/` and its contents. + 14. Set `house2` perms to owner `rw`, group `r`, others none. + 15. Recursively block write access to `projects/` for all. + +### Task 5: Agent Collaboration (Root + Users) +- As root: + 1. Create group `agents`. + 2. Add users `alice` and `bob` to `agents` with home dirs and passwords. + 3. Establish `/data/shared_space`. + 4. Assign group `agents` with `chgrp`. + 5. Set perms to `770`. + 6. Enable SGID with `chmod g+s`. +- As `alice`: Create `alice_file.txt` in `/data/shared_space`. +- As `bob`: Create `bob_file.txt` in `/data/shared_space`. +- Verify: `ls -l` shows both files group-owned by `agents`. + +### Task 6: Cover Your Tracks with Links (as `ec2-user`) +- In `~`: + 1. Try hard-linking `/etc/passwd` to `passwd_hard` (expect denial). + 2. Soft-link `/etc/passwd` to `passwd_link`. + 3. Soft-link `/etc/hosts` without a target dir (default to current dir). +- In `heist_vault`: + 4. Create `evidence.txt`. + 5. Hard-link it to `evidence_copy` and check link count with `ls -l`. + 6. Soft-link it to `evidence_sym`. + 7. Delete `evidence.txt`. + 8. Test `cat evidence_sym` (broken) and `cat evidence_copy` (works). + 9. Restore `evidence.txt` from `evidence_copy` with `ln`. + 10. Verify with `ls -l`. + +### Task 7: Archive the Loot (Root) +- As root: + 1. Archive `/etc` to `/root/etc.tar` (uncompressed). + 2. Check type with `file`. + 3. Compress to `etc.tar.gz` with `gzip`. + 4. List contents with `tar tvf`. + 5. Extract `/etc/hosts` to `/root`. + 6. Verify with `ls -R /root/etc/`. + 7. Decompress `etc.tar.gz`. + 8. Extract `/etc/passwd` to `/tmp` with dir structure. + 9. Confirm with `ls -l /tmp/etc/passwd`. + 10. Create bzip2 archive of `/home` as `/root/homes.tar.bz2`. + 11. Clean up archives from `/root`. + +### Task 8: Root Shell Heist (as `student` → Root) +- As `student`: + 1. Escalate to root with `sudo -i`. + 2. Archive `/home` and `/etc` to `/root/essentials.tar`. + 3. Copy to `/tmp`. + 4. Hard-link it to `/essentials.tar`. + 5. Rename to `/archive.tar`. + 6. Soft-link from `/root/link.tar` to `/archive.tar`. + 7. Delete `/archive.tar` and observe `link.tar`. + 8. Remove `link.tar`. + 9. Compress `/root/essentials.tar` to `.gz`. + +--- + +# Submission Guidelines for Day 10: The Filesystem Heist + +Congrats on cracking the vault! To claim your victory and share your heist mastery, follow these guidelines. Submit your evidence and insights to showcase your filesystem skills! + +## Proof of Completion +Submit these deliverables to confirm you’ve cracked the challenge: +- **Screenshot of EC2 Terminal (Task 1)**: After the final step, run `pwd` to show `/home/ec2-user`, proving you hit `/tmp` and bounced back. +- **Screenshot of EC2 Terminal (Task 2)**: Display `ls -ld ~/heist_vault/newfiles` (expect `drwx------`) and `ls -a ~/heist_vault/newfiles` (showing `.secret`, `decoy.txt`). +- **Screenshot of EC2 Terminal (Task 3)**: As root, run `df -h | grep /data` and `lsblk`, confirming `/data` is mounted with a partition (e.g., `/dev/xvdf1` or `/dev/loop0p1`). +- **Screenshot of EC2 Terminal (Task 4)**: Post-task, show `ls -l ~/projects/house2` (perms `rw-r-----`) and `ls -R ~/projects` (no `doors/` subdir present). +- **Screenshot of EC2 Terminal (Task 5)**: Run `ls -l /data/shared_space`, displaying `alice_file.txt` and `bob_file.txt` with group `agents` and SGID (`drwxrws---`). +- **Screenshot of EC2 Terminal (Task 6)**: After restoring `evidence.txt`, show `ls -l ~/heist_vault` with `evidence_sym -> evidence.txt`, `evidence_copy`, and `evidence.txt` (link count 2). +- **Screenshot of EC2 Terminal (Task 7)**: As root, display `ls -l /tmp/etc/passwd` and `ls -R /root/etc/ | grep hosts`, verifying extracted archive files. +- **Screenshot of EC2 Terminal (Task 8)**: As root, show `ls -l /root/essentials.tar.gz` and `ls -l /root/link.tar` (broken link post-removal). +- **Text Output**: Full contents of `~/heist_vault/theory_answers.txt` with your answers to all 10 theory questions. + +## Documentation +Detail your heist in a text file or markdown doc (e.g., `heist_notes.md`): +- **Steps Taken**: Outline your approach for each task: + - Task 1: Navigating with `cd` and `pwd`. + - Task 2: Setting up `heist_vault` with `mkdir`, `touch`, and `chmod`. + - Task 3: Mounting `/data` with `fdisk`, `mkfs.ext4`, and `mount`. + - Task 4: Managing files in `oldfiles` and `projects` with `cp`, `rm`, `mv`, etc. + - Task 5: Creating users and syncing files in `/data/shared_space`. + - Task 6: Handling hard and soft links in `heist_vault`. + - Task 7: Archiving `/etc` and `/home` with `tar`, `gzip`, and `bzip2`. + - Task 8: Root ops with linking and compression. +- **Challenges & Fixes**: Document hurdles and solutions, e.g.: + - “`fdisk` didn’t save” → “Missed `w`, reran it.” + - “Permission denied on `/etc/passwd` hard link” → “Expected, switched to soft link.” + - “`ls -R` flooded terminal” → “Piped to `less` as instructed.” +- **Optional**: Add a takeaway (e.g., “Hard links survive deletion—mind blown!”). + +## Share Your Progress! +Broadcast your heist success to the world: +- Post on social media with your screenshots, insights, or a quick “I cracked Day 10!” shoutout. +- Use hashtags: `#getfitwithsagar`, `#SRELife`, `#DevOpsForAll`. +- Tag me (@SagarUtekar) for a chance to get featured! + +## **Join Our Community** +Connect with fellow learners: +- **Discord:** [Join here](https://discord.gg/mNDm39qB8t) +- **Google Group:** [Join here](https://groups.google.com/forum/#!forum/daily-devops-sre-challenge-series/join) +- **YouTube:** [Subscribe here](https://www.youtube.com/@Sagar.Utekar) + +--- + +Keep learning and happy exploring! +Sagar Utekar diff --git a/DailyChallenges/Season2/Day11/1_vi_editor_tips.md b/DailyChallenges/Season2/Day11/1_vi_editor_tips.md new file mode 100644 index 0000000..377423e --- /dev/null +++ b/DailyChallenges/Season2/Day11/1_vi_editor_tips.md @@ -0,0 +1,83 @@ +# vi/vim Cheat Sheet – Quick Reference + +## Modes +| Mode | Key | Description | +|-----------------|-------------|------------------------------------| +| Command | Default | For navigation and actions | +| Insert | `i` / `a` / `o` | Enter insert mode (insert, append, open) | +| Return to Command | `Esc` | Switch from insert to command mode | + +## Insert Mode Commands +| Command | Description | +|---------|---------------------------------| +| `i` | Insert before the cursor | +| `a` | Append after the cursor | +| `I` | Insert at the beginning of the line | +| `A` | Append at the end of the line | +| `o` | Open new line below | +| `O` | Open new line above | +| `Esc` | Return to command mode | + +## Navigation +| Key | Moves Cursor To | +|-------------|-------------------------------| +| `h` / `l` | Left / Right one character | +| `j` / `k` | Down / Up one line | +| `w` / `W` | Next word (by punctuation / space) | +| `b` / `B` | Previous word | +| `0` | Start of line | +| `$` | End of line | +| `H` | Top of screen | +| `M` | Middle of screen | +| `L` | Bottom of screen | +| `G` | Last line of file | +| `1G` | First line of file | +| `35G` | Line 35 | +| `Ctrl+f` | Page forward | +| `Ctrl+b` | Page backward | +| `Ctrl+d` | Half page forward | +| `Ctrl+u` | Half page backward | + +## ✂️ Editing Text +| Command | Description | +|-------------|-----------------------------------| +| `x` / `X` | Delete character under / before cursor | +| `dd` | Delete current line | +| `dw` | Delete word after cursor | +| `db` | Delete word before cursor | +| `d$` | Delete from cursor to end of line | +| `c + move` | Change text (e.g., `c$` changes to end) | +| `cc` | Change entire line | +| `yy` | Yank (copy) current line | +| `y}` | Yank paragraph | +| `p` | Paste below/after | +| `P` | Paste above/before | +| `.` | Repeat last change | + +## Searching +| Command | Description | +|------------------|---------------------------------------| +| `/text` | Search forward for "text" | +| `?text` | Search backward for "text" | +| `n` / `N` | Repeat search forward / backward | +| `/The.*end` | Regex: match “The” followed by “end” | +| `?[Pp]rint` | Case-insensitive search for “Print” or “print” | + +## Ex Commands (use `:`) +| Command | Description | +|-------------------|---------------------------------------| +| `:w` | Save file | +| `:q` | Quit if no unsaved changes | +| `:wq` / `ZZ` | Save and quit | +| `:q!` | Quit without saving | +| `:s/old/new/` | Replace first “old” with “new” in current line | +| `:g/old/s//new/g` | Replace all “old” with “new” in file | +| `:g/old/s//new/gp`| Replace and print affected lines | + +## Pro Tips +| Command | Description | +|-----------|---------------------------------------| +| `u` | Undo last change | +| `Ctrl+r` | Redo (undo the undo) | +| `:!command`| Run shell command (e.g., `:!date`, `:!ls`) | +| `Ctrl+G` | Show filename, line number, and position | diff --git a/DailyChallenges/Season2/Day11/2_common_text_commands.md b/DailyChallenges/Season2/Day11/2_common_text_commands.md new file mode 100644 index 0000000..93f8335 --- /dev/null +++ b/DailyChallenges/Season2/Day11/2_common_text_commands.md @@ -0,0 +1,266 @@ +## Essential Tools for Managing Text File Contents +| Command | Explanation | +|---------|-----------------------------------------------------------------------------| +| `less` | Opens a file in a pager for easy reading and scrolling | +| `cat` | Displays the entire file contents on the screen | +| `head` | Shows the first few lines of a file (default: 10) | +| `tail` | Shows the last few lines of a file (default: 10) | +| `cut` | Filters specific columns or characters from a file | +| `sort` | Sorts the file contents alphabetically or numerically | +| `wc` | Counts lines, words, and characters in a file | + +### Versatility with Pipes +These commands shine when paired with pipes: +- `less /etc/passwd`: View the `/etc/passwd` file. +- `ps aux | less`: Pipe process list output to `less` for scrollable reading. + +--- + +## Mastering `less` +The `less` utility is a go-to tool for reading text files as a Linux admin. + +### Usage +- Open a file: `less /etc/passwd` +- Navigate: Use **Page Up**/**Page Down** to scroll, press **`q`** to quit. +- Search: + - Forward: `/sometext` + - Backward: `?sometext` + - Repeat: **`n`** + +### Connection to `vim` and `man` +`less` shares navigation and search mechanics with `vim` and `man` due to their common code base. + +> **Note:** +> `less` was created to outdo `more`, a basic pager, inspiring the phrase "do more with less." Today, both tools have similar features, but `more` exits at file end (use `-p` to change this) and is standard everywhere, while `less` may require installation. + +--- + +## Exercise 1: Applying Basic `less` Skills + +### Objective + +Practice using the `less` command to view and navigate through text files. + +### Steps + +1. **Open a File:** +Open the `/etc/passwd` file using `less`: + +```bash +less /etc/passwd +``` + +2. **Search:** +Type `/root` to search for and highlight all occurrences of "root" in the file. +3. **Go to End:** +Press `G` to jump to the last line of the file. +4. **Quit:** +Press `q` to exit the `less` viewer. +5. **Pipe Output:** +Pipe the output of the `ps aux` command to `less` to browse the process list: + +```bash +ps aux | less +``` + +## Showing File Contents with `cat` + +### Overview + +The `cat` command is a simple utility used to display the contents of a text file directly on the terminal screen. It is most suitable for short files, as it displays the entire file from start to finish without pausing. + +### Using `cat` + +1. **Basic Usage:** +To view the contents of a file, simply type `cat` followed by the filename: + +```bash +cat filename.txt +``` + +2. **Example:** +To display the contents of the `/etc/passwd` file, use: + +```bash +cat /etc/passwd +``` + + +### Limitations + +- **Long Files:** If the file is long, `cat` will display all contents, but only the lines that fit on the terminal screen will be visible. You might miss the beginning of the file as it scrolls by. +- **Viewing First Lines:** If you want to see the first lines of a long file, you can use the `tac` command, which displays the file in reverse order (from end to beginning). + + +### Alternatives + +For longer files, consider using `less` or `more`, which allow you to view the file one page at a time and navigate through it more easily. + +### Tip + +- **Inverse Display:** Use `tac` to display a file in reverse order: + +```bash +tac filename.txt +``` + +6. **Quit Again:** +Press `q` to exit the `less` viewer. + + +## Displaying First or Last Lines with `head` and `tail` + +### Overview + +The `head` and `tail` commands are used to display the beginning or end of a file, respectively. By default, they show the first or last ten lines of a file, but this can be adjusted. + +### Using `head` + +1. **Default Usage:** +To display the first ten lines of a file: + +```bash +head filename.txt +``` + +2. **Specifying Number of Lines:** +To display a specific number of lines, use the `-n` option followed by the number: + +```bash +head -n 5 filename.txt +# Or simply: +head -5 filename.txt +``` + + +### Using `tail` + +1. **Default Usage:** +To display the last ten lines of a file: + +```bash +tail filename.txt +``` + +2. **Specifying Number of Lines:** +To display a specific number of last lines: + +```bash +tail -n 2 filename.txt +# Or simply: +tail -2 filename.txt +``` + +3. **Following a File:** +The `-f` option displays the last ten lines and continues to display new lines as they are added to the file. This is useful for monitoring log files: + +```bash +tail -f /var/log/messages +``` + +Press `Ctrl-C` to stop following the file. + +### Combining `head` and `tail` + +You can combine `head` and `tail` using pipes to extract specific lines from a file. + +1. **Example:** +To display line number 11 of a file: + +```bash +head -n 11 filename.txt | tail -n 1 +``` + +This command first displays the first 11 lines and then extracts the last line from that output, which is line number 11 of the original file. + + +## Filtering Columns with `cut` and Sorting with `sort` + +### Filtering Specific Columns with `cut` + +#### Overview + +The `cut` command is used to extract specific columns (fields) from a text file based on a delimiter. + +#### Usage + +```bash +cut -d <delimiter> -f <field_number> filename.txt +``` + +- `-d <delimiter>`: Specifies the delimiter that separates the fields. +- `-f <field_number>`: Specifies the field number to extract. + + +#### Example + +To extract the first field (username) from the `/etc/passwd` file, where fields are delimited by a colon: + +```bash +cut -d : -f 1 /etc/passwd +``` + + +### Sorting File Contents with `sort` + +#### Overview + +The `sort` command is used to sort the lines of a text file. + +#### Usage + +```bash +sort filename.txt +``` + +- This sorts the file contents in byte order (ASCII order). + + +#### Sorting Output of a Command + +You can sort the output of another command using a pipe: + +```bash +cut -f 1 -d : /etc/passwd | sort +``` + +This extracts the first column from `/etc/passwd` and sorts the result. + +#### Sorting Options + +- `-n`: Sorts numerically. + +```bash +cut -f 3 -d : /etc/passwd | sort -n +``` + +This sorts the third field of `/etc/passwd` in numeric order. +- `-r`: Sorts in reverse order. + +```bash +du -h | sort -rn +``` + +This lists files sorted by size, with the largest file first. +- `-k column_number`: Specifies the column to sort. +- `-t delimiter`: Specifies the field separator. + +```bash +sort -k3 -t : /etc/passwd +``` + +This sorts the `/etc/passwd` file by the third column, using ":" as the delimiter. + + +#### Example: Finding Busiest Processes + +To sort processes by memory usage (4th column in `ps aux` output): + +```bash +ps aux | sort -k 4 -n +``` + +This command displays processes sorted by memory usage, with the process using the least memory listed first. + + + diff --git a/DailyChallenges/Season2/Day11/3_grep_egrep.md b/DailyChallenges/Season2/Day11/3_grep_egrep.md new file mode 100644 index 0000000..f5cffd9 --- /dev/null +++ b/DailyChallenges/Season2/Day11/3_grep_egrep.md @@ -0,0 +1,196 @@ +## Using `grep` to Analyze Text + +### Overview + +`grep` (Global Regular Expression Parser) is a powerful utility for searching plain-text data using regular expressions. It works on files or command outputs. + +### Key Features + +- **Case-sensitive by default** +- **Can search files, directories, and command output** +- **Supports regular and extended regular expressions** +- **Highlights matches using `--color`** + +--- + +### Commonly Used Options + +| Option | Description | +| :-- | :-- | +| `-i` | Case-insensitive matching | +| `-v` | Invert match (show lines that do **not** match) | +| `-r` | Recursively search directories | +| `-l` | Print only filenames with matches | +| `-e` | Use multiple regex patterns | +| `-E` | Interpret pattern as an extended regular expression | +| `-A N` | Show N lines *after* each match | +| `-B N` | Show N lines *before* each match | +| `--color` | Highlight the matching text | + +--- + +### Practical Examples + +#### 1. **Simple Match in File** + +```bash +grep desktop /etc/services +``` + +* Matches exact lowercase `desktop`. + + +#### 2. **Case-Insensitive Match** + +```bash +grep -i desktop /etc/services +``` + +* Matches variations like `Desktop`, `DESKTOP`, etc. + + +#### 3. **Exclude Matches** + +```bash +grep -vi tcp /etc/services +``` + +* Show all lines **except** those containing `tcp`. + + +#### 4. **Recursive Search (Case-Insensitive)** + +```bash +grep -rli peerdns /usr/share/doc/ +``` + +* Search recursively for `peerdns`, show filenames only. + + +#### 5. **Recursive Search with Highlight** + +```bash +grep -ri --color root /etc/sysconfig/ +``` + +* Highlights `root` in red (by default) wherever it appears. + + +#### 6. **Filter Output of Another Command** + +```bash +ip addr show | grep inet +``` + +* Shows only lines with `inet`, useful for filtering IP addresses. + +--- + +### Filtering Comments and Blank Lines + +```bash +grep -v -e '^#' -e '^$' /etc/services +``` + +* Removes comment lines (`#`) and empty lines. + +--- + +### Finding lines starting with a comment (`#`)**: + +```bash +grep ' #' /etc/services +``` + +### Excluding lines starting with a comment**: + +```bash +grep -v '^#' /etc/services +``` + +* Shows only lines that do not start with `#`. +### Showing lines before matches**: + +```bash +grep -v '^#' /etc/services -B 5 +``` +* Shows lines that do not start with `#`, along with the 5 lines before each match. + +## Tips + +- Combine `grep` with `sort`, `uniq`, and `wc` for powerful text processing. +- Use double quotes for patterns with spaces or variables: `grep "$pattern" file.txt`. +- Use `^` and `$` to anchor searches to the beginning or end of lines. + + + +## Using Extended Regular Expressions (EREs) + +### Overview + +Extended Regular Expressions (EREs) provide additional features compared to basic regular expressions. Tools like `grep` require the `-E` option to recognize EREs. + +### Key ERE Features + +| Regular Expression | Use | +| :-- | :-- | +| `^text` | Matches a line that starts with the specified text. | +| `text$` | Matches a line that ends with the specified text. | +| `.` | Wildcard: Matches any single character. | +| `[abc]` | Matches `a`, `b`, or `c`. | +| `?` | Matches zero or one occurrence of the preceding character. | +| `+` | Matches one or more occurrences of the preceding character. | +| `*` | Matches zero to an infinite number of occurrences of the preceding character. | +| `\{2\}` | Matches exactly two occurrences of the preceding character. | +| `\{1,3\}` | Matches a minimum of one and a maximum of three occurrences of the preceding character. | +| `colou?r` | Matches zero or one of the previous character. Matches both "color" and "colour". | +| `(...)` | Groups multiple characters so that the regular expression can be applied to the group. | + +### Examples and Usage + +1. **Enabling ERE with `grep`**: + * Use `grep -E` to interpret patterns as extended regular expressions. +2. **Example Regular Expression**: + * `"/web(/.*)?"` + * Matches `/web`, `/web/`, or `/web/filename`. + +### Demonstrating ERE with `grep` + +1. **Create a text file named `regex.txt`**: + +```text +bat +boot +boat +bt +``` + +2. **Base Regular Expression**: + +```bash +grep 'b.*t' regex.txt +``` + + * Finds any line that starts with `b` and ends with `t`. +3. **Extended Regular Expression (Incorrect)**: + +```bash +grep 'b.+t' regex.txt +``` + + * Without the `-E` option, `grep` does not correctly interpret the `+`. +4. **Extended Regular Expression (Correct)**: + +```bash +grep -E 'b.+t' regex.txt +``` + + * Finds lines that have at least one character between `b` and `t`. + + + +### Examples and Usage + + + + diff --git a/DailyChallenges/Season2/Day11/4_awk_sed.md b/DailyChallenges/Season2/Day11/4_awk_sed.md new file mode 100644 index 0000000..fcc25f2 --- /dev/null +++ b/DailyChallenges/Season2/Day11/4_awk_sed.md @@ -0,0 +1,101 @@ +## Working with Other Useful Text Processing Utilities + +### Overview + +Besides `grep`, `awk` and `sed` are powerful utilities for text processing, particularly useful in scripted environments where direct screen interaction is limited. + +### `awk` + +#### Overview + +`awk` is a versatile utility for text processing, capable of more complex field extraction than `cut`. + +#### Usage + +```bash +awk -F delimiter '{ print $column_number }' filename +``` + +- `-F delimiter`: Specifies the field delimiter. +- `'{ print $column_number }'`: Specifies the column to print. + + +#### Examples + +1. **Printing the fourth field from `/etc/passwd`**: + +```bash +awk -F : '{ print $4 }' /etc/passwd +``` + +2. **Searching for text and printing a field**: + +```bash +awk -F : '/user/ { print $4 }' /etc/passwd +``` + + * Searches `/etc/passwd` for lines containing "user" and prints the fourth field of matching lines. + +### `sed` + +#### Overview + +`sed` (stream editor) is used for filtering and modifying text in files. + +#### Usage + +1. **Printing a specific line**: + +```bash +sed -n line_number;p filename +``` + + * `-n`: Suppresses default output. + * `p`: Print command. + +Example: + +```bash +sed -n 5p /etc/passwd +``` + + * Prints the fifth line from `/etc/passwd`. +2. **Replacing text in a file**: + +```bash +sed -i 's/old-text/new-text/g' filename +``` + + * `-i`: Modifies the file directly. + * `s/old-text/new-text/g`: Substitutes all occurrences of `old-text` with `new-text`. + +Example: + +```bash +sed -i 's/old-text/new-text/g' ~/myfile +``` + + * Replaces all occurrences of "old-text" with "new-text" in `~/myfile`. +3. **Deleting lines by line number**: + +```bash +sed -i -e 'line_number;d' filename +``` + + * `-e`: Specifies an editing command. + * `d`: Delete command. + +Example: + +```bash +sed -i -e '2d' ~/myfile +``` + + * Deletes line 2 from `~/myfile`. +4. **Deleting multiple lines**: + +```bash +sed -i -e '2d;20,25d' ~/myfile +``` + + * Deletes line 2 and lines 20 through 25 in `~/myfile`. diff --git a/DailyChallenges/Season2/Day11/5_locate_find.md b/DailyChallenges/Season2/Day11/5_locate_find.md new file mode 100644 index 0000000..838e2a4 --- /dev/null +++ b/DailyChallenges/Season2/Day11/5_locate_find.md @@ -0,0 +1,57 @@ +## Finding Files in Linux: Summary + +This section covers two main utilities for finding files: `locate` (for fast searches using a database) and `find` (for real-time, flexible searches). + +### 1. `locate`: Quick Database Search + +* **Method:** Searches a pre-built database of filenames. +* **Speed:** Very fast. +* **Database Updates:** The database is updated by `updatedb`, usually run daily. +* **Limitations:** + * Won't find files created since the last `updatedb` run. + * Excludes pruned paths (e.g., `/tmp`, `/media`) defined in `/etc/updatedb.conf`. +* **Permissions:** Respects user permissions. +* **Examples:** + +```bash +locate .bashrc # Finds all .bashrc files +locate -i muttrc # Case-insensitive search +``` + + +### 2. `find`: Real-Time Filesystem Search + +* **Method:** Searches the live filesystem. +* **Speed:** Slower than `locate`, but always up-to-date. +* **Search Criteria:** + * Filename (`-name`, `-iname`) + * Size (`-size +10M`, `-size -1G`) + * Ownership (`-user`, `-group`) + * Permissions (`-perm 755`, `-perm +222`) + * Modification, change, or access times (`-mtime`, `-ctime`, `-atime`, `-mmin`, `-cmin`, `-amin`) +* **Actions:** + * `-exec`: Executes a command on each result (e.g., `-exec du -sh {} \;`) +* **Error Handling:** Redirect errors with `2> /dev/null`. +* **Examples:** + +```bash +find /etc -name passwd # Exact name match +find /etc -iname '*passwd*' # Partial + case-insensitive match +find /home -user chris # Files owned by chris +find /bin -perm 755 # Files with exact permissions +find /var -size +500M # Large files +find /etc -mmin -10 # Modified in last 10 minutes +``` + + +### 3. Combining Conditions + +* `-not`: Excludes specific matches. +* `-or` (`-o`): Includes multiple conditions. +* Parentheses `$ ... $`: For grouping conditions. +* **Examples:** + +```bash +find /var/allusers \( -user joe -o -user chris \) -ls # Files owned by joe OR chris +find /var -user joe -not -group joe # Files owned by joe, NOT in group joe +``` diff --git a/DailyChallenges/Season2/Day11/readme.md b/DailyChallenges/Season2/Day11/readme.md new file mode 100644 index 0000000..8de0b4a --- /dev/null +++ b/DailyChallenges/Season2/Day11/readme.md @@ -0,0 +1,131 @@ +# Daily DevOps + SRE Challenge Series – Season 2 +## Day 11: The Text File Takedown – Wrangle Logs and Configs Like a Pro + +### Introduction +Welcome to Day 11 of the Daily DevOps + SRE Challenge Series – Season 2! 🎉 + +Today, you’re diving into the art of text file mastery—crucial for taming logs, tweaking configs, and slicing through data in Linux. You’ll wield powerful tools to read, filter, sort, and edit text, turning raw files into actionable insights. By the end, you’ll: +- Navigate and extract data from text files with ninja-like precision. +- Transform and analyze logs/configs using command-line magic. +- Build SRE-ready skills for debugging and automation. + +#### Why This Matters? +Text files are the lifeblood of Linux systems—logs in `/var/log`, configs in `/etc`, scripts everywhere. Mastering them is non-negotiable: +- **Universal Power**: `grep`, `awk`, and `sed` work across clouds, containers, and bare metal—your Swiss Army knife for any system. +- **Automation Edge**: `tail -f /var/log/app.log | grep ERROR` feeds live alerts into monitoring scripts. +- **SRE Superpower**: During outages, `sort | uniq -c` on logs spots patterns fast; `sed` fixes configs on the fly. +- **Interview Gold**: “How do you find errors in a 1GB log?” or “Extract IPs from a file?”—you’ll ace these. +- **Real-World Save**: An SRE at a cloud provider used `awk '{print $1}' /var/log/access.log | sort | uniq -c` to pinpoint a DDoS source, throttling it in minutes. + +Grab your terminal—it’s time to take down the text chaos! + + +In UNIX and Linux systems, configuration and information management have traditionally relied on plain-text files. Despite the availability of graphical tools today, editing these files manually with a text editor remains essential, especially for full system control and on servers without GUIs. Aspiring system administrators must learn to use text editors and tools like find (to locate files) and grep (to search within files) to manage and configure Linux systems effectively. + + ## Practical Tasks Setup: Linux Text Avengers Training Camp + +Welcome, recruits! You've been selected to join the Linux Text Avengers, the elite team responsible for manipulating text files to save the world (or at least your systems). Get ready for a series of challenges that will test your text-fu and prepare you for any text-based crisis. + +### 1. Create Your Base of Operations + +* Every hero needs a base! Create your training directory: + +```bash +mkdir ~/textlab +cd ~/textlab +``` + + +### 2. The Intelligence File + +* Your first mission requires analyzing an intelligence file. Create `myfile.txt` with this vital information: + +```bash +cat << EOF > myfile.txt +Line 1: This is a test file. Initializing systems... +Line 2: Administrator access granted. Beware of rogue agents. +Line 3: Memory usage: 500MB. System stable. +Line 4: Error log entry: Unauthorized access attempt detected. +Line 5: IP address: 192.168.1.100. Target server. +Line 6: Password: secret123. DO NOT SHARE. +Line 7: End of file. Mission critical. +EOF +``` + + +### 3. Log Simulation: The Enemy's Activity + +* Simulate enemy activity by creating fake logs. The logs directory will track their actions. + +```bash +mkdir ~/textlab/logs +echo "error: disk full. System compromised!" > ~/textlab/logs/log1.log +echo "ERROR: network down. Communications disrupted!" > ~/textlab/logs/log2.log +echo "warning: low memory. System stability threatened!" > ~/textlab/logs/log3.log +# Make log1.log larger than 1MB to simulate a real log. +for i in {1..10000}; do echo "filler line $i" >> ~/textlab/logs/log1.log; done +``` + + +### 4. System Check + +* Before you begin, ensure you have the necessary tools and access. + * You should be able to read system files like `/etc/passwd` and `/var/log/messages` (some tasks might require sudo). + * Install `mailx` for communication tasks: + +```bash +sudo dnf install mailx # Or sudo apt install mailutils on Debian/Ubuntu +``` + + +### Mission Briefing: Data Analysis + +* Our world depends on your ability to analyze and manipulate text data. Every line in `myfile.txt` is critical: + * Line 1: Initialization status. + * Line 2: Administrator alert, with a hidden warning. + * Line 3: Memory info to optimize system. + * Line 4: Error log for security checks. + * Line 5: Target IP address. + * Line 6: A Secret! + + +## Operation: Text Manipulation + +### Original Tasks: Core Skills Training + +1. **Decoding the Intelligence:** Show line 5 from `~/textlab/myfile.txt` using two different methods. What does this IP address represent? +2. **Hunting the Target:** Locate text files in `~/textlab` containing `"192.168.1.100"`. Is a regex needed for this simple search? +3. **Reversing the Sabotage:** A rogue agent changed `"Administrator"` to `"root"` in `~/textlab/myfile.txt` using `sed`. Can you undo this sabotage *without* a backup? +4. **Memory Optimization:** Sort `myfile.txt` lines, simulating system performance by putting the largest memory value first (based on line 3). This tests ability to analyze numerical info within text. +5. **System Recon:** Extract the sixth column from `ps aux` to understand system usage. (This is a unchanged recon mission, unrelated to the rest of this scenario). +6. **Deleting Sensitive Info:** Remove the potentially compromised "Password" line (line 6) from `~/textlab/myfile.txt`. + +### Additional Interesting Tasks: Advanced Missions + +1. **Swapping Intel:** Swap lines 3 and 4 in `~/textlab/myfile.txt` using `vi/vim` to reorganize mission-critical info. +2. **Archiving Enemy Logs:** Find all `.log` files larger than 1MB under `~/textlab/logs`, and archive them to a `.tar.gz` in `~/textlab` for analysis. Think of this step as containing damage after an attack. +3. **Finding Patterns:** Use `grep`, `sort`, and `uniq` to count unique words with `"error"` (case-insensitive) in `~/textlab/logs/log1.log`. What does that data signify about system stability? +4. **Alerting the Team:** Search `~/textlab` for `"password"`, highlight it in color, and email this vital alert to yourself using `mail`. (Configure `mail` first if you haven't done so) + + +# Submission Guidelines +- Store your findings and execution steps in a markdown file (`solution.md`). +- Submit it in your GitHub repository and share the link. +- Post your experience on social media with **#getfitwithsagar #SRELife #DevOpsForAll**. +- Tag us in your posts for visibility and networking. + +### **Share Your Progress!** +Post your experience on social media with the hashtags: **#getfitwithsagar, #SRELife, #DevOpsForAll** + +--- + +## **Join Our Community** +Connect with fellow learners: +- **Discord:** [Join here](https://discord.gg/mNDm39qB8t) +- **Google Group:** [Join here](https://groups.google.com/forum/#!forum/daily-devops-sre-challenge-series/join) +- **YouTube:** [Subscribe here](https://www.youtube.com/@Sagar.Utekar) + +--- + +Keep learning and happy exploring! +Sagar Utekar diff --git a/DailyChallenges/Season2/Day12/readme.md b/DailyChallenges/Season2/Day12/readme.md new file mode 100644 index 0000000..3dcb5b0 --- /dev/null +++ b/DailyChallenges/Season2/Day12/readme.md @@ -0,0 +1,468 @@ +# Daily DevOps + SRE Challenge Series – Season 2 +## Day 12: The Process Power-Up – Command Your Linux System Like a Pro + +### Introduction +Welcome to **Day 12** of the Daily DevOps + SRE Challenge Series – Season 2! 🎉 + +Today, you’re stepping into the control room of your Linux system, mastering the art of **process management**. Processes are the heartbeat of any system—every command, script, or service runs as a process. Knowing how to monitor, prioritize, and control them is like wielding a superpower for debugging, optimizing, and keeping systems humming. By the end of this challenge, you’ll: +- Monitor running processes with precision using tools like `ps` and `top`. +- Control process priorities to optimize system performance. +- Manage background jobs and terminate rogue processes like a seasoned SRE. + +#### Why This Matters? +Process management is critical for DevOps and SRE roles: +- **System Health**: Spot CPU-hogging processes before they crash your app. +- **Automation**: Script `kill` or `nice` commands to automate resource management. +- **Outage Recovery**: During incidents, `top` and `pkill` help you isolate and stop problematic services. +- **Interview Win**: Questions like “How do you find a zombie process?” or “What’s the difference between `SIGTERM` and `SIGKILL`?” are common. +- **Real-World Impact**: An SRE at a fintech firm used `htop` to trace a memory leak in a trading app, saving millions by preventing downtime. + +Grab your terminal and let’s power up your process-fu! + +--- + +### 1. Understanding Processes +- **Definition**: A process is an instance of a running program, identified by a unique **Process ID (PID)**. Every command (e.g., `ls`), script, or service (e.g., `httpd`) creates a process. +- **Attributes**: + - **PID**: Unique identifier (e.g., 1234). + - **PPID**: Parent Process ID, linking to the process that started it (e.g., `bash` spawns `ls`). + - **User**: The account running the process (e.g., `root` or `user1`). + - **State**: Current status: + - **R** (Running): Actively using CPU. + - **S** (Sleeping): Waiting for an event (e.g., I/O). + - **D** (Uninterruptible Sleep): Waiting for I/O, can’t be killed easily. + - **Z** (Zombie): Defunct, awaiting parent cleanup. + - **T** (Stopped): Paused, often via `Ctrl+Z`. +- **Types**: + - **Foreground**: Occupies the terminal (e.g., `vim`). + - **Background**: Runs without terminal control (e.g., `sleep 100 &`). + - **Daemons**: System services running continuously (e.g., `sshd`). +- **Process Creation**: Processes are forked (copied) from a parent, often `init` (PID 1) or `systemd` in RHEL. + +### 2. Monitoring Processes +To manage processes, you first need to see what’s running. RHEL provides several tools for this purpose. + +- **`ps` (Process Status)**: + - Displays a snapshot of processes. + - **Key Options**: + - `-a`: Show all processes with a terminal (excludes daemons). + - `-u user`: Filter by user (e.g., `ps -u root`). + - `-x`: Include processes without a terminal (e.g., daemons). + - `aux`: BSD-style, shows all processes (user, PID, %CPU, %MEM, command). + - `-e`: Show every process (System V style). + - `-f`: Full format (includes PPID, start time). + - `-ef`: Combines `-e` and `-f` for hierarchy view. + - `-p PID`: Select specific PID (e.g., `ps -p 1234`). + - `-C command`: Select by command name (e.g., `ps -C httpd`). + - `--sort=field`: Sort output (e.g., `ps aux --sort=-%cpu` for highest CPU first). + - `-o field1,field2`: Customize columns (e.g., `ps -o pid,comm,state`). + - `-l`: Long format (includes nice value, priority). + - Common options: + - `ps aux`: Shows all processes in BSD format (user, PID, %CPU, %MEM, command). + - `ps -ef`: Displays all processes in System V format, including PPID. + - `ps -u username`: Lists processes for a specific user. + - `ps -p 1234`: Shows details for PID 1234. + - Example: + ```bash + ps aux | grep httpd + ``` + Lists all `httpd` processes with resource usage. + - **Customization**: Use `ps -o pid,ppid,comm,state` to select specific columns (PID, PPID, command, state). + +- **`top`**: + - Real-time, interactive process viewer. + - Key features: + - Displays CPU, memory, and load averages. + - Sort by CPU (`P`), memory (`M`), or PID (`N`). + - Kill processes: Press `k`, enter PID, select signal (e.g., 15 for `SIGTERM`). + - Adjust priority: Press `r`, enter PID, set nice value. + - Example: + ```bash + top + ``` + Press `q` to quit. + - **Fields**: + - `%CPU`: CPU usage percentage. + - `%MEM`: Memory usage percentage. + - `NI`: Nice value (priority). + - `S`: Process state (R, S, Z, etc.). + +- **`uptime`**: + - Shows system load averages (1, 5, 15 minutes). + - Example: + ```bash + uptime + ``` + Output: `12:34:56 up 2 days, 3:45, 2 users, load average: 0.50, 0.75, 1.00`. + - **Interpretation**: A load average above the number of CPU cores indicates resource contention. + +### 3. Controlling Processes with Signals +Processes communicate via **signals**, which instruct them to perform actions like stopping or restarting. + +- **Common Signals**: + - `SIGTERM` (15): Requests graceful termination (default for `kill`). + - `SIGKILL` (9): Forces immediate termination (use cautiously, as it doesn’t allow cleanup). + - `SIGHUP` (1): Reloads configuration for daemons (e.g., `kill -1 1234`). + - `SIGSTOP` (19): Pauses a process (resumed with `SIGCONT`). + - `SIGCONT` (18): Resumes a paused process. +- **Commands**: + - **`kill`**: Sends a signal to a PID. + ```bash + kill -15 1234 # Graceful stop + kill -9 1234 # Force kill + ``` + - **`killall`**: Sends a signal to all processes matching a name. + ```bash + killall -15 httpd # Stops all httpd processes + ``` + - **`pkill`**: Similar to `killall`, but supports patterns. + ```bash + pkill -u user1 # Terminates all processes for user1 + ``` + +### 4. Managing Process Priorities +CPU resources are allocated based on process priority, controlled by **nice** values (-20 highest, +19 lowest). + +- **`nice`**: + - Sets priority when starting a process. + - Example: + ```bash + nice -n 10 sleep 1000 # Runs with low priority + ``` + - Range: `-20` (high priority) to `+19` (low priority, default 0). + +- **`renice`**: + - Adjusts priority of a running process. + - Example: + ```bash + renice 5 -p 1234 # Sets nice value to 5 for PID 1234 + ``` + - Requires `root` for increasing priority (lowering nice value below 0). + +- **Verification**: + - Check nice value with `ps -l` (column `NI`) or `top` (press `f`, select `NI`). + - Example: + ```bash + ps -lp 1234 + ``` + +### 5. Job Control +Job control manages foreground and background processes within a terminal session. + +- **Foreground Processes**: + - Run interactively, locking the terminal (e.g., `vim`). +- **Background Processes**: + - Run without terminal control, freeing it for other tasks. + - Start with `&`: + ```bash + sleep 1000 & + ``` +- **Key Commands**: + - **`jobs`**: Lists background/stopped jobs. + ```bash + jobs + ``` + Output: `[1]+ Running sleep 1000 &`. + - **`fg`**: Brings a job to the foreground. + ```bash + fg %1 # Brings job 1 to foreground + ``` + - **`bg`**: Resumes a stopped job in the background. + ```bash + bg %1 + ``` + - **Suspend**: Press `Ctrl+Z` to pause a foreground process. + - **Terminate**: Use `Ctrl+C` to stop a foreground process. + +- **Example Workflow**: + ```bash + sleep 1000 # Runs in foreground + Ctrl+Z # Suspends + jobs # Shows [1]+ Stopped + bg %1 # Resumes in background + fg %1 # Brings back to foreground + Ctrl+C # Terminates + ``` + +### 6. Advanced Monitoring Tools +For deeper insights, RHEL offers additional tools. + +- **`pidstat`** (part of `sysstat` package): + - Monitors CPU, memory, or I/O per process. + - Example: + ```bash + pidstat -u 1 5 # CPU usage every second for 5 iterations + ``` +- **`lsof`**: + - Lists open files associated with a process. + - Example: + ```bash + lsof -p 1234 # Files opened by PID 1234 + ``` +- **`free`**: + - Shows system memory usage, useful alongside `top` for memory-intensive processes. + ```bash + free -h + ``` + +### 7. Process Troubleshooting +- **High CPU Usage**: + - Use `top` or `ps aux --sort=-%cpu` to identify culprits. + - Lower priority with `renice` or terminate with `kill`. +- **Zombies**: + - Find with `ps aux | grep Z`. + - Kill parent process to clean up (e.g., `kill -9 `). +- **Out-of-Memory Issues**: + - Check `free -h` and `top` (%MEM column). + - Terminate non-critical processes or adjust `nice` values. + +### 8. Systemd and Processes +- Processes like `httpd` or `sshd` are managed by `systemd` units. +- Check service processes: + ```bash + systemctl status httpd + ``` + Shows main PID and recent logs. +- Stop a service to kill its processes: + ```bash + systemctl stop httpd + ``` + +## Practical Tasks: Operation Server Rescue: Diagnose, Stabilize, and Optimize +As an SRE on call, your task is to diagnose, stabilize, and optimize a production server under heavy load from rogue processes. This guide assumes a Fedora or RHEL system. + +### Setup the Crisis + +1. **Create a Workspace**: + +```bash +mkdir ~/processlab +cd ~/processlab +``` + +2. **Simulate Server Stress**: + - **CPU-intensive script**: + +```bash +echo 'while true; do :; done' > cpu_hog.sh +chmod +x cpu_hog.sh +``` + + - **Memory-heavy script**: + +```bash +echo 'python3 -c "while True: a = [0] * 1000000"' > mem_hog.sh +chmod +x mem_hog.sh +``` + + - **Sleep script**: + +```bash +echo 'sleep 3600' > sleeper.sh +chmod +x sleeper.sh +``` + +3. **Install Tools (if needed)**: + +```bash +sudo dnf install python3 procps-ng sysstat +``` + + +### Task: Stabilize the Server + +#### Assess the Load + +1. **Check Load Averages**: +Run `uptime` to check load averages and save output to `~/processlab/load.txt`. + +```bash +uptime > ~/processlab/load.txt +``` + +2. **Note CPU Cores**: +Use `lscpu` to note the number of CPU cores and save to `~/processlab/cpu_info.txt`. + +```bash +lscpu > ~/processlab/cpu_info.txt +``` + +3. **Compare Load to Cores**: +High load (>2 on 2 cores) confirms stress. + +#### Survey Running Processes + +1. **List All Processes**: +List all processes with `ps aux`, pipe to `less` for paging, and save to `~/processlab/all_processes.txt`. + +```bash +ps aux > ~/processlab/all_processes.txt +``` + +2. **List Processes Sorted by User**: +Append the list of processes sorted by user to `~/processlab/all_processes.txt`. + +```bash +ps aux --sort=user >> ~/processlab/all_processes.txt +``` + +3. **Show Custom Columns**: +Show custom columns (PID, user, group, VSZ, RSS, command) with `ps -e -o pid,user,group,vsz,rss,comm` and save to `~/processlab/custom_processes.txt`. + +```bash +ps -e -o pid,user,group,vsz,rss,comm > ~/processlab/custom_processes.txt +``` + + +#### Launch Rogue Processes + +1. **Start Background Jobs**: + +```bash +./cpu_hog.sh & +./mem_hog.sh & +./sleeper.sh & +``` + +2. **Start a Foreground Job**: + +```bash +./cpu_hog.sh +``` + +Suspend it with `Ctrl+Z`. + +#### Manage Jobs + +1. **List Jobs**: +Run `jobs` to list all jobs (expect 3 running, 1 stopped). +2. **Resume Stopped Job**: +Resume the stopped `cpu_hog.sh` job in the background with `bg`. +3. **Bring Job to Foreground and Terminate**: +Bring the first `sleeper.sh` job to the foreground with `fg %1`, then terminate it with `Ctrl+C`. +4. **Confirm Job Removal**: +Check `jobs` to confirm `sleeper.sh` is gone. + +#### Monitor in Real-Time + +1. **Run Top**: +Run `top`, sort by CPU (P), then memory (M). +2. **Identify mem_hog.sh PID**: +Identify the `mem_hog.sh` PID (high `%MEM`). +3. **Terminate mem_hog.sh**: +In `top`, press `k`, enter the `mem_hog.sh` PID, and send `SIGTERM` (15) to stop it gracefully. +4. **Quit Top**: +Quit `top` with `q`. + +#### Adjust Priorities + +1. **Find PID of cpu_hog.sh**: +Find the PID of one `cpu_hog.sh` with `ps aux | grep cpu_hog`. +2. **Set Nice Value to +5**: + +```bash +renice 5 -p <PID> +``` + +3. **Increase Priority to -5**: + +```bash +sudo renice -5 -p <PID> +``` + +4. **Verify Priority Change**: +Verify with `ps -lp <PID>` (check NI column). Save output to `~/processlab/priority.txt`. + +```bash +ps -lp <PID> > ~/processlab/priority.txt +``` + + +#### Trace Process Hierarchy + +1. **Show Process Hierarchy**: +Run `ps fax | grep -B5 cpu_hog` to show `cpu_hog.sh` processes and their parent shell. +2. **Save Hierarchy**: +Save output to `~/processlab/hierarchy.txt`. + +```bash +ps fax | grep -B5 cpu_hog > ~/processlab/hierarchy.txt +``` + + +#### Clean Up Stragglers + +1. **Start Another Rogue Process**: +From a second terminal, start another rogue process: + +```bash +./cpu_hog.sh & +exit +``` + +2. **Confirm Process Running**: +Back in the first terminal, use `ps aux | grep cpu_hog` to confirm it’s still running. +3. **Terminate All cpu_hog.sh Processes**: + +```bash +killall -15 cpu_hog.sh +``` + +4. **Verify Termination**: +Verify with `ps aux | grep cpu_hog` (no results). + +#### Final Sweep + +1. **Check Load Again**: +Check load again with `uptime`. Append to `~/processlab/load.txt`. + +```bash +uptime >> ~/processlab/load.txt +``` + +2. **Force-Stop Remaining Processes**: +If load is still high, use `killall -9 sleeper.sh` to force-stop any remaining processes. +3. **Save Final Processes**: +Save final `ps aux` output to `~/processlab/final_processes.txt`. + +```bash +ps aux > ~/processlab/final_processes.txt +``` + + +#### Document Findings + +In `~/processlab/notes.txt`, document: + +- Which process caused the most stress (`cpu_hog.sh` or `mem_hog.sh`). +- How load changed after cleanup. +- One lesson learned (e.g., “`SIGTERM` is safer than `SIGKILL`”). + + +## **Submission Guidelines** +- Store your findings and execution steps in a markdown file (`solution.md`). +- Submit it in your GitHub repository and share the link. +- Post your experience on social media with **#getfitwithsagar #SRELife #DevOpsForAll**. +- Tag us in your posts for visibility and networking. + +--- + +## **Join Our Community** + +To make the most of this journey, connect with fellow learners: +- **Discord** – Ask questions and collaborate: [Join here](https://discord.gg/mNDm39qB8t) +- **Google Group** – Get updates and discussions: [Join here](https://groups.google.com/forum/#!forum/daily-devops-sre-challenge-series/join) +- **YouTube** – Watch solution videos and tutorials: [Subscribe here](https://www.youtube.com/@Sagar.Utekar) + +--- + +--- + +## **Stay Motivated!** + +Every challenge you complete brings you closer to mastering Git, DevOps, and SRE practices. Keep pushing your limits and collaborating with the community. + +Happy Learning! + +**Best regards,** +Sagar Utekar diff --git a/DailyChallenges/Season2/Day13/challenge.md b/DailyChallenges/Season2/Day13/challenge.md new file mode 100644 index 0000000..225f3d8 --- /dev/null +++ b/DailyChallenges/Season2/Day13/challenge.md @@ -0,0 +1,449 @@ +# Daily DevOps + SRE Challenge Series – Season 2 +## Day 13: Secure Shell Mastery – SSH, SCP, Hardening & Troubleshooting Like a Pro + +### Introduction +Welcome to **Day 13** of the Daily DevOps + SRE Challenge Series – Season 2! 🔐 + +Today, you’ll master secure remote access and file transfer with **SSH** and **SCP**. You’ll also learn to harden servers, move beyond passwords to keys, and troubleshoot failures like an SRE under pressure. SSH is the backbone of Linux administration, CI/CD deploys, remote debugging, and automated operations. +Fun fact, Google ask these questions for the Cloud Engineer role. + +By the end of today, you’ll confidently connect, transfer files safely, lock down your servers, and fix tricky login issues fast. + +#### Why This Matters? +- **Security**: SSH done wrong is an open door. Harden it to reduce risk. +- **Reliability**: Key-based access avoids password lockouts and human error. +- **Scale**: Client and server configs let you manage fleets and bastions cleanly. +- **Incident Response**: During outages, fast, correct SSH troubleshooting is vital. +- **Interview Win**: “How do you harden sshd?” and “Why key-based auth?” are common. +- **Real-World Impact**: Disabling password logins and enforcing keys can reduce SSH-related incidents dramatically. + +--- + +### 1. Understanding SSH +- **Definition**: SSH (Secure Shell) is a protocol for secure remote login, command execution, and tunneling over untrusted networks. +- **How it stays secure**: + - Host keys: Identify the server; clients cache fingerprints in `~/.ssh/known_hosts`. + - Key exchange + ciphers: Negotiate an encrypted session (confidentiality + integrity). + - Authentication: Password, public key, or other methods (e.g., MFA). +- **Common use cases**: + - Admin access: `ssh user@server` + - Remote commands: `ssh user@server "systemctl status nginx"` + - File transfers: `scp`, `sftp`, `rsync -e ssh` + - Port forwarding and tunneling +- **Ports**: Default server port is 22. Many orgs change it (e.g., 2222) and update firewalls accordingly. + +--- + +### 2. Connecting with SSH +Basic commands: +```bash +# Default port +ssh user@hostname + +# Specify identity key +ssh -i ~/.ssh/id_ed25519 user@hostname + +# Non-default port +ssh -p 2222 user@hostname + +# Run a one-off command +ssh user@hostname "uptime && whoami" +``` + +Helpful client options: +- `-vvv`: Verbose output for troubleshooting (negotiation, auth) +- `-o StrictHostKeyChecking=ask`: Safer first-connection behavior +- `-J bastion`: ProxyJump via bastion (multi-hop) +- ControlMaster/ControlPersist: Multiplex connections for faster repeated SSH + +--- + +### 3. Secure File Transfers: SCP, SFTP, and rsync over SSH +SCP and SFTP both run over SSH (encrypted end-to-end). Modern OpenSSH may implement `scp` via SFTP. + +SCP examples: +```bash +# Upload a file +scp ./app.tar.gz user@server:/opt/releases/ + +# Download a file +scp user@server:/var/log/nginx/access.log ./logs/ + +# Recursive copy (directory) +scp -r ./static/ user@server:/var/www/static/ + +# Non-default port +scp -P 2222 ./file.txt user@server:/tmp/ +``` + +SFTP (interactive or batch): +```bash +sftp -P 2222 -i ~/.ssh/id_ed25519 user@server +sftp> put file.txt /opt/data/ +sftp> get /opt/data/file.txt +``` + +rsync over SSH (efficient sync with deltas): +```bash +rsync -az --progress -e "ssh -p 2222 -i ~/.ssh/id_ed25519" ./site/ user@server:/var/www/site/ +``` + +Integrity verification: +```bash +sha256sum file.txt +ssh user@server "sha256sum /opt/data/file.txt" +``` + +--- + +### 4. How SSH Works (Mermaid Diagram) +```mermaid +sequenceDiagram + participant Client + participant Server + + Client->>Server: Initiates TCP connection on port 22 + Server->>Client: Sends SSH version and server public key + Client->>Server: Negotiates encryption + session key + Client->>Server: Sends authentication request (password or key) + alt Password Auth + Client->>Server: Sends username + password + Server->>Server: Verifies credentials + else Public Key Auth + Client->>Server: Proves possession of private key (signs challenge) + Server->>Server: Verifies using authorized_keys + end + Server-->>Client: Authentication success/failure + Client->>Server: Opens shell session or executes command +``` + +--- + +### 5. How SCP Works (Mermaid Diagram) +```mermaid +sequenceDiagram + participant Client + participant Server + + Client->>Server: Establish SSH connection (port 22) + Client->>Server: Sends SCP command (upload/download) + Server->>Client: Responds with SCP process + Client->>Server: Transfers file chunks (encrypted) + Server->>Client: Sends ACK after completion + Note over Client,Server: Entire session is encrypted with SSH +``` + +--- + +### 6. sshd_config Essentials (Server) +Key directives in `/etc/ssh/sshd_config`: +- `Port 2222`: Change from 22 (remember to open firewall) +- `PermitRootLogin no`: Disable direct root login +- `PasswordAuthentication no`: Enforce key-based auth +- `PubkeyAuthentication yes`: Enable key auth +- `MaxAuthTries 3`: Throttle brute-force attempts +- `PermitEmptyPasswords no`: Disallow empty passwords +- `AllowUsers devops_user`: Restrict to allowed accounts (or `AllowGroups`) +- `ClientAliveInterval 300` + `ClientAliveCountMax 2`: Idle session control +- `LogLevel VERBOSE`: Better audit logs +- Optionally harden crypto (modern OpenSSH defaults are strong): + - `KexAlgorithms curve25519-sha256,curve25519-sha256@libssh.org` + - `Ciphers chacha20-poly1305@openssh.com,aes256-gcm@openssh.com` + - `MACs hmac-sha2-512-etm@openssh.com,hmac-sha2-256-etm@openssh.com` + +Validate and apply: +```bash +sudo sshd -t # syntax check +sudo sshd -T | sort # effective config +sudo systemctl reload sshd # safer than restart +``` + +--- + +### 7. Passwordless Access & Key Management +Generate a key pair (ed25519 recommended): +```bash +ssh-keygen -t ed25519 -C "you@example.com" +``` + +Install your public key: +```bash +ssh-copy-id user@server +# or manual: +cat ~/.ssh/id_ed25519.pub | ssh user@server 'mkdir -p ~/.ssh && chmod 700 ~/.ssh && cat >> ~/.ssh/authorized_keys && chmod 600 ~/.ssh/authorized_keys' +``` + +Login: +```bash +ssh user@server +``` + +Permissions (avoid auth failures): +```bash +chmod 700 ~/.ssh +chmod 600 ~/.ssh/id_ed25519 ~/.ssh/authorized_keys +``` + +Use `ssh-agent` to cache decrypted keys: +```bash +eval "$(ssh-agent -s)" +ssh-add ~/.ssh/id_ed25519 +``` + +Tips: +- Use `IdentitiesOnly yes` per host to avoid “Too many authentication failures” +- Always protect private keys with a passphrase and store backups securely + +--- + +### 8. Server Hardening Simulation +🔴 Buggy server config (for lab): +- `PermitRootLogin yes` +- `PasswordAuthentication yes` +- Weak password + +⚡ Exploit demonstration (lab-only): +```bash +hydra -l root -P rockyou.txt ssh://server-ip +``` + +🟢 Hardened fix: +- `PermitRootLogin no` +- `PasswordAuthentication no` +- `PubkeyAuthentication yes` +- `AllowUsers devops_user` +- Change port (e.g., 2222) & update firewall rules + +UFW example: +```bash +sudo ufw allow 2222/tcp +sudo ufw delete allow 22/tcp +sudo ufw reload +``` + +firewalld example: +```bash +sudo firewall-cmd --add-port=2222/tcp --permanent +sudo firewall-cmd --remove-service=ssh --permanent +sudo firewall-cmd --reload +``` + +--- + +### 9. Troubleshooting – Google-Style +1) Client-Side +- Correct command: `ssh -i key.pem -p 2222 user@ip` +- Debug with `ssh -vvv` +- Key file permissions: +```bash +chmod 700 ~/.ssh +chmod 600 ~/.ssh/id_* ~/.ssh/authorized_keys +``` + +2) Network +- Reachability and port: +```bash +ping -c 2 ip || true +nc -zv ip 2222 +``` +- Firewall/security group rules (local + cloud) + +3) Server +- Service and port: +```bash +sudo systemctl status sshd +sudo ss -tulpn | grep -E ':(22|2222)\s' +``` +- Config validation: +```bash +sudo sshd -t +sudo sshd -T | sort +``` + +4) Authentication +- User exists and has valid shell +- `~/.ssh` ownership and modes; `authorized_keys` present +- `AllowUsers/AllowGroups` not blocking login + +5) Logs +- Debian/Ubuntu: `/var/log/auth.log` +- RHEL/CentOS/Fedora: `/var/log/secure` +- Journal: `sudo journalctl -u sshd -e` + +6) Cloud-Specific +- AWS: Default usernames (`ec2-user`, `ubuntu`, etc.), SG inbound 2222/tcp, PEM permissions +- GCP: OS Login/IAM bindings; project-level keys +- Azure: NSGs; Just-In-Time access + +7) Edge Cases +- Fail2ban blocking IP (unban via `fail2ban-client`) +- Host key mismatch (fix `~/.ssh/known_hosts`) +- SELinux enforcing (check AVC denials; `restorecon -Rv /etc/ssh` after changes) + +--- + +### 10. Troubleshooting Flow (Mermaid Diagram) +```mermaid +flowchart TD + A[SSH Fails] --> B[Check Client Command] + B -->|Wrong user/IP/key| FixClient + B --> C[Enable ssh -vvv debug] + + C --> D[Network Reachable?] + D -->|No| FixNetwork + D -->|Yes| E[Is Port Open?] + + E -->|No| FixFirewall + E -->|Yes| F[Is sshd Running?] + + F -->|No| RestartService + F -->|Yes| G[Check sshd_config] + + G --> H[Auth Issues?] + H -->|Key mismatch| FixKeys + H -->|User restricted| UpdateAllowUsers + + G --> I[Check Logs] + I -->|Errors Found| FixConfig + I -->|Cloud Issue| CheckIAM + + I --> J[Still Broken?] + J --> Escalate +``` + +--- + +## Practical Tasks: Operation Secure Access – Harden, Validate, and Recover +As an SRE, your job is to enable secure, reliable access while minimizing risk. This lab uses Ubuntu/Debian or RHEL/Fedora. Keep a second console open before changing SSH. + +### Setup Prep +1. Create a workspace: +```bash +mkdir -p ~/ssh-lab && cd ~/ssh-lab +``` +2. Confirm current connectivity: +```bash +ssh user@server "whoami && hostname" +``` + +### Task A: Establish Key-Based Login +1. Generate key (if needed): +```bash +ssh-keygen -t ed25519 -C "you@example.com" +``` +2. Install your public key: +```bash +ssh-copy-id user@server +``` +3. Verify: +```bash +ssh -vvv user@server "echo OK && id -u && hostname" +``` +4. Save the line showing “Authentication succeeded (publickey)” to `notes.md`. + +### Task B: Harden sshd +1. Open firewall for new port (2222 shown below). +2. Edit `/etc/ssh/sshd_config` to include: +``` +Port 2222 +PermitRootLogin no +PasswordAuthentication no +PubkeyAuthentication yes +AllowUsers user +MaxAuthTries 3 +PermitEmptyPasswords no +ClientAliveInterval 300 +ClientAliveCountMax 2 +LogLevel VERBOSE +``` +3. Validate and reload: +```bash +sudo sshd -t && sudo systemctl reload sshd +``` +4. Test new port: +```bash +ssh -p 2222 user@server "echo PORT_OK" +``` +5. Remove old port (only after verifying new): +- UFW: `sudo ufw delete allow 22/tcp` +- firewalld: `sudo firewall-cmd --remove-service=ssh --permanent && sudo firewall-cmd --reload` + +Document all commands + outputs in `notes.md`. + +### Task C: Secure File Transfer Roundtrip +1. Create a file and upload/download: +```bash +echo "hello-ssh" > hello.txt +scp -P 2222 hello.txt user@server:/tmp/hello.txt +scp -P 2222 user@server:/tmp/hello.txt hello.remote.txt +``` +2. Verify integrity: +```bash +sha256sum hello.txt hello.remote.txt | tee checksums.txt +``` + +### Task D (Optional): SFTP-Only Restricted User +1. Create group and user (server): +```bash +sudo groupadd -f sftpusers +sudo useradd -m -G sftpusers -s /usr/sbin/nologin sftpuser +``` +2. Prepare chroot: +```bash +sudo mkdir -p /sftp/sftpuser/upload +sudo chown root:root /sftp /sftp/sftpuser +sudo chmod 755 /sftp /sftp/sftpuser +sudo chown sftpuser:sftpusers /sftp/sftpuser/upload +``` +3. Add to `/etc/ssh/sshd_config`: +``` +Subsystem sftp internal-sftp +Match Group sftpusers + ChrootDirectory /sftp/%u + ForceCommand internal-sftp + AllowTCPForwarding no + X11Forwarding no +``` +4. Validate + reload, then test: +```bash +sudo sshd -t && sudo systemctl reload sshd +ssh -p 2222 sftpuser@server # should fail (no shell) +sftp -P 2222 sftpuser@server # should succeed +``` + +### Task E: Induce and Fix Failures (Troubleshooting) +Pick at least one, capture `ssh -vvv` and server logs, and document your fix: +- Wrong key permissions: fix with `chmod 600 ~/.ssh/id_*` +- Blocked port: fix firewall rules and retest with `nc -zv` +- Misconfigured `AllowUsers`: correct user and reload +- Host key mismatch: remove offending line in `~/.ssh/known_hosts` + +--- + +## Submission Guidelines +- Store your findings and execution steps in a markdown file (`solution.md`). +- Include: + - Final `/etc/ssh/sshd_config` + - Commands and outputs for Tasks A–C (and D/E if attempted) + - Notes from troubleshooting (`ssh -vvv` snippets and relevant server logs) +- Submit it in your GitHub repository and share the link. +- Post your experience on social media with **#getfitwithsagar #SRELife #DevOpsForAll**. +- Tag us in your posts for visibility and networking. + +--- + +## Join Our Community +To make the most of this journey, connect with fellow learners: +- **Discord** – Ask questions and collaborate: https://discord.gg/mNDm39qB8t +- **Google Group** – Get updates and discussions: https://groups.google.com/forum/#!forum/daily-devops-sre-challenge-series/join +- **YouTube** – Watch solution videos and tutorials: https://www.youtube.com/@Sagar.Utekar + +--- + +## Stay Motivated! +Every challenge you complete brings you closer to mastering Git, DevOps, and SRE practices. Keep pushing your limits and collaborating with the community. + +Happy Learning! + +**Best regards,** +Sagar Utekar diff --git a/DailyChallenges/Season2/Day14/challenge.md b/DailyChallenges/Season2/Day14/challenge.md new file mode 100644 index 0000000..418327f --- /dev/null +++ b/DailyChallenges/Season2/Day14/challenge.md @@ -0,0 +1,228 @@ +# Daily DevOps + SRE Challenge Series – Season 2 +## Day 14: Linux Users, Groups, and Permissions – Essentials for DevOps/SRE + +### Introduction +Welcome to Day 14 of the Daily DevOps + SRE Challenge Series – Season 2! + +Today you’ll learn the core skills for managing users, groups, and file permissions in Linux. We’ll stay focused on the 20% that covers 80% of real DevOps/SRE work: creating users, adding them to groups, fixing “Permission denied” issues, and setting up safe shared directories for teams. + +By the end of this challenge, you’ll be able to: +- Create and manage users and groups safely +- Read and change permissions with confidence +- Fix ownership/permission issues that break apps and deployments +- Build a team directory that “just works” for collaboration + +Why this matters +- Security: Avoid overexposed files and risky defaults +- Reliability: Services run only when ownership/modes are correct +- Collaboration: The right group setup saves time and tickets +- Troubleshooting: “Permission denied” is one of the most common production errors + +--- + +### 1) The Essentials You’ll Use Daily + +Users and groups (who can do what) +- Users have a username, UID, home, and shell +- Groups let multiple users share access without changing file owners + +Quick commands +```bash +# Users +sudo useradd -m -s /bin/bash riya +sudo passwd riya +id riya +groups riya + +# Groups +sudo groupadd devops +sudo usermod -aG devops riya # append (-a) to Group (-G) +getent group devops +``` + +Ownership (who owns a file/dir) +```bash +sudo chown user:group path +sudo chown -R www-data:www-data /var/www/html +``` + +Permissions (what actions are allowed) +- r (read), w (write), x (execute/traverse) +- Apply to: owner (u), group (g), others (o) +- Numeric: r=4, w=2, x=1 → 640 = u=rw, g=r, o=- + +Common modes you’ll set +```bash +chmod 640 file # configs readable by app group, not world +chmod 750 dir # app can enter/execute; group can traverse; others blocked +chmod 755 dir # world-readable directory (no write) +``` + +Team-friendly directories +- setgid (2) on a directory makes new files inherit the directory’s group +```bash +sudo chmod 2775 /projects/teamspace +``` + +Sticky bit (shared scratch areas like /tmp) +```bash +sudo chmod 1777 /shared # users can’t delete others’ files +``` + +Tip: For most team folders, set owner to root:team, apply 2770/2775, and add members to the team group. + +--- + +### 2) Two Short Mermaid Diagrams + +Permission evaluation (simple mental model) +```mermaid +flowchart TD + A[Request access to path] --> B{Is caller root?} + B -->|Yes| Z[Allow] + B -->|No| C{Is caller the file owner?} + C -->|Yes| D[Check owner bits] + C -->|No| E{Is caller in the file's group?} + E -->|Yes| F[Check group bits] + E -->|No| G[Check others bits] + D --> H{Bits allow action?} + F --> H + G --> H + H -->|Yes| Z + H -->|No| Y[Denied] +``` + +umask (why new files default to 644 and dirs to 755) +```mermaid +flowchart TD + A[Create] --> B{Type?} + B -->|File| C[Base 666] + B -->|Directory| D[Base 777] + C --> E[Subtract umask → mode] + D --> E[Subtract umask → mode] + E --> F[Final permissions] +``` + +--- + +### 3) Practical Scenarios (Do These) + +Scenario A: Create users and a devops group +```bash +# Users +for u in riya amit neha; do sudo useradd -m -s /bin/bash "$u" || true; done +# Group +sudo groupadd devops || true +# Add users to group +for u in riya amit neha; do sudo usermod -aG devops "$u"; done +# Verify +for u in riya amit neha; do id "$u"; done +``` + +Scenario B: Team workspace that “just works” +Goal: Everyone in devops can create/edit files in /projects/infra; others blocked. +```bash +sudo mkdir -p /projects/infra +sudo chown root:devops /projects/infra +sudo chmod 2770 /projects/infra # setgid ensures new files use group devops +# Test (from different users): create files and edit each other’s files +``` + +Scenario C: Fix a broken web app folder +Symptom: Nginx can’t read /var/www/html (403/404). +```bash +# Debian/Ubuntu typical web user +sudo chown -R www-data:www-data /var/www/html +sudo find /var/www/html -type d -exec chmod 755 {} \; +sudo find /var/www/html -type f -exec chmod 644 {} \; +``` + +Scenario D: Quick account hygiene +```bash +# Lock an unused account +sudo usermod -L guest +# Set password policy for riya +sudo chage -M 90 -m 7 -W 14 riya +# See current policy +chage -l riya +``` + +Scenario E: Troubleshoot “Permission denied” +Checklist: +- ls -ld path; stat file +- namei -l /deep/nested/path (shows each directory’s x bit) +- id user (group membership) +- If it’s a team dir, ensure setgid and group membership are correct +Commands: +```bash +namei -l /srv/app/config/app.env +id appsvc +ls -ld /srv /srv/app /srv/app/config +``` + +Optional (nice to know, not required today): +- Default ACLs for always-on group write in team dirs: +```bash +sudo apt-get install -y acl 2>/dev/null || true +sudo setfacl -m g:devops:rwx /projects/infra +sudo setfacl -d -m g:devops:rwx /projects/infra +getfacl /projects/infra +``` + +--- + +## Practical Tasks: Build, Share, Fix + +Do these and capture the commands and outputs. + +Task 1: Users and group +- Create users riya, amit, neha +- Create devops group and add the users +- Save id outputs to ~/perm-lab/users.txt + +Task 2: Team directory +- Create /projects/infra as root:devops with 2770 +- Show that files created by any member are editable by others +- Save ls -l and a short note to ~/perm-lab/teamspace.md + +Task 3: Fix permissions +- Simulate a broken app dir /opt/app owned by root:root with mode 600 +- Create appsvc user (no password needed) +- Fix to appsvc:appsvc, dirs 750, files 640 +- Prove appsvc can read configs; others cannot +- Save commands and stat outputs to ~/perm-lab/appfix.md + +Task 4: One troubleshooting case +- Break something intentionally (e.g., remove x from a parent directory) +- Use namei -l to find the missing execute bit and fix it +- Save before/after to ~/perm-lab/troubleshoot.md + +Deliverables +- A markdown file solution.md with: + - Commands and outputs for Tasks 1–4 + - 3–5 bullet notes on what you learned + +--- + +## Submission Guidelines +- Store your findings and execution steps in solution.md +- Submit it in your GitHub repository and share the link +- Post your experience on social media with #getfitwithsagar #SRELife #DevOpsForAll +- Tag us in your posts for visibility and networking + +--- + +## Join Our Community +- Discord – Ask questions and collaborate: https://discord.gg/mNDm39qB8t +- Google Group – Get updates and discussions: https://groups.google.com/forum/#!forum/daily-devops-sre-challenge-series/join +- YouTube – Watch solution videos and tutorials: https://www.youtube.com/@Sagar.Utekar + +--- + +## Stay Motivated! +Keep it simple. Master the basics. You’ll use these commands every week in real DevOps/SRE work. + +Happy Learning! + +Best regards, +Sagar Utekar diff --git a/DailyChallenges/Season2/Day15/challenge.md b/DailyChallenges/Season2/Day15/challenge.md new file mode 100644 index 0000000..0cb67e2 --- /dev/null +++ b/DailyChallenges/Season2/Day15/challenge.md @@ -0,0 +1,150 @@ +# Daily DevOps + SRE Challenge Series – Season 2 +## Day 15: OSI & TCP/IP Model Foundations for DevOps/SRE + +### Introduction +Welcome to Day 15 of the Daily DevOps + SRE Challenge Series - Season 2! + +Today marks the start of your Networking for DevOps journey. Before diving into commands and configs, you'll build a strong mental model of how networking actually works layer by layer. Understanding the OSI and TCP/IP models will make you a faster troubleshooter, a better cloud/network engineer, and help you communicate clearly with your team. + +By the end of today, you'll be able to: +- Draw and describe both the OSI and TCP/IP models +- Map common protocols and tools (ping, SSH, HTTP, DNS, etc.) to the right layer +- Explain where most real-world DevOps/SRE problems occur +- Use this model to guide troubleshooting (“where is it broken?”) + +Why this matters: +- Troubleshooting: Pinpoint failures (e.g., “Is it a firewall or DNS issue?”) +- Communication: Clearly explain problems to network, security, and cloud teams +- Design: Choose the right tool or protocol for any scenario +- Interviews: “At what layer does X work?” is a common question + +--- + +### 1) The OSI & TCP/IP Models + +#### OSI Model (7 Layers) and Data Formats + +| Layer | Number | Example Protocols | Example Tools | Data Unit (PDU) | +|---------------|--------|----------------------------|--------------------|---------------------------| +| Application | 7 | HTTP, SSH, DNS, SMTP | curl, ssh, dig | Data / Message | +| Presentation | 6 | SSL/TLS, ASCII, JPEG | openssl | Data | +| Session | 5 | NetBIOS, RPC, TLS session | | Data | +| Transport | 4 | TCP, UDP | nc, ss, netstat | Segment (TCP) / Datagram (UDP) | +| Network | 3 | IP, ICMP | ip, ping, traceroute | Packet | +| Data Link | 2 | Ethernet, ARP, MAC | ip link, arp | Frame | +| Physical | 1 | Cables, Wi-Fi, fiber | ethtool, ip link | Bit | + +#### TCP/IP Model (4 Layers) + +| Layer | OSI Layer(s) | Example Protocols/Tools | +|---------------|--------------------------|------------------------------| +| Application | 7, 6, 5 | HTTP, SSH, DNS, TLS, curl | +| Transport | 4 | TCP, UDP, nc, ss | +| Internet | 3 | IP, ICMP, ping, traceroute | +| Network Access| 2, 1 | Ethernet, ARP, ip link, ethtool| + +--- + +### 2) Visual Diagrams + +#### OSI Model Overview +```mermaid +flowchart TD + L7[Application : HTTP, SSH, DNS] + L6[Presentation : TLS, SSL, Encoding] + L5[Session : TLS session, NetBIOS] + L4[Transport : TCP, UDP] + L3[Network : IP, ICMP] + L2[Data Link : Ethernet, ARP] + L1[Physical : Cable, Wi-Fi] + L7 --> L6 --> L5 --> L4 --> L3 --> L2 --> L1 +``` + +#### TCP/IP Model & Protocol Mapping +```mermaid +flowchart TD + App[Application: HTTP, SSH, DNS, SMTP, curl] + Trans[Transport: TCP, UDP, nc, ss] + Net[Internet: IP, ICMP, ping, traceroute] + Link[Network Access: Ethernet, ARP, ip link] + App --> Trans --> Net --> Link +``` + +--- + +### 3) Hands-On Challenge + +#### Scenario A: Draw & Map the Layers + +- Draw both models (on paper or using a tool) +- List at least 2 protocols and 2 tools for each layer (see tables above) +- Save a photo/image/markdown table in your solution + +#### Scenario B: Protocol Mapping Exercise + +Given these tools/protocols, map each to its layer(s): +- ping +- ssh +- dig +- curl +- nc +- ip addr +- arp +- openssl +- traceroute +- tcpdump + +Fill out a table like: +| Tool/Protocol | OSI Layer(s) | TCP/IP Layer | Description | +|---------------|--------------|--------------|-------------| +| ping | 3, 1 | Internet | Tests IP reachability using ICMP | +| ssh | 7, 4 | Application, Transport | Secure shell (needs network and transport) | +| ... | ... | ... | ... | + +#### Scenario C: Mini Incident Simulation + +A developer says: +> “I can’t reach the app at http://10.10.10.20:5000 from my laptop. Ping fails, but DNS resolves.” + +Your task: +1. Identify which layer(s) might be failing based on symptoms +2. List which commands/tools you would use in order to troubleshoot—from bottom to top + +Document your answer in solution.md. + +--- + +### 4) What to Watch For (Common Gotchas) +- Not every protocol is at only one layer (e.g., DNS = Application, but depends on Transport/Network) +- Some tools work at multiple layers (e.g., tcpdump sees almost everything!) +- Real-world cloud infra may “hide” some layers (e.g., you don’t see cables, but you still check link status in VMs) +- The PDU (data format) matters for troubleshooting (e.g., "packet loss" is different from "frame error") + +--- + +## Submission Guidelines +- Create `solution.md` including: + - Your diagrams/tables for OSI & TCP/IP models + - Completed protocol mapping table + - Your troubleshooting approach for the mini incident + - 3–5 short bullet points on what you learned + +- Push to your GitHub repo and share the link +- Post your experience with #getfitwithsagar #SRELife #DevOpsForAll and tag us + +--- + +## Join Our Community +- Discord – Ask questions and collaborate: https://discord.gg/mNDm39qB8t +- Google Group – Updates and discussions: https://groups.google.com/forum/#!forum/daily-devops-sre-challenge-series/join +- YouTube – Tutorials: https://www.youtube.com/@Sagar.Utekar + +--- + +## Stay Motivated! +A solid mental model makes every future network problem easier. Build your foundation today! + +Happy Learning! + +Best regards, +Sagar Utekar diff --git a/DailyChallenges/Season2/Day16/challenge.md b/DailyChallenges/Season2/Day16/challenge.md new file mode 100644 index 0000000..2a513bc --- /dev/null +++ b/DailyChallenges/Season2/Day16/challenge.md @@ -0,0 +1,253 @@ +# Day 16: OSI Layer 1 & 2 – Physical and Data Link – Interfaces, MAC, ARP, VLAN + +## "Getting Connected: Cables, Cards, and Local Communication" + +--- + +## Why This Matters + +Before you can ping, curl, or SSH, your device must be physically and logically present on the network. Whether you’re on a real laptop, a cloud VM, or a VirtualBox guest, understanding these basics will help you fix some of the trickiest “it just doesn’t work” problems in DevOps and SRE. + +--- + +## 1. Concepts in Simple Words + +### Layer 1: Physical Layer +- **What is it?** The wires, Wi-Fi, or virtual “cables” that transmit bits (0s/1s). +- **On Cloud/VMs:** This is simulated—your “virtual cable” can be unplugged/plugged via settings or commands. +- **Problems:** Cable unplugged, “interface down”, VM not attached to network. + +### Layer 2: Data Link Layer +- **What is it?** Local communication (same network/segment). + Every device has a **MAC address**—its unique hardware “name tag.” +- **How do devices discover each other?** + Using ARP (Address Resolution Protocol): + "Who has IP X? Tell me your MAC address!" +- **VLANs:** Like colored wristbands at a party—devices with the same color (VLAN tag) can talk. + +--- + +## 1a. Devices at Layers 1 & 2 – What Hardware (and Virtual Devices) Live Here? + +| Layer | Real-World Devices | Virtual/Cloud Equivalents | What They Do | +|--------|-------------------------------|---------------------------------------------|----------------------------------------------------------| +| 1 | Cables (Ethernet, Fiber), | Virtual cables (VM adapters), | Transmit bits (0s/1s) as electrical or optical signals | +| | Hubs, Network Interface Cards | Virtual NICs, “Attach Network” in settings | or Wi-Fi radio signals. | +| | (NIC), Repeaters | | | +| 2 | Switches, Network Interface | vSwitch, Linux bridge, | Forward frames based on MAC addresses, | +| | Cards (NIC), Bridges | cloud “virtual switches” | manage local network traffic, segment VLANs | + +**Explanation:** +- **Layer 1 (Physical):** + - *Devices:* Cables, connectors, network cards, hubs, repeaters. + - *Virtual:* VM “network cable” or “adapter”, virtual NIC. +- **Layer 2 (Data Link):** + - *Devices:* Network switches, bridges, NICs (MAC logic), wireless access points. + - *Virtual:* Virtual switches (e.g., VirtualBox Host-Only or Bridged Adapter), Linux bridges, cloud VPC “switches”. + +**In the Cloud/VMs:** +- You don’t see the actual switch or cable, but you configure a “virtual” version (attach/detach network, set up vSwitch, assign MAC). +- Cloud providers use massive, invisible Layer 2/3 switches to connect your VMs—sometimes ARP and MAC issues still appear! + +--- + +## 1b. Where Are These Layers in Docker, Kubernetes, and the Cloud? + +Even though you rarely see real cables, switches, or MAC addresses in containers or the cloud, **Layer 1 & 2 concepts still exist**—they’re just handled for you by the platform. + +- **Docker:** + Each container gets its own virtual network interface (veth), a virtual MAC address, and is connected to a virtual bridge (like `docker0`). Docker manages these “cables” and “switches” behind the scenes, but issues like duplicate MACs, interface down, or ARP failures can still break connectivity between containers. + +- **Kubernetes:** + Every Pod has a network namespace with its own virtual interfaces and MAC addresses. The CNI (Container Network Interface) plugin creates and manages these, connecting pods via virtual switches or bridges. If something breaks at Layer 2 (e.g., pod’s veth deleted, wrong bridge config), Pods can’t talk—even if everything “looks fine” at higher layers. + +- **Cloud (AWS/GCP/Azure):** + Your VM’s “NIC” is virtual, but it has a MAC address and a link state (“attached,” “detached,” “UP,” “DOWN”). Cloud providers connect your VM to huge, invisible Layer 2/3 networks. Problems like “interface down,” “wrong MAC,” or ARP not resolving can still cause outages. + +**Why care?** +- Many real-world outages—even in the cloud or containers—are caused by issues at these “hidden” layers! +- Knowing how to check and troubleshoot Layer 1/2 is a real SRE/DevOps superpower. + +--- + +## 2. Data Format at Each Layer + +| Layer | Data Unit | Example | +|-------------------|-----------|------------------------| +| Physical (L1) | Bit | 0, 1 (electrical/light)| +| Data Link (L2) | Frame | Ethernet frame [Dst MAC][Src MAC][Type][Data][FCS] | + +--- + +## 2a. Important Layer 1 & 2 Commands (Linux) + +| Purpose | Command Example | What It Shows/Does | +|-----------------------------------|-------------------------------------------------------|----------------------------------------| +| List all interfaces | `ip link show` | All network interfaces & MAC addresses | +| Show IP addresses & details | `ip addr show` | Interface IPs, MAC, status | +| Bring interface up/down | `sudo ip link set eth0 up` / `sudo ip link set eth0 down` | Enable/disable network interface | +| Show interface statistics | `ip -s link show eth0` | RX/TX stats, errors, dropped packets | +| Check ARP/neighbor cache | `ip neigh show` | IP-to-MAC mappings (ARP table) | +| Show legacy ARP table | `arp -a` | (Sometimes more readable) | +| Flush ARP/neighbor cache | `sudo ip neigh flush all` | Forces ARP to be rebuilt | +| Query ARP for specific IP | `arping ` | Sends ARP request for IP | +| Show/modify MAC address | `ip link set eth0 address aa:bb:cc:dd:ee:ff` | Change interface MAC (careful!) | +| Check link/cable status | `ethtool eth0` | Physical "link detected" etc. | +| Show VLAN interfaces | `ip -d link show` | Shows VLAN info if present | +| Create VLAN interface (if supported) | `sudo ip link add link eth0 name eth0.10 type vlan id 10` | Adds VLAN 10 on eth0 | +| | `sudo ip link set eth0.10 up` | Bring VLAN interface up | + +**Cloud/VM notes:** +- Some commands (like `ethtool`) may not work on all VM types. +- For VirtualBox: "Cable Connected" can be toggled in VM settings (simulates unplug/plug). +- In the cloud, use console/CLI to check NIC status, MAC, and attachment. + +--- + +## 3. Visuals & Analogies + +- **Physical Layer:** The network cable (or “Connect Network Adapter” option in VirtualBox/cloud UI). +- **Data Link Layer:** The “name tag” (MAC) and waving to others to find out who’s who (ARP). +- **VLAN:** Like separate groups at a party wearing colored wristbands. + +```mermaid +flowchart LR + Start[Start of Frame] --> DstMAC[Destination MAC] + DstMAC --> SrcMAC[Source MAC] + SrcMAC --> Type[Type e.g., IPv4] + Type --> Data[Data :Payload] + Data --> FCS[FCS – Error Check] + FCS --> End[End of Frame] +``` + +--- + +## 4. Guided Hands-On: Local, Cloud, and VirtualBox + +### A. Local or VM Interface Basics + +1. **List all interfaces & MAC addresses** + ```bash + ip link show + ``` + - Find your primary network interface and note its MAC. + +2. **Is your interface up?** + ```bash + ip link show + ``` + - If not, bring it up: + ```bash + sudo ip link set up + ``` + +3. **Check statistics** + ```bash + ip -s link show + ``` + - Look for dropped or error packets. + +--- + +### B. Cloud Scenario: AWS/GCP/Azure + +**Scenario:** +You have two cloud VMs in the same subnet. They cannot ping each other. +- Both have IPs in the correct range and interfaces marked UP. +- Both have unique MAC addresses. +- ARP tables on both show the other’s IP as “INCOMPLETE”. + +**Questions:** +1. What might cause this in cloud environments? (Hint: Security Groups, NACLs, subnet config, ENI not attached) +2. What console/CLI checks should you try? +3. What Linux commands confirm interface/MAC status? + +--- + +### C. VirtualBox Scenario + +**Scenario:** +You clone a VM in VirtualBox, but networking is broken: +- `ip addr` shows an IP +- `ip link show` shows interface DOWN, or both VMs have the same MAC + +**Questions:** +1. What Layer 1/2 issues can happen when cloning a VM? +2. How can you fix these in VirtualBox settings and on the VM? +3. What commands would you use to verify? + +--- + +### D. VLANs – (Optional/Conceptual) + +> **Note:** +> True VLAN separation isn’t possible in plain VirtualBox setups without advanced config or special hardware/software switches. +> You can create VLAN interfaces in Linux for practice, but isolation won’t occur. + +- **Task:** Draw a diagram: Two VLANs (10 and 20) on the same switch—who can talk to whom? +- Try to create a VLAN interface on a Linux VM (if supported): + ```bash + sudo ip link add link name .10 type vlan id 10 + sudo ip link set .10 up + ip link show + ``` + +--- + +## 5. ARP Log Analysis: Mini Incident Simulation + +You receive these logs from a VM with network issues: + +``` +$ ip link show eth0 +2: eth0: mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000 + link/ether 52:54:00:ab:cd:ef brd ff:ff:ff:ff:ff:ff + +$ ip addr show eth0 + inet 10.10.10.5/24 brd 10.10.10.255 scope global eth0 + +$ ip neigh show +10.10.10.1 dev eth0 INCOMPLETE +``` + +**Your tasks:** +1. What does the ARP entry “INCOMPLETE” mean? +2. List two possible causes for this on physical, VirtualBox, or cloud VMs. +3. What troubleshooting steps would you take to fix it, layer by layer? + +--- + +## 6. Gotchas & Security Notes + +- Two VMs/devices with the same MAC = unpredictable network issues! +- ARP spoofing: Pretending to be another device’s MAC/IP can redirect traffic-beware! +- In the cloud, even “virtual cables” can be unplugged (e.g., detached NIC, disabled interface). + +--- + +## 7. Submission Guidelines + +- Create `solution.md` and include: + - Outputs from your interface and MAC exploration + - Your answers to **one real-world scenario** (cloud or VirtualBox) + - Your analysis and fix for the ARP mini-incident (log analysis) + - 3–5 “what I learned” bullet points +- Push to your GitHub repo and share the link +- Post with #getfitwithsagar #SRELife #DevOpsForAll + +--- + +## 8. Community & Support + +- [Discord](https://discord.gg/mNDm39qB8t) +- [Google Group](https://groups.google.com/forum/#!forum/daily-devops-sre-challenge-series/join) +- [YouTube](https://www.youtube.com/@Sagar.Utekar) + +--- + +## Remember! +Start troubleshooting from the bottom: Is the “cable” plugged in? Is the interface up? Is the MAC unique? Can ARP resolve? +Master these, and you’re ready for network success anywhere—cloud, VM, or on-prem! + +Happy learning, +Sagar Utekar diff --git a/DailyChallenges/Season2/Day17/challenge.md b/DailyChallenges/Season2/Day17/challenge.md new file mode 100644 index 0000000..523354d --- /dev/null +++ b/DailyChallenges/Season2/Day17/challenge.md @@ -0,0 +1,270 @@ +# **Day 17: OSI Layer 3 – The Network Layer (IP, Subnets, Routing)** + +## **The Internetworking Layer: From Local to Global** + +--- + +## **Why This Matters More Than Ever** + +You've mastered local talk (L2) and physical signals (L1). But true internet magic happens when a request from your laptop in Delhi finds a server in São Paulo, traversing dozens of unknown networks within milliseconds. + +**Layer 3 makes this possible.** It's the post office of the digital world. As an SRE or DevOps engineer, you don't just use this layer—you **design, secure, and troubleshoot** it. Every VPC, firewall rule, Kubernetes network policy, and connectivity ticket ultimately boils down to Layer 3 concepts. + +> **"It's never DNS... until it is. It's never networking... until it is. And then it's everything."** + +--- + +## 1. **Core Concepts Demystified** + +### 1.1 **The IP Address: Your Digital Passport** +- **What it is:** A logical, hierarchical address assigned to a network interface. Unlike a MAC address (permanent, factory-assigned), an IP address is logical and changes with network location. +- **The Hierarchy:** An IP has a **network portion** (like a zip code) and a **host portion** (like a street address). Routers use the network portion to make forwarding decisions. +- **IPv4 vs. IPv6:** + - **IPv4** (e.g. `192.168.1.10`): 32-bit, ~4.3B addresses, running out, kept alive by NAT. + - **IPv6** (e.g. `2001:db8::7334`): 128-bit, essentially unlimited, with built-in security and auto-configuration. + +### 1.2 **Subnetting & CIDR: Drawing the Maps** +- **The Problem:** One giant network is noisy and insecure. +- **The Solution: Subnetting.** Divide large networks into smaller, manageable subnets. +- **CIDR (Classless Inter-Domain Routing):** + - `192.168.1.10/24` means first 24 bits are the network. + - **Subnet Mask:** `/24` = `255.255.255.0` + - **Golden Rule:** Two devices can talk directly **only if on the same subnet**; otherwise, they need a router. + +**Quick Subnetting Reference:** + +| CIDR | Subnet Mask | Usable Hosts | Use Case | +| :--- | :--- | :--- | :--- | +| `/32` | `255.255.255.255` | 1 | Single host (loopback, SGs) | +| `/30` | `255.255.255.252` | 2 | Point-to-point links | +| `/24` | `255.255.255.0` | 254 | Small LANs | +| `/16` | `255.255.0.0` | 65,534 | Large private networks | +| `/8` | `255.0.0.0` | 16.7M | Huge legacy networks | + +### 1.3 **The Default Gateway: Your Door to the World** +- The router on your subnet. Your device's routing table says, "If you're unsure where to send a packet, give it to the gateway." + +### 1.4 **Routing: The Path-Finding Algorithm** +- **The Routing Table:** Every device has one. + - `ip route show` displays yours. + - Answers: "To reach IP X, which next hop/interface should I use?" +- **How Routes are Added:** + - **Direct:** When you assign an IP. + - **Static:** Manually set by admins. + - **Dynamic:** Learned from routing protocols (BGP, OSPF) by routers. + +--- + +## 2. **The IP Packet Deep Dive (The L3 Envelope)** + +The Layer 4 segment is wrapped in an IP Packet. The header carries delivery-critical info. + +| Field | Purpose | Why it Matters | +| :--- | :--- | :--- | +| **Version** | 4 or 6 | IPv4 vs. IPv6 stack | +| **Source IP** | Sender | Used for replies, can be spoofed (security!) | +| **Destination IP** | Receiver | Where the packet is headed | +| **TTL** | Hop Limit | Decremented each router; prevents loops. `traceroute` manipulates TTL. | +| **Protocol** | TCP (6), UDP (17), etc. | Tells OS which L4 protocol to use. | +| **Flags/Fragment Offset** | Fragmentation | Large packets split—can cause issues.| + +```mermaid +graph LR +A[Application Data] --> B[L4: TCP Header + Data] +B --> C[L3: IP Header + Segment] +C --> D[L2: Frame] +``` + +--- + +## 3. **Layer 3 Devices & Cloud Equivalents** + +| Concept | Physical | Virtual/Cloud | Purpose | +| :--- | :--- | :--- | :--- | +| **Router** | Cisco/Juniper router | VPC Route Table, Virtual Gateway | Connect networks, forward packets | +| **L3 Switch** | Switch with routing | Same as Router | High-speed routing in datacenter | +| **Firewall** | Palo Alto/Fortinet | Security Groups, NACLs | Filter traffic at IP/port level | +| **Load Balancer** | F5 BIG-IP | ALB/NLB (AWS), GLB (GCP) | Distribute traffic to backends | +| **Gateway** | Internet Router | Internet Gateway, NAT Gateway | Connect private networks to Internet | + +--- + +## 4. **SRE's Toolkit: Essential Linux Commands** + +| Task | Command | Example Output | +| :--- | :--- | :--- | +| Show IP config | `ip addr show` | `inet 192.168.1.5/24 ...` | +| Show routing table | `ip route show` | `default via 192.168.1.1 dev eth0 ...` | +| Test reachability | `ping ` | `64 bytes from ...` | +| Trace packet path | `traceroute ` or `tracepath` | `1: _gateway ...` | +| Check open ports | `ss -tuln` | `tcp LISTEN ...` | +| Manage routes | `sudo ip route add/del ...` | `sudo ip route add 10.10.0.0/24 via 192.168.1.254` | +| Clear ARP cache | `ip neigh flush all` | (After config changes) | + +--- + +## 5. **Guided Hands-On Lab: Become Your Own Router** + +### **Lab Setup (VirtualBox/VMware)** +1. Create **three** VMs. +2. Set their network adapters to **Internal Network** (e.g., `LAB_NET`). +3. Boot them up (they’ll have no IP initially). + +### **Step 1: Configure the "Router"** +On VM1 (`router`): +```bash +sudo ip addr add 192.168.10.1/24 dev eth0 +sudo ip addr add 192.168.20.1/24 dev eth1 +echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward +``` + +### **Step 2: Configure the "Clients"** +On VM2 (`client-a`): +```bash +sudo ip addr add 192.168.10.10/24 dev eth0 +sudo ip route add default via 192.168.10.1 +``` +On VM3 (`client-b`): +```bash +sudo ip addr add 192.168.20.20/24 dev eth0 +sudo ip route add default via 192.168.20.1 +``` + +### **Step 3: Test the Connectivity** +From `client-a`: +```bash +ping 192.168.20.20 +``` +- If it works: you built a routed network! The router VM forwards between subnets. + +--- + +## 6. **Layer 3 in Action – App, Proxy, & Load Balancer Troubleshooting** + +### **Mini Project: How Does My Traffic Flow?** + +**Setup:** +1. **Deploy Simple Apps:** + - Start two instances of a basic app (Python Flask, Node.js, or simply `python -m http.server`) on two different VMs/containers or on different ports. +2. **Configure NGINX as Load Balancer/Reverse Proxy:** + - On a third VM/host, install NGINX and configure it to proxy traffic to the two app backends. + +**Sample `nginx.conf` for reverse proxy:** +```nginx +http { + upstream myapp { + server 192.168.10.10:8000; + server 192.168.20.20:8000; + } + server { + listen 80; + location / { + proxy_pass http://myapp; + } + } +} +``` +3. **Test Access:** + - Use `curl` or a browser to access the app via NGINX (`http://`). + +**Experiment:** +4. **Simulate Failure:** + - Stop one backend app (`kill` the process or `docker stop`) and access via NGINX again. +5. **Debug Traffic Flow:** + - Use `ip route`, `ss -tuln`, `ping`, `traceroute`, and review NGINX logs. + - What does NGINX do when a backend is down? Does the traffic route to the healthy instance? + +**Questions:** +- How does NGINX know where to send traffic? +- What changes at Layer 3 when a backend is down? +- How do you localise the failure using network commands? +- How does this all appear in your troubleshooting outputs? + +**Submission:** +- Include: + - Your NGINX config + - Outputs from `ip route`, `ss -tuln`, and relevant logs + - A short analysis of what you observed and how Layer 3 knowledge helped you debug + +--- + +## 7. **Real-World Scenarios & Debugging** + +### **Scenario 1: The Cloud VPC Mystery** +**Problem:** A VM in `subnet-private` can't pull updates from the internet, but a VM in `subnet-public` can. + +**Investigation:** +1. **Check Route Tables:** Is the default route in `subnet-private`’s table pointing to a **NAT Gateway**? +2. **Check NACLs/Security Groups:** Is outbound traffic (80/443, 1024-65535) allowed? +3. **Check if the subnet has a route to an Internet Gateway or NAT?** + +### **Scenario 2: The Hybrid Cloud Tunnel** +**Problem:** On-prem app (`10.10.10.5`) cannot reach cloud DB (`10.20.30.5`) after maintenance. + +**Investigation:** +1. **Traceroute:** Where does it stop? +2. **VPN/Direct Connect:** Is BGP up, routes advertised? +3. **Firewalls:** Are ACLs allowing the right traffic? + +--- + +## 8. **Mini Incident Simulation** + +**Alert:** Web servers in availability zone 1a fail health checks from the load balancer in 1b. + +You check a web server: +```bash +$ ip addr show eth0 +inet 10.0.1.25/20 brd 10.0.15.255 scope global eth0 + +$ ip route show +10.0.0.0/20 dev eth0 proto kernel scope link src 10.0.1.25 +default via 10.0.0.1 dev eth0 +``` +The load balancer's IP is `10.0.16.105`. + +**Tasks:** +1. Calculate the subnet range for `10.0.0.0/20`. Is the LB in the same subnet? +2. Where will the web server send packets destined for the LB? +3. Can packets reach it directly? Why or why not? +4. Propose a fix so all resources are routable without a router. + +--- + +## 9. **Advanced Gotchas for SREs** + +- **MTU & Fragmentation:** VPNs or overlays with smaller MTU can fragment packets, hurting performance. Enable MTU Path Discovery or set MTU manually. +- **Asymmetric Routing:** Requests go through firewall A, responses out firewall B—B may drop them. Beware in multi-path topologies. +- **ECMP:** Multiple equal-cost paths can confuse stateful firewalls and impact packet order. +- **Network Namespaces:** Containers/pods have isolated stacks. A `ping` from the host may work, but not from inside a container due to separate routing tables. Debug with `nsenter` or `kubectl exec`. + +--- + +## 10. **Submission Guidelines: Prove Your Mastery** + +**Push `day17solution.md` to your GitHub repo.** + +**Include:** +1. **Screenshots:** Output of `ip a` and `ip r` from your main machine. +2. **Hands-on Proof:** Output from the "Become Your Own Router" lab showing successful ping. +3. **App & LB Lab:** NGINX config, command outputs, and a brief analysis of your troubleshooting. +4. **Incident Response:** Detailed answers for the Mini Incident Simulation. +5. **Reflection:** A paragraph on a networking issue you’ve faced (or might face) and how Layer 3 knowledge helps. + +**Share your journey:** +`#getfitwithsagar #100DaysOfSRE #OSIMastery #Layer3DeepDive #CloudNetworking` + +--- + +## 11. **Community & Support** + +- [Discord](https://discord.gg/mNDm39qB8t) +- [Google Group](https://groups.google.com/forum/#!forum/daily-devops-sre-challenge-series/join) +- [YouTube](https://www.youtube.com/@Sagar.Utekar) + +--- + +**Master Layer 3, and you’ll never be lost in the network again!** + +Happy learning and exploring, +Sagar Utekar diff --git a/DailyChallenges/Season2/Day18/challenge.md b/DailyChallenges/Season2/Day18/challenge.md new file mode 100644 index 0000000..97465db --- /dev/null +++ b/DailyChallenges/Season2/Day18/challenge.md @@ -0,0 +1,205 @@ +# **Day 18: OSI Layer 4 – The Transport Layer (TCP, UDP, Ports, and Real-World Debugging)** + +--- + +## **Why Layer 4 is a Game Changer for SREs & DevOps** + +Once packets have their addresses (Layer 3), they need to get delivered **reliably** (or not!), in the right **order** (or not!), and to the correct **application**. That’s **Layer 4**: the Transport Layer. +Here, you’ll unlock the secrets behind TCP and UDP, ports, connection states, and how to debug the issues that puzzle even experienced engineers. + +> “It’s always the firewall… unless it’s the ephemeral port range, or TCP state bloat, or a rogue UDP flood. Layer 4 is where the real fun (and pain!) starts.” + +--- + +## 1. **Core Concepts Simplified (All-in-One Table)** + +| Topic / Protocol | TCP | UDP | +|---------------------- |-----------------------------------------------------------|---------------------------------------| +| **Type** | Reliable, connection-oriented | Unreliable, connectionless | +| **Guarantees** | Delivery, order, integrity | No guarantees | +| **Handshake?** | Yes (SYN, SYN-ACK, ACK) | No | +| **Teardown?** | Yes (FIN/ACK) | No | +| **Overhead** | Higher (more checks, slower) | Lower (faster, less checks) | +| **Common Uses** | Web, SSH, SMTP, DB | DNS, VoIP, games, streaming | +| **Connection State?** | Yes (tracks state, can get stuck) | No | +| **Good For** | Anything needing accuracy and order | Real-time, stateless, speed | + +> **Tip:** +> - Use TCP for reliability (web, SSH, DB). +> - Use UDP for speed (DNS, games). +> - Ports help traffic reach the right app. + +--- + +## 2. **TCP Three-Way Handshake (Connection Establishment)** +```mermaid +sequenceDiagram + participant Client + participant Server + + Client->>Server: SYN + Server->>Client: SYN-ACK + Client->>Server: ACK + Note over Client,Server: Connection Established +``` + + +### ** Layer 4 Packet Filtering** +```mermaid +flowchart TD + Internet[Internet Traffic] --> FW[Firewall :Layer 4] + FW -->|Allow TCP 80/443| Web[Web Server] + FW -->|Allow TCP 22| SSH[SSH Server] + FW -.->|Block UDP 53| DNS[DNS Server] +``` + +--- + +## 3. **Layer 4 in the Cloud & Modern Architectures** + +- **Security Groups & Firewalls:** + Control access by IP **and** port. Layer 4 rules are often the first line of defense. +- **Load Balancers:** + - **L4 load balancer:** Routes based on IP:port (TCP/UDP). Faster, less aware of app logic. + - **L7 (Application) load balancer:** Routes based on HTTP, cookies, etc. +- **NAT & Port Mapping:** + Translate internal IP:port to public IP:port. Essential for internet access and microservices. + +--- + +## 4. **SRE’s Command-Line Toolkit** + +| Task | Command | Example Output | +|------|---------|---------------| +| List connections | `ss -tuln` or `netstat -tuln` | Show listening TCP/UDP ports | +| Show connection states | `ss -s` or `netstat -an` | SYN_SENT, ESTABLISHED, etc. | +| Check open ports | `lsof -i :PORT` | Processes using a port | +| Test TCP | `nc -vz HOST PORT` | Success/failure of connection | +| Test UDP | `nc -vu HOST PORT` | Send/receive data | +| Capture packets | `sudo tcpdump -i eth0 port 80` | Live packet dump | +| Analyze with Wireshark | Open `.pcap` | Visualize TCP handshakes, retransmits | + +--- + +## 5. **Hands-On Lab: Layer 4 in Action** + +### **Lab 1: TCP/UDP Connections with Netcat** + +**A. TCP Echo Test** +1. On one terminal, run a TCP server: + ```bash + nc -l 9000 + ``` +2. On another terminal, connect as a client: + ```bash + nc 127.0.0.1 9000 + ``` +3. Type messages – verify bidirectional communication. +4. Observe connection with: + ```bash + ss -tna | grep 9000 + ``` + +**B. UDP Echo Test** +1. On one terminal, run a UDP server: + ```bash + nc -lu 9001 + ``` +2. On another terminal, send a UDP message: + ```bash + echo "hello" | nc -u 127.0.0.1 9001 + ``` +3. UDP is stateless—observe the difference in `ss -una | grep 9001`. + +--- + +### **Lab 2: TCP Handshake & State Machine with tcpdump** + +1. In one terminal, start a simple TCP server (e.g., Python): + ```bash + python3 -m http.server 8080 + ``` +2. In another, start packet capture: + ```bash + sudo tcpdump -i lo tcp port 8080 -w handshake.pcap + ``` +3. In a third, connect: + ```bash + curl http://localhost:8080 + ``` +4. Stop tcpdump. Open `handshake.pcap` in Wireshark and analyze: + - Find SYN, SYN-ACK, ACK packets (3-way handshake) + - See FIN/ACK for teardown + +5. List TCP connection states: + ```bash + ss -tan state all | grep 8080 + ``` + +--- + +## 6. **Mini Incident Simulation** + +**Scenario:** +Your web app is intermittently failing to connect to its database. The app logs show `Connection refused` and `Too many open files`. +You run: + +```bash +ss -s +ss -tan | grep 5432 +ulimit -n +``` +- You see hundreds of connections in `TIME_WAIT` to port 5432, and `ulimit -n` is 1024. + +**Tasks:** +1. What is `TIME_WAIT` and why does it happen? +2. How can too many connections in `TIME_WAIT` cause failures? +3. What are two ways to mitigate this issue (at the OS/app level)? +4. What Layer 4 lesson does this teach for SREs? + +--- + +## 7. **Common SRE Pitfalls & Gotchas** + +- **Ephemeral Port Exhaustion:** + Too many outgoing connections can exhaust source ports. Tweak ephemeral range and app keepalive settings. +- **Firewall/Security Group Rules:** + Ensure correct protocol (TCP vs. UDP) and port are allowed—port 53/UDP ≠ port 53/TCP! +- **Stateless vs. Stateful Load Balancing:** + Sticky sessions can break if not handled at Layer 4. +- **NAT & Connection Tracking:** + NAT device may drop idle connections or run out of state table slots. +- **Socket Options:** + Tuning `tcp_tw_reuse`, `tcp_fin_timeout`, and `ulimit` can help, but can also cause subtle bugs. + +--- + +## 8. **Submission Guidelines** + +**Push a file named `solution.md` to your repo. Include:** + +1. **Screenshots/outputs** of: + - `ss -tuln` and `ss -s` or `netstat -an` + - Your netcat and tcpdump/Wireshark experiments +2. **Mini Incident Simulation:** + - Your answers and reasoning +3. **Reflection:** + - Share a real or hypothetical Layer 4 troubleshooting story and how this knowledge would help you. + +**Share your journey:** +`#getfitwithsagar #DevOpsChallenge` + +--- + +## 9. **Community & Support** + +- [Discord](https://discord.gg/mNDm39qB8t) +- [Google Group](https://groups.google.com/forum/#!forum/daily-devops-sre-challenge-series/join) +- [YouTube](https://www.youtube.com/@Sagar.Utekar) + +--- + +**Master Layer 4, and you’ll unlock a new level of systems intuition. Every outage, fix, and performance boost starts making sense.** + +Happy hacking and troubleshooting, +Sagar Utekar diff --git a/DailyChallenges/Season2/Day19/challenge.md b/DailyChallenges/Season2/Day19/challenge.md new file mode 100644 index 0000000..2cb5a3e --- /dev/null +++ b/DailyChallenges/Season2/Day19/challenge.md @@ -0,0 +1,519 @@ +# Day 19: Firewalls, Security Groups & TLS/SSL – Defending Modern Infrastructure + +> “Performance issues are annoying. Misconfigured firewalls and expired TLS certs are outages.” + +--- + +## Why This Matters + +You can design the cleanest microservice architecture, but without deliberate **traffic control (firewalls & security groups)** and **secure channel establishment (TLS/SSL + optional mTLS)**, your stack is a soft target. +Today you will: lock down a host, design a 3‑tier perimeter, enable HTTPS, optionally add mTLS, capture a handshake, and reason about real incident scenarios. + +--- + +## 1. Core Concepts (At a Glance) + +| Concept | Simple Meaning | Why You Care | +|---------|----------------|--------------| +| Firewall | Policy engine (allow/deny) | Blocks unwanted access | +| Stateless Filter | Evaluates each packet isolated | Fast, but no connection awareness | +| Stateful Firewall | Tracks connection state (ESTABLISHED, RELATED) | Fewer rules; safer defaults | +| Security Group (SG) | Cloud stateful L3/L4 micro‑perimeter on ENI | Fine-grained ingress/egress | +| NACL | Stateless subnet ACL | Coarse baseline filtering | +| WAF | Layer 7 HTTP filter | Stops XSS, SQLi, bots | +| TLS/SSL | Encrypted transport with identity | Confidentiality + integrity + auth | +| Certificate | Signed public key + identity metadata | Trust chain & validation | +| Cipher Suite | Algorithms bundle (KEX + auth + cipher + MAC) | Security & performance | +| PFS (Forward Secrecy) | Ephemeral session keys | Limits blast radius of key theft | +| mTLS | Mutual certificate auth | Zero trust / service identity | +| Conntrack | Kernel table tracking flows | Enables stateful decisions | +| Ephemeral Ports | High-numbered client-side ports | Needed for return traffic / NAT | + +--- + +## 2. Packet & Flow Fundamentals + +### 2.1 Linux Simplified Packet Path + +```mermaid +flowchart LR + IN[eth0 Ingress] --> PREROUTING + PREROUTING -->|Local Dest?| INPUT + PREROUTING -->|Forward?| FORWARD + INPUT --> PROC[Local Process] + PROC --> OUTPUT + FORWARD --> POSTROUTING + OUTPUT --> POSTROUTING + POSTROUTING --> OUT[egress iface] +``` + +### 2.2 Stateless vs Stateful + +| Aspect | Stateless (NACL / raw ACL) | Stateful (iptables/nftables, SG) | +|--------|----------------------------|----------------------------------| +| Tracks return flows? | No | Yes (conntrack) | +| Reverse rule needed? | Yes | No | +| Typical placement | Subnet edge / router | Host / cloud instance | +| Logging context | Limited | Rich (state-based decisions) | + +--- + +## 3. Cloud Security Constructs + +| Control | Layer | Stateful | Scope | Typical Purpose | +|---------|-------|----------|-------|-----------------| +| SG | L3/L4 | Yes | ENI/Instance | Instance micro-perimeter | +| NACL | L3/L4 | No | Subnet | Baseline allow/deny | +| WAF | L7 | N/A | HTTP edge | App-layer attack filtering | +| Shield / DDoS | L3–7 | Managed | Edge | Volumetric resilience | +| Host Firewall | L3/L4 | Yes | Kernel/OS | Defense in depth | + +### 3.1 Security Group Statefulness + +```mermaid +sequenceDiagram + participant Client + participant SG as Security Group (Stateful) + participant Server + Client->>SG: SYN (TCP 443) + SG->>Server: SYN (Allowed by inbound rule) + Server->>SG: SYN-ACK (Return allowed via state table) + SG->>Client: SYN-ACK + Client->>SG: ACK (Established) + Note over SG: Reverse flow auto-permitted +``` + +--- + +## 4. TLS Essentials + +### 4.1 Guarantees + +| Guarantee | Meaning | +|-----------|---------| +| Confidentiality | Encrypted payload | +| Integrity | Tamper detection | +| Server Authentication | Proves server identity | +| (Optional) Client Authentication | mTLS mutual identity | + +### 4.2 TLS 1.2 (Simplified) + +```mermaid +sequenceDiagram + participant C as Client + participant S as Server + C->>S: ClientHello + S->>C: ServerHello + Certificate (+ ServerKeyExchange?) + S->>C: ServerHelloDone + C->>S: ClientKeyExchange + C->>S: ChangeCipherSpec + Finished + S->>C: ChangeCipherSpec + Finished +``` + +### 4.3 TLS 1.3 (Fewer Messages) + +```mermaid +sequenceDiagram + participant C as Client + participant S as Server + C->>S: ClientHello (key shares) + S->>C: ServerHello + EncryptedExtensions + Certificate + Finished + C->>S: Finished +``` + +--- + +## 5. Certificates (Types & Use) + +| Type | Use Case | Pros | Cons | +|------|----------|------|------| +| Self-Signed | Local dev | Fast | Not trusted externally | +| Internal CA | Internal services | Control lifecycle | Need trust distribution | +| Let’s Encrypt | Public web | Free + automated | 90‑day renew window | +| Wildcard | Many subdomains | Flexible | Larger key compromise blast | +| EV | High-profile brand | Strong validation chain | Minimal modern UX value | +| mTLS Leaf | Service identity | Strong auth | Ops overhead | + +--- + +## 6. Hands-On Labs + +> Choose either iptables or nftables for host firewall implementation. Both labs are provided. + +### Lab 1: Baseline Inspection + +Commands: +```bash +sudo ss -tulpen +sudo iptables -L -n -v 2>/dev/null || echo "iptables not active" +sudo nft list ruleset 2>/dev/null | head -n 50 || echo "nftables empty" +``` + +Record: +- Listening ports (service, port, protocol) +- Default INPUT/FORWARD policies (ACCEPT/DROP) +- Which framework is currently active? + +### Lab 2: Basic iptables Rule Set (Stateful Minimal Policy) + +Goal: Allow loopback, established/related, SSH(22), HTTP(80), HTTPS(443). Drop other inbound. + +```bash +sudo iptables -P INPUT DROP +sudo iptables -P FORWARD DROP +sudo iptables -P OUTPUT ACCEPT + +sudo iptables -A INPUT -i lo -j ACCEPT +sudo iptables -A INPUT -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT +sudo iptables -A INPUT -p tcp -m multiport --dports 22,80,443 -j ACCEPT + +sudo iptables -L -n -v +``` + +Test blocked port: +```bash +nc -vz 127.0.0.1 8080 || echo "Blocked as expected" +``` + +(Optionally persist): +```bash +sudo sh -c 'iptables-save > /etc/iptables/rules.v4' +``` + +### Lab 3: nftables Equivalent (If You Prefer Modern Stack) + +```bash +sudo nft flush ruleset +sudo tee /tmp/day19.nft <<'EOF' +table inet filter { + chain input { + type filter hook input priority 0; policy drop; + iif lo accept + ct state established,related accept + tcp dport {22,80,443} accept + } + chain forward { type filter hook forward priority 0; policy drop; } + chain output { type filter hook output priority 0; policy accept; } +} +EOF +sudo nft -f /tmp/day19.nft +sudo nft list ruleset +``` + +Optional logging rule: +```bash +sudo nft add rule inet filter input log prefix "DROP_INPUT " counter +journalctl -k -f +``` + +### Lab 4: Prove Statefulness + +```bash +curl -I https://example.com 2>/dev/null | head -n1 +``` +Explain: Why did the response succeed without an inbound ephemeral port rule? (conntrack state ESTABLISHED handles reverse flow.) + +### Lab 5: 3‑Tier Cloud Perimeter Design (Conceptual) + +Create a diagram (mermaid allowed) & a table: + +| Tier | Inbound Source | Ports | Outbound | +|------|----------------|-------|----------| +| ALB | 0.0.0.0/0 | 443 (maybe 80 redirect) | Web SG on 443 | +| Web | ALB SG | 80,443 | App SG:8080; DNS, NTP | +| App | Web SG | 8080 | DB SG:5432 | +| DB | App SG | 5432 | Backup / monitoring only | + +Mermaid: +```mermaid +flowchart LR + User((Internet)) -->|443| ALB[ALB/WAF] + ALB -->|80/443| WEB[Web Tier] + WEB -->|8080| APP[App Tier] + APP -->|5432| DB[(Database)] +``` + +Explain differences: +- SG (stateful) vs NACL (stateless) +- WAF sits before ALB (optional) for L7 inspection + +### Lab 6: Generate Self-Signed TLS & Serve via Nginx + +```bash +sudo apt-get update -y +sudo apt-get install -y nginx openssl + +mkdir -p ~/tlslab && cd ~/tlslab +openssl req -x509 -newkey rsa:4096 -sha256 -days 90 -nodes \ + -keyout server.key -out server.crt -subj "/CN=localhost" +``` + +Nginx site: +```bash +sudo tee /etc/nginx/sites-available/day19 <<'EOF' +server { + listen 443 ssl; + server_name localhost; + ssl_certificate /home/$USER/tlslab/server.crt; + ssl_certificate_key /home/$USER/tlslab/server.key; + + add_header X-Day19 "PerimeterDemo"; + location / { + return 200 "Day19 Secure Hello\n"; + } +} +EOF +sudo ln -s /etc/nginx/sites-available/day19 /etc/nginx/sites-enabled/day19 +sudo nginx -t && sudo systemctl reload nginx +``` + +Test: +```bash +curl -vk https://localhost 2>&1 | head -n 20 +``` + +Questions: +- Why browser/curl trust warning? (Self-signed: subject=issuer) +- Production cert should include proper SANs (Subject Alternative Names) + +### Lab 7: TLS Certificate & Cipher Inspection + +```bash +echo | openssl s_client -connect localhost:443 -servername localhost 2>/dev/null \ + | openssl x509 -noout -subject -issuer -dates + +echo | openssl s_client -connect localhost:443 -servername localhost 2>/dev/null \ + | grep -i "Cipher" +``` +Record: +- Subject / Issuer +- notBefore / notAfter +- Chosen cipher (identify ECDHE for PFS) + +### Lab 8: TLS Handshake Capture (Optional but Recommended) + +```bash +sudo tcpdump -i lo port 443 -w tls_handshake.pcap & +PID=$! +curl -k https://localhost >/dev/null +sleep 1 +sudo kill $PID +``` +View in Wireshark: +- Inspect ClientHello (cipher list, SNI) +- ServerHello (selected cipher) +- Certificate frame + +Document 2–3 handshake observations. + +### Lab 9: Near-Expiry Cert Simulation + +```bash +cd ~/tlslab +openssl req -x509 -newkey rsa:2048 -sha256 -days 1 -nodes \ + -keyout short.key -out short.crt -subj "/CN=localhost-short" +sudo sed -i 's/server.crt/short.crt/; s/server.key/short.key/' /etc/nginx/sites-available/day19 +sudo nginx -t && sudo systemctl reload nginx +echo | openssl s_client -connect localhost:443 -servername localhost 2>/dev/null \ + | openssl x509 -noout -dates +``` + +Restore: +```bash +sudo sed -i 's/short.crt/server.crt/; s/short.key/server.key/' /etc/nginx/sites-available/day19 +sudo nginx -t && sudo systemctl reload nginx +``` + +Explain operational risk of missed renewal; propose monitoring threshold (e.g. alert at <20 days). + +### Lab 10: mTLS Mini Demo (Optional Advanced) + +Internal CA: +```bash +cd ~/tlslab +openssl genrsa -out ca.key 4096 +openssl req -x509 -new -key ca.key -days 365 -out ca.crt -subj "/CN=Day19-CA" +``` + +Server leaf (reissue signed by CA): +```bash +openssl genrsa -out mtls_server.key 4096 +openssl req -new -key mtls_server.key -out mtls_server.csr -subj "/CN=mtls-server" +openssl x509 -req -in mtls_server.csr -CA ca.crt -CAkey ca.key -CAcreateserial \ + -out mtls_server.crt -days 120 -sha256 +``` + +Client cert: +```bash +openssl genrsa -out client.key 4096 +openssl req -new -key client.key -out client.csr -subj "/CN=client1" +openssl x509 -req -in client.csr -CA ca.crt -CAkey ca.key -CAserial ca.srl \ + -out client.crt -days 120 -sha256 +``` + +Update nginx: +```bash +sudo sed -i 's#ssl_certificate .*#ssl_certificate /home/'"$USER"'/tlslab/mtls_server.crt;#' /etc/nginx/sites-available/day19 +sudo sed -i 's#ssl_certificate_key .*#ssl_certificate_key /home/'"$USER"'/tlslab/mtls_server.key;#' /etc/nginx/sites-available/day19 +sudo sed -i '/server_name localhost;/a ssl_client_certificate /home/'"$USER"'/tlslab/ca.crt;\nssl_verify_client on;' /etc/nginx/sites-available/day19 +sudo nginx -t && sudo systemctl reload nginx +``` + +Tests: +```bash +curl --cert ~/tlslab/client.crt --key ~/tlslab/client.key -k https://localhost +curl -k https://localhost -o /dev/null -w "HTTP:%{http_code}\n" +``` + +Record: success vs failure status codes. + +### Lab 11: Incident Simulation & Analysis + +Pick any TWO and write: Timeline → Symptom → Root Cause → Resolution → Prevention + +| Scenario | Symptom | Likely Root Cause | Prevention Idea | +|----------|---------|------------------|-----------------| +| SSH Lockout | Timeout | Dropped port 22 / no ESTABLISHED rule | Pre-change session + rollback script | +| App→DB Fail | ECONNREFUSED | Missing SG rule or wrong CIDR | IaC & peer review | +| Expired Cert | Browser errors / failing jobs | Failed renewal automation | Expiry monitoring + ACME | +| mTLS Fail | 400 / 401 / handshake abort | Wrong CA bundle / mismatched CN | Automate trust distribution | +| Conntrack Exhaustion | Drops / latency | High short-lived connection churn | Tune nf_conntrack_max, reuse pools | +| NAT Port Exhaustion | Outbound stalls | Too many parallel ephemeral flows | Increase ranges / SNAT scaling | +| TLS CPU Spike | High CPU, slow handshakes | No session reuse, old ciphers | TLS 1.3, reuse, offload | + +--- + +## 7. Debugging & Toolbelt + +| Goal | Commands | +|------|----------| +| List iptables | `iptables -L -n -v`, `iptables-save` | +| List nft | `nft list ruleset` | +| Open sockets | `ss -tulpen` | +| Conntrack sample | `sudo conntrack -L | head` | +| Test port | `nc -vz host 443` | +| Quick scan (careful) | `nmap -sS -p 22,80,443 host` | +| TLS details | `openssl s_client -connect host:443 -servername host` | +| Enumerate ciphers | `nmap --script ssl-enum-ciphers -p 443 host` | +| Cert expiry | `echo | openssl s_client -connect host:443 2>/dev/null | openssl x509 -noout -dates` | +| Handshake capture | `tcpdump -n -i eth0 port 443 -w tls.pcap` | + +--- + +## 8. Performance & Pitfalls + +| Pitfall | Cause | Impact | Mitigation | +|---------|-------|--------|-----------| +| Overly permissive SG | Quick testing | Attack surface | Least privilege + IaC | +| Rule sprawl | Manual edits | Complexity / errors | Periodic pruning / versioning | +| Conntrack full | Burst connections | Drops / latency | Tune + scale + pooling | +| NAT ephemeral exhaustion | High outbound fan-out | Stalled flows | Expand range / scale NAT | +| Weak ciphers enabled | Legacy defaults | MITM risk | Harden config / TLS 1.3 | +| TLS handshake CPU | No reuse / RSA | Latency, cost | Session tickets / TLS 1.3 | +| Expired cert | Missed renew job | Outage | Alert < 20–30 days | +| Unlogged drops | No visibility | Slow RCA | Rate-limited logging | + +--- + +## 9. Hardening Checklist + +| Area | Action | +|------|--------| +| Ingress Policy | Default DROP (host & SG) | +| Loopback | Always allow | +| SSH | Restrict CIDR / MFA bastion | +| Services | Bind only required interfaces | +| TLS | Enforce 1.2+ (prefer 1.3), strong ciphers | +| Cert Lifecycle | Auto-renew + expiry alerts | +| Keys | 600 perms; minimal exposure | +| Logging | Drop logging (rate-limited) | +| Observability | Dashboards: conntrack, drops, handshake failures | +| mTLS | Internal sensitive APIs / service mesh | +| IaC | Terraform / Ansible for rules & cert provisioning | + +--- + +## 10. Design Patterns + +| Pattern | Description | Example | +|---------|-------------|---------| +| Choke Point | Single controlled ingress layer | ALB + WAF bundle | +| Defense in Depth | Multiple layered controls | SG + host firewall + app ACL | +| Zero Trust Drift | Auth every hop | mTLS between services | +| Segmentation | Contain lateral movement | Web/App/DB tiering | +| Policy as Code | Versioned reproducible security | Terraform SG modules & nft templates | + +--- + +## 11. Reflection Questions + +1. Why does reply traffic not need an explicit inbound rule on a stateful firewall? +2. One sentence: SG vs NACL difference? +3. Advantage of nftables over iptables (design or management)? +4. Where do you see Perfect Forward Secrecy in the cipher output? +5. Operational risk illustrated by the 1‑day certificate test? +6. Give one concrete internal use case for mTLS. +7. Describe a mitigation for conntrack table exhaustion. +8. Which design pattern (choke point, defense in depth, zero trust) did you partially implement today and how? +9. One automation you’d add tomorrow to reduce perimeter toil? + +--- + +## 12. Submission Guidelines (day19solution.md) + +Include: + +1. Baseline outputs: `ss`, firewall policy, active framework +2. Final ruleset (iptables-save OR `nft list ruleset` excerpt) +3. Statefulness explanation (Lab 4) +4. 3-tier model (table + mermaid or diagram) + SG vs NACL vs WAF one-liners +5. TLS evidence: curl snippet, cert subject/issuer/dates, cipher line +6. Near-expiry (1‑day cert) output + monitoring note +7. (Optional) Handshake capture observations (ClientHello, cipher chosen) +8. (Optional) mTLS success vs failure HTTP codes +9. Two incident analyses (structured) +10. Reflection answers (all) +11. (Optional) Hardening checklist: mark applied items + +Share progress with: +`#getfitwithsagar #100DaysOfSRE #CloudSecurity #TLS #DevSecOps #NetworkDefense` + +--- + +## 13. Safety Notes + +- Use `tmux`/`screen` before changing firewall if remote. +- Prepare rollback (cron or script) when imposing DROP defaults. +- Never commit private keys. +- Only scan systems you own or are authorized to test. +- Keep at least one active session while tightening rules. + +--- + +## 14. Cleanup (Optional) + +```bash +sudo rm /etc/nginx/sites-enabled/day19 +sudo rm /etc/nginx/sites-available/day19 +sudo systemctl reload nginx +# Revert firewall manually if needed (e.g., set INPUT policy ACCEPT) +``` + +--- + +## 15. Success Criteria + +You can: +- Explain stateful vs stateless & show working policy +- Present a 3-tier SG design with correct segmentation rationale +- Serve HTTPS, inspect cert + cipher, identify PFS component +- Simulate expiry risk & propose monitoring +- (Optionally) Demonstrate mTLS acceptance vs rejection +- Produce credible incident root cause analyses + +If yes → You’ve internalized Day 19. + +--- + +Happy securing, +Sagar Utekar diff --git a/DailyChallenges/Season2/Day20/challenge.md b/DailyChallenges/Season2/Day20/challenge.md new file mode 100644 index 0000000..df81412 --- /dev/null +++ b/DailyChallenges/Season2/Day20/challenge.md @@ -0,0 +1,359 @@ +# **Day 20: Application Layer 7 – Reverse Proxies, Caching, Performance & Observability** + +--- + +## Why this Matters + +Most real-world outages aren’t kernel panics; they’re **502s, 503s, latency spikes, cache misconfigurations, bad headers, or upstream timeouts**. +Layer 7 (HTTP) is where reliability, performance, and security meet. Mastering this layer lets you: +- Slash origin load with caching +- Expose reliable APIs behind reverse proxies +- Debug 502 vs 504 vs 499 quickly +- Optimise compression, connection reuse, and multiplexing +- Trace requests across layers with structured logs and headers + +> “If you can read headers, logs, and timings, you can solve 80% of web production issues.” + +--- + +## 1. Core Concepts (At a Glance) + +| Concept | Meaning (Simple) | Why It Matters | +|---------|------------------|----------------| +| Method (GET/POST/etc.) | Action on a resource | Safe vs unsafe; idempotency | +| Status Code Classes | 1xx info, 2xx success, 3xx redirect, 4xx client error, 5xx server error | Fast triage | +| Headers | Metadata before body | Routing, caching, debugging | +| Body | Payload (HTML, JSON, binary) | API / page content | +| Connection Persistence | Keep-Alive vs new TCP per request | Latency + scalability | +| Reverse Proxy | Intermediary forwarding to upstream | Control point (NGINX / Envoy) | +| Gateway / API Gateway | Adds auth, rate limits, transforms | API governance | +| Cache | Store response to reuse later | Performance + cost reduction | +| CDN | Global cache + edge network | Latency + DDoS absorption | +| Observability Fields | latency, upstream time, status, bytes | SLA / SLO measurement | + +--- + +## 2. Reverse Proxy vs Load Balancer vs API Gateway + +| Component | Focus | Examples | Adds | +|-----------|-------|----------|------| +| Reverse Proxy | Fronts and forwards requests to upstreams | NGINX, HAProxy, Caddy | TLS, routing, basic caching | +| Load Balancer | Distributes load across backend pool | ALB, ELB, HAProxy, Envoy | Health checks, balancing algorithms | +| API Gateway | Manages API lifecycle | Kong, APIGW, Tyk, Zuul | Auth, quotas, transforms, versioning | +| Service Mesh Sidecar | Per-service proxy | Envoy/Istio, Linkerd | mTLS, routing policies, telemetry | + +--- + +## 3. Critical HTTP Headers (You Should Recognise Instantly) + +| Category | Header | Purpose | +|----------|--------|---------| +| Client Identity | X-Forwarded-For, X-Real-IP | Preserve original IP through proxy chain | +| Protocol & Host | Host, X-Forwarded-Proto | Virtual hosting, HTTPS redirect logic | +| Caching – Freshness | Cache-Control (max-age, s-maxage), Expires | Decide if cached copy usable | +| Caching – Validation | ETag, Last-Modified, If-None-Match, If-Modified-Since | Conditional revalidation | +| Auth | Authorization, WWW-Authenticate | API / Basic / Bearer tokens | +| Content | Content-Type, Content-Length, Transfer-Encoding | Parse and stream body | +| Compression | Accept-Encoding, Content-Encoding | gzip, br support | +| Security | Strict-Transport-Security, X-Frame-Options, Content-Security-Policy, Set-Cookie (HttpOnly/Secure/SameSite) | Mitigate attacks | +| Control / Debug | Via, X-Trace-Id, X-Request-Id, X-Upstream-Response-Time | Distributed tracing | +| Variation | Vary | Cache key dimension changes | + +--- + +## 4. Observability KPIs (Track These) + +| Metric | Reason | +|--------|--------| +| p50 / p95 / p99 latency | User experience & performance tuning | +| Upstream response time vs total time | Find proxy vs backend delay | +| Active connections | Capacity planning | +| 4xx / 5xx ratio | Error detection | +| Cache HIT ratio | Cost + speed optimization | +| Request size / response size | Abuse detection / tuning | +| 499 count (client closed) | User aborts / timeout mismatch | +| Open file descriptors | Resource limits | + +--- + +## 5. Mermaid Architecture & Flows + +### 5.1 Request Flow (Reverse Proxy + Cache) + +```mermaid +flowchart LR + A[Client] --> B[Reverse Proxy / Cache] + B -->|HIT| C[Serve Cached Response] + B -->|MISS| D[Upstream Pool] + D --> E[App Service] + E --> D --> B --> A +``` + +### 5.2 Cache Decision Logic + +```mermaid +flowchart TD + R[Incoming Request] --> K{Key Exists?} + K -->|No| U[Fetch Upstream] --> S[Store if Cacheable] --> Return + K -->|Yes| F{Fresh?} + F -->|Yes| Return + F -->|No| V{Validatable? :ETag} + V -->|No| U + V -->|Yes| C[Send Conditional Request] + C -->|304| UpdateMeta --> Return + C -->|200| S +``` + +--- + +## 6. Hands-On Labs + +## Problem Statement + +You are the on-call SRE. A small internal service (a simple Python HTTP backend) is getting slower under load, and the product wants: +1. Faster repeated responses +2. Fewer upstream timeouts +3. Basic visibility into performance +4. Ability to explain a 502 vs 504 if it happens + +Your task: Put NGINX in front of the backend as a reverse proxy with minimal caching + basic observability, then demonstrate: +- A normal proxied request +- A cached response +- A slow request + how a timeout manifests (and how you fix it) +- A short log-based latency summary + +You will deliver a single file `solution.md` documenting what you did and the answers to reflection questions. + +--- + +## 1. Setup (Backend + Reverse Proxy) + +### 1.1 Start a very simple backend + +```bash +mkdir -p ~/day20 && cd ~/day20 +cat > app.py <<'EOF' +from http.server import BaseHTTPRequestHandler, HTTPServer +import time, datetime + +class H(BaseHTTPRequestHandler): + def do_GET(self): + # /slow simulates slowness + if self.path.startswith("/slow"): + time.sleep(2) + body = f"PATH={self.path} TIME={datetime.datetime.utcnow().isoformat()}Z\n" + self.send_response(200) + self.send_header("Content-Type", "text/plain") + # Let proxy decide caching; still send something explicit: + if self.path.startswith("/cache"): + self.send_header("Cache-Control", "max-age=30") + else: + self.send_header("Cache-Control", "no-store") + self.end_headers() + self.wfile.write(body.encode()) + +HTTPServer(("127.0.0.1", 9000), H).serve_forever() +EOF + +python3 app.py +``` + +(Leave it running in one terminal.) + +### 1.2 Install and configure NGINX (reverse proxy + tiny cache) + +```bash +sudo apt-get update -y +sudo apt-get install -y nginx +``` + +Create config: + +```bash +sudo tee /etc/nginx/conf.d/day20.conf <<'EOF' +# Simple cache definition +proxy_cache_path /var/cache/nginx/day20 levels=1:2 keys_zone=day20cache:10m max_size=50m inactive=2m use_temp_path=off; + +upstream backend_pool { + server 127.0.0.1:9000; +} + +# Custom log format (pipe-separated so it's easy to parse) +log_format mini '$remote_addr|$time_local|$request|$status|$body_bytes_sent|$request_time|$upstream_response_time|$upstream_cache_status'; + +server { + listen 8080; + server_name localhost; + + access_log /var/log/nginx/day20_access.log mini; + + # Default (no caching) proxy + location / { + proxy_pass http://backend_pool; + proxy_set_header Host $host; + proxy_set_header X-Forwarded-For $remote_addr; + # Short upstream read timeout to show 504 on /slow initially + proxy_read_timeout 1s; + } + + # Caching block (/cache path) + location /cache { + proxy_pass http://backend_pool; + proxy_cache day20cache; + proxy_cache_valid 200 30s; + add_header X-Cache $upstream_cache_status always; + proxy_set_header Host $host; + proxy_set_header X-Forwarded-For $remote_addr; + } +} +EOF + +sudo nginx -t && sudo systemctl reload nginx +``` + +--- + +## 2. Demonstrate Core Behaviours + +In a new terminal: + +### 2.1 Normal proxied request + +```bash +curl -i http://localhost:8080/ +``` + +Expected: 200, no `X-Cache` header (not in /cache), fresh timestamp each call. + +### 2.2 Cached request (HIT vs MISS) + +First call (MISS): +```bash +curl -i http://localhost:8080/cache +``` + +Second call (HIT): +```bash +curl -i http://localhost:8080/cache +``` + +Check headers for: `X-Cache: MISS` then `X-Cache: HIT` + +### 2.3 Trigger a 504 (slow path + tight timeout) + +```bash +curl -i http://localhost:8080/slow +``` + +Because backend sleeps 2s and `proxy_read_timeout` is 1s, you should see a 504 Gateway Timeout. + +### 2.4 Fix the timeout + +Edit just the timeout line: + +```bash +sudo sed -i 's/proxy_read_timeout 1s;/proxy_read_timeout 3s;/' /etc/nginx/conf.d/day20.conf +sudo nginx -t && sudo systemctl reload nginx +curl -i http://localhost:8080/slow +``` + +Now it should succeed with 200 after ~2 seconds. + +--- + +## 3. Lightweight Observability + +Grab last few log lines: + +```bash +tail -n 20 /var/log/nginx/day20_access.log +``` + +Example line format: +``` +IP|TIME|REQUEST|STATUS|BYTES|REQ_TIME|UPSTREAM_TIME|CACHE_STATUS +``` + +Quick status breakdown: + +```bash +awk -F'|' '{print $4}' /var/log/nginx/day20_access.log | sort | uniq -c +``` + +Average request time: + +```bash +awk -F'|' '{sum+=$6; n+=1} END {if(n>0) print "avg_request_time_secs="sum/n;}' /var/log/nginx/day20_access.log +``` + +Cache hits vs misses: + +```bash +awk -F'|' '$8!="" {print $8}' /var/log/nginx/day20_access.log | sort | uniq -c +``` + +--- + +## 4. 502 vs 504 (You Only Simulated 504 Today) + +Explain in your solution: +- 502 = proxy talked to upstream but got a bad/invalid/closed response +- 504 = upstream did not respond before timeout + +(You can optionally fake a 502 by stopping the backend: Ctrl+C backend → `curl http://localhost:8080/` → likely 502.) + +--- + +## 5. What to Include in `day20solution.md` + +Minimum: +1. Commands you ran (copy/paste sequence) +2. Output snippets: + - First MISS then HIT (`X-Cache`) + - 504 example (before fix) + - Successful `/slow` after timeout change +3. At least 5 log lines + your short interpretation (e.g., which was cached? which timed out?) +4. Answers to reflection questions (below) + +Optional: +- A sentence describing how you’d extend this (rate limiting? gzip?). + +--- + +## 6. Reflection Questions (Answer Briefly) + +1. What changed between MISS and HIT responses? +2. Why did `/slow` first return 504? +3. After increasing the timeout, what risk do you accept (hint: tying up connections longer)? +4. How would a 502 differ (root cause wise) from the 504 you saw? +5. How does basic caching reduce backend load in this scenario? + +--- + +## 7. Cleanup (Optional) + +```bash +sudo rm /etc/nginx/conf.d/day20.conf +sudo systemctl reload nginx +``` + +--- + +## 8. Summary (What You Just Learned Quietly) + +| Skill | You Did It By | +|-------|---------------| +| Reverse proxy basics | NGINX forwarding to Python server | +| Caching | `/cache` location using `proxy_cache` | +| Cache diagnostics | Observed `X-Cache` MISS → HIT | +| Timeout handling | Forced 504 then tuned `proxy_read_timeout` | +| Log-based insight | Parsed custom access log for status & timing | +| Error differentiation | Defined 502 vs 504 semantics | + +--- + +**Layer 7 mastery lets you design faster, more reliable, more observable systems.** +Today you built the foundation for API gateways, CDNs, mesh routing, and performance engineering. + +Happy optimizing and debugging, +Sagar Utekar diff --git a/DailyChallenges/Season2/Day7/challenge.md b/DailyChallenges/Season2/Day7/challenge.md new file mode 100644 index 0000000..d7c1b65 --- /dev/null +++ b/DailyChallenges/Season2/Day7/challenge.md @@ -0,0 +1,142 @@ +# Day7: GitHub Self-Hosted Runners + +## Introduction +Welcome to Day7 of the Daily DevOps SRE Daily Challenge Series - Season2! 🎉 + +Today, we focus on **GitHub Self-Hosted Runners**, a powerful way to customize GitHub Actions workflows by running jobs on your own infrastructure. + +By the end of this challenge, you will: +- Understand what GitHub runners are and their types. +- Compare **hosted** vs. **self-hosted** runners with advantages and disadvantages. +- Set up a **self-hosted runner** on an **AWS EC2 instance**. +- **Dockerize and deploy** the Snake game from [this repo](https://github.com/Sagar2366/season2-snake_game.git) using the self-hosted runner. + +--- + +### **Why This Matters?** +Understanding **self-hosted GitHub runners** is critical for enterprises managing private infrastructure, optimizing costs, and ensuring workflow security. + +## 1️⃣ What Are Runners? + +GitHub **runners** are machines that execute workflows defined in **GitHub Actions**. These runners can be: +- Hosted by **GitHub** (default). +- **Self-hosted** (your own machine or cloud instance). + + +## 2️⃣ Types of GitHub Runners + +| Runner Type | Description | Use Cases | Advantages | Disadvantages | +|-----------------|-------------|-----------|-------------|---------------| +| **GitHub-Hosted** | Managed by GitHub with predefined environments. | General CI/CD workflows with minimal setup. | No maintenance, built-in scaling. | Limited customization, restricted software access. | +| **Self-Hosted** | User-managed runner on a personal machine or cloud instance. | Workflows needing custom dependencies or private network access. | Full control, cost-effective for large workloads. | Requires maintenance, security responsibility. | + +### When to Use What? + +| Scenario | Use **GitHub-Hosted** | Use **Self-Hosted** | +|----------|---------------------|---------------------| +| Simple CI/CD workflows | ✅ | ❌ | +| Need custom software/dependencies | ❌ | ✅ | +| Need access to private networks | ❌ | ✅ | +| Cost-effective for large workloads | ❌ | ✅ | +| Security and compliance control | ❌ | ✅ | + +--- + +## 3️⃣ Challenge Breakdown + +### **📝 Theoretical Questions** +Answer the following questions: +1. What is a GitHub runner? +2. How do self-hosted runners differ from GitHub-hosted runners? +3. What security considerations should you take when using self-hosted runners? +4. How can you scale self-hosted runners? +5. Can a single self-hosted runner be used for multiple repositories? Why or why not? + +--- + +### **⚙️ Practical Challenge: Setting Up a Self-Hosted Runner** + +#### **Step 1: Create an EC2 Instance** +1. Launch an AWS EC2 instance (Ubuntu 22.04 recommended). +2. Allow inbound traffic on ports **22 (SSH)** and **80 (HTTP)**. + +#### **Step 2: Configure GitHub Self-Hosted Runner** +1. Navigate to your **GitHub repository** → **Settings** → **Actions** → **Runners**. +2. Click **"New self-hosted runner"**, select **Linux**, and follow the instructions to set it up. +3. SSH into the EC2 instance and install the runner using the provided commands. +4. Start the runner: + ```bash + ./run.sh + ``` + +#### **Step 3: Deploy the Snake Game** + + +https://github.com/user-attachments/assets/3c0e0fc7-a5ef-4285-9b3a-4cdb57915f0e + + +1. Install **Docker** on your EC2 instance: + ```bash + sudo apt update + sudo apt install docker.io -y + ``` +2. Clone the **Snake Game repository**: + ```bash + git clone https://github.com/clear-code-projects/Snake + cd Snake + ``` +3. Build and run the application as a Docker container: + ```bash + docker build -t snake-game . + docker run -d -p 80:5000 snake-game + ``` +4. Confirm the deployment by accessing `http://` in a browser. + +5. You are required to create a GitHub Actions workflow that: + **Builds and pushes a Docker image** to Docker Hub. + **Deploys the application on Self Hosted GitHub Runners** + **Validates that the application is running correctly**. + **Sends an email notification** with the deployment status. + +#### **Step 4: Take Screenshots for Submission** +- Running EC2 instance (`aws ec2 describe-instances`). +- GitHub Actions workflow logs showing execution on the self-hosted runner. +- Webpage running the Snake Game. + +--- + +## 4️⃣ Submission Guidelines + +✅ **Proof of Completion:** +- Screenshot of **EC2 instance running**. +- Screenshot of **GitHub Actions logs showing execution on self-hosted runner**. +- Screenshot of **Snake Game running in the browser**. + +✅ **Documentation:** +- Steps taken to configure the self-hosted runner. +- Learnings and challenges faced. + +--- + +## 5️⃣ Bonus Tasks 🎯 +- Auto-scale self-hosted runners with **multiple EC2 instances**. +- Implement **load balancing** using **Nginx**. + +### **Share Your Progress!** +Post your experience on social media with the hashtags: **#getfitwithsagar, #SRELife, #DevOpsForAll** + +--- + +## **Join Our Community** + +To make the most of this journey, connect with fellow learners: +- **Discord** – Ask questions and collaborate: [Join here](https://discord.gg/mNDm39qB8t) +- **Google Group** – Get updates and discussions: [Join here](https://groups.google.com/forum/#!forum/daily-devops-sre-challenge-series/join) +- **YouTube** – Watch solution videos and tutorials: [Subscribe here](https://www.youtube.com/@Sagar.Utekar) + +--- + +Keep learning and happy automating! + +Kind Regards, +Sagar Utekar and Raihan diff --git a/DailyChallenges/Season2/Day8/readme.md b/DailyChallenges/Season2/Day8/readme.md new file mode 100644 index 0000000..c61ee05 --- /dev/null +++ b/DailyChallenges/Season2/Day8/readme.md @@ -0,0 +1,289 @@ +# Daily DevOps + SRE Challenge Series – Season 2 +## Day 8: Kickstart Your Journey with Linux & Virtualization + +## Introduction +Welcome to Day 8 of the Daily DevOps SRE Challenge Series - Season 2! 🎉 + +Today, we’re starting strong with **Linux and Virtualization**—the bedrock of DevOps and SRE workflows. By the end of this challenge, you will: +- Understand what Unix and Linux are and why they matter. +- Explore **virtualization** and its role in modern infrastructure. +- Compare popular **Linux distributions** and their use cases. +- Set up **AWS EC2 instances** with different Linux distros and install **RHEL** on VirtualBox locally. + +--- + +### **Why This Matters?** +Linux powers the cloud, containers, and automation tools that DevOps and SRE engineers rely on daily. Virtualization is the foundation of scalable infrastructure—think AWS, testing environments, and cost savings. Together, they’re your launchpad to mastering modern IT, unlocking roles like Cloud Engineer, SRE, and System Admin! + +--- + +## 1️⃣ What Are Unix and Linux? +![0_Qqqd7UsfFDPL7WXh](https://github.com/user-attachments/assets/126f72c0-6d92-440a-a42d-c758394f7046) + +- **Unix:** A powerful, multi-user, multitasking OS born in the 1970s at AT&T Bell Labs. It’s the ancestor of many modern systems, built for stability and modularity. +- **Linux:** A Unix-inspired, open-source OS—free, customizable, and community-driven. It runs everywhere: servers, clouds, even your phone! + +**Key Difference:** Unix is often proprietary (e.g., Solaris), while Linux is open-source and adaptable (e.g., Ubuntu, RHEL). + +**Fun Fact:** The `ls` command in Linux? It’s a Unix legacy, reborn for today’s systems! + +--- + +## 2️⃣ Virtualization Basics + +**What is Virtualization?** +Virtualization lets you run multiple **virtual machines (VMs)** on one physical host, splitting resources like CPU, RAM, and storage. It’s like having a dozen computers in one box! + +**How It Works:** +A **hypervisor** manages VMs, abstracting hardware to create isolated environments. + +**Types of Hypervisors:** +![611e5d7e-953a-41db-9171-c7ba1cd6df9c](https://github.com/user-attachments/assets/817b944b-848a-426d-a6a3-1c030ff3adb2) + +| Type | Description | Examples | +|---------|---------------------|---------------------| +| Type 1 | Runs on bare metal | VMware ESXi, Hyper-V | +| Type 2 | Runs on a host OS | VirtualBox, VMware Workstation | + +**Benefits:** +- **Cost Savings:** One server hosts multiple VMs. +- **Flexibility:** Snapshot and restore VMs for backups. +- **Dev/Test/QA:** Spin up isolated environments fast. +- **Cloud Power:** AWS, GCP, and Azure rely on virtualization. + +**Quick Quiz:** Which hypervisor type is better for production servers? *(Hint: Think bare metal!)* + +--- + +## 3️⃣ Types of Linux Distributions + +Linux comes in flavors called **distributions**, each tailored to specific needs. Here’s a rundown: + +| Distribution | Description | Use Cases | Advantages | Disadvantages | +|--------------|------------------------------|---------------------|------------------------|------------------------| +| Ubuntu | User-friendly Linux distro | Beginners, Developers | Easy to use, LTS support | Slightly bloated for servers | +| Debian | Stable, free software-focused| Servers, Advanced Users | Rock-solid, vast repos | Slower release cycle | +| Fedora | Cutting-edge tech showcase | Developers | Latest features, modular | Frequent updates | +| CentOS | Enterprise-grade stability | Servers | RHEL-like, free, SELinux | End-of-life for CentOS 7 | +| Kali Linux | Security and pentesting toolkit | Security Pros | 600+ tools preloaded | Not for general use | +| RHEL | Supported enterprise solution | Enterprises | Secure, scalable, support | Subscription cost | + +### **When to Use What?** +| Scenario | Ubuntu | RHEL | Kali | +|---------------------|--------|------|------| +| Learning Linux | ✅ | ❌ | ❌ | +| Enterprise production | ❌ | ✅ | ❌ | +| Ethical hacking | ❌ | ❌ | ✅ | + +**Pro Tip:** Match your distro to your goal—Ubuntu for learning, RHEL for enterprise skills! + +--- + +## 4️⃣ Why Learn Linux? +Linux is *the* skill for DevOps, SRE, and Cloud Engineering. Here’s why it’s non-negotiable for DevOps and SRE engineers: + +### Why Linux Matters for DevOps Engineers +- **Automation Foundation:** DevOps is all about automating workflows—Linux’s command-line tools (e.g., `bash`, `awk`, `sed`) and scripting capabilities make it ideal for building CI/CD pipelines (Jenkins, GitLab CI). +- **Containerization:** Docker and Kubernetes run on Linux kernels. Understanding Linux (e.g., namespaces, cgroups) is key to managing containers at scale. +- **Cloud Dominance:** Most cloud providers (AWS, GCP, Azure) use Linux VMs. Deploying apps or infrastructure-as-code (Terraform, Ansible) requires Linux fluency. +- **Efficiency:** Linux’s lightweight nature and customization let DevOps engineers optimize systems for performance and cost—critical for microservices and serverless architectures. + +### Why Linux Matters for SRE Engineers +- **Reliability at Scale:** SREs ensure systems are up 99.99% of the time. Linux’s stability (e.g., Debian, RHEL) and tools like `systemd` and `SELinux` help enforce uptime and security SLAs. +- **Monitoring & Debugging:** Tools like Prometheus, Grafana, and `strace` run natively on Linux, letting SREs trace issues (e.g., memory leaks, network latency) in production. +- **Incident Response:** When systems fail, SREs use Linux commands (`top`, `netstat`, `journalctl`) to diagnose and fix problems fast—often under pressure. +- **Infrastructure Mastery:** SREs manage fleets of servers or clusters (e.g., Kubernetes). Linux’s kernel-level control is essential for tuning performance and handling failures gracefully. + +### Broader Impact +- **Cloud:** AWS, Azure, and GCP run on Linux VMs. +- **Containers:** Kubernetes and Docker are Linux-native. +- **Automation:** CI/CD tools thrive on Linux. +- **Monitoring:** Prometheus and Grafana are Linux-friendly. + +### Roles It Unlocks: +- Site Reliability Engineer (SRE) +- Cloud/DevOps Engineer +- System Administrator + +**Challenge Question:** Can you name a DevOps tool that relies on Linux? *(Hint: Think containers!)* + +--- + +## 5️⃣ Challenge Breakdown + +### **📝 Theoretical Questions** +Answer these to solidify your understanding: +1. What is Unix, and how does it relate to Linux? +2. How does Linux’s open-source nature benefit DevOps workflows? +3. What’s the role of the Linux kernel in virtualization? +4. What’s the difference between Type 1 and Type 2 hypervisors? +5. Why might an SRE prefer Linux for monitoring production systems? + +--- + +### **⚙️ Practical Challenge: Mastering Linux & Virtualization** + +#### **Step 1: Launch EC2 Instances with Linux Distros** +**Goal:** Deploy three Linux distros on AWS EC2 to explore their differences. +1. Log into AWS → EC2 → "Launch Instance." +2. Deploy these AMIs (use `t2.micro` for free tier): + - **Ubuntu 22.04 LTS**: Search "Ubuntu 22.04" in AMI catalog. + - **Amazon Linux 2**: Default EC2 option. + - **RHEL 9**: Search "Red Hat Enterprise Linux 9." +3. Set up a key pair (e.g., `mykey.pem`). +4. Configure security group: Allow port 22 (SSH) from your IP. +5. SSH into each: + ```bash + ssh -i mykey.pem ubuntu@ # Ubuntu + ssh -i mykey.pem ec2-user@ # Amazon Linux + ssh -i mykey.pem ec2-user@ # RHEL + +### Connect via PuTTY (Windows) + +**Step-by-step:** + +1. **Install [PuTTY and PuTTYgen](https://www.chiark.greenend.org.uk/~sgtatham/putty/latest.html)** + Download and install both tools on your Windows machine. + +2. **Convert `.pem` to `.ppk` using PuTTYgen** + - Open **PuTTYgen** + - Click **"Load"** and select your `mykey.pem` file + - Click **"Save private key"** + - Save it as `mykey.ppk` (you can skip the passphrase if not required) + +3. **Open PuTTY and configure connection** + - In **Host Name**, enter: + ``` + ubuntu@ # For Ubuntu + ec2-user@ # For Amazon Linux / RHEL + ``` + - **Port:** `22` + - In the left panel, go to: + `Connection` → `SSH` → `Auth` + - Click **Browse**, and attach your `mykey.ppk` file + +4. **Connect** + - Click **"Open"** + - Accept any security alert + - You should now be logged into your EC2 instance 🎉 + +**You are now connected to your VM via PuTTY!** + + +6. Run this on each: + ```bash + cat /etc/os-release + +--- + +### Repeat the Same on Azure & GCP + +--- + +#### **Azure – Launch a Linux VM** + +1. Go to **[Azure Portal](https://portal.azure.com)** → `Virtual Machines` → **Create VM** +2. Choose your desired Linux distribution: + - Ubuntu + - RHEL + - CentOS +3. For free-tier eligible setup: + - **Size:** `Standard_B1s` +4. **Authentication type:** + - Select **SSH public key** + - Upload or generate a new key pair +5. Under **Networking**, open **Port 22 (SSH)** +6. Click **Review + Create**, then **Create** + +**Connect Options:** +- Use Azure Cloud Shell's SSH option +- Or use **PuTTY / terminal** with your `.pem` or converted `.ppk` key (same as AWS SSH steps) + +--- + +#### **GCP – Launch a Linux VM (Compute Engine)** + +1. Go to **[GCP Console](https://console.cloud.google.com)** → `Compute Engine` → `VM Instances` → **Create Instance** +2. Select your preferred OS: + - Ubuntu + - RHEL + - Debian +3. Choose machine type: + - **e2-micro** (Free tier eligible) +4. Scroll to **Security** section: + - Add your **SSH public key** +5. Under **Firewall**, check: + - ✅ Allow HTTP traffic + - ✅ Allow SSH traffic +6. Click **Create** + +**Connect Options:** +- Use GCP’s built-in browser-based SSH +- Or use **external SSH client / PuTTY**: + ```bash + ssh -i your-key.pem username@ + + +**Troubleshooting Tip:** SSH failing? Double-check your security group and key permissions (`chmod 400 mykey.pem`). + +--- + +### **Step 2: Install RHEL on VirtualBox Locally** +**Goal:** Set up a local RHEL VM to grasp virtualization hands-on. + +**Downloads:** +- VirtualBox: [virtualbox.org](https://www.virtualbox.org) +- RHEL ISO: [redhat.com](https://developers.redhat.com/products/rhel/download) (free developer account). + +1. Open VirtualBox → "New" → Name: "RHEL9" → Type: Linux → Version: Red Hat (64-bit). +2. Allocate: + - RAM: 2 GB+ + - Disk: 20 GB+ (dynamic) +3. Mount the RHEL ISO in Settings → Storage. +4. Start the VM and follow the installer (default settings are fine). +5. Post-install, log in and run: + ```bash + sudo dnf update -y + cat /etc/os-release + + +**Pro Tip:** VM won’t boot? Enable VT-x/AMD-V in your BIOS settings. + +--- + +## 6️⃣ Submission Guidelines +✅ **Proof of Completion:** +- Screenshot of AWS EC2 dashboard with all three instances running. +- Screenshot of RHEL VM terminal after login. +- Output of `cat /etc/os-release` from each system (EC2 + VM). + +✅ **Documentation:** +- Steps you took to launch EC2 instances and set up the VM. +- Any challenges faced (e.g., SSH issues, VM crashes) and how you solved them. + +**Example Submission:** +- Ubuntu output: `NAME="Ubuntu" VERSION="22.04.1 LTS (Jammy Jellyfish)"` +- Screenshot showing all instances in "Running" state. + +--- + +## 7️⃣ Bonus Tasks 🎯 +- **Explore:** Install `htop` on your Ubuntu EC2 (`sudo apt install htop -y`) and screenshot the process list. +- **Compare:** Run `uname -r` on all systems to compare kernel versions. What differences do you spot? + +### **Share Your Progress!** +Post your experience on social media with the hashtags: **#getfitwithsagar, #SRELife, #DevOpsForAll** + +--- + +## **Join Our Community** +Connect with fellow learners: +- **Discord:** [Join here](https://discord.gg/mNDm39qB8t) +- **Google Group:** [Join here](https://groups.google.com/forum/#!forum/daily-devops-sre-challenge-series/join) +- **YouTube:** [Subscribe here](https://www.youtube.com/@Sagar.Utekar) + +--- + +Keep learning and happy exploring! +Sagar Utekar + diff --git a/DailyChallenges/Season2/Day9/1_shell_terminal_windows.md b/DailyChallenges/Season2/Day9/1_shell_terminal_windows.md new file mode 100644 index 0000000..a986bf7 --- /dev/null +++ b/DailyChallenges/Season2/Day9/1_shell_terminal_windows.md @@ -0,0 +1,109 @@ +## About Shells and Terminal Windows + +Linux offers multiple ways to access a shell interface, essential for interacting with the system. Here are three common methods: the shell prompt, terminal windows, and virtual consoles. + +--- + +### **Using the Shell Prompt** +If your Linux system boots without a graphical user interface (GUI) or the GUI isn’t working, you’ll see a plaintext login prompt after starting the system, like this: +``` +Red Hat Enterprise Linux Workstation Release 6.1 (Santiago) +Kernel 2.6.32-131... on X86 +joe login: +``` + +- **After Login:** Enter your username and password. You’ll get a shell prompt—`$` for regular users, `#` for root. +- **Example Prompt:** `[jake@pine share]$` shows username (`jake`), hostname (`pine`), and current directory (`/usr/share`). +- **Customization:** You can tweak the prompt to show info like date or directory (see “Setting your prompt” later in *Linux Bible* Ch. 3). +- **Usage:** Type commands (e.g., `ls`, `whoami`) and press Enter. This is your primary interaction method without a GUI. + +**Note:** `$` means any user can run the command; `#` indicates root privileges are needed (common for admin tasks). + +--- + +### **Using a Terminal Window** +With a GUI running, access a shell via a terminal emulator (Terminal window) from the desktop: +- **Launch Methods:** + - **Right-Click Desktop:** Look for “Open in Terminal” or “Terminal Window” in the context menu (if enabled). + - **Panel Menu:** E.g., in GNOME, go to `Applications ➪ System Tools ➪ Terminal`. +- **Default Emulator:** In Fedora/RHEL with GNOME, it’s GNOME Terminal (`gnome-terminal`). +- **Features:** Beyond basic shell access, GNOME Terminal offers: + - Cut/paste text. + - Custom fonts/colors (Edit ➪ Profiles ➪ General/Colors tabs). + - Background images or transparency. + - Scrollback buffer settings. + +**Quick Demo (Fedora/RHEL):** +1. Open `Applications ➪ System Tools ➪ Terminal`. +2. Edit profile: Change font (General tab), tweak colors (Colors tab), or set a background image (Profile window). +3. Close when done—ready to use! + +**Why Use It?** Most GUI users access shells this way for convenience. + +--- + +### **Using Virtual Consoles** +Virtual consoles provide multiple shell sessions alongside the GUI: +- **Access:** Press `Ctrl+Alt+F1` to `F7`. + - **Fedora Example:** `Ctrl+Alt+F1` = GUI, `F2`–`F7` = text-based consoles. + - **Variation:** Some systems use `F5` or `F7` for GUI (e.g., `Ctrl+Alt+F7` to return). +- **Try It:** + 1. Press `Ctrl+Alt+F3` → see a text login prompt. + 2. Log in, run commands (e.g., `whoami`), then `exit`. + 3. Return to GUI with `Ctrl+Alt+F1`. + +**Benefit:** Run multiple shells without a GUI window, ideal for troubleshooting or multi-tasking. + +--- + +### **Key Takeaways** +- **Shell Prompt:** Direct, no-GUI access—perfect for minimal systems. +- **Terminal Window:** GUI-friendly shell with extra features. +- **Virtual Consoles:** Multiple text-based shells, accessible via shortcuts. +Start exploring by booting your Linux system and trying these methods! + + +In most Linux systems, the default shell is **Bash** (`/bin/bash`). +## 🐚 Trying Other Shells + +Other shells (`ksh`, `tcsh`, `csh`, `sh`, `dash`, etc.) might be installed on your system. + +--- + +### 🔍 Test One +- Open a terminal and type the name of the shell (e.g., `ksh`). +- Try a few commands. +- Type `exit` to return to your default shell (e.g., Bash). + +--- + +### 🤔 Why Switch? + +- **Familiarity**: + Users from different UNIX backgrounds may prefer different shells. + - *System V* users → `ksh` + - *Berkeley UNIX* users → `csh` + +- **Scripts**: + Some scripts are written specifically for a certain shell (e.g., `csh` scripts need `csh` to run properly). + +- **Features**: + Certain shells have unique features that may appeal to specific users. + Example: `ksh` offers more flexible alias handling than Bash. + +--- + +### 🧠 Why Bash? + +- **Default**: + Bash is the default shell on most Linux distributions (e.g., Fedora, RHEL). + Exceptions include embedded systems using lighter shells like `dash`. + +- **Features**: + Bash blends features from `sh`, `ksh`, and `csh`, making it powerful and versatile. + +- **Learnability**: + Learning one shell—especially Bash—helps you easily transition to others. + Use the manual pages to explore: + ```bash + man bash diff --git a/DailyChallenges/Season2/Day9/2_recalling_commands_history.md b/DailyChallenges/Season2/Day9/2_recalling_commands_history.md new file mode 100644 index 0000000..cd8a763 --- /dev/null +++ b/DailyChallenges/Season2/Day9/2_recalling_commands_history.md @@ -0,0 +1,71 @@ +## Recalling Commands Using Command History + +Repeating or modifying commands from your shell session is a powerful way to save time and effort. The Bash shell provides a **command history** feature that stores commands you’ve entered, allowing you to recall, edit, and reuse them. This is especially handy for correcting typos in long commands or reusing complex pipelines without retyping. + +--- + +### **Understanding the History List** +The shell maintains a list of your previous commands, accessible via the `history` command: +**View History:** + ```bash + history + ``` +Displays all stored commands (up to a default of 1,000, configurable via HISTSIZE). + +**Limit Output:** +```bash +history 10 +``` +Shows the last 10 commands, each prefixed with a number (e.g., 382 date). + +**Storage:** Commands are held in memory during the session and saved to ~/.bash_history upon exit. +**Example Output:** + + ```bash +382 date +383 ls /usr/bin | sort -f | less +384 history 10 +``` + +### **Command-Line Editing in Bash** +Bash lets you edit recalled commands instead of starting from scratch. +By default, it uses Emacs-style key bindings, but you can switch to Vi-style for a different experience. + +**Set Vi Mode** +Add the following to your ~/.bashrc file: + + ```bash +set -o vi +Then, reload your shell: + + ```bash +source ~/.bashrc +``` +**In Vi mode:** Press Esc to enter command mode, then use Vi keys (e.g., i to insert). + +**Emacs Editing Example** +**Modify this command:** + + ```bash +ls /usr/bin | sort -f | less +``` + +**To:** + + ```bash +ls /bin | sort -f | less +``` + +**Steps:** +Press ↑ (up arrow) to recall the previous command. + +Press Ctrl+A to move to the start of the line. + +Press Ctrl+F (or →) to move under the first /. + +Press Ctrl+D four times to delete /usr. + +Type bin, then press Enter. + + +Screenshot 2025-04-08 at 8 03 36 AM diff --git a/DailyChallenges/Season2/Day9/readme.md b/DailyChallenges/Season2/Day9/readme.md new file mode 100644 index 0000000..ddb593d --- /dev/null +++ b/DailyChallenges/Season2/Day9/readme.md @@ -0,0 +1,311 @@ +# Daily DevOps + SRE Challenge Series – Season 2 +## Day 9: Mastering the Linux Shell + +## Introduction +Welcome to Day 9 of the Daily DevOps SRE Challenge Series - Season 2! 🎉 + +Today, we’re diving into **the Linux Shell**, You’ll explore OS architecture, master Bash commands, edit with Vim, and harness help tools. By the end, you’ll: +- Grasp Linux’s layered structure and the shell’s critical role. +- Master commands, redirection, piping, history, variables, aliases, and Vim. +- Apply these on your Day 1 EC2 instances and RHEL VM. + +--- + +### **Why This Matters?** +The shell is the backbone of Linux system administration, DevOps, and SRE. It’s not just a tool—it’s a philosophy of control and automation. Here’s why: +- **Universal Interface:** Every Linux system—cloud VMs, containers, or bare metal—has a shell. It’s your consistent entry point, unlike GUIs that vary or may not exist. +- **Automation Powerhouse:** Shell scripts drive CI/CD pipelines (e.g., `bash deploy.sh` in Jenkins), cron jobs (`0 0 * * * backup.sh`), and container startups (`CMD ["/bin/bash", "start.sh"]`). +- **SRE Debugging:** During outages, `tail -f log | grep error` or `ps aux | grep nginx` isolates issues in seconds, bypassing slow dashboards. +- **RHCSA Requirement:** The exam tests shell fluency—redirection (`2>&1`), syntax (`ls -l`), file editing (`vim`), and help usage (`man`). +- **Efficiency Multiplier:** One-liners like `find / -name "*.conf" | xargs grep "port"` save hours over manual searches. +- **Career Edge:** Shell mastery signals expertise—recruiters ask, “How’s your Bash?” because it’s foundational to cloud (AWS CLI), Kubernetes (`kubectl exec`), and Ansible (`command` module). + +**Real-World Example:** An SRE at a major cloud provider used `for i in $(cat servers.txt); do ssh $i 'uptime'; done` to check 100 servers in seconds, averting a downtime escalation. + +--- + +## 1️⃣ Linux OS Architecture + +Linux is a modular, layered system, with each layer abstracting complexity for the next: + +![linux](https://github.com/user-attachments/assets/4bba243a-b552-451b-9891-3fdf6896bc6a) + + +| **Layer** | **Role** | **Example** | +|---------------------|---------------------------------------|-----------------------| +| **Hardware** | Physical devices | CPU, RAM, SSDs | +| **Kernel** | Core OS, manages resources | `sched`, `ext4` | +| **Libraries** | Standard functions for apps | `glibc`, `libpthread` | +| **Shell** | Command interpreter | `bash`, `zsh` | +| **Applications** | User tools and services | `ls`, `nginx` | + +- **Hardware:** Raw compute—CPUs execute instructions, disks store data. +- **Kernel:** The brain, handling system calls (e.g., `open()` for files), process scheduling, and device drivers. It’s a translator between software and hardware. +- **Libraries:** Provide reusable code—e.g., `printf()` in `glibc` lets `ls` format output without reinventing string handling. +- **Shell:** Your CLI gateway, parsing commands like `cat file | grep "error"` into kernel instructions via libraries. It’s both interactive (type `whoami`) and scriptable (`#!/bin/bash`). +- **Applications:** Tools like `vim` or `docker`, built on libraries, invoked via the shell. + +**How It Flows:** Typing `echo "hi" > file` in Bash: +1. Shell parses `echo` (internal command) and `>` (redirection). +2. Libraries handle string output (`hi`). +3. Kernel writes to disk via filesystem drivers. + +**Why It’s Elegant:** This separation lets you swap shells (Bash to Zsh), upgrade kernels, or run apps on different hardware without rewriting everything. It’s why Linux powers phones, servers, and supercomputers. + +--- + +## 2️⃣ Why Learn Shell Commands? + +Shell commands are the Swiss Army knife for DevOps and SRE: +- **Granular Control:** `kill -HUP $(pidof nginx)` reloads configs without downtime, unlike restarting via GUI. +- **Automation Foundation:** Scripts like `while true; do curl -s server; sleep 10; done` monitor health, feeding into CI/CD or alerting systems. +- **Incident Response:** `netstat -tulpn | grep 80` finds port conflicts during outages—faster than logs. +- **Data Processing:** `awk '{print $2}' access.log | sort | uniq -c` counts unique IPs in seconds, no Python needed. +- **RHCSA Must-Have:** Exam tasks demand `>`, `|`, `2>`, and correct syntax (e.g., `ls -lR /etc`). +- **Cloud Dominance:** AWS CLI (`aws s3 cp`), Terraform (`local-exec`), and Kubernetes (`kubectl`) lean on shell for execution and scripting. +- **Minimal Footprint:** Shell runs on tiny systems where GUIs can’t—think IoT or containerized apps. + +**Historical Context:** Shells evolved from 1970s Unix, prioritizing text-based efficiency. Today, Bash is ubiquitous because it’s powerful yet simple—learn it once, use it everywhere. + +**Career Impact:** Shell fluency cuts task time (e.g., `find / -mtime -1` vs. manual search), impresses in interviews, and unlocks advanced tools like `jq` or `yq` for JSON/YAML parsing. + +--- + +## 3️⃣ Shell Essentials + +This section unifies *Linux Bible* Chapter 3 and RHCSA Chapter 2, diving deep into Bash’s core features. + +#### **Shell Access** +The shell is your interface to Linux, accessible via: +- **Terminals:** GUI apps like GNOME Terminal or Konsole. +- **SSH:** Remote access (`ssh user@host`). +- **Virtual Consoles:** Ctrl+Alt+F2–F6 on physical machines. +- **TTY Devices:** `/dev/tty1` for console, `/dev/pts/0` for SSH sessions. +**Theory:** Shells like Bash interpret commands, sending them to the kernel. They’re lightweight, running even on minimal systems, unlike resource-heavy GUIs. Bash’s dominance stems from its GNU roots, balancing interactivity and scripting. + +#### **Command Structure** +Commands follow: `command -options arguments`. +- **Examples:** `ls -la /etc` (`-la` options, `/etc` argument), `grep -i "error" log` (`-i` ignores case). +- **Options:** Short (`-l`), long (`--all`), or combined (`-la`). +- **Arguments:** Files, dirs, or text the command acts on. +**Theory:** This structure is universal across Unix-like systems, rooted in 1970s design for flexibility. Options modify behavior (e.g., `ls -R` for recursive), while arguments specify targets. Bash parses these, resolving aliases first, then built-ins, then PATH. + +#### **Command Types** +Bash distinguishes three command types: +- **Aliases:** User-defined shortcuts, e.g., `alias ll='ls -la'`—checked first, overriding others. Stored in memory, reset on logout unless in `~/.bashrc`. +- **Internal (Built-in):** Part of Bash, e.g., `cd`, `echo`. Fast, no disk access. `type cd` shows “builtin”. +- **External:** Executables in `$PATH`, e.g., `/bin/ls`. Slower due to disk I/O. `which ls` finds them. +**Theory:** Aliases prioritize customization, built-ins optimize speed, and externals enable extensibility. This hierarchy ensures flexibility—e.g., `type ls` might show an alias, but `/bin/ls` forces the binary. `$PATH` dictates external command lookup, excluding `.` for security (use `./script`). + +#### **PATH** +`$PATH` lists directories for external commands: `echo $PATH` (`/bin:/usr/bin`). +- **Mechanics:** Bash searches left to right for matches. +- **Security:** Current dir (`.`) isn’t included—prevents accidental script execution. Use `./myscript`. +**Theory:** `$PATH` is a core environment variable, balancing accessibility and safety. Root’s `$PATH` includes `/sbin` for admin tools. Misconfigured `$PATH` can break commands (`command not found`), critical for troubleshooting. + +#### **I/O Redirection** +Redirection manages input/output streams: +- **STDIN (0):** `< file` (e.g., `sort < data.txt`). +- **STDOUT (1):** `> file` (overwrite), `>> file` (append), e.g., `ls > out.txt`. +- **STDERR (2):** `2> file`, `2>&1` (merge with STDOUT), e.g., `ls nofile 2> err.txt`. +- **Examples:** `ls nofile /etc > out.txt 2>&1` (all output to one file), `cat < input.txt > output.txt`. +**Theory:** File descriptors (0, 1, 2) are kernel abstractions, inherited by processes. Redirection lets programs operate agnostically—`ls` doesn’t know it’s writing to `/dev/null` vs. a file. `/dev/null` discards output, `/dev/tty` targets terminals. Redirection is key for automation, logging, and error suppression (e.g., `2> /dev/null` in cron). + +#### **Pipes** +Pipes (`|`) connect STDOUT of one command to STDIN of another: `ls -l | grep ".txt" | sort`. +- **Chaining:** `cat log | grep "error" | wc -l` counts error lines. +- **Power:** Combines simple tools into complex workflows. +**Theory:** Pipes embody Unix’s “do one thing well” philosophy—each command is a filter, transforming data streams. Unlike redirection (file-based), pipes operate in memory, enabling real-time processing (e.g., `tail -f log | grep "fail"`). They’re foundational for log analysis and data munging. + +#### **History Tricks** +Bash stores ~1,000 commands in `~/.bash_history`, updated on session close. +- **View:** `history` (all), `history 10` (last 10). +- **Recall:** `!5` (run #5), `!!` (last), `!ls` (last starting with `ls`). +- **Search:** Ctrl+R, type `ls`, cycle with Ctrl+R. +- **Edit:** `history -d 299` (delete line), `history -c` (clear memory), `history -c; history -w` (clear file too). +**Theory:** History enhances productivity, letting you reuse complex commands (e.g., `!curl` for API tests). Clearing sensitive entries (`history -d`) is crucial for security—passwords typed in cleartext (e.g., `mysql -psecret`) persist otherwise. Saved to disk only on exit, so `history -c` alone isn’t enough. +Screenshot 2025-04-08 at 8 08 03 AM + +#### **Bash Completion** +Tab completion works for commands, files, and variables: +- **Examples:** `ec` → `echo`, `cat /et` → `/etc/`, `$PA` → `$PATH`. +- **Multi-Tab:** `ls --` lists options like `--all`, `--color`. +**Theory:** Completion leverages Bash’s parsing, querying `$PATH`, filesystem, and env vars. It’s programmable—packages like `bash-completion` extend it to tools like `git` or `kubectl`. Speeds up workflows, reduces typos, and reveals options dynamically. + +#### **Variables & Environment** +Variables store dynamic values: `NAME=value`, `echo $NAME`. +- **Types:** + - **Shell Vars:** Local, e.g., `MYDIR=/tmp`. + - **Env Vars:** Exported, e.g., `export PATH`, inherited by child processes. + - You can display the value of a specific environment variable by using the echo command followed by the variable name preceded by a dollar sign ($). For example: +```bash +echo $USER +``` +- To refer to the value of an environment variable in a command simply prefix the variable name with a dollar sign ($). For example: +```bash +echo $USER +``` +- **Key Vars:** `PATH`, `HOME`, `LANG` (e.g., `en_US.UTF-8` for locale), `PS1` (prompt). +- **Config Files:** + - `/etc/profile`: Global login shell settings. + - `/etc/bashrc`: Global subshell settings. + - `~/.bash_profile`: User login shell (calls `~/.bashrc`). + - `~/.bashrc`: User subshells, aliases, custom vars. +Screenshot 2025-04-08 at 8 14 04 AM + +- **Messages:** `/etc/motd` (post-login notice), `/etc/issue` (pre-login banner). +**Theory:** Variables create a tailored environment—`LANG` affects date formats, `PATH` enables commands. Config files build a hierarchy: system-wide (`/etc`) then user-specific (`~`). Subshells (e.g., scripts) inherit login shell vars unless overridden. `/etc/motd` is ideal for admin alerts (e.g., “Maintenance at 2AM”). + +#### **Aliases** +Shortcuts for commands: `alias ll='ls -la'`. +- **Scope:** Session unless in `~/.bashrc`. +- **Check:** `alias` lists all. +**Theory:** Aliases override built-ins/externals, speeding repetitive tasks (e.g., `alias k='kubectl'`). They’re checked first in command resolution, making them powerful but risky—`alias rm='rm -i'` adds safety, but `alias ls='rm'` could be disastrous. + +#### **Vim Basics** +Vim is RHCSA-essential for file editing: +- **Modes:** Command (Esc), Input (`i`, `a`, `o`). +- **Commands:** `:wq` (save+quit), `:q!` (discard), `dd` (delete line), `yy` (copy), `p` (paste), `:%s/old/new/g` (replace all), `u` (undo). +- **Navigation:** `gg` (top), `G` (bottom), `/text` (search). +**Theory:** Vim’s modal design optimizes keystrokes—Command mode for navigation/ops, Input for typing. It’s universal, lightweight, and scriptable (e.g., `vim +:%s/foo/bar/g file`). RHCSA expects fluency—editing `/etc/fstab` under pressure demands `i`, `:wq`, and `:q!`. Syntax highlighting (`vim-enhanced`) aids debugging. + +#### **Help Tools** +Finding command info is critical: +- **`man`:** `man ls` (details), `man -k disk` (search summaries), `man 5 passwd` (file formats). +- **`--help`:** `ls --help` (quick options). +- **`info`:** `info coreutils` (in-depth, hierarchical), better with `pinfo`. +- **`/usr/share/doc`:** Service docs, e.g., `/usr/share/doc/rsyslog`. +- **`mandb`:** Updates `man -k` database (`sudo mandb`). +**Theory:** `man` pages are structured—NAME, SYNOPSIS, DESCRIPTION, EXAMPLES—rooted in Unix tradition. `man -k` searches `mandb`, but stale databases return “nothing appropriate.” `info` suits complex tools (e.g., `gdb`), while `/usr/share/doc` holds configs/samples. RHCSA tests help navigation—know `man -k`, `/example`, and `G` to jump to examples. + +**Pro Tip:** `type -a ls` shows all matches (alias, built-in, external) vs. `which ls` (just `$PATH`)! + +--- + + +## 4️⃣ Challenge Breakdown + +### **📝 Theoretical Questions** +1. How does the shell bridge applications and the kernel? +2. Explain `2>&1` vs. `2> file` with an example. +3. Why does `myscript` fail but `./myscript` work? +4. How does `history -c; history -w` differ from `history -c` alone? +5. Why is `/etc/motd` useful for admins but not GUI users? + +--- + +### **⚙️ Practical Challenge: Shell Skills in Action** + +#### **Step 1: Shell & Redirection on EC2 Instances** +**Goal:** Master commands and I/O on Day 1 EC2s (Ubuntu, Amazon Linux, RHEL). + +1. SSH into each: +``` + ssh -i mykey.pem ubuntu@ # Ubuntu + ssh -i mykey.pem ec2-user@ # Amazon Linux + ssh -i mykey.pem ec2-user@ # RHEL +``` + +2. Run these: +``` +type ls # Alias or binary? +ls -l /etc > out.txt # Redirect STDOUT +ls nofile 2> err.txt # Redirect STDERR +cat out.txt err.txt > combined.txt 2>&1 # Merge outputs +echo "Host: $(hostname)" >> combined.txt # Append +ls -R / | grep "bin" | less # Pipe chain +history -d 2 # Delete 2nd command +``` + +3. Capture combined.txt content. +**Troubleshooting Tip:** ls: command not found? Check $PATH with echo $PATH. +```bash +#!/bin/bash +REPORT="sys_$(date +%Y%m%d).txt" +echo "Check - $(date)" > "$REPORT" +echo "User: $USER" >> "$REPORT" +uptime -p | tee -a "$REPORT" # Pipe and append +df -h / | tail -n 1 >> "$REPORT" 2>/dev/null +``` + +#### **Step 2: Vim & Scripting on RHEL VM** +**Goal:** Edit and automate with Vim on your Day 1 RHEL VM. + +1. Boot RHEL VM, log in. +2. Create/edit a script: +``` +vim ~/monitor.sh +``` + +3. Input (press i): +``` +#!/bin/bash +REPORT="sys_$(date +%Y%m%d).txt" +echo "Check - $(date)" > "$REPORT" +echo "User: $USER" >> "$REPORT" +uptime -p | tee -a "$REPORT" # Pipe and append +df -h / | tail -n 1 >> "$REPORT" 2>/dev/null +``` + +4. Edit: Esc, :3 (line 3), dd (delete), u (undo), :wq (save+quit). +5. Run: +``` +chmod +x monitor.sh +./monitor.sh +cat "$REPORT" +``` + +6. Persist an alias: +``` +echo "alias m='./monitor.sh'" >> ~/.bashrc +source ~/.bashrc +``` + +**Pro Tip:** Vim stuck? Esc, :q! forces exit! + + +## 5️⃣ Submission Guidelines +✅ **Proof of Completion:** +- Screenshot of EC2 terminal with combined.txt content. +- Screenshot of RHEL VM terminal showing sys_*.txt content. +- Text output of both files. +✅ **Documentation:** +- Steps for both tasks. +- Issues (e.g., Permission denied) and fixes (e.g., sudo). + +**Example Submission:** + +**EC2 combined.txt:** +``` +dir1 dir2 file1 +ls: nofile: No such file +Host: ip-10-0-0-1 +``` + +**RHEL VM sys_20250407.txt:** +``` +Check - Mon Apr 07 2025 +User: ec2-user +up 1 hour +/dev/sda1 20G 5G 15G 25% / +``` + +## 6️⃣ Bonus Tasks 🎯 +- Help Hunt: Run man -k disk | grep 8—screenshot admin commands. +- Vim Swap: In monitor.sh, use :%s/User/USER/g—screenshot result. + +### **Share Your Progress!** +Post your experience on social media with the hashtags: **#getfitwithsagar, #SRELife, #DevOpsForAll** + +--- + +## **Join Our Community** +Connect with fellow learners: +- **Discord:** [Join here](https://discord.gg/mNDm39qB8t) +- **Google Group:** [Join here](https://groups.google.com/forum/#!forum/daily-devops-sre-challenge-series/join) +- **YouTube:** [Subscribe here](https://www.youtube.com/@Sagar.Utekar) + +--- + +Keep learning and happy exploring! +Sagar Utekar diff --git a/DailyChallenges/Season2/Linux/Install Linux.txt b/DailyChallenges/Season2/Linux/Install Linux.txt deleted file mode 100644 index 64d637e..0000000 --- a/DailyChallenges/Season2/Linux/Install Linux.txt +++ /dev/null @@ -1,41 +0,0 @@ -Why Virtualization? -Many a times, we need to or prefer running Linux for development or for personal use. But it is not always possible to allocate a seperate machine for running Linux. There are multiple ways of running Linux without having a seperate machine for the same. One of the ways is to dual boot a system wherein you make a choise of which operating system to use when staring a machine. But if you do not want to go the dual boot way, then creating a virtual machine or creating a VM on a cloud system line AWS is alao an option. Mentioned below are the two ways on creating a linux instance for your perusal. - -1. Install using Oracle Virtual Box - 1.1 First step is to download Oracle Virtual Box. - A. Go to https://www.oracle.com/in/virtualization/technologies/vm/downloads/virtualbox-downloads.html. - B. Download the appropriate version on Virtual Box based on the operating system being used. [For this tutorial, we will assume that Windows is the base operating system being used]. - C. Since the operating system is assumed to be Windows, double click on the installer and follow the on-screen instructions for install Oracle Virtual Box. You can also follow the detailed instruction on [https://www.virtualbox.org/manual/] this page. - 1.2 Second Step is to download the latest version of Ubuntu - A. Download the latest version of Ubuntu from https://ubuntu.com/download/desktop - B. By default, on Windows, the ISO image will downloaded under the Download folder. - C. Create a separate folder on your hard disk and move the downloaded file to the same. - 1.3 Creating a New VM - [Before starting this step, please ensure that you have downloaded and installed Oracle Virtual Box as well as downloaded the latest version of Ubuntu and moved the same from the Download folder to a sepearte folder.] - A. Start Oracle Virtual Box. - B. Click on the New Icon or Select Machine -> New from the Menu Bar. - C. Give a suitable to your Virtual Machine. In case you plan to use multiple Linux OS's then naming them makes it easy to access them. In this case, name the virtual machine as Ubuntu. - D. The next step is the specify where will the VM file be stored. In case you have multiple drives, Select/Create a folder on the drive which has a reasonable amount of space. - E. Next step is to select the OS downloade by us as part of Step 1.2. Click on the Down Button and then Click on Other option. Navigate to the folder where the downloade Ubuntu ISO Image has been kept and select the same. - F. Click on Unattended Install Option. This will help us install Ubuntu unattened. - F.1 Enter an User Name. - F.2 Enter and then confirm a password. - G. Select the Hardware Option. Here you can define the amount of Ram and the number of CPU's to be used by Vurtual Box. - H. Select the Hardisk option and define the amount of space to the used on the Hard Disk based on availability. - I. Click on Finish to complete the configuration process. - J. Click on Start button the main screen to start the installation process. Note: This is a one time process that happens when you start the Ubuntu Virtual Machine the first time. - K. You can also refer to this tutorial [https://ubuntu.com/tutorials/how-to-run-ubuntu-desktop-on-a-virtual-machine-using-virtualbox#1-overview] from Ubuntu for a detailed understanding of the process. -2. Creating VM om AWS - 1.1. Create an accoount on AWS. - 1.2. Log in to your AWS account and navigate to the EC2 service in the AWS Management Console - 1.3. Give a relevant name for your instance. [eg: My Linux Machine] - 1.4. Choose an Instance Type. For this tutorial, I have selected an Ubuntu Instance. - 1.5. Choose an appropriate instance type based on your requirement. - 1.6. Create a Key Pair to securly login to your instance. For this click on the Create New Key Pair button and select RSA. Give a relevant Key Pair name to identify the same. Select PEM file for Private Key File Format. This file will be downloaded to your Download directory. Shift the same to a more secure location on your har disk. - 1.7. Under Network Setting, - 1.7.1. Select Create Security Group. This is selected by default. In case, you have already created a Security Group, you can use the same. - 1.7.2. Select the Allow SSH Traffic From Check Box. Here, select My IP from the drop down. - 1.8. Configure the storage in case your use case demands additional storage requirment. If not, go with the default. - 1.9. Click on Launch Instance to start the Ubuntu Machine. - 1.10. Click on the connect button to connect to the Ubuntu Instance. - diff --git a/QOTD/answers.md b/QOTD/answers.md new file mode 100644 index 0000000..0ae3820 --- /dev/null +++ b/QOTD/answers.md @@ -0,0 +1,19 @@ +=Question: What is blue-green Deployment? How is it useful? + +Answer: Blue-green deployment is a methodology for releasing software in a controlled and low-risk manner. It involves creating two environments, referred to as blue and green. The blue environment is the live production environment, serving all client traffic, while the green environment is an identical clone used to test and validate the new version of the application. + +Once the new software version is ready and fully tested on the green environment, the traffic is switched (or gradually moved in case of canary testing) from the blue environment to the green environment. With this method, if there's a need to rollback, it's as simple as switching back the traffic to the previous version. + +This strategy is useful because it minimizes downtime during application deployment. The application remains available to users throughout the entire process. It also offers a failsafe—should something go wrong, rollback is easy and instantaneous by re-routing the traffic back to the blue environment (previous version). + +Diagram/Example: + + +In this diagram, initially, all requests are routed to the blue environment. When a new software version is ready, it is deployed to the green environment. After it passes all tests, the router starts routing the requests to the green environment. If the new version fails in production for some reason, the router goes back to routing requests to the blue environment. +``` mermaid +graph LR +A[User] -- Request --> B{{Router}} +B -- Route to --> C[Blue Environment] +B -- Route to --> D[Green Environment] +style C fill:#0000ff,stroke:#333,stroke-width:4px +style D fill:#008000,stroke:#333,stroke-width:4px \ No newline at end of file diff --git a/submission_tracker.md b/submission_tracker.md deleted file mode 100644 index e3ba9f7..0000000 --- a/submission_tracker.md +++ /dev/null @@ -1 +0,0 @@ -# Day 2 submissions list