Configuration Checks and Resource Validation for HA Configuration Validation (#146)

Author: devanshjainms

.ansible-lint

@@ -4,6 +4,7 @@ exclude_paths:
   - .github
   - src/roles/ha_db_hana/tasks/files/
   - src/roles/ha_scs/tasks/files/
+  - src/roles/configuration_checks/tasks/files/
 
 skip_list:
   - no-handler

README.md

@@ -11,52 +11,65 @@
 
 ## 🔍 Overview
 
-The SAP Testing Automation Framework is an open-source orchestration tool designed to validate SAP deployments on Microsoft Azure. It enables you to assess system configurations against SAP on Azure best practices and guidelines. Additionally, the framework facilitates automation for various testing scenarios, including High Availability (HA) functional testing.
-
-> **NOTE**: This repository is currently in public preview and is intended for testing and feedback purposes. As this is an early release, it is not yet production-ready, and breaking changes can be introduced at any time.
+The SAP Testing Automation Framework is an open-source orchestration tool designed to validate SAP deployments on Microsoft Azure. It enables you to assess system configurations against SAP on Azure best practices and guidelines, and facilitates automation for various testing scenarios.
 
 ![SAP Testing Automation Framework](./docs/images/sap-testing-automation-framework.png)
 
-## Supported Configuration Matrix
+## 📊 Key Scenarios
+
+SAP Testing Automation is designed as a scalable framework to orchestrate and validate a wide spectrum of SAP landscape scenarios through repeatable, policy-driven test modules. The framework currently offers following scenarios -
+
+> [!NOTE]
+>
+> The High Availability testing scenario in the SAP Testing Framework is **generally available (GA)**, while the Configuration Checks scenario is currently in **public preview**.
 
-The following SAP components are supported in a two-node Pacemaker cluster running on SUSE Linux Enterprise Server (SLES) or Red Hat Enterprise Linux (RHEL):
+### High Availability Testing
 
-- **SAP HANA Scale-Up**
-- **SAP Central Services**
+In the SAP Testing Automation Framework, thorough validation of high availability SAP HANA scale-up and SAP Central Services failover mechanism in a two node pacemaker cluster can be performed, ensuring the system operates correctly across different situations.
 
-For additional information on supported configuration patterns, such as cluster types (Azure Fence Agent or SBD) and storage options (Azure Files or Azure NetApp Files) in this automated testing framework, refer to [supported high availability configuration](./docs/HIGH_AVAILABILITY.md).
+- **High Availability Configuration Validation:** The framework helps to ensure that SAP HANA scale-up and SAP Central Services configurations and load balancer settings are compliant with SAP on Azure high availability configuration guidelines.
+- **Functional Testing:** The framework executes series of real-world scenarios based on the SAP HANA and SAP Central Services high availability setup to identify potential issues, whether during a new system deployment or before implementing cluster changes in a production environment. The test cases are based on what is documented in how-to guides for SAP HANA and SAP Central Services configuration.
+- **Offline configuration validation:** Offline validation is a mode of the framework that validates SAP HANA and SAP Central Services high availability cluster configurations without establishing a live SSH connection to the production cluster. Instead, it analyzes captured cluster information base (CIB) XML files exported from each cluster node.
 
-## 📊 Key Features
+### Configuration Checks (Preview)
 
-- **High Availability Testing** - Thorough validation of the SAP HANA scale-up and SAP Central Services failover mechanism in a two node pacemaker cluster, ensuring the system operates correctly across various test cases.
-  - **Configuration Validation** - Ensures that SAP HANA scale-up and SAP Central Services configurations comply with SAP on Azure best practices and guidelines.
-  - **Functional Testing** - Executes test scenarios on the high availability setup to identify potential issues, whether during a new system deployment or before implementing cluster changes in a production environment.
-- **Detailed Reporting** - Generates comprehensive reports, highlighting configuration mismatch or deviations from recommended best practices. Includes failover test outcomes, any failures encountered, and logs with insights to aid in troubleshooting identified issues.
+The framework performs comprehensive configuration checks to ensure that the SAP system and its components are set up according to [SAP on Azure best practice](https://learn.microsoft.com/azure/sap/). This includes validating infrastructure settings, operating system parameter configurations, and network settings, in addition to the cluster configuration, to identify any deviations that could impact system performance or reliability.
+
+- **Infrastructure Validation:** This includes validating the underlying infrastructure components, such as virtual machines, load balancer, and other resource configurations, to ensure they meet the requirements for running SAP workloads on Azure.
+- **Storage Configuration Checks:** It validates settings of disks, storage accounts, Azure NetApp Files, including throughput, performance, and stripe size.
+- **Operating System and SAP Parameter Validation:** The framework checks critical operating system parameters and SAP kernel settings to ensure they align with recommended configurations.
+- **Cluster Configuration Validation:** This framework ensures that the high availability cluster resource settings adhere to best practices for high availability and failover scenarios.
+
+The framework generates comprehensive reports, highlighting configuration mismatch or deviations from recommended best practices. For high availability scenarios, the report includes failover test outcomes, any failures encountered, and logs with insights to aid in troubleshooting identified issues.
 
 ## 🏆 Purpose
 
 Testing is crucial for keeping SAP systems running smoothly, especially for critical business operations. This framework helps by addressing key challenges:
 
-- **Preventing Risks** - It simulates system failures like node crashes, network issues, and storage failures to check if recovery mechanisms work properly, helping to catch problems before they affect real operations.
-- **Meeting Compliance Requirements** - Many businesses need to prove their SAP systems are reliable. This framework provides detailed reports and logs that help with audits and ensure compliance with internal and regulatory standards.
-- **Ensuring Quality** -  The framework runs automated tests to verify whether the failover behavior of SAP components functions as expected on Azure across various test scenarios. It also ensures that the cluster and resource configurations are set up correctly, helping to maintain system reliability.
-- **Automating Testing**: Manually testing high availability (HA) setups is slow and error-prone. This framework automates the process—from setup to reporting—saving time and ensuring more accurate and consistent results.
+- **Risk Prevention** - The high availability testing helps simulate system failures like node crashes, network issues, and storage failures to check if recovery mechanisms work properly, helping to catch problems before they affect real operations. Configuration validation detects misalignments with SAP on Azure best practices early.
+- **Compliance Requirements** - Many businesses need to prove their SAP systems are reliable. This framework provides detailed reports and logs that help with audits and ensure compliance with internal and regulatory standards.
+- **Quality Assurance** - The framework runs automated tests to verify whether the failover behavior of SAP components functions as expected on Azure across various test scenarios. It also ensures that the cluster and resource configurations are set up correctly, helping to maintain system reliability.
+- **Test Automation** - Manually validating overall SAP systems' configurations and high availability (HA) setup is slow and error-prone. This framework automates the process, from setup to reporting, saving time and ensuring more accurate and consistent results.
 
-## 🚦 Get Started
+## 🏗️ Architecture and Components
+
+To learn how the framework works, refer to the [architecture and components](./docs/ARCHITECTURE.md) documentation.
 
-There are two primary ways to get started with the SAP Testing Automated Framework. Choose the path that best fits your current environment and objectives:
+## 🚦 Get Started
 
-### Option 1: [Integration with SAP Deployment Automation Framework (SDAF)](./docs/SDAF_INTEGRATION.md)
+There are two primary ways to get started with the SAP Testing Automation Framework. You can choose the path that best fits your current environment and objectives:
 
-If you already have [SDAF](https://github.com/Azure/sap-automation) environment set up, integrating the SAP Testing Automation Framework is a natural extension that allows you to leverage existing deployment pipelines and configurations.
+### Option 1: Standalone Setup of SAP Testing Automation Framework
 
-### Option 2: [Getting Started with High Availability Testing (Standalone)](./docs/HIGH_AVAILABILITY.md)
+For users focused solely on validating SAP functionality and configurations, the standalone approach offers a streamlined process to test critical SAP components without the complexity of full deployment integration. For more details on the setup, see following documents to get started -
 
-For users focused solely on validating SAP functionality and configurations, the standalone approach offers a streamlined process to test critical SAP components without the complexity of full deployment integration.
+- Configure management server following the document [Setup Guide for SAP Testing Automation Framework](https://github.com/Azure/sap-automation-qa/blob/main/docs/SETUP.MD).
+  - For high availability testing scenarios, see [High Availability documentation](./docs/HIGH_AVAILABILITY.md).
+  - For Configuration Checks and Testing details, see the [Configuration Checks documentation](./docs/CONFIGURATION_CHECKS.md).
 
-## 🏗️ Architecture and Components
+### Option 2: Integration with SAP Deployment Automation Framework (SDAF)
 
-To learn how the framework works, refer to the [architecture and components](./docs/ARCHITECTURE.md) documentation.
+If you already have an [SAP Deployment Automation Framework](https://learn.microsoft.com/azure/sap/automation/deployment-framework) environment set up, integrating the SAP Testing Automation Framework is a natural extension that allows you to leverage existing deployment pipelines and configurations. For more details on the setup, see [Setup Guide for SAP Testing Automation Framework](./docs/SETUP.MD).
 
 ## 🆘Support
 
@@ -68,7 +81,8 @@ For support and questions, please:
 ## 📚 Additional Resources
 
 - [Azure SAP Documentation](https://docs.microsoft.com/azure/sap)
-- [SAP on Azure: High Availability Guide](https://docs.microsoft.com/azure/sap/workloads/sap-high-availability-guide-start)
+- [Configuration Checks Guide](./docs/CONFIGURATION_CHECKS.md)
+- [High Availability Testing Guide](./docs/HIGH_AVAILABILITY.md)
 
 ## 🤝 Contributing
 

docs/CONFIGURATION_CHECKS.md

@@ -0,0 +1,129 @@
+# SAP Configuration Checks
+
+SAP Configuration Checks is an integral part of the SAP Testing Automation framework, providing comprehensive validation of SAP system configurations on Azure infrastructure. This module ensures that SAP Database and SAP Central Services deployments meet enterprise requirements for compliance before entering production. This tool is designed to identify misconfigurations, deviations from best practices, and potential issues that could impact system's stability and performance.
+
+This tool is a new version of the existing [Quality Checks scripts](https://github.com/Azure/SAP-on-Azure-Scripts-and-Utilities/tree/main/QualityCheck), re-architected to provide a extensible, and maintainable solution. It leverages Python for core logic and Ansible for orchestration.
+
+## Overview
+
+Configuration validation serves as a critical quality gate in the SAP deployment lifecycle by:
+
+- **Validating Azure Infrastructure**: Ensuring compute, storage, and network configurations align with SAP best practices
+- **Verifying SAP Parameters**: Checking critical SAP HANA and application server settings
+- **Assessing Cluster Health**: Validating Pacemaker configurations and resource constraints
+- **Ensuring Compliance**: Confirming adherence to organizational and SAP security standards
+
+## Configuration Check Categories
+
+The configuration checks are organized into logical groups that can be executed independently or all at once. The main categories are:
+
+1. **Infrastructure**
+   - While not a separate execution category, infrastructure checks are performed as part of the other categories.
+   - **Azure Compute**: VM SKU, Accelerated Networking, Availability Set/Zone, Proximity Placement Group.
+   - **Storage**: Use of Premium SSD/Ultra Disk, Write Accelerator, disk caching policies, and redundancy settings.
+
+2. **Database**
+   - Validates SAP HANA or IBM DB2 specific settings.
+   - **SAP HANA**: Checks memory allocation, system replication parameters, and Pacemaker cluster configurations (resource agents, fencing, constraints).
+   - **IBM DB2**: Verifies hardware requirements, system language, and OS tuning parameters.
+
+3. **Central Services**
+   - Validates the configuration of ASCS (ABAP SAP Central Services) and ERS (Enqueue Replication Server) instances.
+   - Checks for virtual hostname configuration, file system mount options, and service startup ordering.
+
+4. **Application Servers**
+   - Validates the configuration of the application server instances.
+
+> [!NOTE]
+>
+> High Availability (HA) configuration checks and functional tests are currently supported only for SAP HANA databases. For IBM DB2 databases, only non-HA configuration checks are available.
+
+## Pre-requisites
+
+### 1. Setup Configuration
+
+Follow the steps (1.1 - 1.5) in [Setup Guide for SAP Testing Automation Framework](./SETUP.MD) to set up the framework on a management server.
+
+### 2. System Configuration
+
+Update the `TEST_TYPE` parameter in [`vars.yaml`](./../vars.yaml) file to `ConfigurationChecks` to enable the Configuration Checks test scenarios.
+
+Follow the steps (2.1 - 2.2) in [Setup Guide for SAP Testing Automation Framework](./SETUP.MD#2-system-configuration) to configure your system details.
+
+### 3. Required Access and Permissions (required)
+
+Effective configuration validation requires that the management server's managed identity (system or user assigned) has read permissions on all target Azure resources. This allows the framework to inspect the settings of services including, but not limited to, Azure Load Balancers, storage solutions (Managed Disks, Azure Files, Azure NetApp Files), and network infrastructure. Lacking the necessary access will prevent the configuration checks from identifying potential misconfigurations in the environment. For more details on configuring system assigned managed identity vs user assigned managed identity, see [Setup Guide for SAP Testing Automation Framework](./SETUP.MD#4-identity-and-authorization).
+
+1. Depending on the type of managed identity method you want to use, configure managed identity on management server
+   - [Configuring access using user-assigned managed identity](./SETUP.MD#option-1-user-assigned-managed-identity).
+   - [Configuring access using system-assigned managed identity](./SETUP.MD#option-2-system-assigned-managed-identity).
+2. Grant the managed identity (system- or user-assigned) the built-in **Reader** role on every resource group that contains SAP system components (VMs, managed disks, load balancers, virtual network, shared storage such as Azure NetApp Files or Azure Files). If everything resides in a single resource group, one assignment there is sufficient; if components are split across multiple resource groups, add a **Reader** role assignment on each resource group to allow the configuration checks to read and validate all infrastructure settings.
+
+## Test Execution
+
+You first need to make sure that you are logged into Azure CLI on the management server VM with the appropriate subscription context:
+
+```bash
+# Login to Azure using System Assigned Managed Identity
+az login --identity
+
+# Login to Azure using User Assigned Managed Identity
+az login --identity -u <client-id-of-user-assigned-managed-identity>
+
+# Set the desired subscription context
+az account set --subscription <subscription-id>
+```
+
+To execute the script, run following command:
+
+```bash
+# Help option
+./scripts/sap_automation_qa.sh --help
+
+# Run all the configuration checks with default parameters
+./scripts/sap_automation_qa.sh
+
+# Run checks with verbose logging
+./scripts/sap_automation_qa.sh -vv
+
+# Run only Database configuration checks (supports both HANA and DB2)
+./scripts/sap_automation_qa.sh --extra-vars='{"configuration_test_type":"Database"}'
+
+# Run only ASCS/ERS configuration checks
+./scripts/sap_automation_qa.sh --extra-vars='{"configuration_test_type":"CentralServiceInstances"}'
+
+# Run only Application Server configuration checks
+./scripts/sap_automation_qa.sh --extra-vars='{"configuration_test_type":"ApplicationInstances"}'
+```
+
+## Viewing Test Results
+
+After the test execution completes, a detailed HTML report is generated. The report provide the summary of each test cases that got executed for each VM.
+
+**To locate and view your test report:**
+
+1. **Navigate to your SAP system’s workspace directory:**
+
+   Replace `<SYSTEM_CONFIG_NAME>` with the name of your SAP system configuration (for example, `DEV-WEEU-SAP01-X00`):
+
+   ```bash
+   cd WORKSPACES/SYSTEM/<SYSTEM_CONFIG_NAME>/quality_assurance/
+   ```
+  
+2. **Find your report file:**
+
+   The report file is named using the following format:
+
+   ```html
+   all_{DISTRO}_{INVOCATION_ID}.html
+   ```
+
+   - `DISTRO`: Linux distribution (SLES or RHEL)
+
+   - `INVOCATION_ID`: A unique identifier (Group invocation ID) for the test run which is logged at the end of test execution. Find example screenshot below:
+
+   ![Test Execution Completion Screenshot](./images/execution_screenshot.png)
+
+3. **View the report**
+
+   You can open the HTML report in any web browser to review the results and logs.