diff --git a/content/learning-paths/servers-and-cloud-computing/flyte-with-grpc/_index.md b/content/learning-paths/servers-and-cloud-computing/flyte-with-grpc/_index.md new file mode 100644 index 0000000000..aadd7e47cf --- /dev/null +++ b/content/learning-paths/servers-and-cloud-computing/flyte-with-grpc/_index.md @@ -0,0 +1,68 @@ +--- +title: Build ML Workflow Pipelines with Flyte and gRPC on Google Cloud C4A Axion processors + +minutes_to_complete: 30 + +who_is_this_for: This is an introductory topic for developers, data engineers, and ML engineers who want to build scalable machine learning workflow pipelines on Arm64-based Google Cloud C4A Axion processors using Flyte workflow orchestration and gRPC-based microservices. + +learning_objectives: + - Deploy Flyte workflow pipelines on Google Cloud C4A Axion processors + - Build distributed machine learning pipelines using Flyte tasks + - Implement gRPC-based services for feature engineering + - Integrate Flyte workflows with distributed services + - Run scalable ML pipelines on Arm-based cloud infrastructure + +prerequisites: + - A [Google Cloud Platform (GCP)](https://cloud.google.com/free) account with billing enabled + - Basic familiarity with Python + - Basic understanding of machine learning pipelines + - Familiarity with Linux command-line operations + +author: Pareena Verma + +##### Tags +skilllevels: Introductory +subjects: ML +cloud_service_providers: +- Google Cloud + +armips: +- Neoverse + +tools_software_languages: +- Flyte +- Python +- gRPC + +operatingsystems: +- Linux + +# ================================================================================ +# FIXED, DO NOT MODIFY +# ================================================================================ + +further_reading: + - resource: + title: Google Cloud documentation + link: https://cloud.google.com/docs + type: documentation + + - resource: + title: Flyte documentation + link: https://docs.flyte.org/ + type: documentation + + - resource: + title: gRPC documentation + link: https://grpc.io/docs/ + type: documentation + + - resource: + title: Flyte GitHub repository + link: https://github.com/flyteorg/flyte + type: documentation + +weight: 1 +layout: "learningpathall" +learning_path_main_page: yes +--- diff --git a/content/learning-paths/servers-and-cloud-computing/flyte-with-grpc/_next-steps.md b/content/learning-paths/servers-and-cloud-computing/flyte-with-grpc/_next-steps.md new file mode 100644 index 0000000000..c3db0de5a2 --- /dev/null +++ b/content/learning-paths/servers-and-cloud-computing/flyte-with-grpc/_next-steps.md @@ -0,0 +1,8 @@ +--- +# ================================================================================ +# FIXED, DO NOT MODIFY THIS FILE +# ================================================================================ +weight: 21 # Set to always be larger than the content in this path to be at the end of the navigation. +title: "Next Steps" # Always the same, html page title. +layout: "learningpathall" # All files under learning paths have this same wrapper for Hugo processing. +--- diff --git a/content/learning-paths/servers-and-cloud-computing/flyte-with-grpc/architecture.md b/content/learning-paths/servers-and-cloud-computing/flyte-with-grpc/architecture.md new file mode 100644 index 0000000000..b52414d60c --- /dev/null +++ b/content/learning-paths/servers-and-cloud-computing/flyte-with-grpc/architecture.md @@ -0,0 +1,166 @@ +--- +title: ML Pipeline Architecture +weight: 8 + +### FIXED, DO NOT MODIFY +layout: learningpathall +--- + +# ML Pipeline Architecture + +In this section, you explore the architecture behind the distributed machine learning pipeline built using Flyte and gRPC on Google Axion Arm-based infrastructure. + +This architecture demonstrates how modern ML workflows are orchestrated using workflow engines while delegating specific tasks to distributed services. + +Flyte manages the pipeline orchestration, while gRPC enables efficient communication between workflow tasks and external services. + + +## System architecture + +The ML pipeline consists of several tasks executed sequentially within the Flyte workflow. + +```text +Flyte Workflow Engine + │ + ▼ +Dataset Loader Task + │ + ▼ +Data Preprocessing Task + │ + ▼ +Feature Engineering Service (gRPC) + │ + ▼ +Model Training Task + │ + ▼ +Model Evaluation Task + │ + ▼ +Pipeline Result +``` + +Each component in the workflow performs a specific function within the machine learning pipeline. + +## Components + +### Flyte workflow engine +Flyte orchestrates the pipeline execution. It manages task dependencies, workflow execution, and data flow between tasks. + +Key capabilities include: + +- defining ML pipelines as Python workflows +- managing task dependencies +- enabling reproducible ML experiments +- scaling pipeline execution + +### Dataset loader +The dataset loader task simulates loading a training dataset that will be used for model training. + +In real ML systems, this step might include: + +- loading datasets from object storage +- retrieving data from data lakes +- accessing distributed datasets + +### Data preprocessing +Data preprocessing transforms raw data into a format suitable for model training. + +Typical preprocessing steps include: + +- cleaning data +- normalizing values +- handling missing data +- encoding categorical variables + +### Feature engineering service (gRPC) +Feature engineering is implemented as a gRPC microservice. + +This design allows feature-generation logic to run independently of the workflow engine. + +Benefits include: + +- scalable feature generation +- reusable feature services +- independent scaling of compute resources +- low-latency communication using gRPC + +### Model training +The training task uses generated features to train a machine learning model. + +In production systems, this stage might include: + +- training regression models +- training classification models +- training deep learning models + +### Model evaluation +The evaluation step measures model performance. + +Typical evaluation metrics include: + +- accuracy +- precision +- recall +- F1 score + +Based on the results, the workflow can determine whether to retrain the model. + +## Pipeline execution flow + +The ML pipeline follows this execution sequence. + +```text +Load Dataset + │ + ▼ +Preprocess Data + │ + ▼ +Feature Engineering (gRPC Service) + │ + ▼ +Model Training + │ + ▼ +Model Evaluation + │ + ▼ +Pipeline Result +``` + +Each task executes sequentially while Flyte manages the workflow orchestration. + +## Benefits of this architecture + +This architecture provides several advantages: + +- scalable ML pipeline orchestration +- distributed feature engineering services +- modular pipeline components +- efficient task communication using gRPC +- reproducible machine learning workflows + +## Running on Axion +This example demonstrates how machine learning workflows can run efficiently on Google Axion Arm-based processors. + +Benefits include: + +- high performance per watt +- efficient execution of data pipelines +- scalable infrastructure for ML workloads +- optimized performance for modern cloud applications + +## What you've learned + +In this section, you explored the architecture behind the ML training pipeline. + +You learned how: + +- Flyte orchestrates ML workflows +- gRPC services enable distributed feature engineering +- pipeline tasks interact through workflow dependencies +- ML pipelines can scale across distributed infrastructure + +This architecture underpins modern distributed machine learning systems running on Arm-based cloud infrastructure. diff --git a/content/learning-paths/servers-and-cloud-computing/flyte-with-grpc/background.md b/content/learning-paths/servers-and-cloud-computing/flyte-with-grpc/background.md new file mode 100644 index 0000000000..37a57d3758 --- /dev/null +++ b/content/learning-paths/servers-and-cloud-computing/flyte-with-grpc/background.md @@ -0,0 +1,35 @@ +--- +title: Get started with Flyte ML Workflow Pipelines with gRPC on Google Axion C4A +weight: 2 + +layout: "learningpathall" +--- + +## Explore Axion C4A Arm instances in Google Cloud + +Google Axion C4A is a family of Arm-based virtual machines built on Google’s custom Axion CPU, which is based on Arm Neoverse-V2 cores. Designed for high-performance and energy-efficient computing, these virtual machines offer strong performance for data-intensive and analytics workloads such as big data processing, in-memory analytics, columnar data processing, and high-throughput data services. + +The C4A series provides a cost-effective alternative to x86 virtual machines while leveraging the scalability, SIMD acceleration, and memory bandwidth advantages of the Arm architecture in Google Cloud. + +These characteristics make Axion C4A instances well-suited for modern analytics stacks that rely on columnar data formats and memory-efficient execution engines. + +To learn more, see the Google blog [Introducing Google Axion Processors, our new Arm-based CPUs](https://cloud.google.com/blog/products/compute/introducing-googles-new-arm-based-cpu). + +## Explore Flyte ML Workflow Pipelines with gRPC on Google Axion C4A (Arm Neoverse V2) + +Flyte is an open-source workflow orchestration platform used to build scalable and reproducible data and machine learning pipelines. It allows developers to define workflows as Python tasks, simplifying the management of complex ML processes such as data preparation, feature engineering, and model training. + +gRPC enables fast communication between distributed services within these pipelines. Running Flyte with gRPC on Google Axion C4A Arm-based processors provides efficient, scalable infrastructure for executing modern ML workflows and distributed data processing tasks. + +To learn more, visit the [Flyte documentation](https://docs.flyte.org/) and explore the [gRPC documentation](https://grpc.io/docs/) to understand how distributed service communication enables scalable machine learning workflows. + +## What you've learned and what's next + +In this section, you learned about: + +* Google Axion C4A Arm-based VMs and their performance characteristics +* Flyte as a workflow orchestration platform for machine learning pipelines +* gRPC as a communication layer for distributed services +* How Flyte and gRPC can be used together to build scalable ML training pipelines + +Next, You will deploy Flyte tools, create a gRPC-based feature engineering service, and build a distributed ML workflow pipeline that orchestrates data processing and model training tasks on Axion infrastructure. diff --git a/content/learning-paths/servers-and-cloud-computing/flyte-with-grpc/create-grpc-service.md b/content/learning-paths/servers-and-cloud-computing/flyte-with-grpc/create-grpc-service.md new file mode 100644 index 0000000000..0b75d1d4b4 --- /dev/null +++ b/content/learning-paths/servers-and-cloud-computing/flyte-with-grpc/create-grpc-service.md @@ -0,0 +1,166 @@ +--- +title: Create Feature Engineering gRPC Service +weight: 5 + +### FIXED, DO NOT MODIFY +layout: learningpathall +--- + +# Create Feature Engineering gRPC Service + + +In modern machine learning pipelines, feature engineering is often implemented as a separate service so it can scale independently from the training workflow. + +In this section, you create a **gRPC-based feature engineering service** that generates features used by the machine learning pipeline. + +This service will later be called by the Flyte workflow during pipeline execution. + +## Architecture overview + +The feature engineering service acts as an external microservice used by the ML workflow. + +```text +Flyte Workflow + | + v +Feature Engineering Service (gRPC) + | + v +Generated Features for Model Training +``` + +## Create project directory +Create a directory for the ML workflow project. + +```bash +mkdir flyte-ml-pipeline +cd flyte-ml-pipeline +``` + +## Create protobuf definition +Create the gRPC service definition file. + +```bash +vi feature.proto +``` + +Add the following code. + +```python +syntax = "proto3"; + +service FeatureService { + rpc GenerateFeatures (FeatureRequest) returns (FeatureResponse); +} + +message FeatureRequest { + int32 value = 1; +} + +message FeatureResponse { + int32 feature = 1; +} +``` + +This file defines the service interface used by the workflow and the feature service. + +## Generate gRPC code +Compile the protobuf file to generate Python client and server code. + + +```python +python3.11 -m grpc_tools.protoc \ +-I. \ +--python_out=. \ +--grpc_python_out=. \ +feature.proto +``` + +This generates the following files: + +```text +feature_pb2.py +feature_pb2_grpc.py +``` + +These files contain the Python classes used by the gRPC server and client. + +Why this matters: + +- Protobuf defines a strongly typed service interface +- Generated code simplifies client-server communication +- Enables efficient RPC communication using gRPC + +## Create the feature engineering service +Create the server implementation. + +```bash +vi feature_server.py +``` + +Add the following code. + +```python +import grpc +from concurrent import futures +import feature_pb2 +import feature_pb2_grpc + + +class FeatureService(feature_pb2_grpc.FeatureServiceServicer): + + def GenerateFeatures(self, request, context): + + value = request.value + feature = value * 10 + + print("Generating feature for:", value) + + return feature_pb2.FeatureResponse(feature=feature) + + +def serve(): + + server = grpc.server(futures.ThreadPoolExecutor(max_workers=10)) + + feature_pb2_grpc.add_FeatureServiceServicer_to_server( + FeatureService(), server + ) + + server.add_insecure_port("[::]:50051") + + server.start() + + print("Feature gRPC service running on port 50051") + + server.wait_for_termination() + + +if __name__ == "__main__": + serve() +``` + +This service receives a value from the workflow and generates a derived feature used during model training. + +## Run the feature service + +Start the gRPC service. + +```bash +python3.11 feature_server.py +``` + +The output is similar to: +```output +Feature gRPC service running on port 50051 +``` + +## What you've learned and what's next + +- Created a project directory for the ML workflow +- Defined a gRPC service using protobuf +- Generated Python client and server code +- Implemented a feature engineering microservice +- Started the gRPC feature service + +In the next section, you will create a Flyte ML training workflow that calls this feature engineering service during pipeline execution. diff --git a/content/learning-paths/servers-and-cloud-computing/flyte-with-grpc/create-ml-workflow.md b/content/learning-paths/servers-and-cloud-computing/flyte-with-grpc/create-ml-workflow.md new file mode 100644 index 0000000000..b6ba33751b --- /dev/null +++ b/content/learning-paths/servers-and-cloud-computing/flyte-with-grpc/create-ml-workflow.md @@ -0,0 +1,182 @@ +--- +title: Create ML Training Workflow +weight: 6 + +### FIXED, DO NOT MODIFY +layout: learningpathall +--- + +## Create a machine learning workflow using Flyte + +In this section, you create a machine learning workflow pipeline using **Flyte**. Flyte workflows allow developers to define machine learning pipelines as Python tasks with explicit dependencies. This enables reproducible and scalable pipeline execution. + +The workflow you build will perform the following steps: + +- Load a dataset +- Preprocess the data +- Generate features using a gRPC service +- Train a machine learning model +- Evaluate the model performance + +This demonstrates how Flyte orchestrates tasks across distributed services within a machine learning pipeline. + +## Architecture overview + +The Flyte workflow interacts with the gRPC feature engineering service created in the previous section. + +```text +Flyte Workflow + | + v +Dataset Loader Task + | + v +Data Preprocessing Task + | + v +Feature Engineering (gRPC Service) + | + v +Model Training Task + | + v +Model Evaluation Task + | + v +Pipeline Result +``` + +This architecture separates workflow orchestration from feature generation, allowing different components of the pipeline to scale independently. + +## Create workflow script + +Create the workflow file. + +```bash +vi workflow.py +``` + +Add the following code. + +```python +from flytekit import task, workflow +import grpc +import feature_pb2 +import feature_pb2_grpc + + +@task +def load_dataset() -> int: + + print("Loading dataset") + + return 10 + + +@task +def preprocess_data(size: int) -> int: + + print("Preprocessing dataset:", size) + + return size * 2 + + +@task +def generate_features(data: int) -> int: + + channel = grpc.insecure_channel("localhost:50051") + + stub = feature_pb2_grpc.FeatureServiceStub(channel) + + response = stub.GenerateFeatures( + feature_pb2.FeatureRequest(value=data) + ) + + return response.feature + + +@task +def train_model(feature: int) -> float: + + print("Training model with feature:", feature) + + accuracy = feature / 20.0 + + return accuracy + + +@task +def evaluate_model(acc: float) -> str: + + print("Model accuracy:", acc) + + if acc > 0.5: + return "Model performance good" + else: + return "Model performance needs improvement" + + +@workflow +def ml_pipeline() -> str: + + data = load_dataset() + + processed = preprocess_data(size=data) + + feature = generate_features(data=processed) + + accuracy = train_model(feature=feature) + + result = evaluate_model(acc=accuracy) + + return result + + +if __name__ == "__main__": + + result = ml_pipeline() + + print("Pipeline result:", result) +``` + +## ML pipeline tasks + +The workflow consists of several tasks: + +- **Load dataset:** Simulates loading a training dataset. +- **Preprocess data:** Performs preprocessing on the dataset before training. +- **Feature engineering:** Calls the gRPC service to generate features used for model training. +- **Model training:** Simulates training a machine learning model. +- **Model evaluation**: Evaluates the model and determines whether the model performance is acceptable. + +## Workflow execution flow + +```text +Load Dataset + │ + ▼ +Preprocess Data + │ + ▼ +Feature Engineering (gRPC Service) + │ + ▼ +Model Training + │ + ▼ +Model Evaluation + │ + ▼ +Pipeline Result +``` + +## What you've learned and what's next + +In this section, you learned how to: + +- Create ML workflow tasks using Flyte +- Define dependencies between pipeline steps +- Integrate Flyte tasks with a gRPC microservice +- Orchestrate ML pipeline execution + +In the next section, you will run the co**mplete ML training pipeline and observe how Flyte interacts with the feature engineering service during workflow execution**. diff --git a/content/learning-paths/servers-and-cloud-computing/flyte-with-grpc/images/gcp-pubip-ssh.png b/content/learning-paths/servers-and-cloud-computing/flyte-with-grpc/images/gcp-pubip-ssh.png new file mode 100644 index 0000000000..558745de3e Binary files /dev/null and b/content/learning-paths/servers-and-cloud-computing/flyte-with-grpc/images/gcp-pubip-ssh.png differ diff --git a/content/learning-paths/servers-and-cloud-computing/flyte-with-grpc/images/gcp-shell.png b/content/learning-paths/servers-and-cloud-computing/flyte-with-grpc/images/gcp-shell.png new file mode 100644 index 0000000000..7e2fc3d1b5 Binary files /dev/null and b/content/learning-paths/servers-and-cloud-computing/flyte-with-grpc/images/gcp-shell.png differ diff --git a/content/learning-paths/servers-and-cloud-computing/flyte-with-grpc/images/gcp-vm.png b/content/learning-paths/servers-and-cloud-computing/flyte-with-grpc/images/gcp-vm.png new file mode 100644 index 0000000000..0d1072e20d Binary files /dev/null and b/content/learning-paths/servers-and-cloud-computing/flyte-with-grpc/images/gcp-vm.png differ diff --git a/content/learning-paths/servers-and-cloud-computing/flyte-with-grpc/install-flyte.md b/content/learning-paths/servers-and-cloud-computing/flyte-with-grpc/install-flyte.md new file mode 100644 index 0000000000..3eb37af0cb --- /dev/null +++ b/content/learning-paths/servers-and-cloud-computing/flyte-with-grpc/install-flyte.md @@ -0,0 +1,135 @@ +--- +title: Install Flyte and Dependencies +weight: 4 + +### FIXED, DO NOT MODIFY +layout: learningpathall +--- + + +In this section, you prepare a SUSE Linux Enterprise Server (SLES) arm64 virtual machine and install the core components required to build machine learning workflow pipelines using Flyte. + +Flyte provides workflow orchestration for scalable ML pipelines, while gRPC enables efficient communication between distributed services used within those pipelines. + +This environment ensures that the workflow orchestration tools and communication libraries run natively on Arm-based Axion processors. + +## Architecture overview + +This architecture represents a single-node development environment used to build and run distributed machine learning workflows. + +```text +SUSE Linux Enterprise Server (arm64) + │ + ▼ +Python 3.11 Environment + │ + ▼ +Flyte SDK + │ + ▼ +gRPC Communication Libraries +``` + +## Update the system + +Update the system packages. + +```bash +sudo zypper refresh +sudo zypper update -y +``` + +## Install system dependencies + +Install Python, development tools, and system libraries required to run Flyte workflows. + +```bash +sudo zypper install -y \ +python311 python311-devel python311-pip \ +gcc gcc-c++ make \ +git curl +``` + +**Verify Python installation:** + +```bash +python3.11 --version +``` + +The output is similar to: + +```output +Python 3.11.x +``` + +**Why this matters:** + +- Python 3.11 provides improved performance and memory efficiency +- Modern workflow libraries are optimized for Python 3.11 +- Ensures compatibility with Flyte and gRPC libraries + +## Install Flyte SDK + +Install the Flyte Python SDK used to define and execute workflows. + +```bash +python3.11 -m pip install --upgrade pip +python3.11 -m pip install flytekit +``` + +**Verify installation:** + +```bash +python3.11 -c "import flytekit; print(flytekit.__version__)" +``` + +This confirms that Flyte is correctly installed. + +## Install gRPC libraries + +Install the libraries required for communication between distributed services. + +```bash +python3.11 -m pip install grpcio grpcio-tools protobuf +``` + +These libraries enable remote procedure calls between workflow tasks and microservices. + +## Install Flyte CLI + +Download and install the Flyte command-line tool. + +```bash +curl -L https://github.com/flyteorg/flytectl/releases/latest/download/flytectl_Linux_arm64.tar.gz -o flytectl.tar.gz +tar -xzf flytectl.tar.gz +sudo mv flytectl /usr/local/bin/ +``` + +**Verify Flyte CLI installation** + +```bash +flytectl version +``` +The output is similar to: + +```output +{ + "App": "flytectl", + "Build": "0a0cbce", + "Version": "0.8.18", + "BuildTime": "2026-03-16 11:18:15.958506423 +0000 UTC m=+0.010167662" +} +``` +This confirms that the Flyte CLI is correctly installed. + +## What you've learned and what's next + +In this section, you learned how to: + +- Prepared a SUSE arm64 environment for ML workflow development +- Installed Python 3.11 and development dependencies +- Installed Flyte SDK for workflow orchestration +- Installed gRPC libraries for service communication +- Configured Flyte CLI tools + +In the next section, you will create a gRPC-based feature engineering service that will be integrated with the Flyte ML workflow pipeline. diff --git a/content/learning-paths/servers-and-cloud-computing/flyte-with-grpc/instance.md b/content/learning-paths/servers-and-cloud-computing/flyte-with-grpc/instance.md new file mode 100644 index 0000000000..7572dec1e5 --- /dev/null +++ b/content/learning-paths/servers-and-cloud-computing/flyte-with-grpc/instance.md @@ -0,0 +1,57 @@ +--- +title: Create a Google Axion C4A arm64 virtual machine on GCP +weight: 3 + +### FIXED, DO NOT MODIFY +layout: learningpathall +--- + +## Provision a Google Axion C4A arm64 virtual machine + +In this section you'll create a Google Axion C4A arm64 virtual machine on Google Cloud Platform. You'll use the `c4a-standard-4` machine type, which provides 4 vCPUs and 16 GB of memory. This virtual machine hosts your Flyte ML Workflow with gRPC applications. + +{{% notice Note %}} +For help with GCP setup, see the Learning Path [Getting started with Google Cloud Platform](/learning-paths/servers-and-cloud-computing/csp/google/). +{{% /notice %}} + +## Provision the virtual machine in Google Cloud Console + +To create a virtual machine based on the C4A instance type: + +* Navigate to the [Google Cloud Console](https://console.cloud.google.com/). +* Go to **Compute Engine > VM Instances** and select **Create Instance**. +* Under **Machine configuration**: + * Populate fields such as **Instance name**, **Region**, and **Zone**. + * Set **Series** to `C4A`. + * Select `c4a-standard-4` for machine type. + +![Screenshot of the Google Cloud Console showing the Machine configuration section. The Series dropdown is set to C4A and the machine type c4a-standard-4 is selected alt-txt#center](images/gcp-vm.png "Configuring machine type to C4A in Google Cloud Console") + + +* Under **OS and storage**, select **Change**, and then choose an arm64 operating system image. + * For this Learning Path, select **SUSE Linux Enterprise Server**. + * For the license type, choose **Pay as you go**. + * Increase **Size (GB)** from **10** to **100** to allocate sufficient disk space. + * Select **Choose** to apply the changes. +* Under **Networking**, enable **Allow HTTP traffic** and **Allow HTTPS traffic** +* Also, add the following tag: `allow-flyte-with-grpc` so this virtual machine matches the firewall rule you created in the previous section. +* For some organizations not using the **'default'** network interface, you may need to select the network appropriate for your organization. +* Select **Create** to launch the virtual machine. + +After the instance starts, select **SSH** next to the VM in the instance list to open a browser-based terminal session. + +![Google Cloud Console VM instances page displaying running instance with green checkmark and SSH button in the Connect column alt-txt#center](images/gcp-pubip-ssh.png "Connecting to a running C4A VM using SSH") + +A new browser window opens with a terminal connected to your VM. + +![Browser-based SSH terminal window with black background showing Linux command prompt and Google Cloud branding at top alt-txt#center](images/gcp-shell.png "Terminal session connected to the VM") + +## What you've learned and what's next + +In this section: + +* You provisioned a Google Axion C4A arm64 virtual machine with 4 vCPUs and 16 GB of memory +* You configured the VM with SUSE Linux Enterprise Server and 100 GB of storage +* You connected to your VM using SSH through the Google Cloud Console + +Your virtual machine is now ready to host Flyte ML Workflow with gRPC. diff --git a/content/learning-paths/servers-and-cloud-computing/flyte-with-grpc/run-ml-pipeline.md b/content/learning-paths/servers-and-cloud-computing/flyte-with-grpc/run-ml-pipeline.md new file mode 100644 index 0000000000..5b1291d1a1 --- /dev/null +++ b/content/learning-paths/servers-and-cloud-computing/flyte-with-grpc/run-ml-pipeline.md @@ -0,0 +1,117 @@ +--- +title: Run ML Training Pipeline +weight: 7 + +### FIXED, DO NOT MODIFY +layout: learningpathall +--- + +# Run ML Training Pipeline + +In this section, you execute the distributed machine learning pipeline built using **Flyte and gRPC**. + +The ML workflow will: + +- load a dataset +- Preprocess the data +- generate features using a gRPC microservice +- train a model +- evaluate model performance + +The feature engineering service runs independently and communicates with the workflow using **gRPC remote procedure calls**. + +## Start the feature engineering service + +First, start the feature engineering service that was created in the previous section. + +```bash +python3.11 feature_server.py +``` + +The output is similar to: +```output +Feature gRPC service running on port 50051 +``` + +Leave this terminal running because the ML pipeline will send requests to this service. + +## Run the ML workflow pipeline + +Open a new terminal session. Navigate to the project directory. + +```bash +cd ~/flyte-ml-pipeline +``` + +**Run the workflow:** + +```bash +python3.11 workflow.py +``` + +## Example pipeline execution output + +The output is similar to: + +```output +Loading dataset +Preprocessing dataset: 10 +Training model with feature: 200 +Model accuracy: 10.0 +Pipeline result: Model performance good +``` + +## What happens during execution + +During pipeline execution the following steps occur: + +1. The dataset is loaded by the Flyte task. +2. The dataset is preprocessed. +3. The workflow sends a request to the gRPC feature engineering service. +4. The gRPC service generates features. +5. The workflow uses the generated features to simulate model training. +6. The model performance is evaluated. +7. The pipeline returns the final result. + +## Pipeline execution flow + +```text +Load Dataset + │ + ▼ +Preprocess Data + │ + ▼ +Feature Engineering (gRPC Service) + │ + ▼ +Model Training + │ + ▼ +Model Evaluation + │ + ▼ +Pipeline Result +``` + +## Verify the gRPC service interaction + +You can observe activity in the terminal running the feature service. When the workflow sends a request, the service prints a message similar to: + +```output +Feature gRPC service running on port 50051 +Generating feature for: 20 +``` + +This confirms that the Flyte workflow successfully communicated with the gRPC service. + +## What you've learned and what's next + +In this section, you learned how to: + +- Start the gRPC feature engineering service +- Execute the Flyte ML workflow pipeline +- Observe task execution across distributed services +- Verify communication between the workflow and the microservice + +In the next section, you will explore the architecture of a distributed ML training pipeline implemented with Flyte and gRPC on Axion infrastructure.