From c86f38610e233c448f5d3a527eb2d0d2cc429bd5 Mon Sep 17 00:00:00 2001 From: Shreyanshu Ghosh Date: Mon, 2 Feb 2026 21:51:25 +0530 Subject: [PATCH] test_pr --- ui/sdk/README.md | 263 +++++++++++++++++++++++++++++++++++++---------- 1 file changed, 210 insertions(+), 53 deletions(-) diff --git a/ui/sdk/README.md b/ui/sdk/README.md index af067b3bf..15da16a6b 100644 --- a/ui/sdk/README.md +++ b/ui/sdk/README.md @@ -17,76 +17,233 @@ specific language governing permissions and limitations under the License. --> -# Apache Hamilton UI SDK: Client Code & Related +# Apache Hamilton UI -> **Disclaimer** -> -> Apache Hamilton is an effort undergoing incubation at the Apache Software Foundation (ASF), sponsored by the Apache Incubator PMC. -> -> Incubation is required of all newly accepted projects until a further review indicates that the infrastructure, communications, and decision making process have stabilized in a manner consistent with other successful ASF projects. -> -> While incubation status is not necessarily a reflection of the completeness or stability of the code, it does indicate that the project has yet to be fully endorsed by the ASF. +> A unified operational UI for managing and observing all your dataflows -Welcome to using the Apache Hamilton UI! +The Apache Hamilton UI is a comprehensive system for monitoring, debugging, and managing Apache Hamilton dataflows throughout the entire software development lifecycle—from development to production. -Here are instructions on how to get started with tracking, and managing your Apache Hamilton -DAGs with the Apache Hamilton UI. +## 📚 Documentation & Resources -## Getting Started +- **[Getting Started Guide](https://hamilton.apache.org/concepts/ui)** - Complete setup and feature overview +- **[Blog Post](https://blog.dagworks.io/p/hamilton-ui-streamlining-metadata)** - In-depth introduction +- **[Quick Video Demo](https://youtu.be/0VIVSeN7Ij8?si=maeV0zdzTPSqUl1N)** - Feature walkthrough +- **[Setup Video Tutorial](https://youtu.be/DPfxlTwaNsM)** - Installation guide -For the latest documentation, please consult our -[Apache Hamilton documentation](https://hamilton.apache.org/) under `Apache Hamilton UI`. +--- -For a quick overview of Apache Hamilton, we suggest [tryhamilton.dev](https://www.tryhamilton.dev/). +## ✨ Key Features -## Using the HamiltonTracker +The Apache Hamilton UI provides four core capabilities: -First, you'll need to install the Apache Hamilton SDK package. Assuming you're using pip, you -can do this with: +### 1. **Execution Tracking** +Track and store execution metadata in a persistent database with a server for reading, writing, and authentication. + +### 2. **Data & Artifact Observability** +Monitor Apache Hamilton executions and inspect specific function results and code through an intuitive web interface. + +### 3. **Lineage & Provenance** +Quickly understand how code and data are connected throughout your dataflows. + +### 4. **Catalog** +Search and discover all observed dataflows with comprehensive execution history. + +--- + +## 🎯 Feature Highlights + +### Execution Tracking + +

+ Waterfall view showing execution performance + Graph view highlighting errors + Comparing execution performance +

+

+ Identify bottlenecks (left) • Pinpoint errors (middle) • Compare performance (right) +

+ +### Data & Artifact Observability + +

+ Data visualization for a run + Code tracking view + Data comparison across executions +

+

+ Visualize run data (left) • Track code versions (middle) • Compare executions (right) +

+ +### Lineage & Provenance + +

+ Lineage dependencies view + Code view grouped by module +

+

+ Trace dependencies (left) • Navigate code visually (right) +

+ +### Catalog + +

+ Artifact catalog view + Transform catalog view +

+

+ Browse artifacts (left) • Discover features and usage (right) +

+ +--- + +## 🚀 Quick Start + +### Prerequisites + +- Docker installed and running on your system + +### Installation Steps + +```bash +# 1. Clone the repository +git clone https://github.com/apache/hamilton + +# 2. Navigate to the UI directory +cd hamilton/ui + +# 3. Start the application +./run.sh +``` + +### Initial Setup + +1. Open your browser and navigate to **http://localhost:8242** +2. Create an account with your email +3. Create a new project +4. Follow the on-screen instructions to integrate with Apache Hamilton + +For a complete setup guide, visit the [documentation](https://hamilton.apache.org/concepts/ui). + +--- + +## 🏗️ Architecture + +The Hamilton UI follows a simple, scalable architecture: + +![Architecture Diagram](./hamilton-ui-architecture.png) + +**Components:** + +- **Tracking Server**: Stores execution data in PostgreSQL and binary objects in S3 (or Docker volumes in local mode) +- **Frontend**: React-based web application +- **Authentication**: Configurable authentication with local/unauthenticated mode as default + +> **Note**: For custom authentication requirements, please contact the maintainers. + +--- + +## 🛠️ Development + +### Project Structure + +The project uses a symlink from `backend/hamilton_ui` to `backend/server` to maintain Django's structure while enabling import as `hamilton_ui`. + +> ⚠️ *This structure may be refactored in future versions.* + +### Building for Development + +```bash +cd hamilton/ui + +# Build everything from scratch +./dev.sh --build + +# Use local code with pulled Docker images +./dev.sh +``` + +**Important**: Docker requires **at least 9GB of memory** allocated to build the frontend. Increase your Docker memory allocation if you encounter build issues. + +### Building for Production ```bash -# install the package & cli into your favorite python environment. -pip install "sf-hamilton[sdk]" +cd hamilton/ui + +# Pull and run production images +./run.sh -# And validate -- this should not error. -python -c "from hamilton_sdk import adapters" +# Rebuild production images +./run.sh --build ``` -Next, you'll need to modify your Apache Hamilton driver. You'll only need to use one line of code to -replace your driver with ours: - -```python -from hamilton_sdk import adapters -from hamilton import driver - -tracker = adapters.HamiltonTracker( - project_id=PROJECT_ID, # modify this as needed - username=YOUR_EMAIL, # modify this as needed - dag_name="my_version_of_the_dag", - tags={"environment": "DEV", "team": "MY_TEAM", "version": "X"} -) -dr = ( - driver.Builder() - .with_config(your_config) - .with_modules(*your_modules) - .with_adapters(tracker) - .build() -) -# to run call .execute() or .materialize() on the driver +**Before building**: Clean the `backend/dist/` directory to avoid adding unnecessary files to the Docker image. + +### Deployment Process + +Use the `admin.py` script in the UI directory to deploy: + +1. Builds the frontend +2. Copies build artifacts to the `build/` directory +3. Publishes to PyPI as [`sf-hamilton-ui`](https://pypi.org/project/sf-hamilton-ui/) + +**Installation**: +```bash +# Install the UI +pip install sf-hamilton[ui] + +# Run the UI +hamilton ui + +# Install SDK for API communication +pip install sf-hamilton[sdk] +``` + +--- + +## 🐳 Docker Management + +### Publishing Backend Images + +```bash +# Tag the image with version +docker tag local-image:tagname dagworks/ui-backend:VERSION + +# Push versioned image +docker push dagworks/ui-backend:VERSION + +# Tag as latest +docker tag dagworks/ui-backend:VERSION dagworks/ui-backend:latest + +# Push latest +docker push dagworks/ui-backend:latest +``` + +### Publishing Frontend Images + +```bash +# Tag the image with version +docker tag local-image:tagname dagworks/ui-frontend:VERSION + +# Push versioned image +docker push dagworks/ui-frontend:VERSION + +# Tag as latest +docker tag dagworks/ui-frontend:VERSION dagworks/ui-frontend:latest + +# Push latest +docker push dagworks/ui-frontend:latest ``` -*Project ID*: You'll need a project ID. Create a project if you don't have one, and take the ID from that. -*username*: This is the email address you used to set up the Apache Hamilton UI. +--- -*dag_name*: for a project, the DAG name is the top level way to group DAGs. -E.g. ltv_model, us_sales, etc. +## 📦 Package Information -*tags*: these are optional are string key value paris. They allow you to filter and curate -various DAG runs. +- **UI Package**: `sf-hamilton-ui` - The web interface +- **SDK Package**: `sf-hamilton[sdk]` - Required for API communication +- **Full Installation**: `sf-hamilton[ui]` - Includes everything needed to run the UI -Then run Apache Hamilton as normal! Each DAG run will be tracked, and you'll have access to it in the -Apache Hamilton UI. After spinning up the Apache Hamilton UI application, visit it to see your projects & DAGs. +--- +## 📄 License -# License -The code here is licensed under the Apache 2.0 license. See the main repository [LICENSE](../../LICENSE) for details. +Licensed under the Apache License, Version 2.0. See the license header for full details.