Skip to content

Improve Apache Hamilton UI README for Better Readability and Navigation #1464

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
ShreyanshuGhosh wants to merge 1 commit into
base: main
Choose a base branch
from
+210 −53
Open

Improve Apache Hamilton UI README for Better Readability and Navigation #1464

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
263 changes: 210 additions & 53 deletions ui/sdk/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,76 +17,233 @@ specific language governing permissions and limitations
under the License.
-->

# Apache Hamilton UI SDK: Client Code & Related
# Apache Hamilton UI

> **Disclaimer**
>
> Apache Hamilton is an effort undergoing incubation at the Apache Software Foundation (ASF), sponsored by the Apache Incubator PMC.
>
> Incubation is required of all newly accepted projects until a further review indicates that the infrastructure, communications, and decision making process have stabilized in a manner consistent with other successful ASF projects.
>
> While incubation status is not necessarily a reflection of the completeness or stability of the code, it does indicate that the project has yet to be fully endorsed by the ASF.
> A unified operational UI for managing and observing all your dataflows

Welcome to using the Apache Hamilton UI!
The Apache Hamilton UI is a comprehensive system for monitoring, debugging, and managing Apache Hamilton dataflows throughout the entire software development lifecycle—from development to production.

Here are instructions on how to get started with tracking, and managing your Apache Hamilton
DAGs with the Apache Hamilton UI.
## 📚 Documentation & Resources

## Getting Started
- **[Getting Started Guide](https://hamilton.apache.org/concepts/ui)** - Complete setup and feature overview
- **[Blog Post](https://blog.dagworks.io/p/hamilton-ui-streamlining-metadata)** - In-depth introduction
- **[Quick Video Demo](https://youtu.be/0VIVSeN7Ij8?si=maeV0zdzTPSqUl1N)** - Feature walkthrough
- **[Setup Video Tutorial](https://youtu.be/DPfxlTwaNsM)** - Installation guide

For the latest documentation, please consult our
[Apache Hamilton documentation](https://hamilton.apache.org/) under `Apache Hamilton UI`.
---

For a quick overview of Apache Hamilton, we suggest [tryhamilton.dev](https://www.tryhamilton.dev/).
## ✨ Key Features

## Using the HamiltonTracker
The Apache Hamilton UI provides four core capabilities:

First, you'll need to install the Apache Hamilton SDK package. Assuming you're using pip, you
can do this with:
### 1. **Execution Tracking**
Track and store execution metadata in a persistent database with a server for reading, writing, and authentication.

### 2. **Data & Artifact Observability**
Monitor Apache Hamilton executions and inspect specific function results and code through an intuitive web interface.

### 3. **Lineage & Provenance**
Quickly understand how code and data are connected throughout your dataflows.

### 4. **Catalog**
Search and discover all observed dataflows with comprehensive execution history.

---

## 🎯 Feature Highlights

### Execution Tracking

<p align="center">
<img src="./screenshots/execution_waterfall_view.png" alt="Waterfall view showing execution performance" width="30%" style="margin-right: 20px;"/>
<img src="./screenshots/execution_graph_error.png" alt="Graph view highlighting errors" width="30%" style="margin-right: 20px;"/>
<img src="./screenshots/execution_comparison_waterfall.png" alt="Comparing execution performance" width="30%"/>
</p>
<p align="center">
<em>Identify bottlenecks (left) • Pinpoint errors (middle) • Compare performance (right)</em>
</p>

### Data & Artifact Observability

<p align="center">
<img src="./screenshots/execution_data_view.png" alt="Data visualization for a run" width="30%"/>
<img src="./screenshots/execution_code_view.png" alt="Code tracking view" width="30%" style="margin-right: 20px;"/>
<img src="./screenshots/execution_data_comparison.png" alt="Data comparison across executions" width="30%" style="margin-right: 20px;"/>
</p>
<p align="center">
<em>Visualize run data (left) • Track code versions (middle) • Compare executions (right)</em>
</p>

### Lineage & Provenance

<p align="center">
<img src="./screenshots/lineage_view.png" alt="Lineage dependencies view" width="30%"/>
<img src="./screenshots/lineage_code_view_grouped_by_module.png" alt="Code view grouped by module" width="30%" style="margin-right: 20px;"/>
</p>
<p align="center">
<em>Trace dependencies (left) • Navigate code visually (right)</em>
</p>

### Catalog

<p align="center">
<img src="./screenshots/catalog_artifact.png" alt="Artifact catalog view" width="30%"/>
<img src="./screenshots/catalog_transform.png" alt="Transform catalog view" width="30%" style="margin-right: 20px;"/>
</p>
<p align="center">
<em>Browse artifacts (left) • Discover features and usage (right)</em>
</p>

---

## 🚀 Quick Start

### Prerequisites

- Docker installed and running on your system

### Installation Steps

```bash
# 1. Clone the repository
git clone https://github.com/apache/hamilton

# 2. Navigate to the UI directory
cd hamilton/ui

# 3. Start the application
./run.sh
```

### Initial Setup

1. Open your browser and navigate to **http://localhost:8242**
2. Create an account with your email
3. Create a new project
4. Follow the on-screen instructions to integrate with Apache Hamilton

For a complete setup guide, visit the [documentation](https://hamilton.apache.org/concepts/ui).

---

## 🏗️ Architecture

The Hamilton UI follows a simple, scalable architecture:

![Architecture Diagram](./hamilton-ui-architecture.png)

**Components:**

- **Tracking Server**: Stores execution data in PostgreSQL and binary objects in S3 (or Docker volumes in local mode)
- **Frontend**: React-based web application
- **Authentication**: Configurable authentication with local/unauthenticated mode as default

> **Note**: For custom authentication requirements, please contact the maintainers.

---

## 🛠️ Development

### Project Structure

The project uses a symlink from `backend/hamilton_ui` to `backend/server` to maintain Django's structure while enabling import as `hamilton_ui`.

> ⚠️ *This structure may be refactored in future versions.*

### Building for Development

```bash
cd hamilton/ui

# Build everything from scratch
./dev.sh --build

# Use local code with pulled Docker images
./dev.sh
```

**Important**: Docker requires **at least 9GB of memory** allocated to build the frontend. Increase your Docker memory allocation if you encounter build issues.

### Building for Production

```bash
# install the package & cli into your favorite python environment.
pip install "sf-hamilton[sdk]"
cd hamilton/ui

# Pull and run production images
./run.sh

# And validate -- this should not error.
python -c "from hamilton_sdk import adapters"
# Rebuild production images
./run.sh --build
```

Next, you'll need to modify your Apache Hamilton driver. You'll only need to use one line of code to
replace your driver with ours:

```python
from hamilton_sdk import adapters
from hamilton import driver

tracker = adapters.HamiltonTracker(
project_id=PROJECT_ID, # modify this as needed
username=YOUR_EMAIL, # modify this as needed
dag_name="my_version_of_the_dag",
tags={"environment": "DEV", "team": "MY_TEAM", "version": "X"}
)
dr = (
driver.Builder()
.with_config(your_config)
.with_modules(*your_modules)
.with_adapters(tracker)
.build()
)
# to run call .execute() or .materialize() on the driver
**Before building**: Clean the `backend/dist/` directory to avoid adding unnecessary files to the Docker image.

### Deployment Process

Use the `admin.py` script in the UI directory to deploy:

1. Builds the frontend
2. Copies build artifacts to the `build/` directory
3. Publishes to PyPI as [`sf-hamilton-ui`](https://pypi.org/project/sf-hamilton-ui/)

**Installation**:
```bash
# Install the UI
pip install sf-hamilton[ui]

# Run the UI
hamilton ui

# Install SDK for API communication
pip install sf-hamilton[sdk]
```

---

## 🐳 Docker Management

### Publishing Backend Images

```bash
# Tag the image with version
docker tag local-image:tagname dagworks/ui-backend:VERSION

# Push versioned image
docker push dagworks/ui-backend:VERSION

# Tag as latest
docker tag dagworks/ui-backend:VERSION dagworks/ui-backend:latest

# Push latest
docker push dagworks/ui-backend:latest
```

### Publishing Frontend Images

```bash
# Tag the image with version
docker tag local-image:tagname dagworks/ui-frontend:VERSION

# Push versioned image
docker push dagworks/ui-frontend:VERSION

# Tag as latest
docker tag dagworks/ui-frontend:VERSION dagworks/ui-frontend:latest

# Push latest
docker push dagworks/ui-frontend:latest
```
*Project ID*: You'll need a project ID. Create a project if you don't have one, and take the ID from that.

*username*: This is the email address you used to set up the Apache Hamilton UI.
---

*dag_name*: for a project, the DAG name is the top level way to group DAGs.
E.g. ltv_model, us_sales, etc.
## 📦 Package Information

*tags*: these are optional are string key value paris. They allow you to filter and curate
various DAG runs.
- **UI Package**: `sf-hamilton-ui` - The web interface
- **SDK Package**: `sf-hamilton[sdk]` - Required for API communication
- **Full Installation**: `sf-hamilton[ui]` - Includes everything needed to run the UI

Then run Apache Hamilton as normal! Each DAG run will be tracked, and you'll have access to it in the
Apache Hamilton UI. After spinning up the Apache Hamilton UI application, visit it to see your projects & DAGs.
---

## 📄 License

# License
The code here is licensed under the Apache 2.0 license. See the main repository [LICENSE](../../LICENSE) for details.
Licensed under the Apache License, Version 2.0. See the license header for full details.