ci config tool work 062325 (includes taskcluster cytoscape viewer)

ci-configuration-tools/.gitignore

@@ -0,0 +1,5 @@
+__pycache__
+
+*.txt
+*.json
+*.zip

ci-configuration-tools/docker-worker-report.sh

@@ -0,0 +1,5 @@
+#!/usr/bin/env bash
+
+set -e
+
+./pools_images_count_linux.sh | grep docker-worker

ci-configuration-tools/docker-worker-report2.sh

@@ -0,0 +1,7 @@
+#!/usr/bin/env bash
+
+set -e
+
+./pools_images.py | grep docker-worker
+
+./pools_images.py | grep docker-worker | wc -l

ci-configuration-tools/pools_images_count_linux.sh

@@ -3,4 +3,4 @@
 set -e
 #set -x
 
-./pools_images.py | cut -f 2 -d ':' | sort | uniq -c | sort | grep -v win
+./pools_images.py | grep -v win | cut -f 2 -d ':' | sort | uniq -c | sort | grep -v win

ci-configuration-tools/pyproject.toml

@@ -0,0 +1,19 @@
+[project]
+name = "ci-configuration-tools"
+version = "0.1.0"
+description = ""
+authors = [{name = "Andrew Erickson",email = "aerickson@mozilla.com"}]
+license = {text = "MPL 2.0"}
+readme = "README.md"
+requires-python = ">=3.12"
+dependencies = ["pyyaml (>=6.0.2,<7.0.0)"]
+
+[tool.poetry]
+
+[tool.poetry.group.dev.dependencies]
+pytest = "^8.4.1"
+pytest-watch = "^4.2.0"
+
+[build-system]
+requires = ["poetry-core>=2.0.0,<3.0.0"]
+build-backend = "poetry.core.masonry.api"

ci-configuration-tools/taskcluster_cytoscape/README.md

@@ -0,0 +1,35 @@
+# taskcluster_cytoscape
+
+## Overview
+
+Visualizes Firefox Taskcluster tasks, worker pools, image aliases, and images using [Cytoscope.js](https://js.cytoscape.org/).
+
+Task data is from [Firefox](https://github.com/mozilla-firefox/firefox)'s Taskgraph. All other data is from [fxci-config](http://github.com/mozilla-releng/fxci-config).
+
+This software consists of a Python script to generate a Cytoscape.js JSON file (`graph.py`) and the associated HTML viewer for the JSON file (see the `cytoscape_viewer/` directory).
+
+## Generation of JSON
+
+1. Run `poetry shell && poetry install` one directory up.
+1. Update firefox repo and generate tasks.json in firefox repo.
+1. Update the fxci-config repo.
+1. Run `./generate_graph.sh` (paths may need to be edited).
+
+## Viewing
+
+See the README.md inside the `cytoscape_viewer` directory.
+
+## TODOs
+
+### Viewer
+
+1. Deeplinking of search query and 'show only linked' params.
+1. Clicking on a search result will center view on it.
+1. Clicking search will run 'clear highlights' first. Currently accumulates.
+
+### Graph
+
+1. For step 'Generation of JSON', for tasks.json, just fetch from 'https://firefox-ci-tc.services.mozilla.com/api/index/v1/task/gecko.v2.mozilla-central.latest.taskgraph.decision/artifacts/public%2Ftask-graph.json'.
+1. Manage repos (don't take paths to them, do shallow clones)
+1. Fetch tasks.json from tc index task:
+    - https://firefox-ci-tc.services.mozilla.com/api/index/v1/task/gecko.v2.mozilla-central.latest.taskgraph.decision/artifacts/public%2Ftask-graph.json

ci-configuration-tools/taskcluster_cytoscape/cytoscape_viewer/.gitignore

@@ -0,0 +1 @@
+old/

ci-configuration-tools/taskcluster_cytoscape/cytoscape_viewer/README.md

@@ -0,0 +1,71 @@
+# Taskcluster Cytoscape Viewer
+
+Visualizes Firefox Taskcluster tasks, worker pools, image aliases, and images using [Cytoscope.js](https://js.cytoscape.org/).
+
+Task data is from [Firefox](https://github.com/mozilla-firefox/firefox)'s Taskgraph. All other data is from [fxci-config](http://github.com/mozilla-releng/fxci-config).
+
+## Why?
+
+Taskgraph is complicated and we frequently get questions like 'What tasks run on these systems?' and 'What pools use these images?'.
+
+This tool aims at helping people find those answers themselves.
+
+## Caveats
+
+- We lack task data for other repositories that submit tasks to fxci. For example, the translations GPU workers don't have any tasks in the dataset. Similarly, the VPN workers don't have any tasks.
+- Known Firefox DOM rendering bug (https://bugzilla.mozilla.org/show_bug.cgi?id=1974854). Chrome doesn't have this issue.
+- Hardware pools without tasks won't be visible. Consider scanning fxci taskcluster? Could be useful to be able to identify them.
+
+## How To Run on Web
+
+Currently deployed to: [https://aerickson.github.io/gh_pages_test/viewer.html](https://aerickson.github.io/gh_pages_test/viewer.html)
+
+## How To Run Locally
+
+### Unix
+
+```bash
+./serve.sh
+# surf to http://localhost:8080/viewer.html
+```
+
+### Windows
+
+Untested. Please report back.
+
+```bash
+# in powershell
+serve.ps1
+# surf to http://localhost:8080/viewer.html
+```
+
+## Usage
+
+You scroll around the display area by clicking and dragging.
+
+Zooming also works (two finger scrolling on Mac). Make sure to hover over the elements you want to zoom in on (it zooms exactly on the cursor location).
+
+Click around and explore for now. Send me feedback.
+
+### Color Guide
+
+```
+Purple: tasks
+Light Green: cloud pool
+Dark Green: hw pool
+Light Blue: image alias
+Light Yellow: l1 image
+Light Pink: l3 image
+```
+
+### Example 1: Find out where `perftest-*-summarizer*` tasks run.
+
+1. Type `summarizer` into search. Press return.
+2. Click the `Show only linked` button.
+3. Make it easier to view by Layout>Tidytree then click `Apply Layout`.
+
+### Example 2: Find out which pools and tasks use the `docker-worker-gcp-u14-04-2025-06-16` (docker-worker) image.
+
+1. Type `fxci-level1-gcp: docker-worker-gcp-u14-04-2025-06-16` into search. Press return.
+2. Click the `Show only linked` button.
+3. Make it easier to view by Layout>Tidytree then click `Apply Layout`.

ci-configuration-tools/taskcluster_cytoscape/cytoscape_viewer/serve.ps1

@@ -0,0 +1,2 @@
+# Start a simple HTTP server on port 8080 using Python 3
+python -m http.server 8080

ci-configuration-tools/taskcluster_cytoscape/cytoscape_viewer/serve.sh

@@ -0,0 +1,6 @@
+#!/usr/bin/env bash
+
+set -e
+
+python3 -m http.server 8080
+