Skip to content

fix: switch copaw-worker runtime entrypoint to qwenpaw#950

Open
LUOSENGWA wants to merge 4 commits into
agentscope-ai:mainfrom
LUOSENGWA:fix/copaw-to-qwenpaw-migration
Open

fix: switch copaw-worker runtime entrypoint to qwenpaw#950
LUOSENGWA wants to merge 4 commits into
agentscope-ai:mainfrom
LUOSENGWA:fix/copaw-to-qwenpaw-migration

Conversation

@LUOSENGWA

Copy link
Copy Markdown

Summary

Switch copaw-worker from copaw.app to qwenpaw.app as the runtime
entrypoint. qwenpaw is already present in the image as a dependency
of copaw; this change makes Workers use it directly, enabling
spawn_subagent and the full qwenpaw multi-agent feature set.

Changes (6 files, +31/-12)

  • worker.py: import path + uvicorn target → qwenpaw.app
  • bridge.py: inject agents.profiles into config.json (qwenpaw requirement)
  • channel.py: channel base import → qwenpaw.app
  • Dockerfile: overlay matrix channel → qwenpaw path; +pip install qwenpaw
  • health.py: docstring sync
  • test_worker_health.py: mock path sync

Validation

  • 18/18 Workers migrated in production, all Up, zero errors
  • spawn_subagent E2E: subagent creation → task execution → result returned
  • Matrix messaging (Element) + QQ channel both verified
  • All Workers running >24h with no regression

Relation to TeamHarness

This PR is a prerequisite for TeamHarness Phase 4 (QwenPaw Worker
integration). The adapter code in plugins/teamharness/adapters/qwenpaw/
already imports from qwenpaw.* modules and calls qwenpaw plugin install,
but those can't work until Workers actually start qwenpaw.app.

The agents.profiles injection in bridge.py feeds directly into
TeamHarness's _iter_qwenpaw_agents().

What this does NOT change

  • Runtime name "copaw" in CRDs (stable API, unchanged)
  • Controller image selection logic
  • HICLAW_COPAW_WORKER_IMAGE default value
  • No Go controller code touched

@github-actions

github-actions Bot commented Jun 21, 2026

Copy link
Copy Markdown
Contributor

📊 CI Metrics Report

Summary

Metric Current Baseline Change
LLM Calls 200 81 +119 ↑ +146.9%
Input Tokens 8252547 2803871 +5448676 ↑ +194.3%
Output Tokens 45191 16791 +28400 ↑ +169.1%
Total Tokens 8297738 2820662 +5477076 ↑ +194.2%

By Role

Role Metric Current Baseline Change
🧠 Manager LLM Calls 151 68 +83 ↑ +122.1%
Input Tokens 6901075 2502214 +4398861 ↑ +175.8%
Output Tokens 32486 13725 +18761 ↑ +136.7%
Total Tokens 6933561 2515939 +4417622 ↑ +175.6%
🔧 Workers LLM Calls 49 13 +36 ↑ +276.9%
Input Tokens 1351472 301657 +1049815 ↑ +348.0%
Output Tokens 12705 3066 +9639 ↑ +314.4%
Total Tokens 1364177 304723 +1059454 ↑ +347.7%

Per-Test Breakdown

Test Mgr Calls Wkr Calls Δ Calls Mgr In Wkr In Mgr Out Wkr Out Δ Tokens Trend
02-create-worker 10 0 -2 ↓ -16.7% 290220 0 1685 0 -66717 ↓ -18.6% ✅ improved
03-assign-task 10 7 +2 ↑ +13.3% 330465 165856 1448 1501 +25614 ↑ +5.4% ⚠️ regressed
04-human-intervene 23 0 +10 ↑ +76.9% 739236 0 4038 0 +310276 ↑ +71.7% ⚠️ regressed
05-heartbeat 15 0 +8 ↑ +114.3% 625275 0 3392 0 +353415 ↑ +128.4% ⚠️ regressed
06-multi-worker 93 42 +101 ↑ +297.1% 4915879 1185616 21923 11204 +4854488 ↑ +379.2% ⚠️ regressed

Trends

1 test(s) improved (fewer LLM calls)
⚠️ 4 test(s) regressed (more LLM calls)


Generated by HiClaw CI on 2026-07-07 17:48:55 UTC


📦 Download debug logs & test artifacts

@github-actions

github-actions Bot commented Jun 21, 2026

Copy link
Copy Markdown
Contributor

❌ Integration Tests Failed (controller-cr-2 / mgr=copaw / wk=copaw)

Commit: 87e6087
Workflow run: #1492

Test Results
No test output captured.
Debug Log (tail)
No debug logs available.

📦 Download full debug logs & test artifacts

@github-actions

github-actions Bot commented Jun 21, 2026

Copy link
Copy Markdown
Contributor

❌ Integration Tests Failed (llm-interaction-2 / mgr=copaw / wk=copaw)

Commit: 87e6087
Workflow run: #1492

Test Results
No test output captured.
Debug Log (tail)
No debug logs available.

📦 Download full debug logs & test artifacts

@github-actions

github-actions Bot commented Jun 21, 2026

Copy link
Copy Markdown
Contributor

❌ Integration Tests Failed (llm-interaction-2 / mgr=copaw / wk=hermes)

Commit: 87e6087
Workflow run: #1492

Test Results
No test output captured.
Debug Log (tail)
No debug logs available.

📦 Download full debug logs & test artifacts

@github-actions

github-actions Bot commented Jun 21, 2026

Copy link
Copy Markdown
Contributor

❌ Integration Tests Failed (llm-interaction / mgr=copaw / wk=copaw)

Commit: 87e6087
Workflow run: #1492

Test Results
No test output captured.
Debug Log (tail)
No debug logs available.

📦 Download full debug logs & test artifacts

@github-actions

github-actions Bot commented Jun 21, 2026

Copy link
Copy Markdown
Contributor

❌ Integration Tests Failed (llm-interaction / mgr=copaw / wk=hermes)

Commit: 9d7f67a
Workflow run: #1494

Test Results
No test output captured.
Debug Log (tail)
No debug logs available.

📦 Download full debug logs & test artifacts

@flystar32 flystar32 requested a review from maplefeng-a June 22, 2026 02:17
@flystar32

flystar32 commented Jun 22, 2026

Copy link
Copy Markdown
Collaborator

这个迁移方向是合理的,但当前实现还有几个合并前需要补齐的运行时接线问题。

  1. Worker 运行目录目前只接到了 copaw,没有明确接到 qwenpaw

copaw/scripts/copaw-worker-entrypoint.sh 只导出了 COPAW_WORKING_DIRcopaw/src/copaw_worker/bridge.py 也只 patch 了 copaw.constant / copaw.providers.store / copaw.envs.store。切到 qwenpaw.app._app:app 后,如果 qwenpaw 使用自己的 working dir 环境变量或 qwenpaw.* 模块状态,就可能读写默认目录,而不是 Worker 的 .copaw runtime dir。

建议二选一:

  • 同步导出 qwenpaw 对应的 working-dir 环境变量,并 patch qwenpaw.* 路径模块;
  • 或补充测试/说明,证明 qwenpaw.app 完全复用现有 copaw.* 路径状态。
  1. Matrix config overlay 可能还在写旧包路径

Dockerfile 已经把 Matrix channel 覆盖到 $SITE/qwenpaw/app/channels/,但 config.py 仍然复制到 $SITE/copaw/config/config.py。如果 qwenpaw.app 加载的是 qwenpaw.config.config,channel 和 config schema 会在运行时不一致。需要确认 import 路径,必要时也把 config overlay 写到 qwenpaw 包下。

  1. qwenpaw CLI 没有暴露到 PATH

PR 描述里说 TeamHarness adapter 会调用 qwenpaw plugin install,但镜像里目前只把 /opt/venv/copaw/bin/copaw symlink 到 /usr/local/bin/copaw。如果 qwenpaw console script 安装在 venv 里,运行时仍然可能找不到命令。建议暴露 venv bin 目录,或补一个 qwenpaw symlink。

  1. agents.profiles 注入逻辑太浅

现在用的是 cfg.setdefault("agents", {...}),只在 agents 整个不存在时生效。如果已有 config.json 里存在 agents 但缺少 profiles,qwenpaw 仍然拿不到必要的 profile 引用。建议改成嵌套 merge,在保留用户字段的同时确保这些字段存在:

  • agents.active_agent
  • agents.profiles.default.id
  • agents.profiles.default.workspace_dir
  1. 需要补 changelog

这个 PR 修改了 copaw/ 下会进入镜像的内容,按仓库规则需要在 changelog/current.md 记录一条变更。

另外,CI 评论里有旧提交上的集成测试失败,以及 06-multi-worker 的 token/call 明显上涨。建议在合并前重新跑最新 head,并明确给出 mgr=copaw / wk=copaw 路径的结果,因为这是本 PR 直接影响的路径。

最后,@maplefeng-a 也在做相关的工作,可以看看是否存在逻辑上不一致的地方。


This migration direction is reasonable, but the current implementation still has several runtime wiring issues that need to be patched before merging.

  1. The Worker running directory currently only receives copaw, but does not explicitly receive qwenpaw

copaw/scripts/copaw-worker-entrypoint.sh only exports COPAW_WORKING_DIR, and copaw/src/copaw_worker/bridge.py only patches copaw.constant / copaw.providers.store / copaw.envs.store. After switching to qwenpaw.app._app:app, if qwenpaw uses its own working dir environment variable or qwenpaw.* module state, it may read and write the default directory instead of the Worker's .copaw runtime dir.

It is recommended to choose one of the two:

  • Synchronously export the working-dir environment variable corresponding to qwenpaw, and patch the qwenpaw.* path module;
  • Or additional tests/explanations to demonstrate that qwenpaw.app fully reuses existing copaw.* path state.
  1. Matrix config overlay may still be writing the old package path

The Dockerfile has overwritten the Matrix channel to $SITE/qwenpaw/app/channels/, but config.py is still copied to $SITE/copaw/config/config.py. If qwenpaw.app loads qwenpaw.config.config, the channel and config schema will be inconsistent at runtime. You need to confirm the import path and write the config overlay to the qwenpaw package if necessary.

  1. qwenpaw CLI is not exposed to PATH

The PR description says that the TeamHarness adapter will call qwenpaw plugin install, but the image currently only symlinks /opt/venv/copaw/bin/copaw to /usr/local/bin/copaw. If the qwenpaw console script is installed in the venv, the command may still not be found when running. It is recommended to expose the venv bin directory or add a qwenpaw symlink.

  1. The injection logic of agents.profiles is too shallow

Currently, cfg.setdefault("agents", {...}) is used, which only takes effect when agents does not exist. If agents already exists in config.json but profiles is missing, qwenpaw still cannot get the necessary profile reference. It is recommended to change to a nested merge to ensure that these fields exist while retaining user fields:

  • agents.active_agent
  • agents.profiles.default.id
  • agents.profiles.default.workspace_dir
  1. Need to add changelog

This PR modifies the content that will enter the image under copaw/. According to the warehouse rules, a change needs to be recorded in changelog/current.md.

In addition, there are integration test failures on old commits in the CI comments, and the token/call of 06-multi-worker has increased significantly. It is recommended to re-run the latest head before merging, and clearly give the results of the mgr=copaw / wk=copaw path, because this is the path directly affected by this PR.

Finally, @mapplefeng-a is also doing related work to see if there are any logical inconsistencies.

LUOSENGWA pushed a commit to LUOSENGWA/HiClaw that referenced this pull request Jun 22, 2026
1. Working dir: export QWENPAW_WORKING_DIR alongside COPAW_WORKING_DIR
   so qwenpaw.constant resolves the runtime dir without relying on
   legacy COPAW_ fallback alone.

2. Dockerfile: add qwenpaw CLI symlink to PATH for TeamHarness
   adapter compatibility.

3. bridge.py: change agents.profiles injection from shallow
   setdefault() to nested merge. Previously, if config.json already
   had an 'agents' key, profiles/active_agent/workspace_dir were
   not injected. Now each sub-key is setdefault'd individually.

4. chanhelog: add PR agentscope-ai#950 entries for copaw/ image-affecting changes.

Note on Matrix config overlay: the maintainer asked about potential
inconsistency between the config.py overlay (copied to copaw/config/)
and qwenpaw.config.config. Analysis confirms no inconsistency:
- Matrix channel.py (overlaid to qwenpaw/app/channels/matrix/) does
  not import from config.py at all — it only depends on BaseChannel
  and agentscope_runtime, both available via the dual fallback.
- The HiClaw config overlay (copaw/src/matrix/config.py) adds
  HiClaw-specific channel types to copaw.config.config and is only
  needed by copaw.app.  qwenpaw.app loads qwenpaw.config.config
  (its own module), and the channel layer is config-agnostic.

@maplefeng-a maplefeng-a left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the follow-up. I checked the latest head, and this PR still has conflicts with main (copaw/Dockerfile, copaw/scripts/copaw-worker-entrypoint.sh, copaw/src/copaw_worker/bridge.py, and copaw/src/copaw_worker/worker.py).

Please rebase or merge main and resolve those conflicts first. Once the branch is clean again, I can continue the code review on the updated diff.


我看了最新 head,目前这个 PR 仍然和 main 有冲突,涉及 copaw/Dockerfilecopaw/scripts/copaw-worker-entrypoint.shcopaw/src/copaw_worker/bridge.pycopaw/src/copaw_worker/worker.py

请先 rebase 或 merge main 并解决冲突;分支干净后我再继续基于最新 diff 做代码 review。

LUOSENGWA pushed a commit to LUOSENGWA/HiClaw that referenced this pull request Jul 3, 2026
1. Working dir: export QWENPAW_WORKING_DIR alongside COPAW_WORKING_DIR
   so qwenpaw.constant resolves the runtime dir without relying on
   legacy COPAW_ fallback alone.

2. Dockerfile: add qwenpaw CLI symlink to PATH for TeamHarness
   adapter compatibility.

3. bridge.py: change agents.profiles injection from shallow
   setdefault() to nested merge. Previously, if config.json already
   had an 'agents' key, profiles/active_agent/workspace_dir were
   not injected. Now each sub-key is setdefault'd individually.

4. chanhelog: add PR agentscope-ai#950 entries for copaw/ image-affecting changes.

Note on Matrix config overlay: the maintainer asked about potential
inconsistency between the config.py overlay (copied to copaw/config/)
and qwenpaw.config.config. Analysis confirms no inconsistency:
- Matrix channel.py (overlaid to qwenpaw/app/channels/matrix/) does
  not import from config.py at all — it only depends on BaseChannel
  and agentscope_runtime, both available via the dual fallback.
- The HiClaw config overlay (copaw/src/matrix/config.py) adds
  HiClaw-specific channel types to copaw.config.config and is only
  needed by copaw.app.  qwenpaw.app loads qwenpaw.config.config
  (its own module), and the channel layer is config-agnostic.
@LUOSENGWA LUOSENGWA force-pushed the fix/copaw-to-qwenpaw-migration branch from 3014704 to 9911d12 Compare July 3, 2026 13:09
@github-actions

github-actions Bot commented Jul 3, 2026

Copy link
Copy Markdown
Contributor

❌ Integration Tests Failed (controller-cr / mgr=copaw / wk=hermes)

Commit: 9898301
Workflow run: #1483

Test Results
No test output captured.
Debug Log (tail)
No debug logs available.

📦 Download full debug logs & test artifacts

@github-actions

github-actions Bot commented Jul 3, 2026

Copy link
Copy Markdown
Contributor

❌ Integration Tests Failed (controller-cr / mgr=copaw / wk=copaw)

Commit: 9898301
Workflow run: #1483

Test Results
No test output captured.
Debug Log (tail)
No debug logs available.

📦 Download full debug logs & test artifacts

@github-actions

github-actions Bot commented Jul 3, 2026

Copy link
Copy Markdown
Contributor

❌ Integration Tests Failed (controller-cr / mgr=openclaw / wk=openclaw)

Commit: 9898301
Workflow run: #1483

Test Results
No test output captured.
Debug Log (tail)
No debug logs available.

📦 Download full debug logs & test artifacts

LUOSENGWA pushed a commit to LUOSENGWA/HiClaw that referenced this pull request Jul 4, 2026
… logging (agentscope-ai#950)

- Replace qwenpaw health API (false positive: token check != sync complete)
  with Matrix /_matrix/client/v3/joined_rooms polling in background task.
  Marker file written when >= 1 room joined → true readiness signal.

- Add logger output to build_worker_readiness() model check to eliminate
  diagnostic blind spot (previously model check was silent during probes).

- test-helpers.sh: print last readyz JSON on probe timeout; increase
  docker logs --tail from 100 to 200 to preserve startup diagnostics.
@LUOSENGWA LUOSENGWA force-pushed the fix/copaw-to-qwenpaw-migration branch from 833b17c to 59782d5 Compare July 4, 2026 17:52
LUOSENGWA pushed a commit to LUOSENGWA/HiClaw that referenced this pull request Jul 4, 2026
… logging (agentscope-ai#950)

- Replace qwenpaw health API (false positive: token check != sync complete)
  with Matrix /_matrix/client/v3/joined_rooms polling in background task.
  Marker file written when >= 1 room joined -> true readiness signal.

- Add logger output to build_worker_readiness() model check to eliminate
  diagnostic blind spot (previously model check was silent during probes).

- test-helpers.sh: print last readyz JSON on probe timeout; increase
  docker logs --tail from 100 to 200 to preserve startup diagnostics.
LUOSENGWA pushed a commit to LUOSENGWA/HiClaw that referenced this pull request Jul 5, 2026
1. Working dir: export QWENPAW_WORKING_DIR alongside COPAW_WORKING_DIR
   so qwenpaw.constant resolves the runtime dir without relying on
   legacy COPAW_ fallback alone.

2. Dockerfile: add qwenpaw CLI symlink to PATH for TeamHarness
   adapter compatibility.

3. bridge.py: change agents.profiles injection from shallow
   setdefault() to nested merge. Previously, if config.json already
   had an 'agents' key, profiles/active_agent/workspace_dir were
   not injected. Now each sub-key is setdefault'd individually.

4. chanhelog: add PR agentscope-ai#950 entries for copaw/ image-affecting changes.

Note on Matrix config overlay: the maintainer asked about potential
inconsistency between the config.py overlay (copied to copaw/config/)
and qwenpaw.config.config. Analysis confirms no inconsistency:
- Matrix channel.py (overlaid to qwenpaw/app/channels/matrix/) does
  not import from config.py at all — it only depends on BaseChannel
  and agentscope_runtime, both available via the dual fallback.
- The HiClaw config overlay (copaw/src/matrix/config.py) adds
  HiClaw-specific channel types to copaw.config.config and is only
  needed by copaw.app.  qwenpaw.app loads qwenpaw.config.config
  (its own module), and the channel layer is config-agnostic.
LUOSENGWA pushed a commit to LUOSENGWA/HiClaw that referenced this pull request Jul 5, 2026
… logging (agentscope-ai#950)

- Replace qwenpaw health API (false positive: token check != sync complete)
  with Matrix /_matrix/client/v3/joined_rooms polling in background task.
  Marker file written when >= 1 room joined -> true readiness signal.

- Add logger output to build_worker_readiness() model check to eliminate
  diagnostic blind spot (previously model check was silent during probes).

- test-helpers.sh: print last readyz JSON on probe timeout; increase
  docker logs --tail from 100 to 200 to preserve startup diagnostics.
@LUOSENGWA LUOSENGWA force-pushed the fix/copaw-to-qwenpaw-migration branch 2 times, most recently from e69bcea to b791cd6 Compare July 5, 2026 10:56
@github-actions

github-actions Bot commented Jul 5, 2026

Copy link
Copy Markdown
Contributor

❌ Integration Tests Failed (llm-interaction / mgr=openclaw / wk=openclaw)

Commit: f098ebd
Workflow run: #1490

Test Results
No test output captured.
Debug Log (tail)
No debug logs available.

📦 Download full debug logs & test artifacts

@LUOSENGWA LUOSENGWA force-pushed the fix/copaw-to-qwenpaw-migration branch from 9898301 to 9bd3cb2 Compare July 6, 2026 16:42
Worker changes:
- copaw/Dockerfile: install qwenpaw after copaw-worker (standard venv)
- entrypoint.sh: set QWENPAW_WORKING_DIR for qwenpaw config path
- worker.py: switch copaw.app._app → qwenpaw.app._app (console + headless)
- bridge.py: inject agents.profiles into config.json (required by qwenpaw)
- matrix_channel.py: dual fallback — try qwenpaw BaseChannel first, then copaw
- health.py: update docstring reference

Manager changes:
- Dockerfile.copaw: install qwenpaw after copaw-worker overlay (agentscope 1.0.20)
- start-copaw-manager.sh: export QWENPAW_WORKING_DIR + launch python3 -m qwenpaw app

CI health check changes:
- test-helpers.sh: pgrep checks for 'copaw app' OR 'qwenpaw app'
- test-01-manager-boot.sh: same dual pgrep pattern
@LUOSENGWA LUOSENGWA force-pushed the fix/copaw-to-qwenpaw-migration branch from 87e6087 to b3d1e5b Compare July 7, 2026 14:01
LUOSENGWA added 3 commits July 7, 2026 15:32
…e, qwenpaw symlink

- changelog/current.md: add copaw→qwenpaw migration entry
- copaw/Dockerfile: ln -sf qwenpaw to /usr/local/bin for CLI PATH
- bridge.py: nested setdefault merge for agents.profiles (handle
  partially-populated blocks instead of only top-level setdefault)
The worker Dockerfile has two venvs: standard (console mode) and
lite (headless mode, default when no CONSOLE_PORT is set). Both
worker.py code paths (_run_copaw_with_console and _run_copaw_headless)
now import from qwenpaw, but qwenpaw was only installed in the
standard venv. Lite mode would crash with ImportError on startup.

CI only tests standard mode (with console port), so this bug was
invisible in CI. Production Workers without console ports would be
affected.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants