Support rooted task and split CI tests #12
Conversation
…r execution - Separate unit tests from integration tests into distinct jobs - Add pytest-split to run integration tests in 3 parallel groups - Move Docker services to GitHub Actions service containers - Unit tests: 3 jobs (Python 3.11, 3.12, 3.13) - Integration tests: 18 jobs (Python × Hatchet × 3 groups)
There was a problem hiding this comment.
Pull request overview
This PR adds support for rooted task functionality to mageflow and restructures the CI pipeline to run tests in parallel. The rooted task feature allows tasks to be marked as "root tasks" that automatically create and manage a swarm context for child tasks. The CI improvements split unit tests from integration tests and parallelize integration tests across 3 groups using pytest-split, reducing overall CI runtime.
Key Changes
- Added root task infrastructure with
RootTaskSignatureclass and context management - Split CI tests into separate unit and integration test jobs with parallel execution
- Reorganized imports across the codebase to follow consistent alphabetical ordering
Reviewed changes
Copilot reviewed 47 out of 48 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
mageflow/root/model.py |
New RootTaskSignature class with swarm lifecycle management |
mageflow/root/context.py |
Context variable for tracking current root swarm |
mageflow/root/consts.py |
Constants for root task markers and config |
mageflow/signature/model.py |
Added root swarm integration to task execution |
mageflow/signature/creator.py |
Modified sign() to create RootTaskSignature for root tasks |
mageflow/client.py |
Added root_task() decorator for marking tasks as root |
mageflow/callbacks.py |
Added end_task() call in task lifecycle |
mageflow/invokers/hatchet.py |
Added start_task() and end_task() lifecycle hooks |
mageflow/task/model.py |
Added is_root_task and root_task_config fields |
mageflow/startup.py |
Extended REGISTERED_TASKS to include root task metadata |
.github/workflows/ci.yml |
Split into unit-tests and integration-tests jobs with parallelization |
tox.ini |
Separated unit and integration test environments |
pyproject.toml |
Added pytest-split dependency |
| Various test files | Import reorganization (alphabetical ordering) |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| root_swarm = await SwarmTaskSignature.get_safe(self.swarm_id) | ||
| if root_swarm: | ||
| await root_swarm.close_swarm() | ||
| if not success: |
There was a problem hiding this comment.
If root_swarm is None (when self.swarm_id is None or the swarm doesn't exist), calling await root_swarm.suspend() on line 40 will raise an AttributeError. Add a check to ensure root_swarm is not None before calling methods on it.
| if not success: | |
| if not success and root_swarm: |
mageflow/callbacks.py
Outdated
| await invoker.remove_task(with_error=False) | ||
| raise | ||
| else: | ||
| await invoker.end_task(success=True) |
There was a problem hiding this comment.
The invoker.end_task() method is being called with a keyword argument success=True, but the method signature in HatchetInvoker.end_task() (line 36 of mageflow/invokers/hatchet.py) does not accept any parameters. This will cause a TypeError at runtime. Either remove the keyword argument here or update the HatchetInvoker.end_task() method signature to accept a success parameter.
| await invoker.end_task(success=True) | |
| await invoker.end_task() |
mageflow/signature/model.py
Outdated
| async def aio_run_no_wait(self, msg: BaseModel, **kwargs): | ||
| root_swarm = current_root_swarm.get() | ||
| if root_swarm: | ||
| batch_item = await swarm.add_task(self) |
There was a problem hiding this comment.
The swarm module is used here but is not imported. This will cause a NameError at runtime when a root swarm context is active. Add the appropriate import statement at the top of the file.
| from threading import Thread | ||
| from typing import Generator, Callable, AsyncGenerator | ||
|
|
||
| import mageflow |
There was a problem hiding this comment.
Module 'mageflow' is imported with both 'import' and 'import from'.
| import mageflow |
…tead of GitHub services GitHub Actions service containers don't support depends_on, causing hatchet-lite to fail connecting to postgres before it's ready. Using docker-compose properly handles service dependencies.
|
This pull request sets up GitHub code scanning for this repository. Once the scans have completed and the checks have passed, the analysis results for this pull request branch will appear on this overview. Once you merge this pull request, the 'Security' tab will show more code scanning analysis results (for example, for the default branch). Depending on your configuration and choice of analysis tool, future pull requests will be annotated with code scanning analysis results. For more information about GitHub code scanning, check out the documentation. |
This workflow runs Bandit, a security linter for Python, on codebase changes.
Summary
Rooted Task Features
CI Improvements
Test plan