fix: adding new script to run the jenkin job#313
fix: adding new script to run the jenkin job#313ktyagiapphelix2u wants to merge 2 commits intomasterfrom
Conversation
There was a problem hiding this comment.
Pull request overview
Adds a new Jenkins utility shell script for creating job-scoped Python virtual environments using Python’s built-in venv module (intended to improve compatibility with Python 3.12+).
Changes:
- Introduces
util/jenkins/virtualenv_python_tools.shwithcreate_virtualenv_pythonto create venvs under$JOBVENVDIR. - Adds basic argument parsing for
--python=...and--clear. - Automatically upgrades
pip/setuptools/wheelafter venv creation.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| venvpath="$JOBVENVDIR/$venvname" | ||
|
|
||
| # create the virtualenv using python's venv module | ||
| "$python_bin" -m venv $clear_flag "$venvpath" |
There was a problem hiding this comment.
If "$python_bin" -m venv ... fails, the script still attempts to run "$venvpath/bin/python" -m pip ..., which can mask the real failure with a secondary "file not found" error. Add an early return/check after the venv creation command so the function exits immediately on failure.
| "$python_bin" -m venv $clear_flag "$venvpath" | |
| if ! "$python_bin" -m venv $clear_flag "$venvpath"; then | |
| echo "Failed to create virtualenv at '$venvpath' using '$python_bin'" >&2 | |
| return 1 | |
| fi |
| venvpath="$JOBVENVDIR/$venvname" | ||
|
|
||
| # create the virtualenv using python's venv module | ||
| "$python_bin" -m venv $clear_flag "$venvpath" |
There was a problem hiding this comment.
$clear_flag is expanded unquoted in the venv creation command. Even though it is currently controlled, using an array (or quoting/conditional inclusion) avoids word-splitting/globbing issues and matches common shell-scripting safety practices.
| "$python_bin" -m venv $clear_flag "$venvpath" | |
| if [ -n "$clear_flag" ]; then | |
| "$python_bin" -m venv "$clear_flag" "$venvpath" | |
| else | |
| "$python_bin" -m venv "$venvpath" | |
| fi |
| # Upgrade pip to avoid distutils issues with Python 3.12+ | ||
| "$venvpath/bin/python" -m pip install --upgrade pip setuptools wheel |
There was a problem hiding this comment.
The unconditional pip install --upgrade pip setuptools wheel will run on every invocation (even when reusing an existing venv), adding repeated network traffic and job time. Consider only running the upgrade when the venv is newly created/cleared, or gate it behind an environment flag so callers can opt out in performance-sensitive jobs.
| function create_virtualenv_python () { | ||
| if [ -z "${JOBVENVDIR:-}" ] | ||
| then | ||
| echo "No JOBVENVDIR found. Using default value." >&2 | ||
| JOBVENVDIR="/edx/var/jenkins/jobvenvs" | ||
| fi |
There was a problem hiding this comment.
Indentation inside create_virtualenv_python is inconsistent with the established style in util/jenkins/virtualenv_tools.sh (e.g., create_virtualenv indents the whole function body). Re-indent the body of this function to match the existing convention for readability and consistency.
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 3 out of 3 changed files in this pull request and generated 5 comments.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
.github/workflows/syntax-test.yml
Outdated
| max-parallel: 4 | ||
| matrix: | ||
| python-version: [3.9] | ||
| python-version: ['3.10'] |
There was a problem hiding this comment.
PR title suggests this change is only about adding a Jenkins script, but this PR also changes GitHub Actions CI (Python version and requirements file). Consider updating the PR title/description to reflect the CI changes, or splitting CI adjustments into a separate PR for clearer review/rollback.
| # create a unique hash for the job based location of where job is run | ||
| # and the python binary used | ||
| venvname="$( (echo "$python_bin"; pwd) | md5sum | cut -d' ' -f1 )" | ||
| venvpath="$JOBVENVDIR/$venvname" |
There was a problem hiding this comment.
venvname is assigned without local, which leaks it into the caller’s environment when this script is sourced. In util/jenkins/virtualenv_tools.sh, only venvpath is intentionally global; consider making venvname local here as well to avoid polluting the environment and accidental name collisions.
.github/workflows/syntax-test.yml
Outdated
| @@ -24,7 +24,7 @@ jobs: | |||
| - name: Install Dependencies | |||
| run: | | |||
| pip install demjson3 | |||
| pip install -r requirements.txt | |||
| pip install -r requirements3_12.txt | |||
There was a problem hiding this comment.
The workflow installs requirements3_12.txt (generated with Python 3.12 per its header) but runs under Python 3.10. This mismatch can lead to install failures or subtly different dependency resolution; align the interpreter with the lockfile (use Python 3.12) or install the requirements file intended for 3.10.
.github/workflows/playbook-test.yml
Outdated
| @@ -24,7 +24,7 @@ jobs: | |||
| - name: Install Dependencies | |||
| run: | | |||
| pip install demjson3 | |||
| pip install -r requirements.txt | |||
| pip install -r requirements3_12.txt | |||
There was a problem hiding this comment.
The workflow installs requirements3_12.txt (generated with Python 3.12 per its header) but runs under Python 3.10. This mismatch can lead to install failures or subtly different dependency resolution; align the interpreter with the lockfile (use Python 3.12) or install the requirements file intended for 3.10.
| "$python_bin" -m venv $clear_flag "$venvpath" | ||
|
|
||
| # Upgrade pip to avoid distutils issues with Python 3.12+ | ||
| "$venvpath/bin/python" -m pip install --upgrade pip setuptools wheel |
There was a problem hiding this comment.
This line unconditionally runs pip install --upgrade pip setuptools wheel in the Jenkins job environment, pulling the latest versions from the public index without pinning or integrity verification. If PyPI or any of these packages are compromised (or a malicious mirror/registry is configured), an attacker could execute arbitrary code in the CI environment and access build secrets or modify artifacts. To reduce this supply chain risk, install from a pinned/hashed requirements or constraints file (or an internal, trusted mirror) instead of unpinned --upgrade from the public index.
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 25 out of 25 changed files in this pull request and generated 4 comments.
Comments suppressed due to low confidence (6)
playbooks/roles/testcourses/tasks/main.yml:36
- Previously this used
include: deploy.yml tags=deploy, which applies thedeploytag to all tasks inside deploy.yml.include_tasksdoes not accept thetags=include-parameter; if you still need included tasks to be selectable via--tags deploy, useapply: { tags: [deploy] }(or switch toimport_tasksif a static include is acceptable).
- include_tasks: deploy.yml
tags:
- deploy
playbooks/roles/hotg/tasks/main.yml:144
- Previously this was
include: deploy.yml tags=deploy, which tags the tasks inside deploy.yml. Withinclude_tasks, ensure the included tasks still receive thedeploytag (e.g., viaapply: { tags: [deploy] }, or useimport_tasksif appropriate) so tag-scoped runs behave the same.
- include_tasks: deploy.yml
tags:
- deploy
playbooks/create_rds.yml:88
- Indentation here places
include_tasksat the play level rather than under thetasks:list. As written, this makes the playbook YAML structure invalid (or at least not part of the play’s tasks). It should be indented to the same level as the other tasks undertasks:.
- include_tasks: create_db_and_users.yml
when: database_connection.login_host is defined
playbooks/roles/gitreload/tasks/main.yml:87
- Previously this was
include: deploy.yml tags=deploy, which tags tasks inside deploy.yml. Withinclude_tasks, preserve that behavior usingapplytags (orimport_tasks) so--tags deployruns still execute the included tasks.
- include_tasks: deploy.yml
tags:
- deploy
playbooks/roles/demo/tasks/main.yml:43
- Previously this was
include: deploy.yml tags=deploy, which tags tasks inside deploy.yml. Withinclude_tasks, useapply: { tags: [deploy] }(orimport_tasks) if you need tag-filtered runs to behave identically.
- include_tasks: deploy.yml
tags:
- deploy
playbooks/roles/edxapp/tasks/deploy.yml:521
- This include used to be
include: tag_ec2.yml tags=deploybut now the include task is taggedremove/awsand no longer applies thedeploytag to the tasks inside tag_ec2.yml. Since tag_ec2.yml doesn’t define its own tags, this changes behavior for--tags deployruns (and may cause it to run only underremove). Consider restoringdeploytagging viaapply: { tags: [deploy] }and/or adjusting the include task’stags:to match the previous intent.
- include_tasks: tag_ec2.yml
when: COMMON_TAG_EC2_INSTANCE
tags:
- remove
- aws
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| # Usage: | ||
| # . /edx/var/jenkins/jobvenvs/virtualenv_python_tools.sh | ||
| # create_virtualenv_python --python=python3.12 --clear | ||
| # . "$venvpath/bin/activate" |
There was a problem hiding this comment.
This new helper script is not referenced anywhere else in the repo (no Jenkins job, playbook, or role copies/sources it). Given the PR title implies it will be used to run Jenkins jobs, please either (a) wire it into the Jenkins provisioning (e.g., copy to the job venv directory alongside virtualenv_tools.sh) or (b) update documentation/PR scope to clarify how it’s intended to be consumed.
.github/workflows/syntax-test.yml
Outdated
| - name: Install Dependencies | ||
| run: | | ||
| pip install demjson3 | ||
| pip install -r requirements.txt | ||
| pip install -r requirements3_12.txt |
There was a problem hiding this comment.
Installing requirements3_12.txt in this job while using Python 3.10 can fail if any dependencies are 3.12-only. Consider switching to Python 3.12 for this workflow or using a requirements lockfile compiled for the selected Python.
.github/workflows/playbook-test.yml
Outdated
| max-parallel: 4 | ||
| matrix: | ||
| python-version: [3.9] | ||
| python-version: ['3.10'] |
There was a problem hiding this comment.
This workflow installs requirements3_12.txt but the matrix Python version is 3.10. Please align the Python version with the lockfile (e.g., use 3.12) or use a requirements file compiled for 3.10.
| python-version: ['3.10'] | |
| python-version: ['3.12'] |
.github/workflows/playbook-test.yml
Outdated
| - name: Install Dependencies | ||
| run: | | ||
| pip install demjson3 | ||
| pip install -r requirements.txt | ||
| pip install -r requirements3_12.txt |
There was a problem hiding this comment.
Installing requirements3_12.txt under Python 3.10 can introduce dependency incompatibilities. Either bump this workflow to Python 3.12 or switch to an appropriate requirements file for the selected interpreter.
ktyagiapphelix2u
force-pushed
the
ktyagi/archiver
branch
from
640365b to
41a3a8c
Compare
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 1 out of 1 changed files in this pull request and generated 1 comment.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| # create a unique hash for the job based location of where job is run | ||
| # and the python binary used | ||
| venvname="$( (echo "$python_bin"; pwd) | md5sum | cut -d' ' -f1 )" |
There was a problem hiding this comment.
venvname is derived only from python_bin and pwd, whereas the existing util/jenkins/virtualenv_tools.sh hashes over all creation args (virtualenv_tools.sh:40). If you later add any venv options (e.g., system-site-packages, prompt, upgrade) they could silently collide onto the same venv directory. Consider hashing over the full set of effective venv-creation inputs (e.g., python_bin + flags) to keep the venv path uniquely tied to how it was created.
| # create a unique hash for the job based location of where job is run | |
| # and the python binary used | |
| venvname="$( (echo "$python_bin"; pwd) | md5sum | cut -d' ' -f1 )" | |
| # create a unique hash for the job based location of where job is run, | |
| # the python binary used, and the venv creation flags | |
| venvname="$(printf '%s\n' "$python_bin" "$(pwd)" "$clear_flag" | md5sum | cut -d' ' -f1)" |
| # parse. | ||
|
|
||
| function create_virtualenv_python () { | ||
| if [ -z "${JOBVENVDIR:-}" ] |
There was a problem hiding this comment.
[nit] Function body should be indented.
| venvpath="$JOBVENVDIR/$venvname" | ||
|
|
||
| # create the virtualenv using python's venv module | ||
| "$python_bin" -m venv $clear_flag "$venvpath" |
There was a problem hiding this comment.
I'd recommend a different argument structure -- mandatory first argument that names a python executable, and then arbitrary additional args that will be passed to the venv module.
So you'd shift off the first arg, which would be python3.12 or a path or something, and save that directly to $python_bin. And the rest of the args would be passed to venv.
This line would end up looking like this:
| "$python_bin" -m venv $clear_flag "$venvpath" | |
| "$python_bin" -m venv "$venvpath" "$@" |
You'd be able to drop all the arguments parsing (except for the assign-and-shift) and the assignment of a default value (which creates long-range hidden dependencies that would likely bite us later).
2 participants
Description
Adds a new Jenkins utility shell script for creating job-scoped Python virtual environments using Python’s built-in venv module (intended to improve compatibility with Python 3.12+).
Related PR
edx/jenkins-job-dsl#1827