Merge pull request #39 from nexB/update-skeleton

Update skeleton

Author: JonoYang

.gitignore

@@ -12,6 +12,7 @@
 /Lib
 /pip-selfcheck.json
 /tmp
+/venv
 .Python
 /include
 /Include

.travis.yml

@@ -19,4 +19,4 @@ python:
 install: ./configure --dev
 
 # Scripts to run at script stage
-script: tmp/bin/pytest
+script: venv/bin/pytest

README.rst

@@ -2,7 +2,7 @@ A Simple Python Project Skeleton
 ================================
 This repo attempts to standardize our python repositories using modern python
 packaging and configuration techniques. Using this `blog post`_ as inspiration, this
-repository will serve as the base for all new python projects and will be adopted to all 
+repository will serve as the base for all new python projects and will be adopted to all
 our existing ones as well.
 
 .. _blog post: https://blog.jaraco.com/a-project-skeleton-for-python-projects/
@@ -33,18 +33,148 @@ Update an existing project
 
 This is also the workflow to use when updating the skeleton files in any given repository.
 
-
 Customizing
 -----------
 
 You typically want to perform these customizations:
 
 - remove or update the src/README.rst and tests/README.rst files
+- set project info and dependencies in setup.cfg
 - check the configure and configure.bat defaults
 
+Initializing a project
+----------------------
+
+All projects using the skeleton will be expected to pull all of it dependencies
+from thirdparty.aboutcode.org/pypi or the local thirdparty directory, using
+requirements.txt and/or requirements-dev.txt to determine what version of a
+package to collect. By default, PyPI will not be used to find and collect
+packages from.
+
+In the case where we are starting a new project where we do not have
+requirements.txt and requirements-dev.txt and whose dependencies are not yet on
+thirdparty.aboutcode.org/pypi, we run the following command after adding and
+customizing the skeleton files to your project:
+
+.. code-block:: bash
+
+    ./configure --init
+
+This will initialize the virtual environment for the project, pull in the
+dependencies from PyPI and add them to the virtual environment.
+
+Generating requirements.txt and requirements-dev.txt
+----------------------------------------------------
+
+After the project has been initialized, we can generate the requirements.txt and
+requirements-dev.txt files.
+
+Ensure the virtual environment is enabled.
+
+.. code-block:: bash
+
+    source venv/bin/activate
+
+To generate requirements.txt:
+
+.. code-block:: bash
+
+    python etc/scripts/gen_requirements.py -s venv/lib/python<version>/site-packages/
+
+Replace \<version\> with the version number of the Python being used, for example: ``venv/lib/python3.6/site-packages/``
+
+To generate requirements-dev.txt after requirements.txt has been generated:
+
+.. code-block:: bash
+    ./configure --init --dev
+    python etc/scripts/gen_requirements_dev.py -s venv/lib/python<version>/site-packages/
+
+Note: on Windows, the ``site-packages`` directory is located at ``venv\Lib\site-packages\``
+
+.. code-block:: bash
+
+    python .\\etc\\scripts\\gen_requirements.py -s .\\venv\\Lib\\site-packages\\
+    .\configure --init --dev
+    python .\\etc\\scripts\\gen_requirements_dev.py -s .\\venv\\Lib\\site-packages\\
+
+Collecting and generating ABOUT files for dependencies
+------------------------------------------------------
+
+Ensure that the dependencies used by ``etc/scripts/bootstrap.py`` are installed:
+
+.. code-block:: bash
+
+    pip install -r etc/scripts/requirements.txt
+
+Once we have requirements.txt and requirements-dev.txt, we can fetch the project
+dependencies as wheels and generate ABOUT files for them:
+
+.. code-block:: bash
+
+    python etc/scripts/bootstrap.py -r requirements.txt -r requirements-dev.txt --with-deps
+
+There may be issues with the generated ABOUT files, which will have to be
+corrected. You can check to see if your corrections are valid by running:
+
+.. code-block:: bash
+
+    python etc/scripts/check_thirdparty.py -d thirdparty
+
+Once the wheels are collected and the ABOUT files are generated and correct,
+upload them to thirdparty.aboutcode.org/pypi by placing the wheels and ABOUT
+files from the thirdparty directory to the pypi directory at
+https://github.com/nexB/thirdparty-packages
+
+
+Usage after project initialization
+----------------------------------
+
+Once the ``requirements.txt`` and ``requirements-dev.txt`` have been generated
+and the project dependencies and their ABOUT files have been uploaded to
+thirdparty.aboutcode.org/pypi, you can configure the project without using the
+``--init`` option.
+
+If the virtual env for the project becomes polluted, or you would like to remove
+it, use the ``--clean`` option:
+
+.. code-block:: bash
+
+    ./configure --clean
+
+Then you can run ``./configure`` again to set up the project virtual environment.
+
+To set up the project for development use:
+
+.. code-block:: bash
+
+    ./configure --dev
+
+To update the project dependencies (adding, removing, updating packages, etc.),
+update the dependencies in ``setup.cfg``, then run:
+
+.. code-block:: bash
+
+    ./configure --clean # Remove existing virtual environment
+    ./configure --init # Create project virtual environment, pull in new dependencies
+    source venv/bin/activate # Ensure virtual environment is activated
+    python etc/scripts/gen_requirements.py -s venv/lib/python<version>/site-packages/ # Regenerate requirements.txt
+    python etc/scripts/gen_requirements_dev.py -s venv/lib/python<version>/site-packages/ # Regenerate requirements-dev.txt
+    pip install -r etc/scripts/requirements.txt # Install dependencies needed by etc/scripts/bootstrap.py
+    python etc/scripts/bootstrap.py -r requirements.txt -r requirements-dev.txt --with-deps # Collect dependency wheels and their ABOUT files
+
+Ensure that the generated ABOUT files are valid, then take the dependency wheels
+and ABOUT files and upload them to thirdparty.aboutcode.org/pypi.
 
 Release Notes
--------------
+=============
+
+- 2021-09-03:
+    - ``configure`` now requires pinned dependencies via the use of ``requirements.txt`` and ``requirements-dev.txt``
+    - ``configure`` can now accept multiple options at once
+    - Add utility scripts from scancode-toolkit/etc/release/ for use in generating project files
+    - Rename virtual environment directory from ``tmp`` to ``venv``
+    - Update README.rst with instructions for generating ``requirements.txt`` and ``requirements-dev.txt``,
+      as well as collecting dependencies as wheels and generating ABOUT files for them.
 
-- 2021-05-11: adopt new configure scripts from ScanCode TK that allows correct
-  configuration of which Python version is used.
+- 2021-05-11:
+    - Adopt new configure scripts from ScanCode TK that allows correct configuration of which Python version is used.

azure-pipelines.yml

@@ -13,52 +13,52 @@ jobs:
           image_name: ubuntu-16.04
           python_versions: ['3.6', '3.7', '3.8', '3.9']
           test_suites:
-              all: tmp/bin/pytest -vvs
+              all: venv/bin/pytest -vvs
 
     - template: etc/ci/azure-posix.yml
       parameters:
           job_name: ubuntu18_cpython
           image_name: ubuntu-18.04
           python_versions: ['3.6', '3.7', '3.8', '3.9']
           test_suites:
-              all: tmp/bin/pytest -n 2 -vvs
+              all: venv/bin/pytest -n 2 -vvs
 
     - template: etc/ci/azure-posix.yml
       parameters:
           job_name: ubuntu20_cpython
           image_name: ubuntu-20.04
           python_versions: ['3.6', '3.7', '3.8', '3.9']
           test_suites:
-              all: tmp/bin/pytest -n 2 -vvs
+              all: venv/bin/pytest -n 2 -vvs
 
     - template: etc/ci/azure-posix.yml
       parameters:
           job_name: macos1014_cpython
           image_name: macos-10.14
           python_versions: ['3.6', '3.7', '3.8', '3.9']
           test_suites:
-              all: tmp/bin/pytest -n 2 -vvs
+              all: venv/bin/pytest -n 2 -vvs
 
     - template: etc/ci/azure-posix.yml
       parameters:
           job_name: macos1015_cpython
           image_name: macos-10.15
           python_versions: ['3.6', '3.7', '3.8', '3.9']
           test_suites:
-              all: tmp/bin/pytest -n 2 -vvs
+              all: venv/bin/pytest -n 2 -vvs
 
     - template: etc/ci/azure-win.yml
       parameters:
           job_name: win2016_cpython
           image_name: vs2017-win2016
           python_versions: ['3.6', '3.7', '3.8', '3.9']
           test_suites:
-              all: tmp\Scripts\pytest -n 2 -vvs
+              all: venv\Scripts\pytest -n 2 -vvs
 
     - template: etc/ci/azure-win.yml
       parameters:
           job_name: win2019_cpython
           image_name: windows-2019
           python_versions: ['3.6', '3.7', '3.8', '3.9']
           test_suites:
-              all: tmp\Scripts\pytest -n 2 -vvs
+              all: venv\Scripts\pytest -n 2 -vvs

configure

@@ -26,16 +26,16 @@ CLI_ARGS=$1
 ################################
 
 # Requirement arguments passed to pip and used by default or with --dev.
-REQUIREMENTS="--editable ."
-DEV_REQUIREMENTS="--editable .[testing]"
+REQUIREMENTS="--editable . --constraint requirements.txt"
+DEV_REQUIREMENTS="--editable .[testing] --constraint requirements.txt --constraint requirements-dev.txt"
 
 # where we create a virtualenv
-VIRTUALENV_DIR=tmp
+VIRTUALENV_DIR=venv
 
 # Cleanable files and directories with the --clean option
 CLEANABLE="
     build
-	tmp"
+	venv"
 
 # extra  arguments passed to pip
 PIP_EXTRA_ARGS=" "
@@ -50,8 +50,9 @@ VIRTUALENV_PYZ_URL=https://bootstrap.pypa.io/virtualenv.pyz
 CFG_ROOT_DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"
 CFG_BIN_DIR=$CFG_ROOT_DIR/$VIRTUALENV_DIR/bin
 
-# Find packages from the local thirdparty directory or from pypi
-PIP_EXTRA_ARGS="--find-links $CFG_ROOT_DIR/thirdparty"
+# Find packages from the local thirdparty directory or from thirdparty.aboutcode.org
+PIP_EXTRA_ARGS="--find-links $CFG_ROOT_DIR/thirdparty --find-links https://thirdparty.aboutcode.org/pypi"
+
 
 ################################
 # Set the quiet flag to empty if not defined
@@ -80,10 +81,14 @@ cli_help() {
     echo "  usage: ./configure [options]"
     echo
     echo The default is to configure for regular use. Use --dev for development.
+    echo Use the --init option if starting a new project and the project
+    echo dependencies are not available on thirdparty.aboutcode.org/pypi/
+    echo and requirements.txt and/or requirements-dev.txt has not been generated.
     echo
     echo The options are:
     echo " --clean: clean built and installed files and exit."
     echo " --dev:   configure the environment for development."
+    echo " --init:  pull dependencies from PyPI. Used when first setting up a project."
     echo " --help:  display this help message and exit."
     echo
     echo By default, the python interpreter version found in the path is used.
@@ -157,12 +162,22 @@ install_packages() {
 # Main command line entry point
 CFG_DEV_MODE=0
 CFG_REQUIREMENTS=$REQUIREMENTS
-
-case "$CLI_ARGS" in
-    --help)    cli_help;;
-    --clean)   clean;;
-    --dev)     CFG_REQUIREMENTS="$DEV_REQUIREMENTS" && CFG_DEV_MODE=1;;
-esac
+NO_INDEX="--no-index"
+
+# We are using getopts to parse option arguments that start with "-"
+while getopts :-: optchar; do
+    case "${optchar}" in
+        -)
+            case "${OPTARG}" in
+                help  ) cli_help;;
+                clean ) clean;;
+                dev   ) CFG_REQUIREMENTS="$DEV_REQUIREMENTS" && CFG_DEV_MODE=1;;
+                init  ) NO_INDEX="";;
+            esac;;
+    esac
+done
+
+PIP_EXTRA_ARGS="$PIP_EXTRA_ARGS $NO_INDEX"
 
 create_virtualenv "$VIRTUALENV_DIR"
 install_packages "$CFG_REQUIREMENTS"

configure.bat

@@ -24,14 +24,14 @@
 @rem ################################
 
 @rem # Requirement arguments passed to pip and used by default or with --dev.
-set "REQUIREMENTS=--editable ."
-set "DEV_REQUIREMENTS=--editable .[testing]"
+set "REQUIREMENTS=--editable . --constraint requirements.txt"
+set "DEV_REQUIREMENTS=--editable .[testing] --constraint requirements.txt --constraint requirements-dev.txt"
 
 @rem # where we create a virtualenv
-set "VIRTUALENV_DIR=tmp"
+set "VIRTUALENV_DIR=venv"
 
 @rem # Cleanable files and directories to delete with the --clean option
-set "CLEANABLE=build tmp"
+set "CLEANABLE=build venv"
 
 @rem # extra  arguments passed to pip
 set "PIP_EXTRA_ARGS= "
@@ -49,7 +49,7 @@ set "CFG_BIN_DIR=%CFG_ROOT_DIR%\%VIRTUALENV_DIR%\Scripts"
 
 @rem ################################
 @rem # Thirdparty package locations and index handling
-set "PIP_EXTRA_ARGS=--find-links %CFG_ROOT_DIR%\thirdparty"
+set "PIP_EXTRA_ARGS=--find-links %CFG_ROOT_DIR%\thirdparty --find-links https://thirdparty.aboutcode.org/pypi" & %INDEX_ARG%
 @rem ################################
 
 
@@ -64,19 +64,26 @@ if not defined CFG_QUIET (
 @rem # Main command line entry point
 set CFG_DEV_MODE=0
 set "CFG_REQUIREMENTS=%REQUIREMENTS%"
-
-if "%1" EQU "--help"   (goto cli_help)
-if "%1" EQU "--clean"  (goto clean)
-if "%1" EQU "--dev"    (
-    set "CFG_REQUIREMENTS=%DEV_REQUIREMENTS%"
-    set CFG_DEV_MODE=1
-)
-if "%1" EQU "--python" (
-    echo "The --python option is now DEPRECATED. Use the PYTHON_EXECUTABLE environment"
-    echo "variable instead. Run configure --help for details."
-    exit /b 0
+set "NO_INDEX=--no-index"
+
+:again
+if not "%1" == "" (
+    if "%1" EQU "--help"   (goto cli_help)
+    if "%1" EQU "--clean"  (goto clean)
+    if "%1" EQU "--dev"    (
+        set "CFG_REQUIREMENTS=%DEV_REQUIREMENTS%"
+        set CFG_DEV_MODE=1
+    )
+    if "%1" EQU "--init"   (
+        set "NO_INDEX= "
+    )
+    shift
+    goto again
 )
 
+set "PIP_EXTRA_ARGS=%PIP_EXTRA_ARGS% %NO_INDEX%"
+
+
 @rem ################################
 @rem # find a proper Python to run
 @rem # Use environment variables or a file if available.
@@ -165,10 +172,14 @@ exit /b 0
     echo "  usage: configure [options]"
     echo " "
     echo The default is to configure for regular use. Use --dev for development.
+    echo Use the --init option if starting a new project and the project
+    echo dependencies are not available on thirdparty.aboutcode.org/pypi/
+    echo and requirements.txt and/or requirements-dev.txt has not been generated.
     echo " "
     echo The options are:
     echo " --clean: clean built and installed files and exit."
     echo " --dev:   configure the environment for development."
+    echo " --init:  pull dependencies from PyPI. Used when first setting up a project."
     echo " --help:  display this help message and exit."
     echo " "
     echo By default, the python interpreter version found in the path is used.