feat: improved builder

README.md

@@ -36,7 +36,9 @@ Available APIs at the moment:
 
 ### Build API
 
-`POST /system/api/v1/build` - Perform the build of a custom image and push it to repository.
+`POST /system/api/v1/build/start` - Perform the build of a custom image and push it to repository.
+
+`POST /system/api/v1/build/cleanup` - Perform cleanup of build jobs older than 24 hours (or different number of hours if otherwise specified)
 
 More informations [Here](docs/DEPLOYER.md)
 
@@ -71,6 +73,7 @@ Taskfile supports the following tasks:
 * buildx:                      Build the docker image using buildx. Set PUSH=1 to push the image to the registry. 
 * docker-login:                Login to the docker registry. Set REGISTRY=ghcr or REGISTRY=dockerhub in .env to use the respective registry. 
 * image-tag:                   Create a new tag for the current git commit.       
+* builder:clean:               Cleanup old build jobs via api
 * builder:cleanjobs:           Clean up old jobs
 * builder:delete-image:        Delete an image from the registry
 * builder:get-image:           Get an image from the registry

TODO.md

@@ -20,10 +20,12 @@
 # TODO
 
 ## Tests
-Add integration and unit tests
+- [ ] Add integration tests
+- [X] Add unit tests
+- [ ] Add more unit tests
 
 ## Various
 
 - [ ] `openserverless.common.whis_user_data.py` - Add `with_` blocks for other new OpenServerless Services
 - [ ] `openserverless.common.whisk_user_generator` - Check if `generate_whisk_user_yaml` is complete
-- [ ] cleanup config maps and builds
+- [X] cleanup config maps and builds

TaskfileBuilder.yml

@@ -31,14 +31,32 @@ tasks:
       - if test -z "{{.KIND}}"; then echo "KIND IS NOT SET" && exit 1; fi
       - |
         echo '{"source": "{{.SOURCE}}", "target": "{{.TARGET}}", "kind": "{{.KIND}}", "file": "{{.REQUIREMENTS}}" }' | \
-        curl -X POST $ADMIN_API_URL/api/v1/build -H "Content-Type: application/json" -H "Authorization: {{.AUTH}}" -d @-
+        curl -X POST $ADMIN_API_URL/api/v1/build/start -H "Content-Type: application/json" -H "Authorization: {{.AUTH}}" -d @-
       - sleep 5
       - task: logs
     deps:
       - cleanjobs
       # - updatetoml
     silent: true
 
+  clean:
+    desc: Cleanup old build jobs via api
+    vars:
+      AUTH:
+        sh: cat ~/.wskprops | grep "AUTH" | cut -d'=' -f2 | xargs -I {}
+      MAX_AGE_HOURS:
+        sh: |
+          if test -z "{{.MAX_AGE_HOURS}}"; 
+          then echo "24";
+          else echo "{{.MAX_AGE_HOURS}}"; 
+          fi        
+    cmds:
+      - |
+        echo '{"max_age_hours": "{{.MAX_AGE_HOURS}}" }' | \
+        curl -X POST $ADMIN_API_URL/api/v1/build/cleanup -H "Content-Type: application/json" -H "Authorization: {{.AUTH}}" -d @-      
+    silent: false
+
+
   logs:
     desc: Show logs of the last build job
     cmds:
@@ -66,7 +84,7 @@ tasks:
     desc: List catalogs in the registry
     cmds:
       - curl -u $REGISTRY_USER:$REGISTRY_PASS $REGISTRY_HOST/v2/_catalog
-    silent: false
+    silent: true
 
   list-images:
     desc: List images in a specific catalog
@@ -75,7 +93,7 @@ tasks:
     cmds:
       - if test -z "{{.CATALOG}}"; then echo "CATALOG IS NOT SET" && exit 1; fi
       - curl -u $REGISTRY_USER:$REGISTRY_PASS $REGISTRY_HOST/v2/{{.CATALOG}}/tags/list
-    silent: false
+    silent: true
 
   get-image:
     desc: Get an image from the registry

docs/DEPLOYER.md

@@ -19,15 +19,75 @@
 -->
 # Deployer
 
-These tasks are useful to interact with OpenServerless Admin Api Builder
+Deployer is the implementation of the feature described 
+in [OpenServerless Issue 156](https://github.com/apache/openserverless/issues/156).
 
-There are some tasks to interact with OpenServerless internal registry too.
+Specifically, the deployer can extend a default runtime with user-defined 
+"requirements" by generating a new "extended" user runtime and pushing it to 
+OpenServerless’ internal Docker registry.
+
+Actually, the supported "requirements" are listed in the following table:
+
+|  kind  | requirement file |
+|:-------|:-----------------|
+| go     | go.mod           |
+| java   | pom.xml          |
+| nodejs | package.json     |
+| php    | composer.json    |
+| python | requirements.txt |
+| ruby   | Gemfile          |
+| dotnet | project.json     |
+
+*NOTE*: this list will be improved when new extendible runtimes will be ready.
+
+The "requirement" can be passed as base64 encoded string inside the `file` attribute
+of the json body payload:
+
+```json
+{
+  "source": "apache/openserverless-runtime-python:v3.13-2506091954", 
+  "target": "devel:python3.12-custom", 
+  "kind": "python", 
+  "file": "Z25ld3MKYmVhdXRpZnVsc291cDQ="
+}
+```
+
+By default the deployer will push to OpenServerless internal docker registry.
+To detect the host, it will use the `registry_host` inside the Operator's config 
+map.
+To authenticate, it will use the imagePullSecret named `registry-pull-secret` 
+(these credentials are valid to push and pull from the internal registry).
+
+The deployer supports also pushing to an external private docker registry, using 
+ops env:
+
+- `REGISTRY_HOST` - put here the hostname:port of the external private registry.
+- `REGISTRY_SECRET` - put here the name of a kubernetes secret containing an 
+imagePullSecret able to push to the registry specified by `REGISTRY_HOST`.
+
+This project has also support tasks:
+
+- to test the build.
+- to interact with OpenServerless internal registry too.
+
+See [Examples](#examples) section
+
+## Endpoints
+
+`POST /system/api/v1/build/start` - Perform the build of a custom image and push it to repository.
+
+`POST /system/api/v1/build/cleanup` - Perform cleanup of build jobs older than 24 hours (or different number of hours if otherwise specified)
+
+Both endpoints requires the wsk token in an `authorization` header.
+The token will be used to check the user (the target image hash needs to be 
+always in the format `user:image-tag`).
 
 ## Available tasks
 
 task: Available tasks for this project:
 
 ```
+* builder:clean:               Cleanup old build jobs via api
 * builder:cleanjobs:           Clean up old jobs
 * builder:delete-image:        Delete an image from the registry
 * builder:get-image:           Get an image from the registry
@@ -44,6 +104,12 @@ task: Available tasks for this project:
 
 `task builder:send SOURCE=apache/openserverless-runtime-python:v3.13-2506091954 TARGET=devel:python3.13-custom KIND=python REQUIREMENTS=$(base64 -i deploy/samples/requirements.txt)`
 
+### Clenaup of old jobs via API
+
+`task builder:clean MAX_AGE_HOURS=2`
+
+MAX_AGE_HOURS, if not specified, has a default value of 24.
+
 ### List images for the user
 
 `task builder:list-images CATALOG=devel`

openserverless/common/kube_api_client.py

@@ -118,7 +118,7 @@ def create_whisk_user(self, whisk_user_dict, namespace="nuvolaris"):
             logging.error("create_whisk_user %s", ex)
             return False
 
-    def delete_whisk_user(self, username, namespace="nuvolaris"):
+    def delete_whisk_user(self, username: str, namespace="nuvolaris"):
         """ "
         Delete a whisk user using a DELETE operation
         param: username of the whisksusers resource to delete
@@ -147,7 +147,7 @@ def delete_whisk_user(self, username, namespace="nuvolaris"):
             logging.error(f"delete_whisk_user {ex}")
             return False
 
-    def get_whisk_user(self, username, namespace="nuvolaris"):
+    def get_whisk_user(self, username: str, namespace="nuvolaris"):
         """ "
         Get a whisk user using a GET operation
         param: username of the whisksusers resource to delete
@@ -210,7 +210,7 @@ def update_whisk_user(self, whisk_user_dict, namespace="nuvolaris"):
             logging.error(f"update_whisk_user {ex}")
             return False
 
-    def get_config_map(self, cm_name, namespace="nuvolaris"):
+    def get_config_map(self, cm_name: str, namespace="nuvolaris"):
         """
         Get a ConfigMap by name.
         :param cm_name: Name of the ConfigMap.
@@ -238,7 +238,7 @@ def get_config_map(self, cm_name, namespace="nuvolaris"):
             logging.error(f"get_config_map {ex}")
             return None
 
-    def post_config_map(self, cm_name, file_or_dir, namespace="nuvolaris"):
+    def post_config_map(self, cm_name: str, file_or_dir: str, namespace="nuvolaris"):
         """        
         Create a ConfigMap from a file or directory.
         :param cm_name: Name of the ConfigMap.
@@ -291,7 +291,7 @@ def post_config_map(self, cm_name, file_or_dir, namespace="nuvolaris"):
             logging.error(f"post_config_map {ex}")
             return None
 
-    def delete_config_map(self, cm_name, namespace="nuvolaris"):
+    def delete_config_map(self, cm_name: str, namespace="nuvolaris"):
         """
         Delete a ConfigMap by name.
         :param cm_name: Name of the ConfigMap to delete.
@@ -320,7 +320,7 @@ def delete_config_map(self, cm_name, namespace="nuvolaris"):
             logging.error(f"delete_config_map {ex}")
             return False
 
-    def get_secret(self, secret_name, namespace="nuvolaris"):
+    def get_secret(self, secret_name: str, namespace="nuvolaris"):
         """
         Get a Kubernetes secret by name.
         :param secret_name: Name of the secret.
@@ -348,7 +348,7 @@ def get_secret(self, secret_name, namespace="nuvolaris"):
             logging.error(f"get_secret {ex}")
             return None
 
-    def post_secret(self, secret_name, secret_data, namespace="nuvolaris"):
+    def post_secret(self, secret_name: str, secret_data: dict, namespace="nuvolaris"):
         """
         Create a Kubernetes secret.
         :param secret_name: Name of the secret.
@@ -385,7 +385,7 @@ def post_secret(self, secret_name, secret_data, namespace="nuvolaris"):
             logging.error(f"post_secret {ex}")
             return None
 
-    def delete_secret(self, secret_name, namespace="nuvolaris"):
+    def delete_secret(self, secret_name: str, namespace="nuvolaris"):
         """
         Delete a Kubernetes secret.
         :param secret_name: Name of the secret to delete.
@@ -412,11 +412,70 @@ def delete_secret(self, secret_name, namespace="nuvolaris"):
         except Exception as ex:
             logging.error(f"delete_secret {ex}")
             return False
+
+    def get_jobs(self, name_filter: str = None, namespace="nuvolaris"):
+        """
+        Get all Kubernetes jobs in a specific namespace.
+        :param namespace: Namespace to list jobs from.
+        :return: List of jobs or None if failed.
+        """
+        url = f"{self.host}/apis/batch/v1/namespaces/{namespace}/jobs"
+        headers = {"Authorization": self.token}
+        try:
+            logging.info(f"GET request to {url}")
+            response = req.get(url, headers=headers, verify=self.ssl_ca_cert)
+
+            if response.status_code in [200, 202]:
+                logging.debug(
+                    f"GET to {url} succeeded with {response.status_code}. Body {response.text}"
+                )
+
+                if name_filter:
+                    jobs = json.loads(response.text)["items"]
+                    filtered_jobs = [job for job in jobs if name_filter in job["metadata"]["name"]]
+                    return filtered_jobs
+
+                return json.loads(response.text)["items"]
 
-    def post_job(self, job_name, job_manifest, namespace="nuvolaris"):
+            logging.error(
+                f"GET to {url} failed with {response.status_code}. Body {response.text}"
+            )
+            return None
+        except Exception as ex:
+            logging.error(f"get_jobs {ex}")
+            return None
+
+    def delete_job(self, job_name: str, namespace="nuvolaris"):
+        """
+        Delete a Kubernetes job by name.
+        :param job_name: Name of the job to delete.
+        :param namespace: Namespace where the job is located.
+        :return: True if deletion was successful, False otherwise.
+        """
+        url = f"{self.host}/apis/batch/v1/namespaces/{namespace}/jobs/{job_name}"
+        headers = {"Authorization": self.token}
+
+        try:
+            logging.info(f"DELETE request to {url}")
+            response = req.delete(url, headers=headers, verify=self.ssl_ca_cert)
+
+            if response.status_code in [200, 202]:
+                logging.debug(
+                    f"DELETE to {url} succeeded with {response.status_code}. Body {response.text}"
+                )
+                return True
+
+            logging.error(
+                f"DELETE to {url} failed with {response.status_code}. Body {response.text}"
+            )
+            return False
+        except Exception as ex:
+            logging.error(f"delete_job {ex}")
+            return False
+
+    def post_job(self, job_manifest: json, namespace="nuvolaris"):
         """
         Create a Kubernetes job.
-        :param job_name: Name of the job.
         :param job_manifest: Dictionary containing the job manifest. 
         :param namespace: Namespace where the job will be created.
         :return: The created job or None if failed.
@@ -440,7 +499,7 @@ def post_job(self, job_name, job_manifest, namespace="nuvolaris"):
             logging.error(f"post_job {ex}")
             return None
 
-    def get_pod_by_job_name(self, job_name, namespace="nuvolaris"):
+    def get_pod_by_job_name(self, job_name: str, namespace="nuvolaris"):
         """
         Get the pod name associated with a job by its name.
         :param job_name: Name of the job.
@@ -474,7 +533,7 @@ def get_pod_by_job_name(self, job_name, namespace="nuvolaris"):
             logging.error(f"get_pod_by_job_name {ex}")
             return None
 
-    def stream_pod_logs(self, pod_name, namespace="nuvolaris"):
+    def stream_pod_logs(self, pod_name: str, namespace="nuvolaris"):
         """
         Stream logs from a specific pod.
         :param pod_name: Name of the pod to stream logs from.
@@ -487,7 +546,7 @@ def stream_pod_logs(self, pod_name, namespace="nuvolaris"):
                 if line:
                     print(line.decode())
 
-    def check_job_status(self, job_name, namespace="nuvolaris"):
+    def check_job_status(self, job_name: str, namespace="nuvolaris"):
         """
         Check the status of a job by its name.
         :param job_name: Name of the job to check.