Overview table from `bot:status last_build` of the bot is incomplete for large numbers of builds

[casparvl]

See EESSI/software-layer#1278 (comment)

It may be that we're exceeding some length when gathering the input data or when producing the overview table?

[casparvl]

From the even handler log:

[20251202-T11:17:10] [handle_bot_command_status]: Overview of finished builds: comment '
This is the status of all the `bot: build` commands:
|on|for|repo|result|date|status|url|
|----|----|----|------|----|------|---|
|`generic`|`aarch64/generic`|eessi.io-2025.06-software|:cry: FAILURE|Oct 30 22:12:07 UTC 2025|finished|https://github.com/EESSI/software-layer/pull/1278#issuecomment-3470371677|
|`generic`|`aarch64/generic`|eessi.io-2023.06-software|:cry: FAILURE|Oct 30 21:41:12 UTC 2025|finished|https://github.com/EESSI/software-layer/pull/1278#issuecomment-3470368618|
|`generic`|`aarch64/generic`, `nvidia/cc70`|eessi.io-2025.06-software|:grin: SUCCESS|Nov 11 23:24:48 UTC 2025|finished|https://github.com/EESSI/software-layer/pull/1278#issuecomment-3519017015|
|`generic`|`aarch64/generic`, `nvidia/cc80`|eessi.io-2025.06-software|:grin: SUCCESS|Nov 12 08:45:50 UTC 2025|finished|https://github.com/EESSI/software-layer/pull/1278#issuecomment-3520560652|
|`neoverse_n1`|`aarch64/neoverse_n1`, `nvidia/cc70`|eessi.io-2025.06-software|:grin: SUCCESS|Nov 11 23:23:36 UTC 2025|finished|https://github.com/EESSI/software-layer/pull/1278#issuecomment-3519017537|
|`neoverse_v1`|`aarch64/neoverse_v1`, `nvidia/cc70`|eessi.io-2025.06-software|:grin: SUCCESS|Nov 11 23:22:25 UTC 2025|finished|https://github.com/EESSI/software-layer/pull/1278#issuecomment-3519018111|
|`nvidia-grace`|`aarch64/nvidia/grace`, `nvidia/cc70`|eessi.io-2025.06-software|:grin: SUCCESS|Nov 11 23:25:39 UTC 2025|finished|https://github.com/EESSI/software-layer/pull/1278#issuecomment-3519015576|
|`nvidia-grace`|`aarch64/nvidia/grace`, `nvidia/cc80`|eessi.io-2025.06-software|:grin: SUCCESS|Nov 12 08:42:57 UTC 2025|finished|https://github.com/EESSI/software-layer/pull/1278#issuecomment-3520559245|
|`nvidia-grace`, `nvidia/cc90`|`aarch64/nvidia/grace`, `nvidia/cc90`|eessi.io-2025.06-software|:grin: SUCCESS|Nov 04 17:01:52 UTC 2025|finished|https://github.com/EESSI/software-layer/pull/1278#issuecomment-3486790412|
|`amd-zen2`|`x86_64/amd/zen2`, `nvidia/cc70`|eessi.io-2025.06-software|:grin: SUCCESS|Nov 11 23:22:27 UTC 2025|finished|https://github.com/EESSI/software-layer/pull/1278#issuecomment-3519014142|
|`amd-zen2`|`x86_64/amd/zen2`, `nvidia/cc80`|eessi.io-2025.06-software|:grin: SUCCESS|Nov 12 08:53:54 UTC 2025|finished|https://github.com/EESSI/software-layer/pull/1278#issuecomment-3520558519|
|`amd-zen3`|`x86_64/amd/zen3`, `nvidia/cc70`|eessi.io-2025.06-software|:grin: SUCCESS|Nov 11 23:18:29 UTC 2025|finished|https://github.com/EESSI/software-layer/pull/1278#issuecomment-3519014649|
|`amd-zen3`, `nvidia/cc80`|`x86_64/amd/zen3`, `nvidia/cc80`|eessi.io-2025.06-software|:grin: SUCCESS|Nov 06 15:28:59 UTC 2025|finished|https://github.com/EESSI/software-layer/pull/1278#issuecomment-3497597096|
|`amd-zen4`|`x86_64/amd/zen4`, `nvidia/cc70`|eessi.io-2025.06-software|:grin: SUCCESS|Nov 11 23:12:37 UTC 2025|finished|https://github.com/EESSI/software-layer/pull/1278#issuecomment-3519015143|
|`amd-zen4`|`x86_64/amd/zen4`, `nvidia/cc80`|eessi.io-2025.06-software|:grin: SUCCESS|Nov 12 08:45:54 UTC 2025|finished|https://github.com/EESSI/software-layer/pull/1278#issuecomment-3520558757|
|`amd-zen4`, `nvidia/cc90`|`x86_64/amd/zen4`, `nvidia/cc90`|eessi.io-2025.06-software|:grin: SUCCESS|Nov 04 17:05:24 UTC 2025|finished|https://github.com/EESSI/software-layer/pull/1278#issuecomment-3486969809|
|`generic`|`x86_64/generic`, `nvidia/cc70`|eessi.io-2025.06-software|:grin: SUCCESS|Nov 11 23:25:51 UTC 2025|finished|https://github.com/EESSI/software-layer/pull/1278#issuecomment-3519016666|
|`generic`|`x86_64/generic`, `nvidia/cc80`|eessi.io-2025.06-software|:grin: SUCCESS|Nov 12 08:49:52 UTC 2025|finished|https://github.com/EESSI/software-layer/pull/1278#issuecomment-3520560132|
|`intel-cascadelake`, `nvidia/cc70`|`x86_64/intel/cascadelake`, `nvidia/cc70`|eessi.io-2025.06-software|:grin: SUCCESS|Nov 06 15:31:03 UTC 2025|finished|https://github.com/EESSI/software-layer/pull/1278#issuecomment-3497596658|
|`intel-cascadelake`|`x86_64/intel/cascadelake`, `nvidia/cc80`|eessi.io-2025.06-software|:grin: SUCCESS|Nov 12 08:47:50 UTC 2025|finished|https://github.com/EESSI/software-layer/pull/1278#issuecomment-3520559533|
|`intel-haswell`|`x86_64/intel/haswell`, `nvidia/cc70`|eessi.io-2025.06-software|:grin: SUCCESS|Nov 11 23:23:39 UTC 2025|finished|https://github.com/EESSI/software-layer/pull/1278#issuecomment-3519015981|
|`intel-haswell`|`x86_64/intel/haswell`, `nvidia/cc80`|eessi.io-2025.06-software|:grin: SUCCESS|Nov 12 08:55:39 UTC 2025|finished|https://github.com/EESSI/software-layer/pull/1278#issuecomment-3520559295|
|`intel-icelake`|`x86_64/intel/icelake`, `nvidia/cc70`|eessi.io-2025.06-software|:grin: SUCCESS|Nov 11 23:21:09 UTC 2025|finished|https://github.com/EESSI/software-layer/pull/1278#issuecomment-3519016205|
|`intel-icelake`, `nvidia/cc80`|`x86_64/intel/icelake`, `nvidia/cc80`|eessi.io-2025.06-software|:grin: SUCCESS|Nov 11 21:35:13 UTC 2025|finished|https://github.com/EESSI/software-layer/pull/1278#issuecomment-3518781820|
|`intel-sapphirerapids`|`x86_64/intel/sapphirerapids`, `nvidia/cc70`|eessi.io-2025.06-software|:grin: SUCCESS|Nov 11 23:29:59 UTC 2025|finished|https://github.com/EESSI/software-layer/pull/1278#issuecomment-3519016411|
|`intel-sapphirerapids`|`x86_64/intel/sapphirerapids`, `nvidia/cc80`|eessi.io-2025.06-software|:grin: SUCCESS|Nov 12 09:22:32 UTC 2025|finished|https://github.com/EESSI/software-layer/pull/1278#issuecomment-3520559758|
|`intel-skylake_avx512`|`x86_64/intel/skylake_avx512`, `nvidia/cc70`|eessi.io-2025.06-software|:grin: SUCCESS|Nov 11 23:23:41 UTC 2025|finished|https://github.com/EESSI/software-layer/pull/1278#issuecomment-3519015587|
|`intel-skylake_avx512`|`x86_64/intel/skylake_avx512`, `nvidia/cc80`|eessi.io-2025.06-software|:grin: SUCCESS|Nov 12 08:49:56 UTC 2025|finished|https://github.com/EESSI/software-layer/pull/1278#issuecomment-3520559044|'
[20251202-T11:17:13] [handle_issue_comment_event]: handling command '' resulted in '
  - added status comment https://github.com/EESSI/software-layer/pull/1278#issuecomment-3601273900'

I guess that's not really surprising, since that is indeed the issue comment that got added...

[casparvl]

From the pyghee.log:

[20251202-T11:17:10] run_subprocess(): 'fetch all comments' by running 'curl -L https://api.github.com/repos/EESSI/software-layer/issues/1278/comments?per_page=100&page=4' in directory '/var/tmp/eessibot/eessi-bot-software-layer'
[20251202-T11:17:10] run_cmd(): Result for running 'curl -L https://api.github.com/repos/EESSI/software-layer/issues/1278/comments?per_page=100&page=4' in 'None
...

And indeed, the only cc90-related things I see are e.g.

  {
    "url": "https://api.github.com/repos/EESSI/software-layer/issues/comments/3486969809",
    "html_url": "https://github.com/EESSI/software-layer/pull/1278#issuecomment-3486969809",
    "issue_url": "https://api.github.com/repos/EESSI/software-layer/issues/1278",
    "id": 3486969809,
    "node_id": "IC_kwDOD58T_M7P1u_R",
    "user": {
      "login": "eessi-bot-surf[bot]",
      "id": 199253719,
      "node_id": "BOT_kgDOC-Be1w",
      "avatar_url": "https://avatars.githubusercontent.com/in/1145565?v=4",
      "gravatar_id": "",
      "url": "https://api.github.com/users/eessi-bot-surf%5Bbot%5D",
      "html_url": "https://github.com/apps/eessi-bot-surf",
      "followers_url": "https://api.github.com/users/eessi-bot-surf%5Bbot%5D/followers",
      "following_url": "https://api.github.com/users/eessi-bot-surf%5Bbot%5D/following{/other_user}",
      "gists_url": "https://api.github.com/users/eessi-bot-surf%5Bbot%5D/gists{/gist_id}",
      "starred_url": "https://api.github.com/users/eessi-bot-surf%5Bbot%5D/starred{/owner}{/repo}",
      "subscriptions_url": "https://api.github.com/users/eessi-bot-surf%5Bbot%5D/subscriptions",
      "organizations_url": "https://api.github.com/users/eessi-bot-surf%5Bbot%5D/orgs",
      "repos_url": "https://api.github.com/users/eessi-bot-surf%5Bbot%5D/repos",
      "events_url": "https://api.github.com/users/eessi-bot-surf%5Bbot%5D/events{/privacy}",
      "received_events_url": "https://api.github.com/users/eessi-bot-surf%5Bbot%5D/received_events",
      "type": "Bot",
      "user_view_type": "public",
      "site_admin": false
    },
    "created_at": "2025-11-04T16:44:28Z",
    "updated_at": "2025-11-04T17:05:27Z",
    "body": "New job on instance `eessi-bot-surf` for repository `eessi.io-2025.06-software`\nBuilding on: `amd-zen4` and accelerator `nvidia/cc90`\nBuilding for: `x86_64/amd/zen4` and accelerator `nvidia/cc90`\nJob dir: `/projects/eessibot/eessi-bot-surf/jobs/2025.11/pr_1278
/15756355`\n|date|job status|comment|\n|----------|----------|------------------------|\n|Nov 04 16:44:26 UTC 2025|submitted|job id `15756355` will be eligible to start in about 20 seconds|\n|Nov 04 16:44:38 UTC 2025|received|job awaits launch by Slurm scheduler|\n|Nov 04 16:45:09 UTC
2025|running|job `15756355` is running|\n|Nov 04 17:05:24 UTC 2025|finished|<details><summary>:grin: SUCCESS _(click triangle for details)_</summary><dl><dt>_Details_</dt><dd>:white_check_mark: job output file <code>slurm-15756355.out</code><br/>:white_check_mark: no message matching <
code>FATAL: </code><br/>:white_check_mark: no message matching <code>ERROR: </code><br/>:white_check_mark: no message matching <code>FAILED: </code><br/>:white_check_mark: no message matching <code> required modules missing:</code><br/>:white_check_mark: found message(s) matching <code
>No missing installations</code><br/>:white_check_mark: found message matching <code>\\.tar\\.gz created!</code><br/></dd><dt>_Artefacts_</dt><dd><details><summary><code>eessi-2025.06-software-linux-x86_64-amd-zen4-accel-nvidia-cc90-17622750940.tar.gz</code></summary>size: 6197 MiB (64
98814027 bytes)<br/>entries: 12593<br/>modules under _2025.06/software/linux/x86_64/amd/zen4/accel/nvidia/cc90/modules/all_<br/><pre><code>CUDA/12.6.0.lua</code><br/><code>CUDA/12.8.0.lua</code><br/><code>cuDNN/9.5.0.50-CUDA-12.6.0.lua</code><br/></pre>software under _2025.06/software/
linux/x86_64/amd/zen4/accel/nvidia/cc90/software_<br/><pre><code>CUDA/12.6.0</code><br/><code>CUDA/12.8.0</code><br/><code>cuDNN/9.5.0.50-CUDA-12.6.0</code><br/></pre>reprod directories under _2025.06/software/linux/x86_64/amd/zen4/accel/nvidia/cc90/reprod_<br/><pre><code>CUDA/12.6.0/2
0251104_164737UTC</code><br/><code>CUDA/12.8.0/20251104_164953UTC</code><br/><code>cuDNN/9.5.0.50-CUDA-12.6.0/20251104_165124UTC</code><br/></pre>other under _2025.06/software/linux/x86_64/amd/zen4/accel/nvidia/cc90_<br/><pre>no other files in tarball<br/></pre></details></dd></dl></de
tails>|\n|Nov 04 17:05:24 UTC 2025|test result|<details><summary>:grin: SUCCESS _(click triangle for details)_</summary><dl><dt>_ReFrame Summary_</dt><dd>[  PASSED  ] Ran 0/0 test case(s) from 0 check(s) (0 failure(s), 0 skipped, 0 aborted)</dd><dt>_Details_</dt><dd>:white_check_mark:
job output file <code>slurm-15756355.out</code><br/>:white_check_mark: no message matching <code>ERROR: </code><br/>:white_check_mark: no message matching <code>\\[\\s\\*FAILED\\s\\*\\].\\*Ran .\\* test case</code><br/></dd></dl></details>|",
    "author_association": "NONE",
    "reactions": {
      "url": "https://api.github.com/repos/EESSI/software-layer/issues/comments/3486969809/reactions",
      "total_count": 0,
      "+1": 0,
      "-1": 0,
      "laugh": 0,
      "hooray": 0,
      "confused": 0,
      "heart": 0,
      "rocket": 0,
      "eyes": 0
    },

and then later

[20251202-T11:17:10] request_bot_build_issue_comments(): found build on architecture: nvidia-grace, with accelerator nvidia/cc90
[20251202-T11:17:10] request_bot_build_issue_comments(): found build for architecture: aarch64/nvidia/grace, with accelerator nvidia/cc90
[20251202-T11:17:10] request_bot_build_issue_comments(): found bot build response in issue, processing...
[20251202-T11:17:10] request_bot_build_issue_comments(): found build for repository: eessi.io-2025.06-software
[20251202-T11:17:10] request_bot_build_issue_comments(): found build on architecture: amd-zen4, with accelerator nvidia/cc90
[20251202-T11:17:10] request_bot_build_issue_comments(): found build for architecture: x86_64/amd/zen4, with accelerator nvidia/cc90

This corresponds to only seeing two types of cc90 builds in the overview, namely for grace+cc90 and zen4+cc90.

[casparvl]

I think I found the issue by manually running it to retrieve the first and fourth pages:

[casparl@tcn427 casparl.16931719]$ curl -o page1 -L https://api.github.com/repos/EESSI/software-layer/issues/1278/comments?per_page=100&page=1
[1] 1609376
[casparl@tcn427 casparl.16931719]$   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  459k  100  459k    0     0  7555k      0 --:--:-- --:--:-- --:--:-- 7665k

[1]+  Done                    curl -o page1 -L https://api.github.com/repos/EESSI/software-layer/issues/1278/comments?per_page=100
[casparl@tcn427 casparl.16931719]$ curl -o page4 -L https://api.github.com/repos/EESSI/software-layer/issues/1278/comments?per_page=100&page=4
[1] 1609378
[casparl@tcn427 casparl.16931719]$   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  459k  100  459k    0     0  7499k      0 --:--:-- --:--:-- --:--:-- 7665k

[1]+  Done                    curl -o page4 -L https://api.github.com/repos/EESSI/software-layer/issues/1278/comments?per_page=100
[casparl@tcn427 casparl.16931719]$ diff page1 page4 | wc -l
0
[casparl@tcn427 casparl.16931719]$

As you can see, it seems to have run in the background, and the command is cut short after the per_page=100. The reason is that the shell is interpreting the & as 'run in the background'. To resolve, it should be escaped:

curl -o page1 -L https://api.github.com/repos/EESSI/software-layer/issues/1278/comments?per_page=100\&page=1
curl -o page4 -L https://api.github.com/repos/EESSI/software-layer/issues/1278/comments?per_page=100\&page=4
diff page1 page4 | wc -l
7078

[casparvl]

A quick search suggests that the only place in which this command is invoked is here

eessi-bot-software-layer/tasks/build.py

Line 1211 in d64a043

curl_cmd = f'curl -L https://api.github.com/repos/{repo_name}/issues/{pr_number}/comments?per_page=100&page={x}'

so should be an easy fix (if that's indeed true)

[casparvl]

Also, the number of pages is now hardcoded:

eessi-bot-software-layer/tasks/build.py

Line 1210 in d64a043

for x in range(1, 5):

I think we should just retrieve pages with a while loop until we get an empty one:

$ curl -o page10 -L https://api.github.com/repos/EESSI/software-layer/issues/1278/comments?per_page=100\&page=10
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100     5  100     5    0     0    152      0 --:--:-- --:--:-- --:--:--   156
[casparl@tcn427 casparl.16931719]$ cat page10
[

]

[casparvl]

Should be as simple as replacing with a while loop and checking that len(comments) for the comments returned by

eessi-bot-software-layer/tasks/build.py

Line 1214 in d64a043

comments = json.loads(curl_output)

is 0. If so, break out of the while loop.