Test bench CI PR

[filchristou]

Not to merge

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 97.29%. Comparing base (de9cac7) to head (62e9733).
Report is 6 commits behind head on master.

❗ Current head 62e9733 differs from pull request most recent head bd5e39a. Consider uploading reports for the commit bd5e39a to get more accurate results

Additional details and impacted files

@@           Coverage Diff           @@
##           master     #352   +/-   ##
=======================================
  Coverage   97.29%   97.29%           
=======================================
  Files         117      117           
  Lines        6814     6814           
=======================================
  Hits         6630     6630           
  Misses        184      184           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[filchristou]

Unfortunately, the PostBenchmarks action from https://github.com/JuliaGraphs/Graphs.jl/blob/master/.github/workflows/postbench.yml was not triggered.

[gdalle]

Any idea why?

[filchristou]

Not sure at the moment.. But it also doesn't work in my fork so I i'll first fix it there and then make a PR here.

[filchristou]

I got the workflow chain to work, but BenchmarkCI.jl is failing.
The problem is that BenchmarkCI.jl is engineered such that it uses the current github workflow event to receive the information about where to post.
Since the new workflow is not triggered with a Pull-Request event the code fails here.
I know I've been supporting BenchmarkCI.jl because it "just works", but it seems that in our case it just doesn't. The options are either to fork it, or make a customized thing out of PkgBenchmark.jl or use something else.

[gdalle]

Sorry that you went through all of this trouble to conclude that BenchmarkCI is not the thing for us. Let's put this on pause for now and think about what other options we have for benchmarking infrastructure.

@willow-ahrens is taking over maintenance of BenchmarkTools, and one of her goals is to figure out the CI conundrum (see the announcement on Discourse). I have explained to her the limits that @filchristou and I have discovered, and she said she'd ping when she has stuff to review in this area.

[willow-ahrens]

Is there a simple fix to benchmarkCI that would make it work, or do we need to rewrite it?