Error: Provider not specified

[pboling]

In GHA everything is success. There are NO upload errors.
In CodeCov's UI everything is broken. I can't understand why.
Just wanted to post it here before I rip CodeCov out of everything.

This same issue seems to be slowly be infecting more and more of my repositories. My best guess is that I'm exporting too many code coverage formats from my SimpleCov config (i.e. all of the major ones: html, xml, rcov, lcov, json) and CodeCov is choking on that. If my hunch is correct, that's a very sad state for CodeCov.

The error has no useful information, at least not useful to me. It is inscrutable.

[spalmurray]

Hey @pboling thanks for your message. Could you point me to a specific repo that is having this issue so we can better investigate?

[pboling]

I am exporting all of the formats I am aware of. I can certainly export fewer. One of the reasons I wanted to do this was precisely for this reason - to see if code coverage tools could handle it. ;)

Is there a particular format I can choose that would work well? I'm not sure what you mean by a simplecov format. The formats I'm exporting are:

• html
• xml
• rcov
• lcov
• json
• tty (outputs results to STDOUT)

In all cases I have turned on branch coverage output for those that support it.

I use a tool I wrote to configure it all, kettle-soup-cover - https://github.com/kettle-rb/kettle-soup-cover

These are the gems it configures:

"simplecov", "~> 0.22" # Includes dependency on simplecov-html
"simplecov-cobertura", "~> 3.0" # Ruby >= 2.5, provides GitLab, Jenkins compatibility (XML)
"simplecov-console", "~> 0.9", ">= 0.9.3" # TTY / Console output
"simplecov-html", "~> 0.13", ">= 0.13.1" # GHA, Human compatibility! (HTML)
"simplecov_json_formatter", "~> 0.1", ">= 0.1.4" # GHA, Jenkins X, CircleCI, Travis CI, BitBucket, CodeClimate compatibility (JSON)
"simplecov-lcov", "~> 0.8" # GHA, Jenkins X, CircleCI, Travis CI, TeamCity, GCOV compatibility
"simplecov-rcov", "~> 0.3", ">= 0.3.7" # Hudson compatibility

NOTE: Conspiculously absent from the comments above is CodeCov, because I couldn't find it documented anywhere which format was preferred.

I control which of those gems actually get loaded and configured, and thus which report files are created, with an ENV variable:

export K_SOUP_COV_FORMATTERS="html,xml,rcov,lcov,json,tty"

[pboling]

A quick fix for you would be to avoid simplecov in favor of other formats

Assuming it is the default HTML formatter that currently (this may change, as it has been extracted to a separate library) ships with simplecov, I've tried turning off HTML.

ruby-openid/ruby-openid#60

[spalmurray]

Oh that's very interesting. Our code would imply simplecov can only be json, which is clearly not the case.

Looking at our docs to try to answer your question, I came across this: https://docs.codecov.com/docs/supported-report-formats#ruby ... I guess we're not breaking new ground here 😆 hopefully this means cobertura or lcov will work out though!

RE: your previous message, json is the problem format at the moment.

[pboling]

That fixed it!

• https://app.codecov.io/github/ruby-openid/ruby-openid2/commit/33796b32f1a025672e42488b897097a67cbbd7f5
• https://github.com/ruby-openid/ruby-openid2/actions/runs/16277379161/job/45959435167

It seems particularly wild that codecov is somehow not compatible with the default format, but I am now very happy with my decision to build kettle-soup-cover! I turned off HTML, and now it works great.

export K_SOUP_COV_FORMATTERS="xml,rcov,lcov,json,tty"

I look forward to hearing about the further fixes going forward, but I'm elated to have a workaround.

[pboling]

And, yeah, JSON is absolutely not the default, and in fact it's a bit of a mine field. One simplecov JSON gem that has been very popular is unmaintained and quite broken. The other is not very actively maintained, but it is serviceable (simplecov_json_formatter).

That's another reason I made kettle-soup-cover. I don't want to have to try to remember what the "state-of-the-art" is for each format everytime I revisit a project. So I host all my knowledge there, and update it there.

[pboling]

@spalmurray Holy crap.

• https://docs.codecov.com/docs/supported-report-formats#non-supported-code-coverage-formats

Non-supported code coverage report format types include:
.xccov (Xcode)
.ec
.exec
.coverage (Python)
.html

So HTML, the default export of simplecov, breaks codecov! Ideally it would be explicitly ignored, since, as the default it takes actual config to turn it off.

[pboling]

Many projects export the HTML as part of the build so they can publish it to a server for consumption on the web. It would be pretty bad if projects had to turn that off so that they could use a code coverage tool... Like 👀

[spalmurray]

Really glad to hear you're unblocked @pboling!

I'll do my best to make sure you're notified of changes that come from this, but here are the issues I made if you want to follow along yourself:

• Simplecov report being matched by VOne matcher before Simplecov matcher can run engineering-team#3591
• Simplecov json report fails to process engineering-team#3592

kettle-soup-cover looks awesome btw 😄

[kristof-mattei]

I am not getting an error like OP, but I am getting "provider not specified":

Action:

      - name: Upload coverage results (to Codecov.io)
        uses: codecov/codecov-action@18283e04ce6e62d37312384ff67231eb8fd56d24 # v5.4.3
        with:
          disable_search: true
          disable_telem: true
          fail_ci_if_error: true
          files: reports/lcov.info
          plugins: ""
          use_oidc: true

Repo: https://app.codecov.io/github/kristof-mattei/rust-seed/commit/b85a3b68ea975989cfd8d354e036f338b8275a11?dropdown=coverage

CodeCov: https://app.codecov.io/github/kristof-mattei/rust-seed/commit/b85a3b68ea975989cfd8d354e036f338b8275a11?dropdown=coverage

[spalmurray]

Looking into this

[kristof-mattei]

@spalmurray let me know if you need me to run a couple of tests

[spalmurray]

@kristof-mattei I just pushed a fix for this. Could you try it out?

[kristof-mattei]

@spalmurray

It works!

[spalmurray]

Awesome! Thanks for checking 😄

[pboling]

@spalmurray So far what I have discovered is that CodeCov is incompatible with the current JSON coverage format producted by the current best JSON formatter for simplecov: https://rubygems.org/gems/simplecov_json_formatter

With a JSON-format coverage file in my coverage/ dir:
https://app.codecov.io/github/appraisal-rb/appraisal2/commit/42b79161872e0e3f0619c175ca9226b6678e10a4

Without a JSON-format coverage file in my coverage/ dir (using LCOV, RCOV, or XML instead):
https://app.codecov.io/github/appraisal-rb/appraisal2/commit/e3ffc2bdd11d070fcdcce41048e74366ce906c6d

The lesson for us users: Turn off JSON coverage in CI if you want to use CodeCov. My kettle-soup-cover gem allows you to turn any format on/off at will via an ENV variable.
The lesson for CodeCov: simplecov_json_formatter has been the best JSON format for simplecov for 4+ years now. Please update!

NOTE: In my marked answer I had previously found that I needed to turn off HTML in order for CodeCov to work.

It is now necessary to turn off both HTML and JSON formats.