Tag jobs that were run from a fork

[AbhiPrasad]

When doing some investigation into flakey tests, we noticed a weird pattern for tests that we couldn't figure out. Upon inspecting the data more closely, we realized these were occuring on forks, and not actually flaking, but happening because the forks did not have correct permissions set up for CI.

Can we add a tag to jobs that say if they are from a forked repo or not? Included the fork name would be even better!

[asottile-sentry]

can you add more details? what job are you referring to and what stats / etc.?

ideally we wouldn't need special permissions for workflows -- and in the case we do need them it should fail more gracefully

[AbhiPrasad]

Yes! So we track our most commonly re-run tests to generally get an idea of tests that need to be re-run a lot.

Here we noticed that the e2e tests were being re-run, but we didn't know why. When asking around, none of the team said the e2e tests were being problematic - it took a little more digging to find out that only e2e tests for workflows started from forks were failing. The problem here was that the e2e tests needed env vars set up, and those weren't enabled for forks. This was an error on our side when setting this up, but if we had this info when searching through Sentry, we could have maybe resolved this faster.

I think it's also interesting to generally compare workflows in forks vs. the main repo - since maybe certain jobs don't run on forks (for us, we do this with the size-check job). That might lead to some interesting analysis.

If this is an anti-pattern (and we should be doing something different), please let me know, happy to adjust things!

[asottile-sentry]

hmmm -- I think the graphs are correct to point out that there's a problematic user experience for contributors. I'd probably work to adjust the pipeline such that it can pass for external contributors or skip it in that case (it seems to use secrets which would only be available to repository maintainers)

I was a little confused about the original issue since the app only tracks runs within the organization and not on actual forks -- the clarifying bit is that these are runs that are from branches outside the repository -- but represent real runs within the org