-
-
Notifications
You must be signed in to change notification settings - Fork 345
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Codecov stuff #3153
Comments
@jakkdl in my experience, one thing that causes Codecov to post statuses prematurely (not just annotations but Checks, comments and their web UI) is allowing it to guess the moment when it's okay to start processing the report files and combining them into one. To fight this, I like setting the config as follows, setting the number of expected uploads explicitly: https://github.com/aio-libs/yarl/blob/f304dd7/.codecov.yml#L3-L8. |
ooh, that'd resolve #2689 that has been bugging me forever. Thanks! But I don't think it'd resolve the problems in the pytest PR where the uncovered code was introduced in one commit, and the tests adding coverage added in a later commit.
yeah the codecov docs say
|
Yep, that assessment is correct. |
I've now edited the org-level yaml, could leave this issue open for a bit and wait for impact. Opened #3156 for the other repo-level improvements |
With #3156 we're gonna start noticing codecov flakiness 🙃
https://github.com/python-trio/trio/actions/runs/12392918438/job/34593105479?pr=3145#step:5:41 |
@webknjaz re: number of uploads - it's not [just] cron, but merge queue + push. Analyzing the list of uploads on codecov.io it appears that macOS pypy-3.10 and windows 3.12 x64 got uploaded twice in the latter (so a total of three times), which is what brings the total from 54 to 56. Looking through the log I'm not seeing any indications why it got uploaded twice, maybe it's a codecov-action@v3 bug /shrug Running codecov twice is a waste, but the root cause there is running the whole CI twice needlessly - so nothing to fix on the codecov end for this one. |
After switching primary branch from master to main we need to set the default branch in codecov.yml: https://docs.codecov.com/docs/codecov-yaml#default-branch (it sets the default branch on first upload, so if you manually change it you need to do the same on codecov) |
@jakkdl oh, I was just about to file an issue about that :) This does not need to go into the config. Instead, I just changed it on the web UI @ https://app.codecov.io/gh/python-trio/trio/config/general. It should be good now. |
Potentially reason why there are duplicate uploads, I've noticed that for pull requests that are branches of https://github.com/python-trio/trio it runs CI both for that branch and for being a pull request |
Yeah, I see 3 uploads for that commit. One of them is from https://github.com/python-trio/trio/actions/runs/12392561462 and then, two are from https://github.com/python-trio/trio/actions/runs/12392781287. Looking into https://github.com/python-trio/trio/actions/runs/12392781287/job/34592683971, there were no reruns. So I'd suspect a bug in the action (IIRC v3 uses https://github.com/codecov/uploader written in TypeScript) — but there are no logs of retries corroborating this. Or, there's a bug within the Codecov platform where they duplicated the entry somehow. I'll try looking into the uploaded payloads to see if they're identical or not.
In this instance, the two workflow runs were triggered by two events — Oh, and before these events, whenever a PR is updated (but is not yet ready for merge), a |
I didn't actually have to download them. I checked that the This means that Codecov's DB displays to upload entries for the same upload. It's their bug. |
Reported @ codecov/feedback#613. |
Experimenting with typical configs from other projects: #3158. |
@webknjaz I also noticed that coverage published from my fork's I'll try downgrading to v3 to see if that persists. |
we should probably bump codecov to v5 sooner rather than later if we're going to play around this much with it, so we don't start reporting v3-only bugs and stuff |
another upload fail https://github.com/python-trio/trio/actions/runs/12436356589/job/34724074506#step:6:43
|
two more upload fails today. I almost wanna disable strict errors, but that could give hard-to-explain coverage drops, soooo fuck codecov 🙃 |
I've been bothered by codecov's github annotations for a long while, where it fairly regularly kept complaining about previously-uncovered code still being uncovered after later commits fixing it. The latest instance of this happened over in pytest but I know it's confused me and others at various points in different trio repositories. (happened in #2812)
When I started digging I found out this functionality is deprecated and has a big red box in the documentation https://docs.codecov.com/docs/github-checks, and I'm fairly certain that more time has been lost collectively from false alarms (and code being near-unreadable in the review window when it's very low on coverage) than from the marginal gain of seeing it in the code review window rather than elsewhere. We'll still have the CI check and everything, and you can easily click through to codecov.io in several places.
I've opened an issue upstream codecov/codecov-action#1710, but in the meantime I think we should edit the YAML at https://app.codecov.io/account/github/python-trio/yaml/
It's a one-line fix, that doesn't even need a PR, but thought I should open it to discussion and make people aware of the change so they don't wonder where it went. And if you personally want the functionality there's a browser extension: https://docs.codecov.com/docs/the-codecov-browser-extension
The text was updated successfully, but these errors were encountered: