Code Coverage: Setup code coverage reports automatically populated by CI #117

New Issue

Bart van der Braak · 2024-08-12T11:12:20+02:00

Bart van der Braak commented

2024-08-12 11:12:20 +02:00

Examples:

github.com/mozilla/code-coverage
docs.codecov.com/docs/installing-codecov-self-hosted
github.com/SonarSource/sonarqube

Questions:

Do we run this daily?
Each track independently?

Examples: - github.com/mozilla/code-coverage - docs.codecov.com/docs/installing-codecov-self-hosted - github.com/SonarSource/sonarqube Questions: - Do we run this daily? - Each track independently?

Bart van der Braak added the

labels 2024-08-12 11:12:20 +02:00

Bart van der Braak added this to the DevOps Progress Board project 2024-08-12 11:12:20 +02:00

Jacques Lucke commented

2024-08-13 16:07:40 +02:00

Thanks for looking into this. In the past few days I worked on a potential code coverage solution that should work well for us. It also seems to be simple enough to deploy.

#126181 adds a tool for generating code coverage reports to Blender. It simply outputs a folder with static html files in the end that can be deployed directly. Maybe deploying to https://coverage.blender.org/ is possible? If not, I guess it can be deployed similiarly to benchmarks.

Here is an example for how the output currently looks like: https://coverage.jlucke.com/

Do we run this daily?

I think running this daily should be good enough for now.

Each track independently?

What's the definition of a track here? This tool only works with GCC, so there might only be a single track?

Thanks for looking into this. In the past few days I worked on a potential code coverage solution that should work well for us. It also seems to be simple enough to deploy. #126181 adds a tool for generating code coverage reports to Blender. It simply outputs a folder with static html files in the end that can be deployed directly. Maybe deploying to `https://coverage.blender.org/` is possible? If not, I guess it can be deployed similiarly to [benchmarks](https://builder.blender.org/download/daily/benchmarks/main-linux-x86_64/report.html). Here is an example for how the output currently looks like: https://coverage.jlucke.com/ > Do we run this daily? I think running this daily should be good enough for now. > Each track independently? What's the definition of a track here? This tool only works with GCC, so there might only be a single track?

Sergey Sharybin commented

2024-08-13 16:14:22 +02:00

I think running this daily should be good enough for now.

If it doesn't take too long to run, then daily is fine.

What's the definition of a track here?

Track is vdev (main branch), v420 (blender-v4.2-release branch), v360 and so on.
I don't think we really need to worry about coverage of non-main branches.

> I think running this daily should be good enough for now. If it doesn't take too long to run, then daily is fine. > What's the definition of a track here? Track is vdev (main branch), v420 (blender-v4.2-release branch), v360 and so on. I don't think we really need to worry about coverage of non-main branches.

Jacques Lucke commented

2024-08-13 16:16:16 +02:00

If it doesn't take too long to run, then daily is fine.

For me, the slow part is running tests in a debug build which takes about 12 min (on 24 threads). Generating the report takes 1-2 min for me.

It would be nice to check if we can speed up the tests in a debug build though. I think some tests are probably needlessly slow.

> If it doesn't take too long to run, then daily is fine. For me, the slow part is running tests in a debug build which takes about 12 min (on 24 threads). Generating the report takes 1-2 min for me. It would be nice to check if we can speed up the tests in a debug build though. I think some tests are probably needlessly slow.

Sergey Sharybin commented

2024-08-13 16:23:27 +02:00

We're 16 threads only. But still seems doable with daily reports.

Bart van der Braak closed this issue

2024-08-14 10:31:39 +02:00

Bart van der Braak reopened this issue

2024-08-14 10:34:32 +02:00

Jacques Lucke commented

2024-08-15 12:21:44 +02:00

@blender#126181 is merged now. Let me know if I can somehow help with the buildbot integration.

[@blender#126181](https://projects.blender.org/blender/blender/pulls/126181) is merged now. Let me know if I can somehow help with the buildbot integration.

Sergey Sharybin commented

2024-08-15 14:33:54 +02:00

@JacquesLucke Technically you can indeed help with the buildbot integration. There is a public repo with the code that takes care of building/delivering: https://projects.blender.org/infrastructure/blender-devops

IT would need to be a new pipeline, similar to the linting from the update/build perspective, and similar to the doc_developer.py from the developer perspective.

If you can have a look at that, it will help. Otherwise we'll handle it with Bart once he is less busy with the chat/DC migration.

@JacquesLucke Technically you can indeed help with the buildbot integration. There is a public repo with the code that takes care of building/delivering: https://projects.blender.org/infrastructure/blender-devops IT would need to be a new pipeline, similar to the linting from the update/build perspective, and similar to the `doc_developer.py` from the developer perspective. If you can have a look at that, it will help. Otherwise we'll handle it with Bart once he is less busy with the chat/DC migration.

Jacques Lucke referenced this issue from infrastructure/blender-devops

2024-08-15 17:41:35 +02:00

WIP: Code Coverage on build-bot #5

Jacques Lucke commented

2024-08-15 17:42:57 +02:00

I started looking into it a bit in infrastructure/blender-devops#5. Unfortunately, I'm a bit limited by my ability to test things. Maybe my initial code is useful anyway.

When I'm in Amsterdam the next time, I might ask for quick tour through the build-bot and the related development set-up.

I started looking into it a bit in https://projects.blender.org/infrastructure/blender-devops/pulls/5. Unfortunately, I'm a bit limited by my ability to test things. Maybe my initial code is useful anyway. When I'm in Amsterdam the next time, I might ask for quick tour through the build-bot and the related development set-up.

Sergey Sharybin commented

2024-08-16 11:29:27 +02:00

@JacquesLucke Cool. Not sure if you've seen https://projects.blender.org/infrastructure/blender-devops/src/branch/main/buildbot/README.md

Been a while since I've been setting things up, so can not give deeper insights atm. But surely when you're here we can have a closer look and make sure others can easily setup an environment.

And surely the work you did is already a good starting point!

@JacquesLucke Cool. Not sure if you've seen https://projects.blender.org/infrastructure/blender-devops/src/branch/main/buildbot/README.md Been a while since I've been setting things up, so can not give deeper insights atm. But surely when you're here we can have a closer look and make sure others can easily setup an environment. And surely the work you did is already a good starting point!

Jacques Lucke commented

2024-08-16 11:44:15 +02:00

Yeah, I've seen that and followed it to some degree. I was able to run the worker script, but anything related to the buildbot like pipelines I didn't get working. Also some of the code assumes the usage of apt, which is fine but didn't work out of the box for me.

However, I think my main problem is that I just don't know what the setup is doing exactly, so I can't troubleshoot issues. For example, I don't have a good understanding for (these things are mostly just assumed to be known by the docs):

What is the buildbot software responsible for and what do we do ourselves?
What exactly is a track/worker/builder/pipeline?
I only know what the buildbot looks online to some degree, no idea how to work with it locally.
How does the delivery work? Where are delivered files stored locally vs. on the server?

I don't really expect you to answer these things now, it's probably easier to just figure these things out when I'm back. Just saying what kinds of bottlenecks I ran into.

Yeah, I've seen that and followed it to some degree. I was able to run the worker script, but anything related to the buildbot like pipelines I didn't get working. Also some of the code assumes the usage of `apt`, which is fine but didn't work out of the box for me. However, I think my main problem is that I just don't know what the setup is doing exactly, so I can't troubleshoot issues. For example, I don't have a good understanding for (these things are mostly just assumed to be known by the docs): * What is the buildbot software responsible for and what do we do ourselves? * What exactly is a track/worker/builder/pipeline? * I only know what the buildbot looks online to some degree, no idea how to work with it locally. * How does the delivery work? Where are delivered files stored locally vs. on the server? I don't really expect you to answer these things now, it's probably easier to just figure these things out when I'm back. Just saying what kinds of bottlenecks I ran into.

🎉 1

Sergey Sharybin commented

2024-08-16 11:57:10 +02:00

That's good points you're rising. Feels like we should expand docs a bit to help others get on-boarded and helping us in return. Will see how it fits into agenda.

👍 2

Bart van der Braak added this to the Buildbot Revamp milestone 2024-10-03 00:35:24 +02:00

Sign in to join this conversation.