Run automated tests on buildbot #69541

New Issue

Brecht Van Lommel · 2019-09-05T15:09:49+02:00

Brecht Van Lommel commented

2019-09-05 15:09:49 +02:00

Automated tests were enabled on buildbot.

- Failure to pass tests should stop the build from being made available for download. We can do this after we verified tests are passing reliably.
- Setup notifications for when buildbot master builds fail
- Automated testing with asserts and address sanitizer?
- Cycles GPU testing
- Eevee/Workbench GPU testing

Automated tests were enabled on buildbot. * - [x] Failure to pass tests should stop the build from being made available for download. We can do this after we verified tests are passing reliably. * - [ ] Setup [notifications ](http://docs.buildbot.net/current/manual/configuration/reporters/index.html) for when buildbot master builds fail * - [ ] Automated testing with asserts and address sanitizer? * - [ ] Cycles GPU testing * - [ ] Eevee/Workbench GPU testing

Brecht Van Lommel commented

2019-09-05 15:09:49 +02:00

Added subscriber: @brecht

Brecht Van Lommel commented

2019-09-05 15:11:48 +02:00

Currently tests are only working on Linux. The issue is that the buildbot installs Blender to a directory separate from the build directory. However the tests run Blender from the build directory, and then will miss the Python executable on Windows and OpenMP library on macOS.

We need to either make the tests run from the install directory, or install Blender to the build directory.

Currently tests are only working on Linux. The issue is that the buildbot installs Blender to a directory separate from the build directory. However the tests run Blender from the build directory, and then will miss the Python executable on Windows and OpenMP library on macOS. We need to either make the tests run from the install directory, or install Blender to the build directory.

blender-admin commented

2019-09-07 14:03:39 +02:00

This issue was referenced by 2028302f44

This issue was referenced by 2028302f4433910dc0841c4033b951275fa3f678

Brecht Van Lommel commented

2019-09-09 13:22:40 +02:00

Tests are now passing on al platforms after 2028302f44 and some additional fixes.

Tests are now passing on al platforms after 2028302f44 and some additional fixes.

Brecht Van Lommel commented

2020-05-05 12:24:47 +02:00

It seems builds are still being uploaded if tests fail.

Himanshi Kalra commented

2021-01-16 19:42:03 +01:00

Added subscriber: @Calra

Himanshi Kalra commented

2021-01-19 16:28:39 +01:00

From the discussion on blender chat, the notifications for failed build can be sent to #blender-coders. Help code snippet which will/can be used:

import requests
import json
url = 'https://blender.chat/api/v1/chat.sendMessage'
# 9p9NPvT7yFseE8ivA = #blender-coders you can get this from /api/v1/channels.list endpoint
payload = { 'message' : { 'rid' : '9p9NPvT7yFseE8ivA', 'msg' : 'test message' }}
# You get both the token and user-id from the personal access token section of the user settings.
headers = {'content-type': 'application/json',
'X-Auth-Token' : 'xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx',
'X-User-Id' : 'xxxxxxxxxxxxxxxxx'
}

r = requests.post(url, data=json.dumps(payload), headers=headers)
print(r.content)```

From the discussion on blender chat, the notifications for failed build can be sent to `#blender-coders`. Help code snippet which will/can be used: ```# rest api documented here https://docs.rocket.chat/api/rest-api import requests import json url = 'https://blender.chat/api/v1/chat.sendMessage' # 9p9NPvT7yFseE8ivA = #blender-coders you can get this from /api/v1/channels.list endpoint payload = { 'message' : { 'rid' : '9p9NPvT7yFseE8ivA', 'msg' : 'test message' }} # You get both the token and user-id from the personal access token section of the user settings. headers = {'content-type': 'application/json', 'X-Auth-Token' : 'xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx', 'X-User-Id' : 'xxxxxxxxxxxxxxxxx' } r = requests.post(url, data=json.dumps(payload), headers=headers) print(r.content)```

Himanshi Kalra commented

2021-01-23 15:31:53 +01:00

Hey @brecht I am kind of getting stuck, could you give some insight?

since they already provide a lot of Reporters (bots) I kind of can't seem to figure out how exactly is the build status being conveyed?
am thinking of creating an instance of StatusBot class and extracting the build information from there? as that is the base class being used by the [IRC]] and [ https:*github.com/buildbot/buildbot/blob/e882d79d71161e471fb63355048f93d4e8c65f96/master/buildbot/reporters/telegram.py | Telegram bot
not sure if there is an obviously easy way, which I can't seem to find?
I also had a few design questions.
we want a simple report failed test status bot?
Not an interactive one? (start build/ stop build etc) or that could be a long term goal?
Using it on IRC was simple

Hey @brecht I am kind of getting stuck, could you give some insight? * since they already provide a lot of Reporters (bots) I kind of can't seem to figure out how exactly is the build status being conveyed? * am thinking of creating an instance of `StatusBot` class and extracting the build information from there? as that is the base class being used by the [IRC]] and [[ https:*github.com/buildbot/buildbot/blob/e882d79d71161e471fb63355048f93d4e8c65f96/master/buildbot/reporters/telegram.py | Telegram](https:*github.com/buildbot/buildbot/blob/a0e1d8840e8856ead136a1ad6e2021931355af15/master/buildbot/reporters/irc.py) bot * not sure if there is an obviously easy way, which I can't seem to find? I also had a few design questions. * we want a simple report failed test status bot? * Not an interactive one? (start build/ stop build etc) or that could be a long term goal? Using it on IRC was simple ![image.png](https://archive.blender.org/developer/F9593960/image.png)

Brecht Van Lommel commented

2021-01-24 00:18:56 +01:00

The functionality I would expect is just a message whenever a build fails. And then perhaps that could be made a bit smarter to group together results from different platforms, filter out reports from the custom branch builder, not report consecutive build failures if it's too noisy, etc. Whatever we find out is a good signal to noise ratio in practice. Maybe just showing every failed build for every platform is fine.

I don't think the bot has to be interactive. It doesn't really solve a problem we have as far as I know, and would mean more maintenance and things that can break. If we do, then StatusBot would make sense since it provides that kind of functionality.

But for our purposes, just subclassing HttpStatusPush seems enough, as is done for the Hipchat bot for example?

The functionality I would expect is just a message whenever a build fails. And then perhaps that could be made a bit smarter to group together results from different platforms, filter out reports from the custom branch builder, not report consecutive build failures if it's too noisy, etc. Whatever we find out is a good signal to noise ratio in practice. Maybe just showing every failed build for every platform is fine. I don't think the bot has to be interactive. It doesn't really solve a problem we have as far as I know, and would mean more maintenance and things that can break. If we do, then `StatusBot` would make sense since it provides that kind of functionality. But for our purposes, just subclassing `HttpStatusPush` seems enough, as is done for the Hipchat bot for example?

Himanshi Kalra commented

2021-01-29 14:54:57 +01:00

Hey @brecht I looked into HttpStatusPush, it seems apt for our use case.

I made one BlenderChatStatusPush subclassing HttpStatusPush. Here is what all I tried:

Directly trying to use HttpStatusPush to send some message to Blender Chat, from the documentation, it says it returns JSON object.
For the generator parameter, I created an object of BuildStatusGenerator class, with mode="all"
For user and password I used my Blender Chat user ID and authorization token.
Subclassing HttpStatusPush and trying to make some changes to the sendMessage, configService and reconfigService method
Explicitly calling sendMessage method but then it requires other parameters like build
Here is my understanding, correct me if I am wrong
All build information is passed internally, we don't need to provide the builder name explicitly (as the IRC bot already had all this info)

I didn't have much success with sending the status to blender.chat or rather any message using the reporter.

Thinking HttpClientService is not receiving proper parameters to make a post request?

I couldn't use Hipchat api to test it, Hipchat has been shutdown, also Hipchat used HttpStatusPushBase which had been deprecated, I mostly toyed around with HttpStatusPush. Could you look into this?
Thanks

Hey @brecht I looked into `HttpStatusPush`, it seems apt for our use case. I made one `BlenderChatStatusPush` subclassing `HttpStatusPush`. Here is what all I tried: * Directly trying to use `HttpStatusPush` to send some message to Blender Chat, from the documentation, it says it returns JSON object. * For the `generator` parameter, I created an object of `BuildStatusGenerator` class, with `mode="all"` * For `user` and `password` I used my Blender Chat user ID and authorization token. * Subclassing `HttpStatusPush` and trying to make some changes to the `sendMessage`, `configService` and `reconfigService` method * Explicitly calling `sendMessage` method but then it requires other parameters like `build` Here is my understanding, correct me if I am wrong * All build information is passed internally, we don't need to provide the builder name explicitly (as the IRC bot already had all this info) I didn't have much success with sending the status to blender.chat or rather any message using the reporter. * Thinking `HttpClientService` is not receiving proper parameters to make a post request? I couldn't use Hipchat api to test it, Hipchat has been shutdown, also Hipchat used `HttpStatusPushBase` which had been deprecated, I mostly toyed around with `HttpStatusPush`. Could you look into this? Thanks

Brecht Van Lommel commented

2021-01-29 15:35:57 +01:00

The authorization token is not an HTTP password and can't be used as such. The user and token need to be put in the header as in the code snippet.

Subclassing just HttpStatusPushBase may be enough. You don't even have to use httpclientservice, using requests as in the code snippet instead is probably fine.

You need to implement the send method (not sendMessage), and convert the build object it receives into a human-readable text message.

The authorization token is not an HTTP password and can't be used as such. The user and token need to be put in the header as in the code snippet. Subclassing just `HttpStatusPushBase` may be enough. You don't even have to use `httpclientservice`, using `requests` as in the code snippet instead is probably fine. You need to implement the `send` method (not `sendMessage`), and convert the `build` object it receives into a human-readable text message.

Himanshi Kalra commented

2021-01-29 16:44:51 +01:00

In #69541#1103566, @brecht wrote:
The authorization token is not an HTTP password and can't be used as such. The user and token need to be put in the header as in the code snippet.

Yes, I passed the authorization token and user id in the headers as well.

Subclassing just HttpStatusPushBase may be enough. You don't even have to use httpclientservice, using requests as in the code snippet instead is probably fine.

Yes, this was my first approach, I tried directly using post method of requests and attempted to send a message to myself on blender.chat inside both send/sendMessage method (separately).

You need to implement the send method (not sendMessage), and convert the build object it receives into a human-readable text message.

I realized this after already commenting, I was implementing it in the wrong method. I will try it again.

> In #69541#1103566, @brecht wrote: > The authorization token is not an HTTP password and can't be used as such. The user and token need to be put in the header as in the code snippet. Yes, I passed the authorization token and user id in the headers as well. > Subclassing just `HttpStatusPushBase` may be enough. You don't even have to use `httpclientservice`, using `requests` as in the code snippet instead is probably fine. Yes, this was my first approach, I tried directly using `post` method of `requests` and attempted to send a message to myself on blender.chat inside both `send/sendMessage` method (separately). > You need to implement the `send` method (not `sendMessage`), and convert the `build` object it receives into a human-readable text message. I realized this after already commenting, I was implementing it in the wrong method. I will try it again.

James Monteath commented

2021-03-23 06:58:28 +01:00

Added subscriber: @ponderz

Philipp Oeser removed the

Download

What's New

Blender Studio

Manual

Developers Blog

Documentation

Benchmark

Blender Conference

Development Fund

One-time Donations

Run automated tests on buildbot #69541