Homepage: Randomize the position of items #215

Pablo Vazquez · 2024-07-18T16:05:51+02:00

Pablo Vazquez commented

2024-07-18 16:05:51 +02:00

Place items randomly on every page load so it is a bit more fair for everyone.

This is only for the 8 add-ons and themes in the homepage. It does not affect other listings since they have custom sorting.

Place items randomly on every page load so it is a bit more fair for everyone. This is only for the 8 add-ons and themes in the homepage. It does not affect other listings since they have custom sorting. <video src="https://projects.blender.org/attachments/503de328-cbb1-405d-839a-4b09bcfc1707" autoplay loop controls></video>

extensions-home-random.mp4

194 KiB

Pablo Vazquez added the

Type

Enhancement

label 2024-07-18 16:05:51 +02:00

Pablo Vazquez added 1 commit 2024-07-18 16:05:53 +02:00

Homepage: Randomize the position of items 764c1c3fa1

Place items randomly on every load so it is a bit more fair
for all the top extensions.

Dalai Felinto approved these changes 2024-07-18 16:17:34 +02:00

Dalai Felinto left a comment

I love the functionality.

Maybe @Oleg-Komarov wants to comment on the code approach?

I love the functionality. Maybe @Oleg-Komarov wants to comment on the code approach?

Oleg-Komarov commented

2024-07-18 19:37:42 +02:00

What is the end goal of this?
I think on a lot of screens all the top 8 cards are visible, so shuffling them doesn't give more exposure to different items - the same ones are shown, just in different order.

Performance-wise, I don't mind this because shuffling the top-8 items in memory is cheap, but I wonder what kind of user behavior change do we expect?

If we have a hypothesis that the first 4 of the top cards get more visits, we can verify this looking at the access log data (before and after this change).
But if we don't have any particular expectations, I wonder why do this?

Current data for all visits (not deduplicated by user) for last 3 days for top 8 extensions:

/add-ons/nd/                            6780
/add-ons/looptools/	                    6571
/add-ons/bool-tool/	                    2910
/add-ons/matalogue/                     2883
/add-ons/io-scene-max/                  2809
/add-ons/ucupaint/	                    2596
/add-ons/bioxelnodes/	                2497
/add-ons/orient-and-origin-to-selected/	2211

Bool-tool is on the 3rd place in visits, but it is the last on the website.

What is the end goal of this? I think on a lot of screens all the top 8 cards are visible, so shuffling them doesn't give more exposure to different items - the same ones are shown, just in different order. Performance-wise, I don't mind this because shuffling the top-8 items in memory is cheap, but I wonder what kind of user behavior change do we expect? If we have a hypothesis that the first 4 of the top cards get more visits, we can verify this looking at the access log data (before and after this change). But if we don't have any particular expectations, I wonder why do this? Current data for all visits (not deduplicated by user) for last 3 days for top 8 extensions: ``` /add-ons/nd/ 6780 /add-ons/looptools/ 6571 /add-ons/bool-tool/ 2910 /add-ons/matalogue/ 2883 /add-ons/io-scene-max/ 2809 /add-ons/ucupaint/ 2596 /add-ons/bioxelnodes/ 2497 /add-ons/orient-and-origin-to-selected/ 2211 ``` Bool-tool is on the 3rd place in visits, but it is the last on the website.

Oleg-Komarov commented

2024-07-18 19:55:15 +02:00

A different idea may be to show completely random extensions every time.
The theory behind this would be to get a more uniform distribution of visits and ratings in the tail of the distribution.

What would be the value for the users in this? Accidentally come across an extension they didn't know they needed?
I'm not sure how to judge if that is valuable - we could see if more extensions get rated.

Implementing this would need slightly more code, I would probably test the following:
select all listed IDs, probably cache them in memory to avoid this select on every page view, and then do a random.sample(cached_addon_ids, 8) so on every page view we would get a random sample from the whole population of the listed add-ons.

A different idea may be to show completely random extensions every time. The theory behind this would be to get a more uniform distribution of visits and ratings in the tail of the distribution. What would be the value for the users in this? Accidentally come across an extension they didn't know they needed? I'm not sure how to judge if that is valuable - we could see if more extensions get rated. Implementing this would need slightly more code, I would probably test the following: select all listed IDs, probably cache them in memory to avoid this select on every page view, and then do a `random.sample(cached_addon_ids, 8)` so on every page view we would get a random sample from the whole population of the listed add-ons.

Pablo Vazquez added 1 commit 2024-07-19 12:33:48 +02:00

Get a longer list but display a smaller sample 34dd1c4a15

Pablo Vazquez commented

2024-07-19 12:35:15 +02:00

What is the end goal of this?
The end goal is to expose users visiting the homepage to popular extensions they might have missed. While being more fair to extensions that do not make the top-left part of the row.

There are other ways to tackle this of course, but randomizing the already visible list I think it could help.

What would be the value for the users in this? Accidentally come across an extension they didn't know they needed?

Yes. That's what the homepage is about, discovery.

I quite like the random sample idea!

Would it be expensive to get a slightly longer list, say 3 times the size of the ones displayed? I just pushed the following change:

Retrieve the top rated 24 add-ons, list 8.
Retrieve the top rated 12 themes, list 4.

    def get_context_data(self, **kwargs):
        q = super().get_queryset().order_by('-rating_sortkey')

        addons_list = list(q.filter(type=EXTENSION_TYPE_CHOICES.BPY)[:24])
        themes_list = list(q.filter(type=EXTENSION_TYPE_CHOICES.THEME)[:12])

        # Randomize a sample of the extensions to list.
        context = {
            'addons': random.sample(addons_list, 8),
            'themes': random.sample(themes_list, 4),
        }

        return context

(already discussed with @dfelinto that the homepage could show more add-ons than themes since they are more popular anyway).

> What is the end goal of this? The end goal is to expose users visiting the homepage to popular extensions they might have missed. While being more fair to extensions that do not make the top-left part of the row. There are other ways to tackle this of course, but randomizing the already visible list I think it could help. > What would be the value for the users in this? Accidentally come across an extension they didn't know they needed? Yes. That's what the homepage is about, discovery. ---- I quite like the random sample idea! Would it be expensive to get a slightly longer list, say 3 times the size of the ones displayed? I just pushed the following change: * Retrieve the top rated **24** add-ons, list **8**. * Retrieve the top rated **12** themes, list **4**. ```python def get_context_data(self, **kwargs): q = super().get_queryset().order_by('-rating_sortkey') addons_list = list(q.filter(type=EXTENSION_TYPE_CHOICES.BPY)[:24]) themes_list = list(q.filter(type=EXTENSION_TYPE_CHOICES.THEME)[:12]) # Randomize a sample of the extensions to list. context = { 'addons': random.sample(addons_list, 8), 'themes': random.sample(themes_list, 4), } return context ``` (already discussed with @dfelinto that the homepage could show more add-ons than themes since they are more popular anyway).

Oleg-Komarov commented

2024-07-19 13:08:40 +02:00

Would it be expensive to get a slightly longer list, say 3 times the size of the ones displayed?

This does a little bit of unnecessary extra work when constructing the objects (with all dependencies) that are then thrown away, but shouldn't be a big deal, I think.
It's possible to optimize it by fetching just the ids, sampling those, then fetching the objects, and shuffling them (because they will be always sorted when fetched by ids).

Let's deploy this and check the latency, we can optimize if needed.

> Would it be expensive to get a slightly longer list, say 3 times the size of the ones displayed? This does a little bit of unnecessary extra work when constructing the objects (with all dependencies) that are then thrown away, but shouldn't be a big deal, I think. It's possible to optimize it by fetching just the ids, sampling those, then fetching the objects, and shuffling them (because they will be always sorted when fetched by ids). Let's deploy this and check the latency, we can optimize if needed.

Oleg-Komarov approved these changes 2024-07-19 13:09:08 +02:00

Dismissed

Oleg-Komarov dismissed Oleg-Komarov’s review 2024-07-19 13:11:18 +02:00

Reason:

tests are failing

Oleg-Komarov commented

2024-07-19 13:12:36 +02:00

We should add a check that we have enough items to sample from - the tests are failing because the number of test extensions is less than the hard-coded constants.

Pablo Vazquez added 1 commit 2024-07-19 14:39:32 +02:00

Fix issue where sample is larger than available extensions 2534781d99

Simply pick the minimum between the desired sample size and length
of the list of available extensions.

Pablo Vazquez commented

2024-07-19 14:41:15 +02:00

Oops sorry, my local db has more extensions than the sample required, and forgot to run the tests.

I've run the tests now and they run okay.

My fix was to get the minimum value between the desired sample size (8 add-ons, 4 themes) and the ones actually available.

        # Get 8 add-ons, 4 themes, or the minimum available.
        addons_sample_size = min(8, len(addons_list))
        themes_sample_size = min(4, len(themes_list))

        # Randomize a sample of the extensions to list.
        context = {
            'addons': random.sample(addons_list, addons_sample_size),
            'themes': random.sample(themes_list, themes_sample_size),
        }

Oops sorry, my local db has more extensions than the sample required, and forgot to run the tests. I've run the tests now and they run okay. My fix was to get the minimum value between the desired sample size (8 add-ons, 4 themes) and the ones actually available. ```python # Get 8 add-ons, 4 themes, or the minimum available. addons_sample_size = min(8, len(addons_list)) themes_sample_size = min(4, len(themes_list)) # Randomize a sample of the extensions to list. context = { 'addons': random.sample(addons_list, addons_sample_size), 'themes': random.sample(themes_list, themes_sample_size), } ```

Oleg-Komarov approved these changes 2024-07-19 14:43:10 +02:00

Oleg-Komarov left a comment

NP, some day we should get a CI to run the tests automatically ;)

Thanks for finishing this!

NP, some day we should get a CI to run the tests automatically ;) Thanks for finishing this!

🎉 1

Pablo Vazquez merged commit e87a583fed into main

2024-07-19 14:46:59 +02:00

Pablo Vazquez deleted branch ui-home-randomize

2024-07-19 14:47:00 +02:00

Pablo Vazquez referenced this issue from a commit

2024-07-19 14:47:01 +02:00