Image Thumbnails not loading when importing Matcaps #106023

New Issue

Shady Hamza · 2023-03-23T09:31:10+01:00

Shady Hamza commented

2023-03-23 09:31:10 +01:00

System Information
Operating system: Windows-10-10.0.22621-SP0 64 Bits
Graphics card: NVIDIA GeForce RTX 2060 SUPER/PCIe/SSE2 NVIDIA Corporation 4.5.0 NVIDIA 531.41

Blender Version
Broken: version: 3.6.0 Alpha, branch: main, commit date: 2023-03-22 23:52, hash: 2de2db0f79d1
Worked: Works in 3.4.1 smoothly

Short description of error
Thumbs, whether matcaps or any high resolution image, sometimes do not load and can even freeze blender as if its loading thumbs.
In those cases we have to refresh.

Exact steps for others to reproduce the error

In the file browser, navigate to a directory containing hundreds of high-resolution EXR images (in my case, 4k HDRIs)
Scroll down to see some textures loading and those that possibly freeze.

**System Information** Operating system: Windows-10-10.0.22621-SP0 64 Bits Graphics card: NVIDIA GeForce RTX 2060 SUPER/PCIe/SSE2 NVIDIA Corporation 4.5.0 NVIDIA 531.41 **Blender Version** Broken: version: 3.6.0 Alpha, branch: main, commit date: 2023-03-22 23:52, hash: `2de2db0f79d1` Worked: Works in 3.4.1 smoothly **Short description of error** Thumbs, whether matcaps or any high resolution image, sometimes do not load and can even freeze blender as if its loading thumbs. In those cases we have to refresh. ![2023-03-24 05-05-07.mp4](https://projects.blender.org/attachments/e17381d9-a3bf-4859-9fbe-ec4556dbb9fa) **Exact steps for others to reproduce the error** - In the file browser, navigate to a directory containing hundreds of high-resolution EXR images (in my case, 4k HDRIs) - Scroll down to see some textures loading and those that possibly freeze.

Shady Hamza added the

labels 2023-03-23 09:31:10 +01:00

Shady Hamza commented

2023-03-23 09:48:04 +01:00

the other sollution i use is that i switch to cpu, and the viewport render works

Germano Cavalcante commented

2023-03-23 21:27:36 +01:00

Could you find simplified steps?

I duplicated this matcap 600 times but could not reproduce the problem.

Aso can you provide clear steps to reproduce the problem?

NOTE: To clear then all I used this command:

import bpy

for i in range(1000):
    bpy.ops.preferences.studiolight_uninstall(index=i)

Could you find simplified steps? I duplicated this matcap 600 times but could not reproduce the problem. Aso can you provide clear steps to reproduce the problem? NOTE: To clear then all I used this command: ``` import bpy for i in range(1000): bpy.ops.preferences.studiolight_uninstall(index=i) ```

0.exr

208 KiB

Germano Cavalcante added

Status

Needs Information from User

and removed

Status

Needs Triage

labels 2023-03-23 21:28:23 +01:00

Shady Hamza commented

2023-03-24 04:15:44 +01:00

also same issue when searching for 4k HDRIs EXR, they freeze blender as if its loading thumbs, but it doesn't then I have to refresh to see the thumbs
its much obvios in 4ks

also same issue when searching for 4k HDRIs EXR, they freeze blender as if its loading thumbs, but it doesn't then I have to refresh to see the thumbs its much obvios in 4ks

2023-03-24 05-05-07.mp4

42 MiB

Germano Cavalcante commented

2023-03-24 19:53:56 +01:00

Can you provide one of these images that do not load?
Maybe if we duplicate it a few times we can replicate the problem.

Can you provide one of these images that do not load? Maybe if we duplicate it a few times we can replicate the problem.

Shady Hamza commented

2023-03-25 02:27:39 +01:00

https://github.com/nidorx/matcaps
I use the 1024 folder

ps: the images load but after many refreshes as shown in video

https://github.com/nidorx/matcaps I use the 1024 folder ps: the images load but after many refreshes as shown in video

Shady Hamza commented

2023-03-25 03:00:07 +01:00

you can test on 4k hdri exrs also

Germano Cavalcante added

Status

Needs Triage

and removed

Status

Needs Information from User

labels 2023-03-28 17:46:26 +02:00

Harley Acheson commented

2023-03-28 18:28:37 +02:00

I have seen this behavior and there appears to be a small bug somewhere, but what is going on might not be exactly what you think it is.

Your video actually shows a combination of "not showing a preview even though there is one" and "preview is not yet done being made". The first is mostly shown when looking at the PNGs, the second when I think you were looking at EXRs.

The first issue is a bug somewhere. This is when you go to a folder that has not been thumbnailed and so those are created while looking at the list. It is possible for all the thumbnails to be created but the list is not properly informed of every creation. You hit refresh, it redraws the list and finds all the thumbnails.

The second isn't a bug. When you first look at a list of big EXRs that all have no thumbnails that thumbnail creation then starts in the background. Each core of your processor working on an image, so your computer becomes quite busy. On my old machine there are 8 processed at once. Because EXRs can take up a lot of space they are each constrained on how much of the image they can do at once to keep the memory used reasonable, but this also slows down the process somewhat. In the second part of your video your computer was still processing some previews when you were clicking "refresh". They just weren't all done yet.

How can you confirm all this?

On Windows the cache of thumbnail images are in a place like C:\Users{username}.thumbnails\large. If you empty that folder before viewing a folder in thumbnail view you can watch each preview being added to that folder as they are made.

No matter how large or complex of a folder I will get every thumbnail of every file eventually. So that process works, even if it can sometimes take longer than expected.

But back to the bug. If I run blender single-threaded then the thumbnail creation is no longer done in background threads, but are just done one at a time. In this case I always see a thumbnail in the File Browser list for every item.

However with normal threading enabled I get times where a thumbnail has been made (and shows up in my thumbnail cache folder) but the File Browser list does not update to reflect that it has been made. So we sometimes miss the signal that a thumbnail has been completed and so the list needs refreshing.

I have seen this behavior and there appears to be a small bug somewhere, but what is going on might not be exactly what you think it is. Your video actually shows a combination of "not showing a preview even though there is one" and "preview is not yet done being made". The first is mostly shown when looking at the PNGs, the second when I think you were looking at EXRs. The first issue is a bug somewhere. This is when you go to a folder that has not been thumbnailed and so those are created while looking at the list. It is possible for all the thumbnails to be created but the list is not properly informed of every creation. You hit refresh, it redraws the list and finds all the thumbnails. The second isn't a bug. When you first look at a list of big EXRs that all have no thumbnails that thumbnail creation then starts in the background. Each core of your processor working on an image, so your computer becomes quite busy. On my old machine there are 8 processed at once. Because EXRs can take up a lot of space they are each constrained on how much of the image they can do at once to keep the memory used reasonable, but this also slows down the process somewhat. In the second part of your video your computer was still processing some previews when you were clicking "refresh". They just weren't all done yet. How can you confirm all this? On Windows the cache of thumbnail images are in a place like C:\Users\{username}\.thumbnails\large\. If you empty that folder before viewing a folder in thumbnail view you can watch each preview being added to that folder as they are made. No matter how large or complex of a folder I will get every thumbnail of every file eventually. So that process works, even if it can sometimes take longer than expected. But back to the bug. If I run blender **single-threaded** then the thumbnail creation is no longer done in background threads, but are just done one at a time. In this case I always see a thumbnail in the File Browser list for every item. However with normal threading enabled I get times where a thumbnail has been made (and shows up in my thumbnail cache folder) but the File Browser list does not update to reflect that it has been made. So we sometimes miss the signal that a thumbnail has been completed and so the list needs refreshing.

Philipp Oeser commented

2023-04-25 12:57:44 +02:00

The first issue is a bug somewhere

OK, lets confirm this then

> The first issue is a bug somewhere OK, lets confirm this then

Philipp Oeser added

and removed

labels 2023-04-25 12:58:06 +02:00

Harley Acheson commented

2023-04-25 20:34:12 +02:00

Not sure what to do with this one.

I have been testing this code today with threading turned on, using my test suite of EXR samples. This is about 80 files (1.83 G that test various features, a bunch from Tears of Steel, and some I picked just because they have large dimensions or file sizes.

I'm just not finding a time when I don't get all the thumbnails properly showing up in my list, so I'm never having to click "refresh".

It is possible that this code has been improved. But it also could be possible that this problem could occur when thumbnailing 300, or 600, or more images at once. But once they are done they are done, so viewing the same folder later would show everything fine.

Without being able to recreate the issue now I am not able find where a problem might be. I'm okay with someone one day sending me a multi-gigabyte zip of files to test against.

Not sure what to do with this one. I have been testing this code today with threading turned on, using my test suite of EXR samples. This is about 80 files (1.83 G that test various features, a bunch from Tears of Steel, and some I picked just because they have large dimensions or file sizes. I'm just not finding a time when I don't get all the thumbnails properly showing up in my list, so I'm never having to click "refresh". It is possible that this code has been improved. But it also could be possible that this problem could occur when thumbnailing 300, or 600, or more images at once. But once they are done they are done, so viewing the same folder later would show everything fine. Without being able to recreate the issue now I am not able find where a problem might be. I'm okay with someone one day sending me a multi-gigabyte zip of files to test against.

Jesse Yurkovich commented

2023-04-26 05:41:33 +02:00

@Harley I have a bit more info in case it helps. First let me apologize because #107212 is going to get in the way of investigating this until I get it fixed. However, it has no bearing on this bug here. This situation is reproducible with 3.5 as well as 3.4 (for me at least).

I have a folder of 477 "simple" .pngs of size 256kb to 280kb.

Turning on browser thumbnails immediately generates 256 thumbnails in the cache folder - waiting longer doesn't load more of them
I scroll down to the very end of the list and now a total of 390 thumbnails are there - again waiting doesn't help
I scroll up, very slowly, to the top of the list but still 390 (??)
I scroll down, very slowly, to the bottom again and now all 477 are cached... But not all of the browser icons are updated
After scrolling up and down a bit the browser icons all eventually refresh

It's almost as if the preview jobs are not being triggered appropriately when scrolled into view.

@Harley I have a bit more info in case it helps. First let me apologize because #107212 is going to get in the way of investigating this until I get it fixed. However, it has no bearing on this bug here. This situation is reproducible with 3.5 as well as 3.4 (for me at least). I have a folder of 477 "simple" .pngs of size 256kb to 280kb. - Turning on browser thumbnails immediately generates 256 thumbnails in the cache folder - waiting longer doesn't load more of them - I scroll down to the very end of the list and now a total of 390 thumbnails are there - again waiting doesn't help - I scroll up, very slowly, to the top of the list but still 390 (??) - I scroll down, very slowly, to the bottom again and now all 477 are cached... But not all of the browser icons are updated - After scrolling up and down a bit the browser icons all eventually refresh It's almost as if the preview jobs are not being triggered appropriately when scrolled into view.

Harley Acheson commented

2023-04-26 16:13:38 +02:00

@deadpin

Thanks for that. I'll give it some testing with a larger sample of simpler PNGs as that will make it easier to test. I recall that we work on a "window" of a subset of longer lists but I haven't looked into it nor did I realize it might be that large at 256. The first thing (for me) to confirm here is whether this can be recreated while single-threaded versus threaded as that would make the problem much easier to see.

@deadpin Thanks for that. I'll give it some testing with a larger sample of simpler PNGs as that will make it easier to test. I recall that we work on a "window" of a subset of longer lists but I haven't looked into it nor did I realize it might be that large at 256. The first thing (for me) to confirm here is whether this can be recreated while single-threaded versus threaded as that would make the problem much easier to see.

Harley Acheson commented

2023-04-26 18:40:18 +02:00

@deadpin - I gathered up 542 png images and thought I could recreate this issue, but it turns out that current main fails the "imb_is_a_png" test for 177 of them. We don't visually differentiate between "thumbnail not yet done" and "thumbnail error" so the two problems just look the same. I added a comment to #107212 with two samples that fail.

Harley Acheson commented

2023-05-06 22:47:38 +02:00

With #107212 committed images are being thumbnailed fine again so we are back to the original complaint. Just some notes:

My test pool of 542 PNG images is perfect for testing this because it is sufficiently more than 256. Each image is small and quick to thumbnail. If I test these with threading turned OFF then everything seems to work perfectly, except for the expected long delay between opening the folder and seeing the final result.

I am testing with a "Lite" build but with threading turned on (WITH_TBB:BOOL=ON in CMakeCache.txt). When I initially load my test folder I see all the previews that are in view very quickly. If I simply wait without scrolling I can check my ".thumbnails" folder to see that 255 previews are created in the background. I can then use the scrollbars to move to the bottom of the folder so quickly that no more previews will be created - and so all of what I see are generic image icons without thumbnails. I wait, nothing changes.

I can also view the folder but then scroll slowly to the bottom. If I do this slowly enough all 542 thumbnails are created by the time I get to the bottom. But if I then scroll quickly to the top it will have long horrible pauses half way there. I can even see some generic icons at the top despite all thumbnails being available.

The "255" above and the behavior seems to come from a sliding window. FileListEntryCache has block_entries, block_start_index, block_end_index, block_center_index, block_cursor to deal with a partial display list.
filelist->filelist_cache.size starts at 1024 (FILELIST_ENTRYCACHESIZE_DEFAULT) but is almost immediately (and fairly repeatedly) set to 256 (in filelist_file_cache_slidingwindow_set) and the cache seems to get cleared very often. filelist_cache_clear gets called a lot more than I would expect. Not allowing the cache size to change from the initial 1024.

If filelist_file_cache_slidingwindow_set is not allowed to change the cache size to 256 from 1024: If I then view the folder without scrolling down then all 542 thumbnails will be made and once they are done I can scroll up and down the list very quickly. None of the pauses and stuttering when doing the same with 256.

But with cache size set to 1024, if I scroll to the bottom as quickly as I can then I get as few as 29 thumbnails no matter how much I scroll up and down after.

At this point I am not sure whether the root of the issue is one or more problems. The sliding window does seem to work correctly when scrolling slowly but then can also seem too small at times. It looks like cache is cleared too often and when it does any unfinished or in-process thumbnails are cancelled. And it seems like once the background processing of thumbnails is stopped it doesn't start again.

I'm guessing so far that this is problems with entries having FILE_ENTRY_PREVIEW_LOADING when they should not. This flag is set when an entry enters the queue for previewing (filelist_cache_previews_push) but is only cleared in filelist_cache_previews_update. So we don't clear this flag when we (quite often) call filelist_cache_previews_clear. But not quite sure yet.

With #107212 committed images are being thumbnailed fine again so we are back to the original complaint. Just some notes: My test pool of 542 PNG images is perfect for testing this because it is sufficiently more than 256. Each image is small and quick to thumbnail. If I test these with threading turned OFF then everything seems to work perfectly, except for the expected long delay between opening the folder and seeing the final result. I am testing with a "Lite" build but with threading turned on (WITH_TBB:BOOL=ON in CMakeCache.txt). When I initially load my test folder I see all the previews that are **in view** very quickly. If I simply wait without scrolling I can check my ".thumbnails" folder to see that 255 previews are created in the background. I can then use the scrollbars to move to the bottom of the folder so quickly that no more previews will be created - and so all of what I see are generic image icons without thumbnails. I wait, nothing changes. I can also view the folder but then scroll slowly to the bottom. If I do this slowly enough all 542 thumbnails are created by the time I get to the bottom. But if I then scroll quickly to the top it will have long horrible pauses half way there. I can even see some generic icons at the top despite all thumbnails being available. The "255" above and the behavior seems to come from a sliding window. FileListEntryCache has block_entries, block_start_index, block_end_index, block_center_index, block_cursor to deal with a partial display list. filelist->filelist_cache.size starts at 1024 (FILELIST_ENTRYCACHESIZE_DEFAULT) but is almost immediately (and fairly repeatedly) set to 256 (in filelist_file_cache_slidingwindow_set) and the cache seems to get cleared very often. filelist_cache_clear gets called a lot more than I would expect. Not allowing the cache size to change from the initial 1024. If filelist_file_cache_slidingwindow_set is not allowed to change the cache size to 256 from 1024: If I then view the folder without scrolling down then all 542 thumbnails will be made and once they are done I can scroll up and down the list very quickly. None of the pauses and stuttering when doing the same with 256. But with cache size set to 1024, if I scroll to the bottom as quickly as I can then I get as few as 29 thumbnails no matter how much I scroll up and down after. At this point I am not sure whether the root of the issue is one or more problems. The sliding window does seem to work correctly when scrolling slowly but then can also seem too small at times. It looks like cache is cleared too often and when it does any unfinished or in-process thumbnails are cancelled. And it seems like once the background processing of thumbnails is stopped it doesn't start again. I'm guessing so far that this is problems with entries having FILE_ENTRY_PREVIEW_LOADING when they should not. This flag is set when an entry enters the queue for previewing (filelist_cache_previews_push) but is only cleared in filelist_cache_previews_update. So we don't clear this flag when we (quite often) call filelist_cache_previews_clear. But not quite sure yet.

Harley Acheson self-assigned this 2023-05-07 03:16:10 +02:00

Harley Acheson referenced a pull request that will close this issue

2023-05-07 03:52:47 +02:00

Fix #106023: File Browser Lists with Incomplete Previews #107698

Harley Acheson referenced this issue

2023-05-08 17:49:21 +02:00

UI: Increase File Browser Minimum Cache size #107711

Harley Acheson closed this issue

2023-05-08 20:53:16 +02:00

Blender Bot added

Status

Resolved