The Blender File Open dialog triggers OneDrive Files On-Demand downloads #87621

System Information

Operating system:	Windows-10-10.0.19041-SP0 64 Bits
	MacOS Big Sur 11.2.3

Graphics card:	NVIDIA GeForce MX250/PCIe/SSE2 NVIDIA Corporation 4.5.0 NVIDIA 466.11

Blender Version
Broken: version: 2.92 - 3.0.0 Alpha
Worked: (To check)

Short description of error
OneDrive has a feature called "Files On-Demand".
This enables the OneDrive folder and its contents to be displayed in the Finder, without actually downloading all files to the local machine.
When you actually access/open a file it gets downloaded to the local machine first and then opened.

The Blender file dialogs trigger this download even when just browsing the OneDrive folder.
So if a OneDrive folder contains files, merely visiting that OneDrive folder using Blender will trigger a download of all those files.
Depending on the size and count of the files that have to be downloaded and the download speed, this also freezes Blender until the downloads have finished.

I noticed this behavior using OneDrive version 21.052.0314.0001.

Other apps and MacOS are perfectly able to navigate/enumerate a OneDrive folder, without downloading its contents.

Update 1: It could be the same problem as described in #87472 (Open file dialog causes application to hang).
Update 2: An possible workaround: on MacOS you can use OneDrive settings to exclude certain apps from triggering OneDrive on-demand downloads.

Exact steps for others to reproduce the error

Go to your OneDrive folder
Create a file and add 1 or 2 images to it
Right-click on the images and click on "Free Space" (The file is cleaned and accessed only in the Cloud)
Open Blender
Open an image file viewer (like "Open" in the Image editor)
Go to the directory you created on OneDrive
Change the view mode to "ThumbNails"
Note that the images are downloaded at that moment.
Apparently the Thumbnails read by Blender are created and saved in C:\Users\%user%\.thumbnails\large.

Added subscriber: @Zandman

Added subscriber: @deadpin

Are you using the Thumbnail view in the File Browser? Locally on Windows+OneDrive here the Blender file browser does not download anything unless I toggle on the thumbnail view. At this point blender needs to load the thumbnail picture that's stored in the .blend file, hence the download.

In #87621#1148561, @deadpin wrote:
Are you using the Thumbnail view in the File Browser?

No, not using thumbnail view. Using the default "Files as a vertical list" view. I just noticed OneDrive also displayed a MacOS notification saying that Blender is downloading files from OneDrive. In the OneDrive settings I can excempt Blender from triggering downloads, but I'd rather just have it working properly.

I also noticed your talking about Windows. This seems to be a MacOS issue.

Here's some prior art: https://answers.microsoft.com/en-us/msoffice/forum/msoffice_onedrivefb-mso_mac-mso_o365b/how-to-avoid-unwanted-downloading-of-files-with/39b2de3f-3444-41b5-b988-64c96e5a1838?auth=1

In any case, it seems Blender's file dialog is enumerating files in a way that triggers their download from OneDrive. Fortunately it only does it for files in the currently folder. It doesn't download sub-folders. But when I navigate to a sub-folder, all files within will be downloaded.

It appears this issue is also happening on Windows: #87472 (Open file dialog causes application to hang)

I did some debugging and I've narrowed it down to line 82 in source/blender/blenlib/intern/storage_apple.mm:

const NSDictionary *resourceKeyValues = [fileURL resourceValuesForKeys:resourceKeys error:nil];

When this line is executed for a file in my OneDrive folder that has not yet been downloaded, it will trigger OneDrive to download the file. When I add a return statement right above line 82 so that the code isn't executed, the file won't be downloaded.

So apparently obtaining info about files whether they are aliases/hidden/readable/writable causes them to be touched in a way that triggers OneDrive. Maybe there's a way to obtain this info without triggering the OneDrive download?

After some more spelunking it appears that the download is triggered specifically for NSURLIsReadableKey and NSURLIsWritableKey. Checking for the other two keys won't trigger a download.

Why does the Blender file dialog even need to know whether a file is readable or writable when enumerating them? Is this info actively used at that stage? Or is it gathering this info so add-ons can use it?

Added subscriber: @mano-wii

Please use Help -> Report a Bug in Blender so there is complete information about the Blender version and OS.
Optionally you could provide the system-info.txt generated in Help -> Save System Info.

Windows and MacOS have a different API for reading files, so it is quite possible that the problem in #87472 is not related.
But it’s a good start to investigate if it’s reproducible on any MacOS.

Changed status from 'Needs Triage' to: 'Confirmed'

I found the steps to reproduce the problem. I can confirm it now.
I suppose it is a high priority as it can negatively affect many users working on OneDrive.

Added subscriber: @ankitm

Why does the Blender file dialog even need to know whether a file is readable or writable when enumerating them? Is this info actively used at that stage? Or is it gathering this info so add-ons can use it?

Because the thumbnail overlays like for non-writable files, aliases depend on that knowledge. The code was added in 1f223b9a1f and afb1a64ccb

Added subscriber: @Harley

The central problems seems to be that users are asking for thumbnail creation for files that are not fully downloaded. So if this issue is fixed we'll get complaints that users cannot see thumbnails for files in OneDrive. LOL

Sounds like we can just ignore the request to make thumbnails if the the file attributes include whatever OneDrive uses for files in that state. Are you able to attrib some of those files while they are not fully downloaded? I'm guessing it uses FILE_ATTRIBUTE_OFFLINE or FILE_ATTRIBUTE_RECALL_ON_DATA_ACCESS or FILE_ATTRIBUTE_RECALL_ON_OPEN or FILE_ATTRIBUTE_SPARSE_FILE or FILE_ATTRIBUTE_TEMPORARY?

I've seen conflicting information about attributes used by OneDrive, including that there might be new ones. FILE_ATTRIBUTE_PINNED, FILE_ATTRIBUTE_UNPINNED, FILE_ATTRIBUTE_RECALL_ON_OPEN, FILE_ATTRIBUTE_RECALL_ON_DATA_ACCESS

I've also read that some programs calling regular API functions can get a filtered version of attributes: https://stackoverflow.com/questions/54560454/getting-full-file-attributes-for-files-managed-by-microsoft-onedrive

If treating these files differently by attribute is not feasible then we could always just deal with everything in special folder locations FOLDERID_SkyDrive, FOLDERID_SkyDriveDocuments, FOLDERID_SkyDrivePictures, FOLDERID_SkyDriveMusic, FOLDERID_SkyDriveCameraRoll

In #87621#1152655, @Harley wrote:
The central problems seems to be that users are asking for thumbnail creation for files that are not fully downloaded. So if this issue is fixed we'll get complaints that users cannot see thumbnails for files in OneDrive. LOL

I was just experimenting with these OneDrive pinning attributes on my local machine. They've changed the implementation a couple of times during these last years, but I think the current Windows 10 implementation is pretty stable.

However, on Windows I cannot reproduce the unwanted downloads. On my system Blender only triggers file downloads when I switch to thumbnails view. Which kinda makes sense. But on macOS downloads are triggered even when in normal list view. So there is different behavior.

MacOS 10.7 and later also have the concept of files on-demand for iCloud. There are also APIs to query the file state. I have to experiment some more to see how this all works. Also, I don't know if iCloud and OneDrive both use the same implementation of this. I don't think so, as iCloud files seem to be renamed to a hidden counterpart (filename starting with a dot) when cloud-only. I don't see this behavior with OneDrive. Would be nice if both OneDrive and iCloud used the same scheme on the Mac. Maybe they will now that Apple seems to have a formalized way to handle cloud files. Here's some info on how you can query iCloud files: https://developer.apple.com/documentation/foundation/filemanager

To summarize: I guess Blender needs to be aware of cloud-files for both OneDrive and iCloud and handle them appropriately.

In addition to that: currently the Blender file dialog doesn't show the OneDrive or iCloud folders in the left side navigation. There's a case to be made to add this, as both Windows and Mac seem to view them as part of the system.

@Zandman - On my system Blender only triggers file downloads when I switch to thumbnails view. Which kinda makes sense.

Yeah, if you want thumbnails of files then they have to be available to load and examine. Doing it "better" in this case would be checking these file attributes to better know their state, and if they are not fully available then not do thumbnailing - despite being in that file view - and show some other type of icon instead perhaps. OneDrive is so integrated, as far as users see, that I'm not sure they would understand why their files were being shown in a different way in this particular place. Some users even change their "My Documents" to use a OneDrive folder. We don't really have control over what they do.

@Zandman - But on macOS downloads are triggered even when in normal list view. So there is different behavior.

It might be the check for aliasing. With an alias (or shortcut) we want to know the actual target. With Windows we only examine the file to find the target if the file has a (hidden) extension of ".lnk". But Mac aliases don't follow the same type of pattern and it might be possible that the process we use to determine "is this an an alias" has to read the file in a way that requires the file to... be there. Not sure as I don't compile for Mac.

To summarize: I guess Blender needs to be aware of cloud-files for both OneDrive and iCloud and handle them appropriately.

That could be difficult without any kind of standard though. Any way we could deal with OneDrive will not apply to Dropbox or Google Drive, Amazon Drive etc. So there are a lot more of these than the "both." And these will differ between platforms as well. Ideally they would just integrate their systems into the OS in a way that would be transparent to client programs. Or users will learn that these file hosting and synchronization services do have some caveats to be aware of.

In addition to that: currently the Blender file dialog doesn't show the OneDrive or iCloud folders in the left side navigation. There's a case to be made to add this, as both Windows and Mac seem to view them as part of the system.

Worth considering adding those shortcuts to the "System" lists if they ever work better, but not sure I'd want to encourage their use while just turning on thumbnailing can render them so sucky. When I was adding most of those Windows "special" folders to the "System" list I did consider "OneDrive". But I was loath to add it as it would promote a Microsoft commercial service over other similar services. And then we'd invariably get complaints of "why isn't DropBox shown?"

But the Blender File Manager is actually aware of the location of the OneDrive folder. If you just browse into your user root folder ("Home" icon in the System list) you will see, among the folders for "Desktop", "Music", etc, one for "OneDrive" and that folder will have a special icon if in thumbnail view. And if you add a bookmark while in that folder you will get an item in your "Favorites" list with a special icon too. On Windows anyway.

> @Zandman - On my system Blender only triggers file downloads when I switch to thumbnails view. Which kinda makes sense. Yeah, if you want thumbnails of files then they have to be available to load and examine. Doing it "better" in this case would be checking these file attributes to better know their state, and if they are not fully available then not do thumbnailing - despite being in that file view - and show some other type of icon instead perhaps. OneDrive is so integrated, as far as users see, that I'm not sure they would understand why their files were being shown in a different way in this particular place. Some users even change their "My Documents" to use a OneDrive folder. We don't really have control over what they do. > @Zandman - But on macOS downloads are triggered even when in normal list view. So there is different behavior. It might be the check for aliasing. With an alias (or shortcut) we want to know the actual target. With Windows we only examine the file to find the target if the file has a (hidden) extension of ".lnk". But Mac aliases don't follow the same type of pattern and it might be possible that the process we use to determine "is this an an alias" has to read the file in a way that requires the file to... be there. Not sure as I don't compile for Mac. > To summarize: I guess Blender needs to be aware of cloud-files for both OneDrive and iCloud and handle them appropriately. That could be difficult without any kind of standard though. Any way we could deal with OneDrive will not apply to Dropbox or Google Drive, Amazon Drive etc. So there are a lot more of these than the "both." And these will differ between platforms as well. Ideally they would just integrate their systems into the OS in a way that would be transparent to client programs. Or users will learn that these file hosting and synchronization services do have some caveats to be aware of. > In addition to that: currently the Blender file dialog doesn't show the OneDrive or iCloud folders in the left side navigation. There's a case to be made to add this, as both Windows and Mac seem to view them as part of the system. Worth considering adding those shortcuts to the "System" lists if they ever work better, but not sure I'd want to encourage their use while just turning on thumbnailing can render them so sucky. When I was adding most of those Windows "special" folders to the "System" list I did consider "OneDrive". But I was loath to add it as it would promote a Microsoft commercial service over other similar services. And then we'd invariably get complaints of "why isn't DropBox shown?" But the Blender File Manager is actually aware of the location of the OneDrive folder. If you just browse into your user root folder ("Home" icon in the System list) you will see, among the folders for "Desktop", "Music", etc, one for "OneDrive" and that folder will have a special icon if in thumbnail view. And if you add a bookmark while in that folder you will get an item in your "Favorites" list with a special icon too. On Windows anyway.

Some further details.

On Windows, files in this "unavailable" state have FILE_ATTRIBUTE_OFFLINE set. There are other file attributes used for this process:

online-only
FILE_ATTRIBUTE_PINNED : False
FILE_ATTRIBUTE_UNPINNED : True
FILE_ATTRIBUTE_RECALL_ON_DATA_ACCESS : True
locally available
FILE_ATTRIBUTE_PINNED : False
FILE_ATTRIBUTE_UNPINNED : False
FILE_ATTRIBUTE_RECALL_ON_DATA_ACCESS : False
always available
FILE_ATTRIBUTE_PINNED : True
FILE_ATTRIBUTE_UNPINNED : False
FILE_ATTRIBUTE_RECALL_ON_DATA_ACCESS : False

But I'd say that none of these new attributes matter. The only thing that matters for this case is that it is marked as offline, using an attribute that has been in place for years. So also likely to be used by other similar systems.

And we are currently checking and storing this one two. Calling BLI_file_attributes() does include FILE_ATTR_OFFLINE (from BLI_fileops.h eFileAttributes) if FILE_ATTRIBUTE_OFFLINE.

So any time we have a file list entry (FileDirEntry or FileListInternEntry) the attributes member can tell us if this is offline. So hopefully we have one when thumbnails are made, or can perhaps call BLI_file_attributes() if not.

Although there is an Mac version of BLI_file_attributes() in storage_apple.mm it does not populate FILE_ATTRIBUTE_OFFLINE, nor could I quickly see that was an option to do with their API.

So there is almost certainly a one-line fix to this.

editors\space_file\filelist.c has a background job that makes preview thumbnails and saves ids to the resulting thumbnail in preview_icon_id member of FileDirEntry. So we just have to not do thumbnail creation if FileDirEntry.attributes contains FILE_ATTRIBUTE_OFFLINE

Not yet understanding that process yet though.

I'd be inclined towards supporting native cloud drives. I could only find API for iCloud drive on mac, and nothing for OneDrive.

Unfortunately, I think this thread has gotten a bit off track by all the responses about thumbnail generation. This issue isn't related to thumbnail generation. Yes, when Blender generates thumbnails it will trigger the download of those cloud files. However, this I could see as perfectly acceptable behavior, because Blender actually has to read the file for thumbnails to be generated.

The main issue I reported however seems to only happen on macOS, where merely enumerating the files in a OneDrive folder triggers the downloading of those files. And this happens because each file is queried specifically for the NSURLIsReadableKey and NSURLIsReadableKey attributes. I've tested this in a separate application and it can be reproduced. I think this is a bug (or at least an unfortunate side-effect) that has to be fixed. Now this might also be a bug in the OneDrive application. Because querying for those other attributes does not trigger a download. I've contacted the OneDrive team about this, but haven't yet heard back from them. A (temporary) workaround might be to only query for those attributes after a user has explicitly selected the thumbnail view.

Now, besides that main issue there's the broader issue of supporting cloud systems like iCloud and OneDrive. And I think those two can be singled out for now, because those two are actually considered system services and by now offer official API support for their respective on-demand files implementation.

It might be the check for aliasing.

Nope, at least not on Mac.

That could be difficult without any kind of standard though.

Indeed. But as I've said both Windows and Mac seem to consider their cloud drive solutions to be part of the system and have added explicit API support for such files. So the way I see it Blender running on Mac should natively support iCloud. And Blender running on Windows should natively support OneDrive.

It gets more difficult when Blender has to support non-native cloud drive solutions. In an ideal world all those cloud drive solutions would use the system-provided APIs. That doesn't seem to be the case yet. But that might change in the near future. So OneDrive on macOS has its own Finder integration and on-demand file scheme. But it might use the iCloud way going forward.

But I was loath to add it as it would promote a Microsoft commercial service over other similar services. And then we'd invariably get complaints of "why isn't DropBox shown?"

As I've said, I consider them system services. So supporting OneDrive on Windows and iCloud on Mac seems perfectly legal to me.

So there is almost certainly a one-line fix to this.

I hope by now you know this isn't about thumbnails and unfortunately there is no one-line fix. Unless the querying for NSURLIsReadableKey and NSURLIsReadableKey gets removed. That worked for me, but I haven't tested the impact of such a change further down the line.

Unfortunately, I think this thread has gotten a bit off track by all the responses about thumbnail generation. This issue isn't related to thumbnail generation. Yes, when Blender generates thumbnails it will trigger the download of those cloud files. However, this I could see as perfectly acceptable behavior, because Blender actually has to read the file for thumbnails to be generated. The main issue I reported however seems to only happen on macOS, where merely enumerating the files in a OneDrive folder triggers the downloading of those files. And this happens because each file is queried specifically for the `NSURLIsReadableKey` and `NSURLIsReadableKey` attributes. I've tested this in a separate application and it can be reproduced. I think this is a bug (or at least an unfortunate side-effect) that has to be fixed. Now this might also be a bug in the OneDrive application. Because querying for those other attributes does not trigger a download. I've contacted the OneDrive team about this, but haven't yet heard back from them. A (temporary) workaround might be to only query for those attributes after a user has explicitly selected the thumbnail view. Now, besides that main issue there's the broader issue of supporting cloud systems like iCloud and OneDrive. And I think those two can be singled out for now, because those two are actually considered system services and by now offer official API support for their respective on-demand files implementation. > It might be the check for aliasing. Nope, at least not on Mac. > That could be difficult without any kind of standard though. Indeed. But as I've said both Windows and Mac seem to consider their cloud drive solutions to be part of the system and have added explicit API support for such files. So the way I see it Blender running on Mac should natively support iCloud. And Blender running on Windows should natively support OneDrive. It gets more difficult when Blender has to support non-native cloud drive solutions. In an ideal world all those cloud drive solutions would use the system-provided APIs. That doesn't seem to be the case yet. But that might change in the near future. So OneDrive on macOS has its own Finder integration and on-demand file scheme. But it might use the iCloud way going forward. > But I was loath to add it as it would promote a Microsoft commercial service over other similar services. And then we'd invariably get complaints of "why isn't DropBox shown?" As I've said, I consider them system services. So supporting OneDrive on Windows and iCloud on Mac seems perfectly legal to me. > So there is almost certainly a one-line fix to this. I hope by now you know this isn't about thumbnails and unfortunately there is no one-line fix. Unless the querying for `NSURLIsReadableKey` and `NSURLIsReadableKey` gets removed. That worked for me, but I haven't tested the impact of such a change further down the line.

@Zandman - I think this thread has gotten a bit off track...main issue I reported however seems to only happen on macOS...

Actually you reported multiple issues on both MacOS and Windows. I'm making notes and will probably make changes for the Windows platform. Someone else will have to address the MacOS-related issues.

Created #87867 (macOS: attribute check for "file on demand" OneDrive files triggers download )

In #87621#1152953, @Harley wrote:
Actually you reported multiple issues on both MacOS and Windows. I'm making notes and will probably make changes for the Windows platform. Someone else will have to address the MacOS-related issues.

Yes, I understand these discussions have surfaced multiple issues on both Windows and Mac. How are you going to track these things? Are you going to create multiple tasks from this task?

Maybe I can fix the Mac issues? I've been looking for a way to contribute to Blender for a while now. For macOS I see the following tasks:

Prevent downloads when merely browsing a OneDrive folder.
Prevent thumbnail generation for files inside OneDrive folder.
Add iCloud Drive on-demand files support to Blender using macOS APIs.
Add iCloud Drive folder to the "System" list.

Do you agree with these?

@Zandman - Are you going to create multiple tasks from this task?

Probably. About to submit a patch that references this one.

Maybe I can fix the Mac issues? I've been looking for a way to contribute to Blender for a while now. For macOS I see the following tasks:

Prevent downloads when merely browsing a OneDrive folder.

Prevent thumbnail generation for files inside OneDrive folder.

Add iCloud Drive on-demand files support to Blender using macOS APIs.

Add iCloud Drive folder to the "System" list.

Do you agree with these?

Those all sounds like great things to address. There is a task you can reference here: https://developer.blender.org/T87867

This issue was referenced by ee5ad46a0e

Changed status from 'Confirmed' to: 'Resolved'

Download

What's New

Blender Studio

Manual

Developers Blog

Documentation

Benchmark

Blender Conference

Development Fund

One-time Donations

The Blender File Open dialog triggers OneDrive Files On-Demand downloads #87621