Some file formats save very slowly over fast VPN link #124908

New Issue

Will Francis · 2024-07-17T18:45:59+02:00

Will Francis commented

2024-07-17 18:45:59 +02:00

System Information
Operating system: Windows-10-10.0.22631-SP0 64 Bits
Graphics card: NVIDIA GeForce RTX 4090/PCIe/SSE2 NVIDIA Corporation 4.6.0 NVIDIA 555.85

Blender Version
Broken: version: 4.1.1, branch: K-Cycles (modified), commit date: 2024-05-04 17:34, hash: 912346ecd58f
Broken: blender-4.3.0-alpha+main.b8eb87258348-windows.amd64-release

Worked: none known

Short description of error
Some file formats save very slowly over fast VPN link. Some save very quickly. I have a list below with my testing results.

Exact steps for others to reproduce the error
This was uncovered during use of the Flamenco render farm software. Using remote computers allows me to have a bigger farm.
Create a wireguard VPN link between two networks on fast (1Gb) connections. The latency is tested to >5ms.
From the remote machine, mount a drive on a NAS across the VPN.
Normal copying of files is quite fast, never slower than 40MB/s
Open either current release (4.1) or the daily build (4.3.0-alpha+main.b8eb87258348)
Render a frame
Save the frame to a drive mounted from the remote NAS. The saving speed can vary dramatically depending on the format. For this testing, I only did one test of each format with a handheld stopwatch. Time is in seconds to save a 11MB(png) file.

png: 45s
bmp: 25s
iris: 21s
jpg: 2s
jpg2000: 2s
targa: 34s
targa raw: 45s
cineon: 31s
dpx: 1s
OpenEXRMultilayer: 3s
OpenEXR: 3s
Radiance: 27s
tiff: 2s
webp: 1s

In my testing with Flamenco, I've seen png files take nearly 1:45s to save, so there may be great variance with further testing. However, it's clear that some file formats are much, much slower to save even over a fairly quick VPN link. Please let me know if you'd like any other information.

**System Information** Operating system: Windows-10-10.0.22631-SP0 64 Bits Graphics card: NVIDIA GeForce RTX 4090/PCIe/SSE2 NVIDIA Corporation 4.6.0 NVIDIA 555.85 **Blender Version** Broken: version: 4.1.1, branch: K-Cycles (modified), commit date: 2024-05-04 17:34, hash: `912346ecd58f` Broken: blender-4.3.0-alpha+main.b8eb87258348-windows.amd64-release Worked: none known **Short description of error** Some file formats save very slowly over fast VPN link. Some save very quickly. I have a list below with my testing results. **Exact steps for others to reproduce the error** This was uncovered during use of the Flamenco render farm software. Using remote computers allows me to have a bigger farm. Create a wireguard VPN link between two networks on fast (1Gb) connections. The latency is tested to >5ms. From the remote machine, mount a drive on a NAS across the VPN. Normal copying of files is quite fast, never slower than 40MB/s Open either current release (4.1) or the daily build (4.3.0-alpha+main.b8eb87258348) Render a frame Save the frame to a drive mounted from the remote NAS. The saving speed can vary dramatically depending on the format. For this testing, I only did one test of each format with a handheld stopwatch. Time is in seconds to save a 11MB(png) file. png: 45s bmp: 25s iris: 21s jpg: 2s jpg2000: 2s targa: 34s targa raw: 45s cineon: 31s dpx: 1s OpenEXRMultilayer: 3s OpenEXR: 3s Radiance: 27s tiff: 2s webp: 1s In my testing with Flamenco, I've seen png files take nearly 1:45s to save, so there may be great variance with further testing. However, it's clear that some file formats are much, much slower to save even over a fairly quick VPN link. Please let me know if you'd like any other information.

Will Francis added the

labels 2024-07-17 18:46:00 +02:00

Richard Antalik commented

2024-07-18 15:40:46 +02:00

I don't quite see, how this is blender bug. IO for network drive is handled by OS and whatever software to do VPN. If I had this problem, I would build Blender from sources, run it from IDE and hit pause button few times to see where it is hanging on. But that would probably end up on some IO call and I would be none the wiser, but it may be worth to give it a shot...

If you have few hours (or minutes depending on internet bandwidth) to set this up, you can follow https://developer.blender.org/docs/handbook/building_blender/windows/

I don't quite see, how this is blender bug. IO for network drive is handled by OS and whatever software to do VPN. If I had this problem, I would build Blender from sources, run it from IDE and hit pause button few times to see where it is hanging on. But that would probably end up on some IO call and I would be none the wiser, but it may be worth to give it a shot... If you have few hours (or minutes depending on internet bandwidth) to set this up, you can follow https://developer.blender.org/docs/handbook/building_blender/windows/

Ray molenkamp commented

2024-07-18 15:53:49 +02:00

Someone will have to take a closer look, gut feeling says It's likely a latency/iops related issue, I suspect the fast formats likely do fewer large writes while the slower formats do frequent small ones. OIIO has the concept of IO Proxies, we could probably insert one that logs how many bytes are written to validate these suspicions.

Ray molenkamp commented

2024-07-18 17:05:32 +02:00

Didn't test all formats but, saving the default cube at 1080p

format	number_of_writes	total_size	average_write
PNG	491	925369	1884
BMP	1096	8294454	7567
TIFF	52	542926	10440
TGA	1286199	3430255	2

There's some buffering by default on file io, (from the top of my head, windows 512 Bytes , mac 1024, linux 8192) so we could try calling setvbuf in oiio to manage this, or supply our own caching IO Proxy. But i'll leave it to @deadpin to decide how / if we want to do anything about this.

Didn't test all formats but, saving the default cube at 1080p |format|number_of_writes|total_size|average_write| |-|-|-|-| |PNG|491|925369|1884| |BMP|1096|8294454|7567| |TIFF|52|542926|10440| |TGA|1286199|3430255|2| There's some buffering by default on file io, (from the top of my head, windows 512 Bytes , mac 1024, linux 8192) so we could try calling `setvbuf` in oiio to manage this, or supply our own caching IO Proxy. But i'll leave it to @deadpin to decide how / if we want to do anything about this.

Jesse Yurkovich commented

2024-07-18 23:48:46 +02:00

I'll put it on the books to take a look into; just unsure when. For any egregious inefficiency I find I would want to upstream that to OIIO directly. As for Blender, the "easiest" thing would be to always write the file locally and then Move to the destination. A quick test here for an 8k png goes from 24 seconds to 3 for a remote folder.

Jesse Yurkovich added

and removed

labels 2024-07-18 23:49:26 +02:00

Iliya Katushenock added