Move Windows dep build from MinGW/GCC to msys/MSVC #105502
No reviewers
Campbell Barton
Brecht Van Lommel
Ray molenkamp
Labels
No Label
Interest
Alembic
Interest
Animation & Rigging
Interest
Asset Browser
Interest
Asset Browser Project Overview
Interest
Audio
Interest
Automated Testing
Interest
Blender Asset Bundle
Interest
BlendFile
Interest
Collada
Interest
Compatibility
Interest
Compositing
Interest
Core
Interest
Cycles
Interest
Dependency Graph
Interest
Development Management
Interest
EEVEE
Interest
EEVEE & Viewport
Interest
Freestyle
Interest
Geometry Nodes
Interest
Grease Pencil
Interest
ID Management
Interest
Images & Movies
Interest
Import Export
Interest
Line Art
Interest
Masking
Interest
Metal
Interest
Modeling
Interest
Modifiers
Interest
Motion Tracking
Interest
Nodes & Physics
Interest
OpenGL
Interest
Overlay
Interest
Overrides
Interest
Performance
Interest
Physics
Interest
Pipeline, Assets & IO
Interest
Platforms, Builds & Tests
Interest
Python API
Interest
Render & Cycles
Interest
Render Pipeline
Interest
Sculpt, Paint & Texture
Interest
Text Editor
Interest
Translations
Interest
Triaging
Interest
Undo
Interest
USD
Interest
User Interface
Interest
UV Editing
Interest
VFX & Video
Interest
Video Sequencer
Interest
Virtual Reality
Interest
Vulkan
Interest
Wayland
Interest
Workbench
Interest: X11
Legacy
Blender 2.8 Project
Legacy
Milestone 1: Basic, Local Asset Browser
Legacy
OpenGL Error
Meta
Good First Issue
Meta
Papercut
Meta
Retrospective
Meta
Security
Module
Animation & Rigging
Module
Core
Module
Development Management
Module
EEVEE & Viewport
Module
Grease Pencil
Module
Modeling
Module
Nodes & Physics
Module
Pipeline, Assets & IO
Module
Platforms, Builds & Tests
Module
Python API
Module
Render & Cycles
Module
Sculpt, Paint & Texture
Module
Triaging
Module
User Interface
Module
VFX & Video
Platform
FreeBSD
Platform
Linux
Platform
macOS
Platform
Windows
Priority
High
Priority
Low
Priority
Normal
Priority
Unbreak Now!
Status
Archived
Status
Confirmed
Status
Duplicate
Status
Needs Info from Developers
Status
Needs Information from User
Status
Needs Triage
Status
Resolved
Type
Bug
Type
Design
Type
Known Issue
Type
Patch
Type
Report
Type
To Do
No Milestone
No project
No Assignees
4 Participants
Notifications
Due Date
No due date set.
Dependencies
No dependencies set.
Reference: blender/blender#105502
Loading…
Reference in New Issue
No description provided.
Delete Branch "Anthony-Roberts/blender:move-to-msys2"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Currently, Windows dependencies are built with MinGW/GCC - this commit removes that, in favour of building them with MSVC, via msys2.
This unifies the Windows build to use MSVC entirely for dependencies.
Done as part of the foundational work for Windows ARM64 platforms.
A note: You may notive the xvid build was removed on Windows - it doesn't even appear to be selectable as a codec in normal release builds (pic attached), and was causing a headache for modern MSVC, so I removed it.
initital pass/lowhanging fruit only, haven't been able to build just yet.
@ -8,3 +4,1 @@# building with mingw, it'll have an unhappy time with that and# we need to clear them out.set(AOM_CMAKE_FLAGS )set(AOM_GENERATOR "Visual Studio 17 2022")you can't hardcode 2022 here, we still support 2019 and you can't link libraries build with 2022 with older compilers, compat only works in one direction there. build in 2019 -> linked with 2022 is fine.. the otherway around is not.
the default generator is already Visual studio (win) and Makefiles (mac+lin), we only had the generator stuff here because we forcibly wanted to use ninja on windows, if you want to stick with visual studio, the custom generator bits can/need to be be removed.
This is another leftover from the ARM64 stuff (which adds support for VS2022 also) - missed it - will remove in next commit.
@ -3,2 +4,3 @@if(WIN32)set(temp_LIBDIR ${mingw_LIBDIR})set(LIBDIR_PREFIX "L")Took me a bit to realize what this did, 'L'? what's 'L'? only when you scroll down you realize it's a flag, I'd prefer
LIBDIR_FLAGand just add the-here as well so it's even clearer that it is indeed a flag.ACK, will fix next commit.
@ -6,2 +6,2 @@# Shared for windows because static libs will drag in a libgcc dependency.set(GMP_OPTIONS --disable-static --enable-shared --enable-fat --host=x86_64-w64-mingw32 --build=x86_64-w64-mingw32)# Borrow a reasonably up-to-date ar-lib from webp to enable MSVCcmake_to_msys_path("${BUILD_DIR}/webp/src/external_webp/ar-lib" arlib_path)This gives me the Heebie-jeebies, i'd rather see this as a patch for gmp or put the script straight up in
build_files\build_environment\patches\the dep between gmp and webp just doesn't sit right with me.
The file in itself is ~270 lines - would you rather it be on it's own ir in a patch file? Can do either.
File: https://github.com/gcc-mirror/gcc/blob/master/ar-lib
If its a fixed URI, we could just grab it in
setup_msys2.cmakeand place it in a convenient location (ie somewhere the msys2 tree) for all deps to use if they need it.Sure, will grap a specific version like I do with gas-preprocessor
@ -2,17 +2,15 @@set(LAME_EXTRA_ARGS)if(MSVC)not super obvious in the online diff viewer, but this produces
which leads to:
ACK, that's a leftover from where I removed ARM64 code from this and missed it, will fix next commit
@ -0,0 +58,4 @@endif()# Refresh pacman and upgrade all packages# Note: We do this regardless of installation status, to get latest packages - pacman works on *rolling releases*That's problematic, i need to be able to rebuild LTS releases for 2 years without getting all kinds of unrelated updates, the old system grabbed the packages once, and for the important ones like compiler versions they were pinned versions so there's no surprise GCC 10->11 update with all kinds of new adventures in it.
So the packages in this case are nothing critical in that way - they are on line 70,
patch m4 coreutils nasm pkgconf make diffutils autoconf-wrapper- the one thing that could be an issue (perl), is on a fixed version.The only one I could see being an issue is NASM, which it may be possible to download a specific version of from elsewhere.
If it's not critical, there is no need to update them no?
You can probably nick the nasm install from the old mingw code.
I mean sure, but rebuilding deps from scratch would get the newest version anyway, as msys2/pacman has no concept of version pinning - I can remove it if you would like though.
ACK on 2nd line, I'll look into using the nasm code from the old release.
one more, after fixing lame manually
Ah, I forgot to stage a file I updated locally - will fix next commit.
missing
endif()I manually added it and looks like things will be building for a while, will let you know if anything else pops up.
@ -932,2 +927,2 @@DEBUG)if(WITH_GMP)missed one :)
@ -992,3 +983,1 @@${LIBDIR}/materialx/bin/MaterialXGenShader_d.dllDEBUG)if(WITH_MATERIALX)and one more
/machine:i386set causing lib/linker errorsVPX_INCLUDE_PATHis set tovpx-vp8-vp9-nopost-x86_64md-vs16-v1.11.0directory is actually calledvpx-vp8-vp9-nodocs-x86_64-win64md-vs16-v1.11.0CUSTOMBUILD : error : aom >= 1.0.0 not found using pkg-config [c:\db\build\S\VS1564R\external_ffmpeg.vcxproj]My bad on ffmpeg and LAME, I forgot to omit the diffs, I was overzealous in removing ARM64 stuff from this locally.
FFMpeg may well need 2 patches (as this one may cause linux issues?), with a seperate one for windows.
@ -3,2 +4,3 @@if(WIN32)set(temp_LIBDIR ${mingw_LIBDIR})set(LIBDIR_FLAG "-L")-Lblahblahdoesn't work, and emits an "unknown flag" warnset(LIBDIR_FLAG "/LIBPATH:")does the trick@ -35,0 +44,4 @@set(FFMPEG_LDFLAGS "\${FFMPEG_LDFLAGS} \-L${temp_LIBDIR}/openjpeg_msvc/lib \use
${LIBDIR_FLAG}rather than-LFinally had some time for another round of (windows) testing on this couple of things stuck out
ZLib had been moved from static to dynamic, while I'm not opposed to that in principle, it does sit at the very bottom of the dependency tree causing a rebuild for virtually all deps which is going to cause more work than I have availability for right now. With zlib in its original configuration (static) the only deps that really need to be rebuild for this to land are sndfile and ffmpeg which is much easier to deal with.
A bunch of the dll/lib filenames changed, that's fair and to be expected, that does however imply that landing this diff and a new set of libraries needs to be done within minutes/seconds from each other, which is a bit tricky. For major changes like this I generally offer an easy upgrade path where I check if the new files exist, if so use those, if not assume the old file names. I generally leave this in place for a version ie 3.5 can build against both an 3.5 and 3.4 lib folder. I'd prefer if this was done here as well, "i have a build error"....."get new libs" gets very old very quickly if you have to tell people multiple times a day.
the
ffmpeg_codecstest has 2 failing testsThe xvid one is to be expected and you can leave for now until the VFX team figures out what to do with xvid, the openjpeg one should be fixed.
Just to give an update why i'm stalling a bit, the change is a little too intrusive to fold in an LTS release we have to maintain for another 2 years, I'm still very interested in landing this but I'd like to target 4.0 as the landing spot.
Do you have a rough date for that? I can try and land the Windows ARM64 patch for then also (which depends on this one)
I do! The 4.0 merge window should open up may 17 could shift a few days depending on what is happening in 3.6 at the time.
Thanks! As far as I am concerned, this is ready to go, unless you have further comments - is there any news on xvid from the relevant team?
WIP: Move Windows dep build from MinGW/GCC to msys/MSVCto Move Windows dep build from MinGW/GCC to msys/MSVC@LazyDodo Ping for another pass at this please!
It's been building for a few hours already, given it was a full clean build, it'll "be a bit"
Just some minor issues at this point
still had some build issues with USD due to things beyond your control, i'll fix those my self, no need to stall this diff on.
@ -33,3 +40,2 @@ExternalProject_Add_Step(external_fftw3_double after_installCOMMAND ${CMAKE_COMMAND} -E copy ${LIBDIR}/fftw3/lib/libfftw3.dll.a ${HARVEST_TARGET}/fftw3/lib/libfftw3-3.libCOMMAND ${CMAKE_COMMAND} -E copy ${LIBDIR}/fftw3/bin/libfftw3-3.dll ${HARVEST_TARGET}/fftw3/lib/libfftw3-3.dllCOMMAND ${CMAKE_COMMAND} -E copy ${LIBDIR}/fftw3/lib/fftw3.lib ${HARVEST_TARGET}/fftw3/lib/libfftw.libJust keep the file names as they are being output by the build, in the past we tried to bend the build system to get it to generate the "old" names, but after a couple of years of that you tend to end up in a situation where you go "but why does it have such a weird name?" and no-one knows. I've been trying to clean up these instances as I update deps.
Done!
ACK, it appears my git client decided a random feature branch was main rather than thr actual main - I'll fix that
ee9106469cto1cbc7cdd64libxvid could be removed under the condition that ffmpeg's internal codepath would be used for this, removing xvid completely from the UI is not what was agreed upon. It would need to stay working
@brecht / @ideasman42 this seems to be getting close to landing, due to the large nature of this diff, can you guys give a whirl on your respective platforms to make sure nothing broke there?
@ -166,18 +207,21 @@ add_dependencies(external_vorbisexternal_oggexternal_lameexternal_aomDon't remove aom here? It seems needed still.
@ -97,3 +141,2 @@CONFIGURE_COMMAND ${CONFIGURE_ENV_NO_PERL} &&CONFIGURE_COMMAND ${FFMPEG_CONFIGURE_COMMAND} &&cd ${BUILD_DIR}/ffmpeg/src/external_ffmpeg/ &&${FFMPEG_ENV} ${CONFIGURE_COMMAND_NO_TARGET} ${FFMPEG_EXTRA_FLAGS}Don't remove
${FFMPEG_ENV}, still needed for Linux and macOS.@ -5,2 +5,3 @@if(NOT WIN32)set(LIBDIR_FLAG "-L")else()set(temp_LIBDIR ${LIBDIR})This should not have been removed, it still needs to be set to
LIBDIRon Linux and macOS, wheremsys2_LIBDIRis not available.@ -6,3 +17,2 @@enabled libopenjpeg && { check_pkg_config libopenjpeg "libopenjp2 >= 2.1.0" openjpeg.h opj_version ||-enabled libopenjpeg && { check_pkg_config libopenjpeg "libopenjp2 >= 2.1.0" openjpeg.h opj_version ||- { require_pkg_config libopenjpeg "libopenjp2 >= 2.1.0" openjpeg.h opj_version -DOPJ_STATIC && add_cppflags -DOPJ_STATIC; } }+ { require_pkg_config libopenjpeg "libopenjp2 >= 2.1.0" openjpeg.h opj_version "-DOPJ_STATIC $pthreads_extralibs $libm_extralibs" && add_cppflags "-DOPJ_STATIC $pthreads_extralibs $libm_extralibs"; } }These openjpeg related changes are still needed, should not be removed for Linux and macOS. You could make a separate diff for Windows and leave the existing one unchanged.
I ran into a cmake error that was already in main, and committed a fix in
e562a8891f.Mostly ran into the same issues as Brecht, noted one extra inline (related to finding libaom).
@ -4,0 +5,4 @@enabled jni && { [ $target_os = "android" ] && check_headers jni.h && enabled pthreads || die "ERROR: jni not found"; }enabled ladspa && require_headers "ladspa.h dlfcn.h"enabled lcms2 && require_pkg_config lcms2 "lcms2 >= 2.13" lcms2.h cmsCreateContext-enabled libaom && require_pkg_config libaom "aom >= 1.0.0" aom/aom_codec.h aom_codec_versionThis change prevents ffmpeg finding
libaom.aon Linux which currently depends on pkgconfig.Thanks for the comments guys, these should all be addressed now! I also merged master to include the mentioned cmake fix in USD
Linux packages now build without problems.
finished final testing, this can land, will land the changed libs on monday