Make update: Bring behavior closer to what it used to be #118812

Sergey Sharybin · 2024-02-27T17:41:25+01:00

Sergey Sharybin commented

2024-02-27 17:41:25 +01:00

The make update will now always update all initialized submodules,
including the library ones.

The make_update.py --no-libraries will neither initialize nor update
pre-compiled libraries or tests.

The `make update` will now always update all initialized submodules, including the library ones. The `make_update.py --no-libraries` will neither initialize nor update pre-compiled libraries or tests.

Sergey Sharybin added 1 commit 2024-02-27 17:41:33 +01:00

23256cb5da Make update: Bring behavior closer to what it used to be

The `make update` will now always update all initialized submodules,
including the library ones.

The `make_update.py --no-libraries` will neither initialize nor update
pre-compiled libraries or tests.

Sergey Sharybin requested review from Brecht Van Lommel 2024-02-27 17:41:42 +01:00

Sergey Sharybin requested review from Ray molenkamp 2024-02-27 17:41:47 +01:00

Brecht Van Lommel approved these changes 2024-02-27 17:57:38 +01:00

Dismissed

Sergey Sharybin added 1 commit 2024-02-27 17:59:19 +01:00

74833cd18d Consider assets a library type of submodule

Sergey Sharybin requested review from Brecht Van Lommel 2024-02-27 17:59:46 +01:00

Ray molenkamp requested changes 2024-02-27 18:09:08 +01:00

Dismissed

Ray molenkamp left a comment

doing the update_code/code_update tries to update the assets folder, previously this updated only things in git, and left all svn data alone, so assets would be skipped.

k:\BlenderGit\blender>make code_update 
No explicit msvc version requested, autodetecting version.
**********************************************************************
** Visual Studio 2019 Developer Command Prompt v16.11.23
** Copyright (c) 2021 Microsoft Corporation
**********************************************************************
[vcvarsall.bat] Environment initialized for: 'x64'
Compiler Detection successful, detected VS2019
C:\Program Files\Git\cmd\git.exe lfs install --skip-repo
Git LFS initialized.

Updating Submodules
===================

* Updating scripts\addons ...
C:\Program Files\Git\cmd\git.exe fetch origin
C:\Program Files\Git\cmd\git.exe checkout blender-v4.1-release
Switched to branch 'blender-v4.1-release'
Your branch is up to date with 'origin/blender-v4.1-release'.
C:\Program Files\Git\cmd\git.exe pull --rebase
Already up to date.
* Updating scripts\addons_contrib ...
C:\Program Files\Git\cmd\git.exe fetch origin
C:\Program Files\Git\cmd\git.exe checkout blender-v4.1-release
Switched to branch 'blender-v4.1-release'
Your branch is up to date with 'origin/blender-v4.1-release'.
C:\Program Files\Git\cmd\git.exe pull --rebase
Already up to date.
* Updating Git submodules
GIT_LFS_SKIP_SMUDGE=1 C:\Program Files\Git\cmd\git.exe submodule update --init --progress release\datafiles\assets
Submodule path 'release/datafiles/assets': checked out '6e43ac4de4474de1c3b00baceaaa85b79d884222'
C:\Program Files\Git\cmd\git.exe -C release\datafiles\assets lfs pull

Update finished with the following messages
===========================================

Blender repository skipped: no remote branch to pull from

the warning is expected since i'm on a local branch pr118812 which has no remote

doing a normal make update

k:\BlenderGit\blender>make update
No explicit msvc version requested, autodetecting version.
**********************************************************************
** Visual Studio 2019 Developer Command Prompt v16.11.23
** Copyright (c) 2021 Microsoft Corporation
**********************************************************************
[vcvarsall.bat] Environment initialized for: 'x64'
Compiler Detection successful, detected VS2019
Submodule path 'lib/windows_x64': checked out '71e6832c48d25b4929bcdfc1637ababab843e551'
C:\Program Files\Git\cmd\git.exe lfs install --skip-repo
Git LFS initialized.

Configuring Precompiled Libraries
=================================

Detected platform     : windows
Detected architecture : x64

C:\Program Files\Git\cmd\git.exe config --local submodule.lib/windows_x64.update checkout

Updating Libraries
==================

GIT_LFS_SKIP_SMUDGE=1 C:\Program Files\Git\cmd\git.exe submodule update --init --progress lib\linux_x64
Skipping submodule 'lib/linux_x64'
C:\Program Files\Git\cmd\git.exe -C lib\linux_x64 lfs pull
GIT_LFS_SKIP_SMUDGE=1 C:\Program Files\Git\cmd\git.exe submodule update --init --progress lib\macos_arm64
Skipping submodule 'lib/macos_arm64'
C:\Program Files\Git\cmd\git.exe -C lib\macos_arm64 lfs pull
GIT_LFS_SKIP_SMUDGE=1 C:\Program Files\Git\cmd\git.exe submodule update --init --progress lib\macos_x64
Skipping submodule 'lib/macos_x64'
C:\Program Files\Git\cmd\git.exe -C lib\macos_x64 lfs pull
GIT_LFS_SKIP_SMUDGE=1 C:\Program Files\Git\cmd\git.exe submodule update --init --progress lib\windows_x64
C:\Program Files\Git\cmd\git.exe -C lib\windows_x64 lfs pull
GIT_LFS_SKIP_SMUDGE=1 C:\Program Files\Git\cmd\git.exe submodule update --init --progress tests\data
Submodule path 'tests/data': checked out '6dd81d3f24ec18cec2ab70d1c4159be3900ccd55'
C:\Program Files\Git\cmd\git.exe -C tests\data lfs pull
Downloading LFS objects: 100% (1/1), 19 KB | 0 B/s
Updating Submodules
===================

* Updating scripts\addons ...
C:\Program Files\Git\cmd\git.exe fetch origin
C:\Program Files\Git\cmd\git.exe pull --rebase
Already up to date.
* Updating scripts\addons_contrib ...
C:\Program Files\Git\cmd\git.exe fetch origin
C:\Program Files\Git\cmd\git.exe pull --rebase
Already up to date.
* Updating Git submodules
GIT_LFS_SKIP_SMUDGE=1 C:\Program Files\Git\cmd\git.exe submodule update --init --progress release\datafiles\assets
C:\Program Files\Git\cmd\git.exe -C release\datafiles\assets lfs pull

Update finished with the following messages
===========================================

Blender repository skipped: no remote branch to pull from

The output gets a bit confusing/noisy, unsure if these are left over debug prints or things are getting executed and failing cause my linux_x64 folder remains empty.

GIT_LFS_SKIP_SMUDGE=1 C:\Program Files\Git\cmd\git.exe submodule update --init --progress lib\linux_x64
Skipping submodule 'lib/linux_x64'
C:\Program Files\Git\cmd\git.exe -C lib\linux_x64 lfs pull

doing the `update_code/code_update` tries to update the assets folder, previously this updated only things in git, and left all svn data alone, so assets would be skipped. ``` k:\BlenderGit\blender>make code_update No explicit msvc version requested, autodetecting version. ********************************************************************** ** Visual Studio 2019 Developer Command Prompt v16.11.23 ** Copyright (c) 2021 Microsoft Corporation ********************************************************************** [vcvarsall.bat] Environment initialized for: 'x64' Compiler Detection successful, detected VS2019 C:\Program Files\Git\cmd\git.exe lfs install --skip-repo Git LFS initialized. Updating Submodules =================== * Updating scripts\addons ... C:\Program Files\Git\cmd\git.exe fetch origin C:\Program Files\Git\cmd\git.exe checkout blender-v4.1-release Switched to branch 'blender-v4.1-release' Your branch is up to date with 'origin/blender-v4.1-release'. C:\Program Files\Git\cmd\git.exe pull --rebase Already up to date. * Updating scripts\addons_contrib ... C:\Program Files\Git\cmd\git.exe fetch origin C:\Program Files\Git\cmd\git.exe checkout blender-v4.1-release Switched to branch 'blender-v4.1-release' Your branch is up to date with 'origin/blender-v4.1-release'. C:\Program Files\Git\cmd\git.exe pull --rebase Already up to date. * Updating Git submodules GIT_LFS_SKIP_SMUDGE=1 C:\Program Files\Git\cmd\git.exe submodule update --init --progress release\datafiles\assets Submodule path 'release/datafiles/assets': checked out '6e43ac4de4474de1c3b00baceaaa85b79d884222' C:\Program Files\Git\cmd\git.exe -C release\datafiles\assets lfs pull Update finished with the following messages =========================================== Blender repository skipped: no remote branch to pull from ``` the warning is expected since i'm on a local branch `pr118812` which has no remote doing a normal `make update` ```plaintext k:\BlenderGit\blender>make update No explicit msvc version requested, autodetecting version. ********************************************************************** ** Visual Studio 2019 Developer Command Prompt v16.11.23 ** Copyright (c) 2021 Microsoft Corporation ********************************************************************** [vcvarsall.bat] Environment initialized for: 'x64' Compiler Detection successful, detected VS2019 Submodule path 'lib/windows_x64': checked out '71e6832c48d25b4929bcdfc1637ababab843e551' C:\Program Files\Git\cmd\git.exe lfs install --skip-repo Git LFS initialized. Configuring Precompiled Libraries ================================= Detected platform : windows Detected architecture : x64 C:\Program Files\Git\cmd\git.exe config --local submodule.lib/windows_x64.update checkout Updating Libraries ================== GIT_LFS_SKIP_SMUDGE=1 C:\Program Files\Git\cmd\git.exe submodule update --init --progress lib\linux_x64 Skipping submodule 'lib/linux_x64' C:\Program Files\Git\cmd\git.exe -C lib\linux_x64 lfs pull GIT_LFS_SKIP_SMUDGE=1 C:\Program Files\Git\cmd\git.exe submodule update --init --progress lib\macos_arm64 Skipping submodule 'lib/macos_arm64' C:\Program Files\Git\cmd\git.exe -C lib\macos_arm64 lfs pull GIT_LFS_SKIP_SMUDGE=1 C:\Program Files\Git\cmd\git.exe submodule update --init --progress lib\macos_x64 Skipping submodule 'lib/macos_x64' C:\Program Files\Git\cmd\git.exe -C lib\macos_x64 lfs pull GIT_LFS_SKIP_SMUDGE=1 C:\Program Files\Git\cmd\git.exe submodule update --init --progress lib\windows_x64 C:\Program Files\Git\cmd\git.exe -C lib\windows_x64 lfs pull GIT_LFS_SKIP_SMUDGE=1 C:\Program Files\Git\cmd\git.exe submodule update --init --progress tests\data Submodule path 'tests/data': checked out '6dd81d3f24ec18cec2ab70d1c4159be3900ccd55' C:\Program Files\Git\cmd\git.exe -C tests\data lfs pull Downloading LFS objects: 100% (1/1), 19 KB | 0 B/s Updating Submodules =================== * Updating scripts\addons ... C:\Program Files\Git\cmd\git.exe fetch origin C:\Program Files\Git\cmd\git.exe pull --rebase Already up to date. * Updating scripts\addons_contrib ... C:\Program Files\Git\cmd\git.exe fetch origin C:\Program Files\Git\cmd\git.exe pull --rebase Already up to date. * Updating Git submodules GIT_LFS_SKIP_SMUDGE=1 C:\Program Files\Git\cmd\git.exe submodule update --init --progress release\datafiles\assets C:\Program Files\Git\cmd\git.exe -C release\datafiles\assets lfs pull Update finished with the following messages =========================================== Blender repository skipped: no remote branch to pull from ``` The output gets a bit confusing/noisy, unsure if these are left over debug prints or things are getting executed and failing cause my `linux_x64` folder remains empty. ``` GIT_LFS_SKIP_SMUDGE=1 C:\Program Files\Git\cmd\git.exe submodule update --init --progress lib\linux_x64 Skipping submodule 'lib/linux_x64' C:\Program Files\Git\cmd\git.exe -C lib\linux_x64 lfs pull ```

Ray molenkamp commented

2024-02-27 18:12:29 +01:00

looks like the assets folder got fixed while i was testing, so that feedback can be ignored, the noise i'd still like to see addressd

Sergey Sharybin commented

2024-02-27 18:46:17 +01:00

GIT_LFS_SKIP_SMUDGE=1 C:\Program Files\Git\cmd\git.exe submodule update --init --progress lib\linux_x64
Skipping submodule 'lib/linux_x64'
C:\Program Files\Git\cmd\git.exe -C lib\linux_x64 lfs pull

it is just for the simplicity of code the make_update.py was relying on Git to do checks whether submodule is enabled or not.

> GIT_LFS_SKIP_SMUDGE=1 C:\Program Files\Git\cmd\git.exe submodule update --init --progress lib\linux_x64 > Skipping submodule 'lib/linux_x64' > C:\Program Files\Git\cmd\git.exe -C lib\linux_x64 lfs pull it is just for the simplicity of code the `make_update.py` was relying on Git to do checks whether submodule is enabled or not.

Sergey Sharybin added 1 commit 2024-02-27 18:47:20 +01:00

1331151cc0 Explicitly check submoduels enabled state prior to update

Avoid possible confusing logs like

```
GIT_LFS_SKIP_SMUDGE=1 C:\Program Files\Git\cmd\git.exe submodule update --init --progress lib\linux_x64
Skipping submodule 'lib/linux_x64'
C:\Program Files\Git\cmd\git.exe -C lib\linux_x64 lfs pull
```

Ray molenkamp approved these changes 2024-02-27 18:47:43 +01:00

Ray molenkamp left a comment

fair enough

Sergey Sharybin commented

2024-02-27 18:53:59 +01:00

@LazyDodo I've tweaked the code, so now it should be less confusing. Not sure you've seen it.

Sergey Sharybin added 1 commit 2024-02-27 19:12:35 +01:00

a6053f8755 Possible fix for failure on windows

git_get_config() expects config file to be str, not Path.

Sergey Sharybin added 1 commit 2024-02-27 19:29:52 +01:00

5079ecff67 Fix incorrect path delimiter used for submodule paths

Sergey Sharybin added 1 commit 2024-02-27 19:35:21 +01:00

fbc85816b9 The POSIX path is to be sued for real

Sergey Sharybin added 1 commit 2024-02-27 19:41:32 +01:00

29423787cc Use list instead of typle/generator

Solves confusion when iteraitng over the list of submodule directories
does not work second time.

Brecht Van Lommel reviewed 2024-02-27 20:00:04 +01:00

build_files/utils/make_utils.py Outdated

						
				@ -134,0 +149,4 @@

				        return False

				    # When the "update" strategy is not provided explicitly in the the local configuration

				    # `git config` returns a non-zero exit code. For those assume the default "chekcout"