WIP: make_update: support Github style of remote organization #104479

Sergey Sharybin · 2023-02-08T18:57:27+01:00

Sergey Sharybin commented

2023-02-08 18:57:27 +01:00

This is rather common workflow in all sort of forges like Github, Gitlab,
perhaps even Gitea. Having better support of our tools for it seems to
have string benefits.

The idea of this change is to make it so make update will perform the
steps needed to update the branch of the fork to the upstream version.

This includes:

Fetch the upstream remote
git merge --ff-only upstream/branch if the current branch exists in
the upstream.
If the branch does not exist in the upstream, then it will be pulled
from the origin instead (the behavior prior to this change).

The same logic is implemented for the submodules.

The make update will also check for the forked submodules within the
same user/organization as the blender.git fork and add the "origin" to
the submodule when such a fork is detected. This allows to fork the
submodule repository after the make update has been run for the first
time.

Some of the new code still follows the old os.path convention. This is
to avoid a bigger refactor or endanger some things not working. Such a
refactor can happen later as a dedicated focused activity of bringing the
Python usage to the more modern level.

This is rather common workflow in all sort of forges like Github, Gitlab, perhaps even Gitea. Having better support of our tools for it seems to have string benefits. The idea of this change is to make it so `make update` will perform the steps needed to update the branch of the fork to the upstream version. This includes: - Fetch the `upstream` remote - `git merge --ff-only upstream/branch` if the current branch exists in the upstream. - If the branch does not exist in the upstream, then it will be pulled from the `origin` instead (the behavior prior to this change). The same logic is implemented for the submodules. The `make update` will also check for the forked submodules within the same user/organization as the blender.git fork and add the "origin" to the submodule when such a fork is detected. This allows to fork the submodule repository after the `make update` has been run for the first time. Some of the new code still follows the old `os.path` convention. This is to avoid a bigger refactor or endanger some things not working. Such a refactor can happen later as a dedicated focused activity of bringing the Python usage to the more modern level.

Sergey Sharybin requested review from Brecht Van Lommel 2023-02-08 18:57:36 +01:00

Sergey Sharybin requested review from Bastien Montagne 2023-02-08 18:57:48 +01:00

Brecht Van Lommel requested changes 2023-02-09 17:21:32 +01:00

Brecht Van Lommel left a comment

When I have the upstream workflow configured for the Blender repo, and then do this to have it reinit the submodules:

git submodule deinit --all
make update

I ended up with main branches in the submodule not tracking any upstream branch. I think they should be tracking origin/main.

When I have the upstream workflow configured for the Blender repo, and then do this to have it reinit the submodules: ``` git submodule deinit --all make update ``` I ended up with `main` branches in the submodule not tracking any upstream branch. I think they should be tracking `origin/main`.

build_files/utils/make_update.py Outdated

						
				@ -187,0 +219,4 @@

				    if not make_utils.git_branch_exists(args.git_command, upstream_branch):

				        return False

				    call((args.git_command, "merge", "--ff-only", upstream_branch))

Brecht Van Lommel commented

2023-02-09 17:00:39 +01:00

This aborts everything if the main branch in the Blender repo has changes. I think we should instead skip just the Blender repo update and continue with the submodules and libraries, same as we already do when there are unstaged changes.

Sergey marked this conversation as resolved

build_files/utils/make_update.py Outdated

						
				@ -187,0 +239,4 @@

				    update_command = [args.git_command, "pull", "--rebase"]

				    if remote_branch:

				        update_command.append(remote_branch)

Brecht Van Lommel commented

2023-02-09 16:46:00 +01:00

Getting this error:

git pull --rebase main
fatal: 'main' does not appear to be a git repository
fatal: Could not read from remote repository.

Can we just leave out the remote branch name here? When not using the upstream workflow it seems reasonable to pull from whatever this branch is tracking.

Getting this error: ``` git pull --rebase main fatal: 'main' does not appear to be a git repository fatal: Could not read from remote repository. ``` Can we just leave out the remote branch name here? When not using the upstream workflow it seems reasonable to pull from whatever this branch is tracking.

Sergey marked this conversation as resolved

build_files/utils/make_update.py Outdated

						
				@ -191,0 +277,4 @@

				    #

				    # - Rename "origin" to "upstream" in the submodule, matching the naming convention in the main

				    #   repository.

				    #   This leaves the submodule with detached head and main branch tracking upstream remote.

Brecht Van Lommel commented

2023-02-09 02:18:51 +01:00

We don't want the main branch tracking the upstream remote as I understand it, both for the main Blender repo and submodules.

So this behavior seems wrong.

We don't want the main branch tracking the upstream remote as I understand it, both for the main Blender repo and submodules. So this behavior seems wrong.

Sergey marked this conversation as resolved

build_files/utils/make_update.py Outdated

						
				@ -191,0 +305,4 @@

				        # The check should not be needed in the final production, but during development it

				        # is possible that the old configuration is still present, despite the deinit, so make

				        # a sanity check that the rename is still needed.

				        if make_utils.git_remote_exist(args.git_command, "origin"):

Brecht Van Lommel commented

2023-02-09 17:17:19 +01:00

Can we add and not make_utils.git_remote_exist(args.git_command, upstream_origin)?

I did rm -rf release/script/addons and then make update. But that didn't remove the remotes for the add-on, so got this error:

git remote rename origin upstream
fatal: remote upstream already exists.

Probably not the right thing to do but still can avoid giving an error.

Can we add `and not `make_utils.git_remote_exist(args.git_command, upstream_origin)? I did `rm -rf release/script/addons` and then `make update`. But that didn't remove the remotes for the add-on, so got this error: ``` git remote rename origin upstream fatal: remote upstream already exists. ``` Probably not the right thing to do but still can avoid giving an error.

Sergey marked this conversation as resolved

build_files/utils/make_update.py Outdated

						
				@ -191,0 +334,4 @@

				        if make_utils.git_remote_exist(args.git_command, "origin"):

				            return

				        if not make_utils.git_is_remote_repository(args.git_command, origin_submodule_url):

Brecht Van Lommel commented

2023-02-09 17:11:42 +01:00

When having only an upstream remote in a submodule and this code runs to add an origin repo, it prints this to the console:

git ls-remote git@projects.blender.org:brecht/blender-addons.git HEAD
Gitea: Unauthorized
fatal: Could not read from remote repository.

It works correctly, but can we suppress writing that out because it looks like something goes wrong.

When having only an `upstream` remote in a submodule and this code runs to add an origin repo, it prints this to the console: ``` git ls-remote git@projects.blender.org:brecht/blender-addons.git HEAD Gitea: Unauthorized fatal: Could not read from remote repository. ``` It works correctly, but can we suppress writing that out because it looks like something goes wrong.

Sergey marked this conversation as resolved

Sergey Sharybin force-pushed make_update from 9b5ae2152c to bec0d5f4d2

2023-02-09 18:04:37 +01:00

Compare

Brecht Van Lommel requested review from Brecht Van Lommel 2023-02-09 18:21:33 +01:00

Brecht Van Lommel approved these changes 2023-02-09 18:52:56 +01:00

Sergey Sharybin changed title from ~~make_update: support Github style of remote organization~~ to WIP: make_update: support Github style of remote organization

2023-02-10 15:46:18 +01:00

Sergey Sharybin commented

2023-02-10 15:47:18 +01:00

Putting on hold until #104573 is signed off, as the submodule part would need to be reconsidered.

Sergey Sharybin commented

2023-02-14 18:21:16 +01:00

Actually, a much bigger change is coming.

Sergey Sharybin closed this pull request

2023-02-14 18:21:16 +01:00

Sergey Sharybin deleted branch make_update

2023-02-14 18:21:25 +01:00

Pull request closed

This pull request cannot be reopened because the branch was deleted.

Sign in to join this conversation.

No reviewers

No Label

Download

What's New

Blender Studio

Manual

Developers Blog

Documentation

Benchmark

Blender Conference

Development Fund

One-time Donations

WIP: make_update: support Github style of remote organization #104479

Pull request closed