Buildbot: Fix bad state within worker repos due to ref error causes #115

New Issue

Bart van der Braak · 2024-07-31T16:19:25+02:00

2024-07-31 16:19:25 +02:00

After commit 811c1d the repository will return an error on each "first time" run when git fetch origin --prune is ran:

INFO: Setting up builder paths from [/home/blender/git/blender-v420]
INFO: Removing /home/blender/git/blender-v420/build_asserts/Testing
INFO: Removing /home/blender/git/blender-v420/build_asserts/bin/tests
INFO: Run command in path [/home/blender/git/blender-v420/blender.git]
git remote set-head origin --auto
origin/HEAD set to main
INFO: Run command in path [/home/blender/git/blender-v420/blender.git]
git remote set-url origin https://projects.blender.org/blender/blender.git
INFO: Run command in path [/home/blender/git/blender-v420/blender.git]
git submodule sync
Synchronizing submodule url for 'lib/linux_x64'
Synchronizing submodule url for 'lib/macos_arm64'
Synchronizing submodule url for 'lib/macos_x64'
Synchronizing submodule url for 'lib/windows_arm64'
Synchronizing submodule url for 'lib/windows_x64'
Synchronizing submodule url for 'release/datafiles/assets'
Synchronizing submodule url for 'tests/data'
INFO: Run command in path [/home/blender/git/blender-v420/blender.git]
git fetch origin --prune
From https://projects.blender.org/blender/blender
   92149a4a284..62bfd0989ef  blender-v4.2-release -> origin/blender-v4.2-release
   a1630792cfa..09bae67c86f  main                 -> origin/main
Fetching submodule tests/data
From https://projects.blender.org/blender/blender-test-data
   01dbd9e..00d24c7  main       -> origin/main
fatal: remote error: upload-pack: not our ref 81c0f9a61c5eca75922688e9695b851fdefbaf7b
Errors during submodule fetch:
	tests/data

Source: https://builder.blender.org/admin/#/builders/264/builds/552

The repository has gotten into a bad state on each worker. After encountering one of these fatal errors, the repository appears to "fix" itself by pruning the bad reference. Re-triggering the Buildbot workers helps, however, since there are a multitude of tracks and a significant amount of workers, I will try to deterministically resolve this using our invoke-workers commands.

After commit [811c1d](https://projects.blender.org/blender/blender/commit/811c1d50f3a44634c5d3ce534ce51fe7074c99b1) the repository will return an error on each "first time" run when `git fetch origin --prune` is ran: ``` INFO: Setting up builder paths from [/home/blender/git/blender-v420] INFO: Removing /home/blender/git/blender-v420/build_asserts/Testing INFO: Removing /home/blender/git/blender-v420/build_asserts/bin/tests INFO: Run command in path [/home/blender/git/blender-v420/blender.git] git remote set-head origin --auto origin/HEAD set to main INFO: Run command in path [/home/blender/git/blender-v420/blender.git] git remote set-url origin https://projects.blender.org/blender/blender.git INFO: Run command in path [/home/blender/git/blender-v420/blender.git] git submodule sync Synchronizing submodule url for 'lib/linux_x64' Synchronizing submodule url for 'lib/macos_arm64' Synchronizing submodule url for 'lib/macos_x64' Synchronizing submodule url for 'lib/windows_arm64' Synchronizing submodule url for 'lib/windows_x64' Synchronizing submodule url for 'release/datafiles/assets' Synchronizing submodule url for 'tests/data' INFO: Run command in path [/home/blender/git/blender-v420/blender.git] git fetch origin --prune From https://projects.blender.org/blender/blender 92149a4a284..62bfd0989ef blender-v4.2-release -> origin/blender-v4.2-release a1630792cfa..09bae67c86f main -> origin/main Fetching submodule tests/data From https://projects.blender.org/blender/blender-test-data 01dbd9e..00d24c7 main -> origin/main fatal: remote error: upload-pack: not our ref 81c0f9a61c5eca75922688e9695b851fdefbaf7b Errors during submodule fetch: tests/data ``` Source: https://builder.blender.org/admin/#/builders/264/builds/552 The repository has gotten into a bad state on each worker. After encountering one of these fatal errors, the repository appears to "fix" itself by pruning the bad reference. Re-triggering the Buildbot workers helps, however, since there are a multitude of tracks and a significant amount of workers, I will try to deterministically resolve this using our `invoke-workers` commands.

Bart van der Braak added this to the DevOps Progress Board project 2024-07-31 16:19:31 +02:00

Bart van der Braak commented

2024-08-01 10:14:24 +02:00

I resolved this issue by running the following for each track on all workers:

try {
    if (Test-Path -Path (Join-Path -Path (Get-Location) -ChildPath 'blender-vdev/blender.git')) {
        Set-Location -Path (Join-Path -Path (Get-Location) -ChildPath 'blender-vdev/blender.git')
        git remote set-head origin --auto
        git remote set-url origin https://projects.blender.org/blender/blender.git
        git submodule sync
        git fetch origin --prune
    } else {
        Write-Host 'Directory not found'
    }
    exit 0
} catch {
    Write-Host 'An error occurred'
    exit 0
}
"

I resolved this issue by running the following for each track on all workers: ```pwsh try { if (Test-Path -Path (Join-Path -Path (Get-Location) -ChildPath 'blender-vdev/blender.git')) { Set-Location -Path (Join-Path -Path (Get-Location) -ChildPath 'blender-vdev/blender.git') git remote set-head origin --auto git remote set-url origin https://projects.blender.org/blender/blender.git git submodule sync git fetch origin --prune } else { Write-Host 'Directory not found' } exit 0 } catch { Write-Host 'An error occurred' exit 0 } " ```

Bart van der Braak closed this issue

2024-08-01 10:14:25 +02:00

Sign in to join this conversation.