studio/flamenco
studio/flamenco
12
30
Fork 38
Code Issues 63 Pull Requests 5 Releases Wiki Activity

Webapp: Clarification of Worker Maintenance section #104229

Merged
Sybren A. Stüvel merged 6 commits from michael-2/flamenco:app_worker_maintenance_caption into main 2023-08-23 15:23:52 +02:00
Conversation 0 Commits 6 Files Changed 1 +904 -582
155 changed files with 904 additions and 582 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files
Showing only changes of commit 9a2d01e2c5 - Show all commits

5
CHANGELOG.md
View File

@ -6,6 +6,8 @@ bugs in actually-released versions.
## 3.3 - in development
- Upgrade bundled FFmpeg from 5.0 to 5.1.
- Rename the add-on download to `flamenco-addon.zip` (it used to be `flamenco3-addon.zip`). It still contains the same files as before, and in Blender the name of the add-on has not changed.
- Improve speed of queueing up >100 simultaneous job deletions.
- Improve logging of job deletion.
- Add Worker Tag support. Workers can be members of any number of tags. Workers will only work on jobs that are assigned to that tag. Jobs that do not have a tag will be available to all workers, regardless of their tag assignment. As a result, tagless workers will only work on tagless jobs.
@ -15,6 +17,9 @@ bugs in actually-released versions.
- Nicer version display for non-release builds. Instead of `3.3-alpha0-v3.2-76-gdd34d538`, show `3.3-alpha0 (v3.2-76-gdd34d538)`.
- Job settings: add a description for the `eval` field. This is shown in the tooltip of the 'set to automatic value' button, to make it clear what that button will do.
- Job settings: make it possible for a setting to be "linked" to its automatic value. For job settings that have this new feature enabled, they will not be editable by default, and the setting will just use its `eval` expression to determine the value. This can be toggled by the user in Blender's submission interface, to still allow manual edits of the value when needed.
- Database integrity tests. These are always run at startup of Flamenco Manager, and by default run periodically every hour. This can be configured by adding/changing the `database_check_period: 1h` setting in `flamenco-manager.yaml`. Setting it to `0` will disable the periodic check. When a database consistency error is found, Flamenco Manager will immediately shut down.
- The webapp automatically reloads after a disconnect, when it reconnects to Flamenco Manager and sees the Manager version changed [#104235](https://projects.blender.org/studio/flamenco/pulls/104235).
- Show the configured Flamenco Manager name in the webapp's browser window title.
## 3.2 - released 2023-02-21

43
Makefile
View File

@ -1,6 +1,6 @@
-include .env
PKG := git.blender.org/flamenco
PKG := projects.blender.org/studio/flamenco
# To update the version number in all the relevant places, update the VERSION
# variable below and run `make update-version`.
@ -10,14 +10,17 @@ RELEASE_CYCLE := alpha
# _GIT_DESCRIPTION_OR_TAG is either something like '16-123abc' (when we're 16
# commits since the last tag) or it's something like `v3.0-beta2` (when exactly
# on a tagged version).
_GIT_DESCRIPTION_OR_TAG := $(subst v${VERSION}-,,$(shell git describe --tag --dirty --always))
_GIT_DESCRIPTION_OR_TAG := $(subst v${VERSION}-,,$(shell git describe --tag --dirty --always --abbrev=9))
# In the above cases, GITHASH is either `16-123abc` (in the same case above) or
# `123abc` (when the tag matches the current commit exactly) or `dirty` (when
# the tag matches the current commit exactly, and there are subsequent
# uncommitted changes). This is done to prevent repetition of the same tag
# in the "extended version" of Flamenco, which combines ${VERSION} and
# ${GITHASH}.
GITHASH := $(subst v${VERSION},$(shell git rev-parse --short HEAD),${_GIT_DESCRIPTION_OR_TAG})
GITHASH := $(subst v${VERSION},$(shell git rev-parse --short=9 HEAD),${_GIT_DESCRIPTION_OR_TAG})
ifeq (${GITHASH},dirty)
GITHASH := $(shell git rev-parse --short=9 HEAD)
endif
LDFLAGS := ${LDFLAGS} -X ${PKG}/internal/appinfo.ApplicationVersion=${VERSION} \
-X ${PKG}/internal/appinfo.ApplicationGitHash=${GITHASH} \
@ -38,11 +41,6 @@ WEB_STATIC=web/static
# Prevent any dependency that requires a C compiler, i.e. only work with pure-Go libraries.
export CGO_ENABLED=0
# FFmpeg version to bundle.
FFMPEG_VERSION=5.0.1
TOOLS=./tools
TOOLS_DOWNLOAD=./tools/download
all: application
# Install generators and build the software.
@ -96,7 +94,7 @@ webapp-static: addon-packer
# in `cmd/flamenco-manager/main.go`
yarn --cwd web/app build --outDir ../static --base=/app/ --logLevel warn
# yarn --cwd web/app build --outDir ../static --base=/app/ --minify false
./addon-packer -filename ${WEB_STATIC}/flamenco3-addon.zip
./addon-packer -filename ${WEB_STATIC}/flamenco-addon.zip
@echo "Web app has been installed into ${WEB_STATIC}"
generate:
@ -264,18 +262,29 @@ tools:
$(MAKE) -s tools-darwin
$(MAKE) -s tools-windows
FFMPEG_PACKAGE_LINUX=$(TOOLS_DOWNLOAD)/ffmpeg-$(FFMPEG_VERSION)-linux-amd64-static.tar.xz
FFMPEG_PACKAGE_DARWIN=$(TOOLS_DOWNLOAD)/ffmpeg-$(FFMPEG_VERSION)-darwin-amd64.zip
FFMPEG_PACKAGE_WINDOWS=$(TOOLS_DOWNLOAD)/ffmpeg-$(FFMPEG_VERSION)-windows-amd64.zip
# FFmpeg version to bundle.
# Version 5.1.3 is the last release in the 5.x series, but binaries have not
# been made available. And then there are different 'latest' binaries for the
# different platforms.
FFMPEG_VERSION_LINUX=5.1.1
FFMPEG_VERSION_WINDOWS=5.1.2
FFMPEG_VERSION_DARWIN=5.1.2
TOOLS=./tools
TOOLS_DOWNLOAD=./tools/download
FFMPEG_PACKAGE_LINUX=$(TOOLS_DOWNLOAD)/ffmpeg-$(FFMPEG_VERSION_LINUX)-linux-amd64-static.tar.xz
FFMPEG_PACKAGE_DARWIN=$(TOOLS_DOWNLOAD)/ffmpeg-$(FFMPEG_VERSION_DARWIN)-darwin-amd64.zip
FFMPEG_PACKAGE_WINDOWS=$(TOOLS_DOWNLOAD)/ffmpeg-$(FFMPEG_VERSION_WINDOWS)-windows-amd64.zip
.PHONY: tools-linux
tools-linux:
[ -e $(FFMPEG_PACKAGE_LINUX) ] || curl \
--create-dirs -o $(FFMPEG_PACKAGE_LINUX) \
https://johnvansickle.com/ffmpeg/releases/ffmpeg-$(FFMPEG_VERSION)-amd64-static.tar.xz
https://www.johnvansickle.com/ffmpeg/old-releases/ffmpeg-$(FFMPEG_VERSION_LINUX)-amd64-static.tar.xz
tar xvf \
$(FFMPEG_PACKAGE_LINUX) \
ffmpeg-$(FFMPEG_VERSION)-amd64-static/ffmpeg \
ffmpeg-$(FFMPEG_VERSION_LINUX)-amd64-static/ffmpeg \
--strip-components=1
mv ffmpeg $(TOOLS)/ffmpeg-linux-amd64
@ -283,7 +292,7 @@ tools-linux:
tools-darwin:
[ -e $(FFMPEG_PACKAGE_DARWIN) ] || curl \
--create-dirs -o $(FFMPEG_PACKAGE_DARWIN) \
https://evermeet.cx/ffmpeg/ffmpeg-$(FFMPEG_VERSION).zip
https://evermeet.cx/ffmpeg/ffmpeg-$(FFMPEG_VERSION_DARWIN).zip
unzip $(FFMPEG_PACKAGE_DARWIN)
mv ffmpeg $(TOOLS)/ffmpeg-darwin-amd64
@ -291,8 +300,8 @@ tools-darwin:
tools-windows:
[ -e $(FFMPEG_PACKAGE_WINDOWS) ] || curl \
--create-dirs -o $(FFMPEG_PACKAGE_WINDOWS) \
https://www.gyan.dev/ffmpeg/builds/packages/ffmpeg-$(FFMPEG_VERSION)-essentials_build.zip
unzip -j $(FFMPEG_PACKAGE_WINDOWS) ffmpeg-5.0.1-essentials_build/bin/ffmpeg.exe -d .
https://www.gyan.dev/ffmpeg/builds/packages/ffmpeg-$(FFMPEG_VERSION_WINDOWS)-essentials_build.zip
unzip -j $(FFMPEG_PACKAGE_WINDOWS) ffmpeg-$(FFMPEG_VERSION_WINDOWS)-essentials_build/bin/ffmpeg.exe -d .
mv ffmpeg.exe $(TOOLS)/ffmpeg-windows-amd64.exe

4
cmd/addon-packer/addon-packer.go
View File

@ -18,7 +18,7 @@ import (
"github.com/rs/zerolog"
"github.com/rs/zerolog/log"
"git.blender.org/flamenco/internal/appinfo"
"projects.blender.org/studio/flamenco/internal/appinfo"
)
var cliArgs struct {
@ -147,7 +147,7 @@ func parseCliArgs() {
flag.BoolVar(&cliArgs.quiet, "quiet", false, "Only log warning-level and worse.")
flag.BoolVar(&cliArgs.debug, "debug", false, "Enable debug-level logging.")
flag.BoolVar(&cliArgs.trace, "trace", false, "Enable trace-level logging.")
flag.StringVar(&cliArgs.filename, "filename", "web/static/flamenco3-addon.zip", "Filename to save the add-on to.")
flag.StringVar(&cliArgs.filename, "filename", "web/static/flamenco-addon.zip", "Filename to save the add-on to.")
flag.Parse()
}

50
cmd/flamenco-manager/main.go
View File

@ -22,23 +22,23 @@ import (
"github.com/rs/zerolog"
"github.com/rs/zerolog/log"
"git.blender.org/flamenco/internal/appinfo"
"git.blender.org/flamenco/internal/manager/api_impl"
"git.blender.org/flamenco/internal/manager/api_impl/dummy"
"git.blender.org/flamenco/internal/manager/config"
"git.blender.org/flamenco/internal/manager/job_compilers"
"git.blender.org/flamenco/internal/manager/job_deleter"
"git.blender.org/flamenco/internal/manager/last_rendered"
"git.blender.org/flamenco/internal/manager/local_storage"
"git.blender.org/flamenco/internal/manager/persistence"
"git.blender.org/flamenco/internal/manager/sleep_scheduler"
"git.blender.org/flamenco/internal/manager/task_logs"
"git.blender.org/flamenco/internal/manager/task_state_machine"
"git.blender.org/flamenco/internal/manager/timeout_checker"
"git.blender.org/flamenco/internal/manager/webupdates"
"git.blender.org/flamenco/internal/own_url"
"git.blender.org/flamenco/internal/upnp_ssdp"
"git.blender.org/flamenco/pkg/shaman"
"projects.blender.org/studio/flamenco/internal/appinfo"
"projects.blender.org/studio/flamenco/internal/manager/api_impl"
"projects.blender.org/studio/flamenco/internal/manager/api_impl/dummy"
"projects.blender.org/studio/flamenco/internal/manager/config"
"projects.blender.org/studio/flamenco/internal/manager/job_compilers"
"projects.blender.org/studio/flamenco/internal/manager/job_deleter"
"projects.blender.org/studio/flamenco/internal/manager/last_rendered"
"projects.blender.org/studio/flamenco/internal/manager/local_storage"
"projects.blender.org/studio/flamenco/internal/manager/persistence"
"projects.blender.org/studio/flamenco/internal/manager/sleep_scheduler"
"projects.blender.org/studio/flamenco/internal/manager/task_logs"
"projects.blender.org/studio/flamenco/internal/manager/task_state_machine"
"projects.blender.org/studio/flamenco/internal/manager/timeout_checker"
"projects.blender.org/studio/flamenco/internal/manager/webupdates"
"projects.blender.org/studio/flamenco/internal/own_url"
"projects.blender.org/studio/flamenco/internal/upnp_ssdp"
"projects.blender.org/studio/flamenco/pkg/shaman"
)
var cliArgs struct {
@ -139,12 +139,6 @@ func runFlamencoManager() bool {
persist := openDB(*configService)
defer persist.Close()
// Disabled for now. `VACUUM` locks the database, which means that other
// queries can fail with a "database is locked (5) (SQLITE_BUSY)" error. This
// situation should be handled gracefully before reinstating the vacuum loop.
//
// go persist.PeriodicMaintenanceLoop(mainCtx)
timeService := clock.New()
compiler, err := job_compilers.Load(timeService)
if err != nil {
@ -196,6 +190,16 @@ func runFlamencoManager() bool {
lastRender.Run(mainCtx)
}()
// Run a periodic integrity check on the database.
// When that check fails, the entire application should shut down.
wg.Add(1)
go func() {
defer wg.Done()
persist.PeriodicIntegrityCheck(mainCtx,
configService.Get().DBIntegrityCheck,
mainCtxCancel)
}()
// Start the web server.
wg.Add(1)
go func() {

20
cmd/flamenco-manager/webservice.go
View File

@ -19,13 +19,13 @@ import (
"github.com/rs/zerolog/log"
"github.com/ziflex/lecho/v3"
"git.blender.org/flamenco/internal/manager/api_impl"
"git.blender.org/flamenco/internal/manager/local_storage"
"git.blender.org/flamenco/internal/manager/swagger_ui"
"git.blender.org/flamenco/internal/manager/webupdates"
"git.blender.org/flamenco/internal/upnp_ssdp"
"git.blender.org/flamenco/pkg/api"
"git.blender.org/flamenco/web"
"projects.blender.org/studio/flamenco/internal/manager/api_impl"
"projects.blender.org/studio/flamenco/internal/manager/local_storage"
"projects.blender.org/studio/flamenco/internal/manager/swagger_ui"
"projects.blender.org/studio/flamenco/internal/manager/webupdates"
"projects.blender.org/studio/flamenco/internal/upnp_ssdp"
"projects.blender.org/studio/flamenco/pkg/api"
"projects.blender.org/studio/flamenco/web"
)
func buildWebService(
@ -136,7 +136,11 @@ func buildWebService(
})
// Serve the Blender add-on. It's contained in the static files of the webapp.
e.GET("/flamenco3-addon.zip", echo.WrapHandler(webAppHandler))
e.GET("/flamenco-addon.zip", echo.WrapHandler(webAppHandler))
e.GET("/flamenco3-addon.zip", func(c echo.Context) error {
return c.Redirect(http.StatusPermanentRedirect, "/flamenco-addon.zip")
})
// The favicons are also in the static files of the webapp.
e.GET("/favicon.png", echo.WrapHandler(webAppHandler))
e.GET("/favicon.ico", echo.WrapHandler(webAppHandler))

4
cmd/flamenco-worker/find_exes.go
View File

@ -9,8 +9,8 @@ import (
"github.com/rs/zerolog/log"
"git.blender.org/flamenco/internal/find_blender"
"git.blender.org/flamenco/internal/find_ffmpeg"
"projects.blender.org/studio/flamenco/internal/find_blender"
"projects.blender.org/studio/flamenco/internal/find_ffmpeg"
)
// findFFmpeg tries to find FFmpeg, in order to show its version (if found) or a warning (if not).

6
cmd/flamenco-worker/main.go
View File

@ -20,9 +20,9 @@ import (
"github.com/rs/zerolog"
"github.com/rs/zerolog/log"
"git.blender.org/flamenco/internal/appinfo"
"git.blender.org/flamenco/internal/worker"
"git.blender.org/flamenco/internal/worker/cli_runner"
"projects.blender.org/studio/flamenco/internal/appinfo"
"projects.blender.org/studio/flamenco/internal/worker"
"projects.blender.org/studio/flamenco/internal/worker/cli_runner"
)
var (

10
cmd/job-creator/main.go
View File

@ -18,11 +18,11 @@ import (
"github.com/rs/zerolog"
"github.com/rs/zerolog/log"
"git.blender.org/flamenco/internal/appinfo"
"git.blender.org/flamenco/internal/manager/config"
"git.blender.org/flamenco/internal/manager/job_compilers"
"git.blender.org/flamenco/internal/manager/persistence"
"git.blender.org/flamenco/pkg/api"
"projects.blender.org/studio/flamenco/internal/appinfo"
"projects.blender.org/studio/flamenco/internal/manager/config"
"projects.blender.org/studio/flamenco/internal/manager/job_compilers"
"projects.blender.org/studio/flamenco/internal/manager/persistence"
"projects.blender.org/studio/flamenco/pkg/api"
)
var cliArgs struct {

8
cmd/shaman-checkout-id-setter/main.go
View File

@ -18,10 +18,10 @@ import (
"github.com/rs/zerolog"
"github.com/rs/zerolog/log"
"git.blender.org/flamenco/internal/appinfo"
"git.blender.org/flamenco/internal/manager/config"
"git.blender.org/flamenco/internal/manager/persistence"
"git.blender.org/flamenco/pkg/api"
"projects.blender.org/studio/flamenco/internal/appinfo"
"projects.blender.org/studio/flamenco/internal/manager/config"
"projects.blender.org/studio/flamenco/internal/manager/persistence"
"projects.blender.org/studio/flamenco/pkg/api"
)
func main() {

4
cmd/stresser/main.go
View File

@ -15,8 +15,8 @@ import (
"github.com/rs/zerolog"
"github.com/rs/zerolog/log"
"git.blender.org/flamenco/internal/appinfo"
"git.blender.org/flamenco/internal/stresser"
"projects.blender.org/studio/flamenco/internal/appinfo"
"projects.blender.org/studio/flamenco/internal/stresser"
)
var cliArgs struct {

2
deploy.sh
View File

@ -1,7 +1,7 @@
#!/bin/bash
MY_DIR="$(dirname "$(readlink -e "$0")")"
ADDON_ZIP="$MY_DIR/web/static/flamenco3-addon.zip"
ADDON_ZIP="$MY_DIR/web/static/flamenco-addon.zip"
WORKER_TARGET=/shared/software/flamenco3-worker/flamenco-worker
TIMESTAMP=$(date +'%Y-%m-%d-%H%M%S')

2
go.mod
View File

@ -1,4 +1,4 @@
module git.blender.org/flamenco
module projects.blender.org/studio/flamenco
go 1.20

4
internal/find_blender/find_blender.go
View File

@ -12,9 +12,9 @@ import (
"strings"
"time"
"git.blender.org/flamenco/pkg/api"
"git.blender.org/flamenco/pkg/crosspath"
"github.com/rs/zerolog/log"
"projects.blender.org/studio/flamenco/pkg/api"
"projects.blender.org/studio/flamenco/pkg/crosspath"
)
var (

2
internal/manager/api_impl/api_impl.go
View File

@ -11,8 +11,8 @@ import (
"sync"
"time"
"git.blender.org/flamenco/pkg/api"
"github.com/labstack/echo/v4"
"projects.blender.org/studio/flamenco/pkg/api"
)
type Flamenco struct {

4
internal/manager/api_impl/dummy/shaman.go
View File

@ -7,8 +7,8 @@ import (
"errors"
"io"
"git.blender.org/flamenco/internal/manager/api_impl"
"git.blender.org/flamenco/pkg/api"
"projects.blender.org/studio/flamenco/internal/manager/api_impl"
"projects.blender.org/studio/flamenco/pkg/api"
)
// DummyShaman implements the Shaman interface from `internal/manager/api_impl/interfaces.go`

22
internal/manager/api_impl/interfaces.go
View File

@ -13,20 +13,20 @@ import (
"github.com/benbjohnson/clock"
"github.com/rs/zerolog"
"git.blender.org/flamenco/internal/manager/config"
"git.blender.org/flamenco/internal/manager/job_compilers"
"git.blender.org/flamenco/internal/manager/job_deleter"
"git.blender.org/flamenco/internal/manager/last_rendered"
"git.blender.org/flamenco/internal/manager/persistence"
"git.blender.org/flamenco/internal/manager/sleep_scheduler"
"git.blender.org/flamenco/internal/manager/task_state_machine"
"git.blender.org/flamenco/internal/manager/webupdates"
"git.blender.org/flamenco/pkg/api"
"git.blender.org/flamenco/pkg/shaman"
"projects.blender.org/studio/flamenco/internal/manager/config"
"projects.blender.org/studio/flamenco/internal/manager/job_compilers"
"projects.blender.org/studio/flamenco/internal/manager/job_deleter"
"projects.blender.org/studio/flamenco/internal/manager/last_rendered"
"projects.blender.org/studio/flamenco/internal/manager/persistence"
"projects.blender.org/studio/flamenco/internal/manager/sleep_scheduler"
"projects.blender.org/studio/flamenco/internal/manager/task_state_machine"
"projects.blender.org/studio/flamenco/internal/manager/webupdates"
"projects.blender.org/studio/flamenco/pkg/api"
"projects.blender.org/studio/flamenco/pkg/shaman"
)
// Generate mock implementations of these interfaces.
//go:generate go run github.com/golang/mock/mockgen -destination mocks/api_impl_mock.gen.go -package mocks git.blender.org/flamenco/internal/manager/api_impl PersistenceService,ChangeBroadcaster,JobCompiler,LogStorage,ConfigService,TaskStateMachine,Shaman,LastRendered,LocalStorage,WorkerSleepScheduler,JobDeleter
//go:generate go run github.com/golang/mock/mockgen -destination mocks/api_impl_mock.gen.go -package mocks projects.blender.org/studio/flamenco/internal/manager/api_impl PersistenceService,ChangeBroadcaster,JobCompiler,LogStorage,ConfigService,TaskStateMachine,Shaman,LastRendered,LocalStorage,WorkerSleepScheduler,JobDeleter
type PersistenceService interface {
StoreAuthoredJob(ctx context.Context, authoredJob job_compilers.AuthoredJob) error

12
internal/manager/api_impl/jobs.go
View File

@ -15,12 +15,12 @@ import (
"github.com/labstack/echo/v4"
"github.com/rs/zerolog"
"git.blender.org/flamenco/internal/manager/job_compilers"
"git.blender.org/flamenco/internal/manager/persistence"
"git.blender.org/flamenco/internal/manager/webupdates"
"git.blender.org/flamenco/internal/uuid"
"git.blender.org/flamenco/pkg/api"
"git.blender.org/flamenco/pkg/crosspath"
"projects.blender.org/studio/flamenco/internal/manager/job_compilers"
"projects.blender.org/studio/flamenco/internal/manager/persistence"
"projects.blender.org/studio/flamenco/internal/manager/webupdates"
"projects.blender.org/studio/flamenco/internal/uuid"
"projects.blender.org/studio/flamenco/pkg/api"
"projects.blender.org/studio/flamenco/pkg/crosspath"
)
// JobFilesURLPrefix is the URL prefix that the Flamenco API expects to serve

6
internal/manager/api_impl/jobs_query.go
View File

@ -9,9 +9,9 @@ import (
"github.com/labstack/echo/v4"
"github.com/rs/zerolog"
"git.blender.org/flamenco/internal/manager/persistence"
"git.blender.org/flamenco/internal/uuid"
"git.blender.org/flamenco/pkg/api"
"projects.blender.org/studio/flamenco/internal/manager/persistence"
"projects.blender.org/studio/flamenco/internal/uuid"
"projects.blender.org/studio/flamenco/pkg/api"
)
// fetchJob fetches the job from the database, and sends the appropriate error

4
internal/manager/api_impl/jobs_query_test.go
View File

@ -7,10 +7,10 @@ import (
"testing"
"time"
"git.blender.org/flamenco/internal/manager/persistence"
"git.blender.org/flamenco/pkg/api"
"github.com/golang/mock/gomock"
"github.com/stretchr/testify/assert"
"projects.blender.org/studio/flamenco/internal/manager/persistence"
"projects.blender.org/studio/flamenco/pkg/api"
)
func TestQueryJobs(t *testing.T) {

12
internal/manager/api_impl/jobs_test.go
View File

@ -9,15 +9,15 @@ import (
"os"
"testing"
"git.blender.org/flamenco/internal/manager/config"
"git.blender.org/flamenco/internal/manager/job_compilers"
"git.blender.org/flamenco/internal/manager/last_rendered"
"git.blender.org/flamenco/internal/manager/persistence"
"git.blender.org/flamenco/pkg/api"
"git.blender.org/flamenco/pkg/moremock"
"github.com/golang/mock/gomock"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"projects.blender.org/studio/flamenco/internal/manager/config"
"projects.blender.org/studio/flamenco/internal/manager/job_compilers"
"projects.blender.org/studio/flamenco/internal/manager/last_rendered"
"projects.blender.org/studio/flamenco/internal/manager/persistence"
"projects.blender.org/studio/flamenco/pkg/api"
"projects.blender.org/studio/flamenco/pkg/moremock"
)
func ptr[T any](value T) *T {

20
internal/manager/api_impl/meta.go
View File

@ -14,18 +14,18 @@ import (
"strconv"
"strings"
"git.blender.org/flamenco/internal/appinfo"
"git.blender.org/flamenco/internal/find_blender"
"git.blender.org/flamenco/internal/manager/config"
"git.blender.org/flamenco/pkg/api"
"github.com/labstack/echo/v4"
"projects.blender.org/studio/flamenco/internal/appinfo"
"projects.blender.org/studio/flamenco/internal/find_blender"
"projects.blender.org/studio/flamenco/internal/manager/config"
"projects.blender.org/studio/flamenco/pkg/api"
)
func (f *Flamenco) GetVersion(e echo.Context) error {
return e.JSON(http.StatusOK, api.FlamencoVersion{
Version: appinfo.ExtendedVersion(),
Shortversion: appinfo.ApplicationVersion,
Name: appinfo.ApplicationName,
Name: f.config.Get().ManagerName,
Git: appinfo.ApplicationGitHash,
})
}
@ -71,18 +71,12 @@ func (f *Flamenco) GetVariables(e echo.Context, audience api.ManagerVariableAudi
func (f *Flamenco) GetSharedStorage(e echo.Context, audience api.ManagerVariableAudience, platform string) error {
location := f.config.EffectiveStoragePath()
feeder := make(chan string, 1)
receiver := make(chan string, 1)
feeder <- location
close(feeder)
f.config.ExpandVariables(feeder, receiver, config.VariableAudience(audience), config.VariablePlatform(platform))
varExpand := f.config.NewVariableExpander(config.VariableAudience(audience), config.VariablePlatform(platform))
return e.JSON(http.StatusOK, api.SharedStorageLocation{
Audience: audience,
Platform: platform,
Location: <-receiver,
Location: varExpand.Expand(location),
ShamanEnabled: f.isShamanEnabled(),
})
}

85
internal/manager/api_impl/meta_test.go
View File

@ -11,12 +11,12 @@ import (
"runtime"
"testing"
"git.blender.org/flamenco/internal/manager/config"
"git.blender.org/flamenco/pkg/api"
"github.com/golang/mock/gomock"
"github.com/labstack/echo/v4"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"projects.blender.org/studio/flamenco/internal/manager/config"
"projects.blender.org/studio/flamenco/pkg/api"
)
func TestGetVariables(t *testing.T) {
@ -90,12 +90,10 @@ func TestGetSharedStorage(t *testing.T) {
mf.config.EXPECT().EffectiveStoragePath().Return(`S:\storage\flamenco`).AnyTimes()
{ // Test user client on Linux.
// Defer to the actual ExpandVariables() implementation of the above config.
mf.config.EXPECT().
ExpandVariables(gomock.Any(), gomock.Any(), config.VariableAudienceUsers, config.VariablePlatformLinux).
Do(func(inputChannel <-chan string, outputChannel chan<- string, audience config.VariableAudience, platform config.VariablePlatform) {
// Defer to the actual ExpandVariables() implementation of the above config.
conf.ExpandVariables(inputChannel, outputChannel, audience, platform)
})
NewVariableExpander(config.VariableAudienceUsers, config.VariablePlatformLinux).
DoAndReturn(conf.NewVariableExpander)
mf.shaman.EXPECT().IsEnabled().Return(false)
echoCtx := mf.prepareMockedRequest(nil)
@ -109,12 +107,10 @@ func TestGetSharedStorage(t *testing.T) {
}
{ // Test worker client on Linux with Shaman enabled.
// Defer to the actual ExpandVariables() implementation of the above config.
mf.config.EXPECT().
ExpandVariables(gomock.Any(), gomock.Any(), config.VariableAudienceWorkers, config.VariablePlatformLinux).
Do(func(inputChannel <-chan string, outputChannel chan<- string, audience config.VariableAudience, platform config.VariablePlatform) {
// Defer to the actual ExpandVariables() implementation of the above config.
conf.ExpandVariables(inputChannel, outputChannel, audience, platform)
})
NewVariableExpander(config.VariableAudienceWorkers, config.VariablePlatformLinux).
DoAndReturn(conf.NewVariableExpander)
mf.shaman.EXPECT().IsEnabled().Return(true)
echoCtx := mf.prepareMockedRequest(nil)
@ -129,12 +125,10 @@ func TestGetSharedStorage(t *testing.T) {
}
{ // Test user client on Windows.
// Defer to the actual ExpandVariables() implementation of the above config.
mf.config.EXPECT().
ExpandVariables(gomock.Any(), gomock.Any(), config.VariableAudienceUsers, config.VariablePlatformWindows).
Do(func(inputChannel <-chan string, outputChannel chan<- string, audience config.VariableAudience, platform config.VariablePlatform) {
// Defer to the actual ExpandVariables() implementation of the above config.
conf.ExpandVariables(inputChannel, outputChannel, audience, platform)
})
NewVariableExpander(config.VariableAudienceUsers, config.VariablePlatformWindows).
DoAndReturn(conf.NewVariableExpander)
mf.shaman.EXPECT().IsEnabled().Return(false)
echoCtx := mf.prepareMockedRequest(nil)
@ -149,6 +143,63 @@ func TestGetSharedStorage(t *testing.T) {
}
// Test shared storage sitting on /mnt/flamenco, where that's mapped to F:\ for Windows.
func TestGetSharedStorageDriveLetterRoot(t *testing.T) {
mockCtrl := gomock.NewController(t)
defer mockCtrl.Finish()
mf := newMockedFlamenco(mockCtrl)
conf := config.GetTestConfig(func(c *config.Conf) {
// Test with a Manager on Linux.
c.MockCurrentGOOSForTests("linux")
// Set up a two-way variable to do the mapping.
c.Variables["shared_storage_mapping"] = config.Variable{
IsTwoWay: true,
Values: []config.VariableValue{
{Value: "/mnt/flamenco", Platform: config.VariablePlatformLinux, Audience: config.VariableAudienceAll},
{Value: `F:\`, Platform: config.VariablePlatformWindows, Audience: config.VariableAudienceAll},
},
}
})
mf.config.EXPECT().Get().Return(&conf).AnyTimes()
mf.config.EXPECT().EffectiveStoragePath().Return(`/mnt/flamenco`).AnyTimes()
{ // Test user client on Linux.
mf.config.EXPECT().
NewVariableExpander(config.VariableAudienceUsers, config.VariablePlatformLinux).
DoAndReturn(conf.NewVariableExpander)
mf.shaman.EXPECT().IsEnabled().Return(false)
echoCtx := mf.prepareMockedRequest(nil)
err := mf.flamenco.GetSharedStorage(echoCtx, api.ManagerVariableAudienceUsers, "linux")
require.NoError(t, err)
assertResponseJSON(t, echoCtx, http.StatusOK, api.SharedStorageLocation{
Location: "/mnt/flamenco",
Audience: api.ManagerVariableAudienceUsers,
Platform: "linux",
})
}
{ // Test user client on Windows.
mf.config.EXPECT().
NewVariableExpander(config.VariableAudienceUsers, config.VariablePlatformWindows).
DoAndReturn(conf.NewVariableExpander)
mf.shaman.EXPECT().IsEnabled().Return(false)
echoCtx := mf.prepareMockedRequest(nil)
err := mf.flamenco.GetSharedStorage(echoCtx, api.ManagerVariableAudienceUsers, "windows")
require.NoError(t, err)
assertResponseJSON(t, echoCtx, http.StatusOK, api.SharedStorageLocation{
Location: `F:\`,
Audience: api.ManagerVariableAudienceUsers,
Platform: "windows",
})
}
}
func TestCheckSharedStoragePath(t *testing.T) {
mf, finish := metaTestFixtures(t)
defer finish()

64
internal/manager/api_impl/mocks/api_impl_mock.gen.go generated
View File

@ -1,5 +1,5 @@
// Code generated by MockGen. DO NOT EDIT.
// Source: git.blender.org/flamenco/internal/manager/api_impl (interfaces: PersistenceService,ChangeBroadcaster,JobCompiler,LogStorage,ConfigService,TaskStateMachine,Shaman,LastRendered,LocalStorage,WorkerSleepScheduler,JobDeleter)
// Source: projects.blender.org/studio/flamenco/internal/manager/api_impl (interfaces: PersistenceService,ChangeBroadcaster,JobCompiler,LogStorage,ConfigService,TaskStateMachine,Shaman,LastRendered,LocalStorage,WorkerSleepScheduler,JobDeleter)
// Package mocks is a generated GoMock package.
package mocks
@ -9,13 +9,13 @@ import (
io "io"
reflect "reflect"
config "git.blender.org/flamenco/internal/manager/config"
job_compilers "git.blender.org/flamenco/internal/manager/job_compilers"
last_rendered "git.blender.org/flamenco/internal/manager/last_rendered"
persistence "git.blender.org/flamenco/internal/manager/persistence"
api "git.blender.org/flamenco/pkg/api"
gomock "github.com/golang/mock/gomock"
zerolog "github.com/rs/zerolog"
config "projects.blender.org/studio/flamenco/internal/manager/config"
job_compilers "projects.blender.org/studio/flamenco/internal/manager/job_compilers"
last_rendered "projects.blender.org/studio/flamenco/internal/manager/last_rendered"
persistence "projects.blender.org/studio/flamenco/internal/manager/persistence"
api "projects.blender.org/studio/flamenco/pkg/api"
)
// MockPersistenceService is a mock of PersistenceService interface.
@ -841,18 +841,6 @@ func (m *MockConfigService) EXPECT() *MockConfigServiceMockRecorder {
return m.recorder
}
// ConvertTwoWayVariables mocks base method.
func (m *MockConfigService) ConvertTwoWayVariables(arg0 <-chan string, arg1 chan<- string, arg2 config.VariableAudience, arg3 config.VariablePlatform) {
m.ctrl.T.Helper()
m.ctrl.Call(m, "ConvertTwoWayVariables", arg0, arg1, arg2, arg3)
}
// ConvertTwoWayVariables indicates an expected call of ConvertTwoWayVariables.
func (mr *MockConfigServiceMockRecorder) ConvertTwoWayVariables(arg0, arg1, arg2, arg3 interface{}) *gomock.Call {
mr.mock.ctrl.T.Helper()
return mr.mock.ctrl.RecordCallWithMethodType(mr.mock, "ConvertTwoWayVariables", reflect.TypeOf((*MockConfigService)(nil).ConvertTwoWayVariables), arg0, arg1, arg2, arg3)
}
// EffectiveStoragePath mocks base method.
func (m *MockConfigService) EffectiveStoragePath() string {
m.ctrl.T.Helper()
@ -867,18 +855,6 @@ func (mr *MockConfigServiceMockRecorder) EffectiveStoragePath() *gomock.Call {
return mr.mock.ctrl.RecordCallWithMethodType(mr.mock, "EffectiveStoragePath", reflect.TypeOf((*MockConfigService)(nil).EffectiveStoragePath))
}
// ExpandVariables mocks base method.
func (m *MockConfigService) ExpandVariables(arg0 <-chan string, arg1 chan<- string, arg2 config.VariableAudience, arg3 config.VariablePlatform) {
m.ctrl.T.Helper()
m.ctrl.Call(m, "ExpandVariables", arg0, arg1, arg2, arg3)
}
// ExpandVariables indicates an expected call of ExpandVariables.
func (mr *MockConfigServiceMockRecorder) ExpandVariables(arg0, arg1, arg2, arg3 interface{}) *gomock.Call {
mr.mock.ctrl.T.Helper()
return mr.mock.ctrl.RecordCallWithMethodType(mr.mock, "ExpandVariables", reflect.TypeOf((*MockConfigService)(nil).ExpandVariables), arg0, arg1, arg2, arg3)
}
// ForceFirstRun mocks base method.
func (m *MockConfigService) ForceFirstRun() {
m.ctrl.T.Helper()
@ -920,6 +896,34 @@ func (mr *MockConfigServiceMockRecorder) IsFirstRun() *gomock.Call {
return mr.mock.ctrl.RecordCallWithMethodType(mr.mock, "IsFirstRun", reflect.TypeOf((*MockConfigService)(nil).IsFirstRun))
}
// NewVariableExpander mocks base method.
func (m *MockConfigService) NewVariableExpander(arg0 config.VariableAudience, arg1 config.VariablePlatform) *config.VariableExpander {
m.ctrl.T.Helper()
ret := m.ctrl.Call(m, "NewVariableExpander", arg0, arg1)
ret0, _ := ret[0].(*config.VariableExpander)
return ret0
}
// NewVariableExpander indicates an expected call of NewVariableExpander.
func (mr *MockConfigServiceMockRecorder) NewVariableExpander(arg0, arg1 interface{}) *gomock.Call {
mr.mock.ctrl.T.Helper()
return mr.mock.ctrl.RecordCallWithMethodType(mr.mock, "NewVariableExpander", reflect.TypeOf((*MockConfigService)(nil).NewVariableExpander), arg0, arg1)
}
// NewVariableToValueConverter mocks base method.
func (m *MockConfigService) NewVariableToValueConverter(arg0 config.VariableAudience, arg1 config.VariablePlatform) *config.ValueToVariableReplacer {
m.ctrl.T.Helper()
ret := m.ctrl.Call(m, "NewVariableToValueConverter", arg0, arg1)
ret0, _ := ret[0].(*config.ValueToVariableReplacer)
return ret0
}
// NewVariableToValueConverter indicates an expected call of NewVariableToValueConverter.
func (mr *MockConfigServiceMockRecorder) NewVariableToValueConverter(arg0, arg1 interface{}) *gomock.Call {
mr.mock.ctrl.T.Helper()
return mr.mock.ctrl.RecordCallWithMethodType(mr.mock, "NewVariableToValueConverter", reflect.TypeOf((*MockConfigService)(nil).NewVariableToValueConverter), arg0, arg1)
}
// ResolveVariables mocks base method.
func (m *MockConfigService) ResolveVariables(arg0 config.VariableAudience, arg1 config.VariablePlatform) map[string]config.ResolvedVariable {
m.ctrl.T.Helper()

32
internal/manager/api_impl/mocks/varrepl.gen.go generated
View File

@ -1,5 +1,5 @@
// Code generated by MockGen. DO NOT EDIT.
// Source: git.blender.org/flamenco/internal/manager/api_impl (interfaces: VariableReplacer)
// Source: projects.blender.org/studio/flamenco/internal/manager/api_impl (interfaces: VariableReplacer)
// Package mocks is a generated GoMock package.
package mocks
@ -7,8 +7,8 @@ package mocks
import (
reflect "reflect"
config "git.blender.org/flamenco/internal/manager/config"
gomock "github.com/golang/mock/gomock"
config "projects.blender.org/studio/flamenco/internal/manager/config"
)
// MockVariableReplacer is a mock of VariableReplacer interface.
@ -34,28 +34,32 @@ func (m *MockVariableReplacer) EXPECT() *MockVariableReplacerMockRecorder {
return m.recorder
}
// ConvertTwoWayVariables mocks base method.
func (m *MockVariableReplacer) ConvertTwoWayVariables(arg0 <-chan string, arg1 chan<- string, arg2 config.VariableAudience, arg3 config.VariablePlatform) {
// NewVariableExpander mocks base method.
func (m *MockVariableReplacer) NewVariableExpander(arg0 config.VariableAudience, arg1 config.VariablePlatform) *config.VariableExpander {
m.ctrl.T.Helper()
m.ctrl.Call(m, "ConvertTwoWayVariables", arg0, arg1, arg2, arg3)
ret := m.ctrl.Call(m, "NewVariableExpander", arg0, arg1)
ret0, _ := ret[0].(*config.VariableExpander)
return ret0
}
// ConvertTwoWayVariables indicates an expected call of ConvertTwoWayVariables.
func (mr *MockVariableReplacerMockRecorder) ConvertTwoWayVariables(arg0, arg1, arg2, arg3 interface{}) *gomock.Call {
// NewVariableExpander indicates an expected call of NewVariableExpander.
func (mr *MockVariableReplacerMockRecorder) NewVariableExpander(arg0, arg1 interface{}) *gomock.Call {
mr.mock.ctrl.T.Helper()
return mr.mock.ctrl.RecordCallWithMethodType(mr.mock, "ConvertTwoWayVariables", reflect.TypeOf((*MockVariableReplacer)(nil).ConvertTwoWayVariables), arg0, arg1, arg2, arg3)
return mr.mock.ctrl.RecordCallWithMethodType(mr.mock, "NewVariableExpander", reflect.TypeOf((*MockVariableReplacer)(nil).NewVariableExpander), arg0, arg1)
}
// ExpandVariables mocks base method.
func (m *MockVariableReplacer) ExpandVariables(arg0 <-chan string, arg1 chan<- string, arg2 config.VariableAudience, arg3 config.VariablePlatform) {
// NewVariableToValueConverter mocks base method.
func (m *MockVariableReplacer) NewVariableToValueConverter(arg0 config.VariableAudience, arg1 config.VariablePlatform) *config.ValueToVariableReplacer {
m.ctrl.T.Helper()
m.ctrl.Call(m, "ExpandVariables", arg0, arg1, arg2, arg3)
ret := m.ctrl.Call(m, "NewVariableToValueConverter", arg0, arg1)
ret0, _ := ret[0].(*config.ValueToVariableReplacer)
return ret0
}
// ExpandVariables indicates an expected call of ExpandVariables.
func (mr *MockVariableReplacerMockRecorder) ExpandVariables(arg0, arg1, arg2, arg3 interface{}) *gomock.Call {
// NewVariableToValueConverter indicates an expected call of NewVariableToValueConverter.
func (mr *MockVariableReplacerMockRecorder) NewVariableToValueConverter(arg0, arg1 interface{}) *gomock.Call {
mr.mock.ctrl.T.Helper()
return mr.mock.ctrl.RecordCallWithMethodType(mr.mock, "ExpandVariables", reflect.TypeOf((*MockVariableReplacer)(nil).ExpandVariables), arg0, arg1, arg2, arg3)
return mr.mock.ctrl.RecordCallWithMethodType(mr.mock, "NewVariableToValueConverter", reflect.TypeOf((*MockVariableReplacer)(nil).NewVariableToValueConverter), arg0, arg1)
}
// ResolveVariables mocks base method.

4
internal/manager/api_impl/shaman.go
View File

@ -7,8 +7,8 @@ import (
"github.com/labstack/echo/v4"
"git.blender.org/flamenco/pkg/api"
"git.blender.org/flamenco/pkg/shaman/fileserver"
"projects.blender.org/studio/flamenco/pkg/api"
"projects.blender.org/studio/flamenco/pkg/shaman/fileserver"
)
func (f *Flamenco) isShamanEnabled() bool {

16
internal/manager/api_impl/support_test.go
View File

@ -17,10 +17,10 @@ import (
"github.com/labstack/echo/v4"
"github.com/stretchr/testify/assert"
"git.blender.org/flamenco/internal/manager/api_impl/mocks"
"git.blender.org/flamenco/internal/manager/config"
"git.blender.org/flamenco/internal/manager/persistence"
"git.blender.org/flamenco/pkg/api"
"projects.blender.org/studio/flamenco/internal/manager/api_impl/mocks"
"projects.blender.org/studio/flamenco/internal/manager/config"
"projects.blender.org/studio/flamenco/internal/manager/persistence"
"projects.blender.org/studio/flamenco/pkg/api"
)
type mockedFlamenco struct {
@ -101,8 +101,8 @@ func (mf *mockedFlamenco) expectExpandVariables(
// Defer the mocked call to the fake configuration.
return mf.config.EXPECT().
ExpandVariables(gomock.Any(), gomock.Any(), expectAudience, expectPlatform).
DoAndReturn(c.ExpandVariables)
NewVariableExpander(expectAudience, expectPlatform).
DoAndReturn(c.NewVariableExpander)
}
func (mf *mockedFlamenco) expectConvertTwoWayVariables(
@ -126,8 +126,8 @@ func (mf *mockedFlamenco) expectConvertTwoWayVariables(
// Defer the mocked call to the fake configuration.
return mf.config.EXPECT().
ConvertTwoWayVariables(gomock.Any(), gomock.Any(), expectAudience, expectPlatform).
DoAndReturn(c.ConvertTwoWayVariables)
NewVariableToValueConverter(expectAudience, expectPlatform).
DoAndReturn(c.NewVariableToValueConverter)
}
// prepareMockedJSONRequest returns an `echo.Context` that has a JSON request body attached to it.

67
internal/manager/api_impl/varrepl.go
View File

@ -3,45 +3,35 @@ package api_impl
// SPDX-License-Identifier: GPL-3.0-or-later
import (
"sync"
"git.blender.org/flamenco/internal/manager/config"
"git.blender.org/flamenco/internal/manager/persistence"
"git.blender.org/flamenco/pkg/api"
"projects.blender.org/studio/flamenco/internal/manager/config"
"projects.blender.org/studio/flamenco/internal/manager/persistence"
"projects.blender.org/studio/flamenco/pkg/api"
)
//go:generate go run github.com/golang/mock/mockgen -destination mocks/varrepl.gen.go -package mocks git.blender.org/flamenco/internal/manager/api_impl VariableReplacer
//go:generate go run github.com/golang/mock/mockgen -destination mocks/varrepl.gen.go -package mocks projects.blender.org/studio/flamenco/internal/manager/api_impl VariableReplacer
type VariableReplacer interface {
ExpandVariables(inputChannel <-chan string, outputChannel chan<- string, audience config.VariableAudience, platform config.VariablePlatform)
NewVariableExpander(audience config.VariableAudience, platform config.VariablePlatform) *config.VariableExpander
ResolveVariables(audience config.VariableAudience, platform config.VariablePlatform) map[string]config.ResolvedVariable
ConvertTwoWayVariables(inputChannel <-chan string, outputChannel chan<- string, audience config.VariableAudience, platform config.VariablePlatform)
NewVariableToValueConverter(audience config.VariableAudience, platform config.VariablePlatform) *config.ValueToVariableReplacer
}
// replaceTaskVariables performs variable replacement for worker tasks.
func replaceTaskVariables(replacer VariableReplacer, task api.AssignedTask, worker persistence.Worker) api.AssignedTask {
feeder := make(chan string, 1)
receiver := make(chan string, 1)
wg := sync.WaitGroup{}
wg.Add(1)
go func() {
defer wg.Done()
replacer.ExpandVariables(feeder, receiver,
config.VariableAudienceWorkers, config.VariablePlatform(worker.Platform))
}()
varExpander := replacer.NewVariableExpander(
config.VariableAudienceWorkers,
config.VariablePlatform(worker.Platform),
)
for cmdIndex, cmd := range task.Commands {
for key, value := range cmd.Parameters {
switch v := value.(type) {
case string:
feeder <- v
task.Commands[cmdIndex].Parameters[key] = <-receiver
task.Commands[cmdIndex].Parameters[key] = varExpander.Expand(v)
case []string:
replaced := make([]string, len(v))
for idx := range v {
feeder <- v[idx]
replaced[idx] = <-receiver
replaced[idx] = varExpander.Expand(v[idx])
}
task.Commands[cmdIndex].Parameters[key] = replaced
@ -50,8 +40,7 @@ func replaceTaskVariables(replacer VariableReplacer, task api.AssignedTask, work
for idx := range v {
switch itemValue := v[idx].(type) {
case string:
feeder <- itemValue
replaced[idx] = <-receiver
replaced[idx] = varExpander.Expand(itemValue)
default:
replaced[idx] = itemValue
}
@ -64,10 +53,6 @@ func replaceTaskVariables(replacer VariableReplacer, task api.AssignedTask, work
}
}
close(feeder)
wg.Wait()
close(receiver)
return task
}
@ -78,16 +63,10 @@ func replaceTaskVariables(replacer VariableReplacer, task api.AssignedTask, work
//
// NOTE: this updates the job in place.
func replaceTwoWayVariables(replacer VariableReplacer, job *api.SubmittedJob) {
feeder := make(chan string, 1)
receiver := make(chan string, 1)
wg := sync.WaitGroup{}
wg.Add(1)
go func() {
defer wg.Done()
replacer.ConvertTwoWayVariables(feeder, receiver,
config.VariableAudienceWorkers, config.VariablePlatform(job.SubmitterPlatform))
}()
valueToVariable := replacer.NewVariableToValueConverter(
config.VariableAudienceWorkers,
config.VariablePlatform(job.SubmitterPlatform),
)
// Only replace variables in settings and metadata, not in other job fields.
if job.Settings != nil {
@ -96,18 +75,14 @@ func replaceTwoWayVariables(replacer VariableReplacer, job *api.SubmittedJob) {
if !ok {
continue
}
feeder <- stringValue
job.Settings.AdditionalProperties[settingKey] = <-receiver
newValue := valueToVariable.Replace(stringValue)
job.Settings.AdditionalProperties[settingKey] = newValue
}
}
if job.Metadata != nil {
for metaKey, metaValue := range job.Metadata.AdditionalProperties {
feeder <- metaValue
job.Metadata.AdditionalProperties[metaKey] = <-receiver
newValue := valueToVariable.Replace(metaValue)
job.Metadata.AdditionalProperties[metaKey] = newValue
}
}
close(feeder)
wg.Wait()
close(receiver)
}

8
internal/manager/api_impl/varrepl_test.go
View File

@ -9,10 +9,10 @@ import (
"github.com/stretchr/testify/assert"
"git.blender.org/flamenco/internal/manager/config"
"git.blender.org/flamenco/internal/manager/persistence"
"git.blender.org/flamenco/pkg/api"
"git.blender.org/flamenco/pkg/crosspath"
"projects.blender.org/studio/flamenco/internal/manager/config"
"projects.blender.org/studio/flamenco/internal/manager/persistence"
"projects.blender.org/studio/flamenco/pkg/api"
"projects.blender.org/studio/flamenco/pkg/crosspath"
)
func varreplTestTask() api.AssignedTask {

2
internal/manager/api_impl/worker_auth.go
View File

@ -13,7 +13,7 @@ import (
"github.com/labstack/echo/v4"
"golang.org/x/crypto/bcrypt"
"git.blender.org/flamenco/internal/manager/persistence"
"projects.blender.org/studio/flamenco/internal/manager/persistence"
)
type workerContextKey string

8
internal/manager/api_impl/worker_mgt.go
View File

@ -6,11 +6,11 @@ import (
"errors"
"net/http"
"git.blender.org/flamenco/internal/manager/persistence"
"git.blender.org/flamenco/internal/manager/webupdates"
"git.blender.org/flamenco/internal/uuid"
"git.blender.org/flamenco/pkg/api"
"github.com/labstack/echo/v4"
"projects.blender.org/studio/flamenco/internal/manager/persistence"
"projects.blender.org/studio/flamenco/internal/manager/webupdates"
"projects.blender.org/studio/flamenco/internal/uuid"
"projects.blender.org/studio/flamenco/pkg/api"
)
func (f *Flamenco) FetchWorkers(e echo.Context) error {

4
internal/manager/api_impl/worker_mgt_test.go
View File

@ -12,8 +12,8 @@ import (
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"git.blender.org/flamenco/internal/manager/persistence"
"git.blender.org/flamenco/pkg/api"
"projects.blender.org/studio/flamenco/internal/manager/persistence"
"projects.blender.org/studio/flamenco/pkg/api"
)
func TestFetchWorkers(t *testing.T) {

11
internal/manager/api_impl/worker_sleep_schedule.go
View File

@ -6,10 +6,10 @@ import (
"errors"
"net/http"
"git.blender.org/flamenco/internal/manager/persistence"
"git.blender.org/flamenco/internal/uuid"
"git.blender.org/flamenco/pkg/api"
"github.com/labstack/echo/v4"
"projects.blender.org/studio/flamenco/internal/manager/persistence"
"projects.blender.org/studio/flamenco/internal/uuid"
"projects.blender.org/studio/flamenco/pkg/api"
)
func (f *Flamenco) FetchWorkerSleepSchedule(e echo.Context, workerUUID string) error {
@ -65,11 +65,11 @@ func (f *Flamenco) SetWorkerSleepSchedule(e echo.Context, workerUUID string) err
DaysOfWeek: schedule.DaysOfWeek,
}
if err := dbSchedule.StartTime.Scan(schedule.StartTime); err != nil {
logger.Warn().Err(err).Msg("bad request received, cannot parse schedule start time")
logger.Warn().Interface("schedule", schedule).Err(err).Msg("bad request received, cannot parse schedule start time")
return sendAPIError(e, http.StatusBadRequest, "invalid format for schedule start time")
}
if err := dbSchedule.EndTime.Scan(schedule.EndTime); err != nil {
logger.Warn().Err(err).Msg("bad request received, cannot parse schedule end time")
logger.Warn().Interface("schedule", schedule).Err(err).Msg("bad request received, cannot parse schedule end time")
return sendAPIError(e, http.StatusBadRequest, "invalid format for schedule end time")
}
@ -84,6 +84,5 @@ func (f *Flamenco) SetWorkerSleepSchedule(e echo.Context, workerUUID string) err
return sendAPIError(e, http.StatusInternalServerError, "error fetching sleep schedule: %v", err)
}
logger.Info().Interface("schedule", schedule).Msg("worker sleep schedule updated")
return e.NoContent(http.StatusNoContent)
}

6
internal/manager/api_impl/worker_task_updates.go
View File

@ -12,9 +12,9 @@ import (
"github.com/labstack/echo/v4"
"github.com/rs/zerolog"
"git.blender.org/flamenco/internal/manager/persistence"
"git.blender.org/flamenco/internal/uuid"
"git.blender.org/flamenco/pkg/api"
"projects.blender.org/studio/flamenco/internal/manager/persistence"
"projects.blender.org/studio/flamenco/internal/uuid"
"projects.blender.org/studio/flamenco/pkg/api"
)
func (f *Flamenco) TaskUpdate(e echo.Context, taskID string) error {

6
internal/manager/api_impl/worker_task_updates_test.go
View File

@ -9,9 +9,9 @@ import (
"github.com/golang/mock/gomock"
"github.com/stretchr/testify/assert"
"git.blender.org/flamenco/internal/manager/config"
"git.blender.org/flamenco/internal/manager/persistence"
"git.blender.org/flamenco/pkg/api"
"projects.blender.org/studio/flamenco/internal/manager/config"
"projects.blender.org/studio/flamenco/internal/manager/persistence"
"projects.blender.org/studio/flamenco/pkg/api"
)
func TestTaskUpdate(t *testing.T) {

12
internal/manager/api_impl/workers.go
View File

@ -13,12 +13,12 @@ import (
"github.com/labstack/echo/v4"
"github.com/rs/zerolog"
"git.blender.org/flamenco/internal/manager/last_rendered"
"git.blender.org/flamenco/internal/manager/persistence"
"git.blender.org/flamenco/internal/manager/task_state_machine"
"git.blender.org/flamenco/internal/manager/webupdates"
"git.blender.org/flamenco/internal/uuid"
"git.blender.org/flamenco/pkg/api"
"projects.blender.org/studio/flamenco/internal/manager/last_rendered"
"projects.blender.org/studio/flamenco/internal/manager/persistence"
"projects.blender.org/studio/flamenco/internal/manager/task_state_machine"
"projects.blender.org/studio/flamenco/internal/manager/webupdates"
"projects.blender.org/studio/flamenco/internal/uuid"
"projects.blender.org/studio/flamenco/pkg/api"
)
// rememberableWorkerStates contains those worker statuses that should be

8
internal/manager/api_impl/workers_test.go
View File

@ -13,10 +13,10 @@ import (
"github.com/labstack/echo/v4"
"github.com/stretchr/testify/assert"
"git.blender.org/flamenco/internal/manager/config"
"git.blender.org/flamenco/internal/manager/last_rendered"
"git.blender.org/flamenco/internal/manager/persistence"
"git.blender.org/flamenco/pkg/api"
"projects.blender.org/studio/flamenco/internal/manager/config"
"projects.blender.org/studio/flamenco/internal/manager/last_rendered"
"projects.blender.org/studio/flamenco/internal/manager/persistence"
"projects.blender.org/studio/flamenco/pkg/api"
)
func TestTaskScheduleHappy(t *testing.T) {

64
internal/manager/config/config.go
View File

@ -20,9 +20,9 @@ import (
"github.com/rs/zerolog/log"
yaml "gopkg.in/yaml.v2"
"git.blender.org/flamenco/internal/appinfo"
"git.blender.org/flamenco/pkg/crosspath"
shaman_config "git.blender.org/flamenco/pkg/shaman/config"
"projects.blender.org/studio/flamenco/internal/appinfo"
"projects.blender.org/studio/flamenco/pkg/crosspath"
shaman_config "projects.blender.org/studio/flamenco/pkg/shaman/config"
)
// configFilename is used to specify where flamenco will write its config file.
@ -73,8 +73,11 @@ type Base struct {
Meta ConfMeta `yaml:"_meta"`
ManagerName string `yaml:"manager_name"`
DatabaseDSN string `yaml:"database"`
Listen string `yaml:"listen"`
DatabaseDSN string `yaml:"database"`
DBIntegrityCheck time.Duration `yaml:"database_check_period"`
Listen string `yaml:"listen"`
SSDPDiscovery bool `yaml:"autodiscoverable"`
@ -469,57 +472,6 @@ func (c *Conf) ExpandVariables(inputChannel <-chan string, outputChannel chan<-
}
}
// ConvertTwoWayVariables converts the value of a variable with "{variable
// name}", but only for two-way variables. The function iterates over all
// strings provided by the input channel, and sends the expanded result into the
// output channel. It will return when the input channel is closed.
func (c *Conf) ConvertTwoWayVariables(inputChannel <-chan string, outputChannel chan<- string,
audience VariableAudience, platform VariablePlatform) {
// Get the variables for the given audience & platform.
twoWayVars := c.GetTwoWayVariables(audience, platform)
if len(twoWayVars) == 0 {
log.Debug().
Str("audience", string(audience)).
Str("platform", string(platform)).
Msg("no two-way variables defined for this platform given this audience")
}
doValueReplacement := func(valueToConvert string) string {
for varName, varValue := range twoWayVars {
if !isValueMatch(valueToConvert, varValue) {
continue
}
valueToConvert = fmt.Sprintf("{%s}%s", varName, valueToConvert[len(varValue):])
}
return valueToConvert
}
for valueToExpand := range inputChannel {
outputChannel <- doValueReplacement(valueToExpand)
}
}
// isValueMatch returns whether `valueToMatch` starts with `variableValue`.
// When `variableValue` is a Windows path (with backslash separators), it is
// also tested with forward slashes against `valueToMatch`.
func isValueMatch(valueToMatch, variableValue string) bool {
if strings.HasPrefix(valueToMatch, variableValue) {
return true
}
// If the variable value has a backslash, assume it is a Windows path.
// Convert it to slash notation just to see if that would provide a
// match.
if strings.ContainsRune(variableValue, '\\') {
slashedValue := crosspath.ToSlash(variableValue)
return strings.HasPrefix(valueToMatch, slashedValue)
}
return false
}
// getVariables returns the variable values for this (audience, platform) combination.
// If no variables are found, just returns an empty map. If a value is defined
// for both the "all" platform and specifically the given platform, the specific

31
internal/manager/config/config_test.go
View File

@ -17,31 +17,8 @@ func TestVariablesWithBackslashes(t *testing.T) {
assert.Equal(t, expectSingle, vars["single-backslash"]["blender"])
assert.Equal(t, expectDouble, vars["double-backslash"]["blender"])
assert.Equal(t, expectSingle, vars["quoted-double-backslash"]["blender"])
}
func TestReplaceTwowayVariables(t *testing.T) {
c := DefaultConfig(func(c *Conf) {
c.Variables["shared"] = Variable{
IsTwoWay: true,
Values: []VariableValue{
{Value: "/shared/flamenco", Platform: VariablePlatformLinux},
{Value: `Y:\shared\flamenco`, Platform: VariablePlatformWindows},
},
}
})
feeder := make(chan string, 2)
receiver := make(chan string, 2)
feeder <- `Y:\shared\flamenco\shot\file.blend`
// This is the real reason for this test: forward slashes in the path should
// still be matched to the backslashes in the variable value.
feeder <- `Y:/shared/flamenco/shot/file.blend`
close(feeder)
c.ConvertTwoWayVariables(feeder, receiver, VariableAudienceUsers, VariablePlatformWindows)
assert.Equal(t, `{shared}\shot\file.blend`, <-receiver)
assert.Equal(t, `{shared}/shot/file.blend`, <-receiver)
assert.Equal(t, `C:\Downloads\tab\newline.exe`, vars["single-backslash-common-escapechar"]["blender"])
assert.Equal(t, `C:\Downloads\blender-1.0\`, vars["single-backslash-trailing"]["blender"])
assert.Equal(t, `F:\`, vars["single-backslash-drive-only"]["blender"])
}

5
internal/manager/config/defaults.go
View File

@ -4,7 +4,7 @@ import (
"runtime"
"time"
shaman_config "git.blender.org/flamenco/pkg/shaman/config"
shaman_config "projects.blender.org/studio/flamenco/pkg/shaman/config"
)
// SPDX-License-Identifier: GPL-3.0-or-later
@ -16,10 +16,11 @@ var defaultConfig = Conf{
Base: Base{
Meta: ConfMeta{Version: latestConfigVersion},
ManagerName: "Flamenco Manager",
ManagerName: "Flamenco",
Listen: ":8080",
// ListenHTTPS: ":8433",
DatabaseDSN: "flamenco-manager.sqlite",
DBIntegrityCheck: 1 * time.Hour,
SSDPDiscovery: true,
LocalManagerStoragePath: "./flamenco-manager-storage",
SharedStoragePath: "", // Empty string means "first run", and should trigger the config setup assistant.

10
internal/manager/config/service.go
View File

@ -70,12 +70,12 @@ func (s *Service) Save() error {
return nil
}
// Expose some functions on Conf here, for easier mocking of functionality via interfaces.
func (s *Service) ExpandVariables(inputChannel <-chan string, outputChannel chan<- string, audience VariableAudience, platform VariablePlatform) {
s.config.ExpandVariables(inputChannel, outputChannel, audience, platform)
// Expose some functions of Conf here, for easier mocking of functionality via interfaces.
func (s *Service) NewVariableExpander(audience VariableAudience, platform VariablePlatform) *VariableExpander {
return s.config.NewVariableExpander(audience, platform)
}
func (s *Service) ConvertTwoWayVariables(inputChannel <-chan string, outputChannel chan<- string, audience VariableAudience, platform VariablePlatform) {
s.config.ConvertTwoWayVariables(inputChannel, outputChannel, audience, platform)
func (s *Service) NewVariableToValueConverter(audience VariableAudience, platform VariablePlatform) *ValueToVariableReplacer {
return s.config.NewVariableToValueConverter(audience, platform)
}
func (s *Service) ResolveVariables(audience VariableAudience, platform VariablePlatform) map[string]ResolvedVariable {
return s.config.ResolveVariables(audience, platform)

2
internal/manager/config/settings_test.go
View File

@ -6,8 +6,8 @@ import (
"sync"
"testing"
"git.blender.org/flamenco/pkg/crosspath"
"github.com/stretchr/testify/assert"
"projects.blender.org/studio/flamenco/pkg/crosspath"
)
func TestDefaultSettings(t *testing.T) {

6
internal/manager/config/test_files/config_with_backslashes.yaml
View File

@ -22,3 +22,9 @@ variables:
value: C:\\Downloads\\blender-1.0\\blender.exe
- platform: quoted-double-backslash
value: "C:\\Downloads\\blender-1.0\\blender.exe"
- platform: single-backslash-common-escapechar
value: C:\Downloads\tab\newline.exe
- platform: single-backslash-trailing
value: C:\Downloads\blender-1.0\
- platform: single-backslash-drive-only
value: F:\

155
internal/manager/config/variables.go Normal file
View File

@ -0,0 +1,155 @@
package config
import (
"fmt"
"strings"
"github.com/rs/zerolog/log"
"projects.blender.org/studio/flamenco/pkg/crosspath"
)
type ValueToVariableReplacer struct {
twoWayVars map[string]string // Mapping from variable name to value.
}
// VariableExpander expands variables and applies two-way variable replacement to the values.
type VariableExpander struct {
oneWayVars map[string]string // Mapping from variable name to value.
managerTwoWayVars map[string]string // Mapping from variable name to value for the Manager platform.
targetTwoWayVars map[string]string // Mapping from variable name to value for the target platform.
targetPlatform VariablePlatform
}
// NewVariableToValueConverter returns a ValueToVariableReplacer for the given audience & platform.
func (c *Conf) NewVariableToValueConverter(audience VariableAudience, platform VariablePlatform) *ValueToVariableReplacer {
// Get the variables for the given audience & platform.
twoWayVars := c.GetTwoWayVariables(audience, platform)
if len(twoWayVars) == 0 {
log.Debug().
Str("audience", string(audience)).
Str("platform", string(platform)).
Msg("no two-way variables defined for this platform given this audience")
}
return &ValueToVariableReplacer{
twoWayVars: twoWayVars,
}
}
// NewVariableExpander returns a new VariableExpander for the given audience & platform.
func (c *Conf) NewVariableExpander(audience VariableAudience, platform VariablePlatform) *VariableExpander {
// Get the variables for the given audience & platform.
varsForPlatform := c.getVariables(audience, platform)
if len(varsForPlatform) == 0 {
log.Warn().
Str("audience", string(audience)).
Str("platform", string(platform)).
Msg("no variables defined for this platform given this audience")
}
return &VariableExpander{
oneWayVars: varsForPlatform,
managerTwoWayVars: c.GetTwoWayVariables(audience, c.currentGOOS),
targetTwoWayVars: c.GetTwoWayVariables(audience, platform),
targetPlatform: platform,
}
}
// ValueToVariableReplacer replaces any variable values it recognises in
// valueToConvert to the actual variable. For example, `/path/to/file.blend` can
// be changed to `{my_storage}/file.blend`.
func (vvc *ValueToVariableReplacer) Replace(valueToConvert string) string {
result := valueToConvert
for varName, varValue := range vvc.twoWayVars {
if !isValueMatch(result, varValue) {
continue
}
result = vvc.join(varName, result[len(varValue):])
}
log.Debug().
Str("from", valueToConvert).
Str("to", result).
Msg("first step of two-way variable replacement")
return result
}
func (vvc *ValueToVariableReplacer) join(varName, value string) string {
return fmt.Sprintf("{%s}%s", varName, value)
}
// isValueMatch returns whether `valueToMatch` starts with `variableValue`.
// When `variableValue` is a Windows path (with backslash separators), it is
// also tested with forward slashes against `valueToMatch`.
func isValueMatch(valueToMatch, variableValue string) bool {
if strings.HasPrefix(valueToMatch, variableValue) {
return true
}
// If the variable value has a backslash, assume it is a Windows path.
// Convert it to slash notation just to see if that would provide a
// match.
if strings.ContainsRune(variableValue, '\\') {
slashedValue := crosspath.ToSlash(variableValue)
return strings.HasPrefix(valueToMatch, slashedValue)
}
return false
}
// Replace converts "{variable name}" to the value that belongs to the audience and platform.
func (ve *VariableExpander) Expand(valueToExpand string) string {
expanded := valueToExpand
// Expand variables from {varname} to their value for the target platform.
for varname, varvalue := range ve.oneWayVars {
placeholder := fmt.Sprintf("{%s}", varname)
expanded = strings.Replace(expanded, placeholder, varvalue, -1)
}
// Go through the two-way variables, to make sure that the result of
// expanding variables gets the two-way variables applied as well. This is
// necessary to make implicitly-defined variable, which are only defined for
// the Manager's platform, usable for the target platform.
//
// Practically, this replaces "value for the Manager platform" with "value
// for the target platform".
isPathValue := false
for varname, managerValue := range ve.managerTwoWayVars {
targetValue, ok := ve.targetTwoWayVars[varname]
if !ok {
continue
}
if !isValueMatch(expanded, managerValue) {
continue
}
expanded = ve.join(targetValue, expanded[len(managerValue):], ve.targetPlatform)
// Since two-way variables are meant for path replacement, we know this
// should be a path.
isPathValue = true
}
if isPathValue {
expanded = crosspath.ToPlatform(expanded, string(ve.targetPlatform))
}
return expanded
}
func (ve *VariableExpander) join(valueFromVariable, suffix string, platform VariablePlatform) string {
result := valueFromVariable + suffix
if platform == VariablePlatformWindows {
// 'result' may now be of the form `F:some\path\to\file`, where `F:` comes
// from `valueFromVariable` and the rest is the suffix. This is not an
// absolute path, and needs a separator between the drive letter and the
// rest of the path.
return crosspath.EnsureDriveAbsolute(result)
}
return result
}

26
internal/manager/config/variables_test.go Normal file
View File

@ -0,0 +1,26 @@
package config
import (
"testing"
"github.com/stretchr/testify/assert"
)
func TestReplaceTwowayVariables(t *testing.T) {
c := DefaultConfig(func(c *Conf) {
c.Variables["shared"] = Variable{
IsTwoWay: true,
Values: []VariableValue{
{Value: "/shared/flamenco", Platform: VariablePlatformLinux},
{Value: `Y:\shared\flamenco`, Platform: VariablePlatformWindows},
},
}
})
replacer := c.NewVariableToValueConverter(VariableAudienceUsers, VariablePlatformWindows)
// This is the real reason for this test: forward slashes in the path should
// still be matched to the backslashes in the variable value.
assert.Equal(t, `{shared}\shot\file.blend`, replacer.Replace(`Y:\shared\flamenco\shot\file.blend`))
assert.Equal(t, `{shared}/shot/file.blend`, replacer.Replace(`Y:/shared/flamenco/shot/file.blend`))
}

4
internal/manager/job_compilers/author.go
View File

@ -10,8 +10,8 @@ import (
"github.com/dop251/goja"
"github.com/rs/zerolog/log"
"git.blender.org/flamenco/internal/uuid"
"git.blender.org/flamenco/pkg/api"
"projects.blender.org/studio/flamenco/internal/uuid"
"projects.blender.org/studio/flamenco/pkg/api"
)
// Author allows scripts to author tasks and commands.

4
internal/manager/job_compilers/job_compilers.go
View File

@ -19,8 +19,8 @@ import (
"github.com/dop251/goja_nodejs/require"
"github.com/rs/zerolog/log"
"git.blender.org/flamenco/internal/uuid"
"git.blender.org/flamenco/pkg/api"
"projects.blender.org/studio/flamenco/internal/uuid"
"projects.blender.org/studio/flamenco/pkg/api"
)
var ErrJobTypeUnknown = errors.New("job type unknown")

2
internal/manager/job_compilers/job_compilers_test.go
View File

@ -13,7 +13,7 @@ import (
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"git.blender.org/flamenco/pkg/api"
"projects.blender.org/studio/flamenco/pkg/api"
)
// The example job is expected to result in these arguments for FFmpeg.

2
internal/manager/job_compilers/js_path.go
View File

@ -3,9 +3,9 @@ package job_compilers
// SPDX-License-Identifier: GPL-3.0-or-later
import (
"git.blender.org/flamenco/pkg/crosspath"
"github.com/dop251/goja"
"github.com/rs/zerolog/log"
"projects.blender.org/studio/flamenco/pkg/crosspath"
)
// PathModule provides file path manipulation functions by wrapping Go's `path`.

12
internal/manager/job_deleter/interfaces.go
View File

@ -5,15 +5,15 @@ package job_deleter
import (
"context"
"git.blender.org/flamenco/internal/manager/local_storage"
"git.blender.org/flamenco/internal/manager/persistence"
"git.blender.org/flamenco/internal/manager/webupdates"
"git.blender.org/flamenco/pkg/api"
"git.blender.org/flamenco/pkg/shaman"
"projects.blender.org/studio/flamenco/internal/manager/local_storage"
"projects.blender.org/studio/flamenco/internal/manager/persistence"
"projects.blender.org/studio/flamenco/internal/manager/webupdates"
"projects.blender.org/studio/flamenco/pkg/api"
"projects.blender.org/studio/flamenco/pkg/shaman"
)
// Generate mock implementations of these interfaces.
//go:generate go run github.com/golang/mock/mockgen -destination mocks/interfaces_mock.gen.go -package mocks git.blender.org/flamenco/internal/manager/job_deleter PersistenceService,Storage,ChangeBroadcaster,Shaman
//go:generate go run github.com/golang/mock/mockgen -destination mocks/interfaces_mock.gen.go -package mocks projects.blender.org/studio/flamenco/internal/manager/job_deleter PersistenceService,Storage,ChangeBroadcaster,Shaman
type PersistenceService interface {
FetchJob(ctx context.Context, jobUUID string) (*persistence.Job, error)

8
internal/manager/job_deleter/job_deleter.go
View File

@ -16,12 +16,12 @@ import (
"fmt"
"time"
"git.blender.org/flamenco/internal/manager/persistence"
"git.blender.org/flamenco/internal/manager/webupdates"
"git.blender.org/flamenco/pkg/api"
"git.blender.org/flamenco/pkg/shaman"
"github.com/rs/zerolog"
"github.com/rs/zerolog/log"
"projects.blender.org/studio/flamenco/internal/manager/persistence"
"projects.blender.org/studio/flamenco/internal/manager/webupdates"
"projects.blender.org/studio/flamenco/pkg/api"
"projects.blender.org/studio/flamenco/pkg/shaman"
)
// jobDeletionQueueSize determines how many job deletion requests can be kept in

6
internal/manager/job_deleter/job_deleter_test.go
View File

@ -7,11 +7,11 @@ import (
"errors"
"testing"
"git.blender.org/flamenco/internal/manager/job_deleter/mocks"
"git.blender.org/flamenco/internal/manager/persistence"
"git.blender.org/flamenco/pkg/shaman"
"github.com/golang/mock/gomock"
"github.com/stretchr/testify/assert"
"projects.blender.org/studio/flamenco/internal/manager/job_deleter/mocks"
"projects.blender.org/studio/flamenco/internal/manager/persistence"
"projects.blender.org/studio/flamenco/pkg/shaman"
)
type JobDeleterMocks struct {

6
internal/manager/job_deleter/mocks/interfaces_mock.gen.go generated
View File

@ -1,5 +1,5 @@
// Code generated by MockGen. DO NOT EDIT.
// Source: git.blender.org/flamenco/internal/manager/job_deleter (interfaces: PersistenceService,Storage,ChangeBroadcaster,Shaman)
// Source: projects.blender.org/studio/flamenco/internal/manager/job_deleter (interfaces: PersistenceService,Storage,ChangeBroadcaster,Shaman)
// Package mocks is a generated GoMock package.
package mocks
@ -8,9 +8,9 @@ import (
context "context"
reflect "reflect"
persistence "git.blender.org/flamenco/internal/manager/persistence"
api "git.blender.org/flamenco/pkg/api"
gomock "github.com/golang/mock/gomock"
persistence "projects.blender.org/studio/flamenco/internal/manager/persistence"
api "projects.blender.org/studio/flamenco/pkg/api"
)
// MockPersistenceService is a mock of PersistenceService interface.

2
internal/manager/last_rendered/last_rendered_test.go
View File

@ -9,8 +9,8 @@ import (
"path/filepath"
"testing"
"git.blender.org/flamenco/internal/manager/local_storage"
"github.com/stretchr/testify/assert"
"projects.blender.org/studio/flamenco/internal/manager/local_storage"
)
func TestNew(t *testing.T) {

2
internal/manager/local_storage/local_storage.go
View File

@ -9,8 +9,8 @@ import (
"os"
"path/filepath"
"git.blender.org/flamenco/pkg/crosspath"
"github.com/rs/zerolog/log"
"projects.blender.org/studio/flamenco/pkg/crosspath"
)
type StorageInfo struct {

4
internal/manager/persistence/db.go
View File

@ -11,12 +11,10 @@ import (
"github.com/rs/zerolog/log"
"gorm.io/gorm"
// sqlite "git.blender.org/flamenco/pkg/gorm-modernc-sqlite"
// sqlite "projects.blender.org/studio/flamenco/pkg/gorm-modernc-sqlite"
"github.com/glebarez/sqlite"
)
const checkPeriod = 1 * time.Hour
// DB provides the database interface.
type DB struct {
gormDB *gorm.DB

54
internal/manager/persistence/integrity.go
View File

@ -12,7 +12,9 @@ import (
var ErrIntegrity = errors.New("database integrity check failed")
const integrityCheckTimeout = 2 * time.Second
const (
integrityCheckTimeout = 2 * time.Second
)
type PragmaIntegrityCheckResult struct {
Description string `gorm:"column:integrity_check"`
@ -25,6 +27,38 @@ type PragmaForeignKeyCheckResult struct {
FKID int `gorm:"column:fkid"`
}
// PeriodicIntegrityCheck periodically checks the database integrity.
// This function only returns when the context is done.
func (db *DB) PeriodicIntegrityCheck(
ctx context.Context,
period time.Duration,
onErrorCallback func(),
) {
if period == 0 {
log.Info().Msg("database: periodic integrity check disabled")
return
}
log.Info().
Stringer("period", period).
Msg("database: periodic integrity check starting")
defer log.Debug().Msg("database: periodic integrity check stopping")
for {
select {
case <-ctx.Done():
return
case <-time.After(period):
}
ok := db.performIntegrityCheck(ctx)
if !ok {
log.Error().Msg("database: periodic integrity check failed")
onErrorCallback()
}
}
}
// performIntegrityCheck uses a few 'pragma' SQL statements to do some integrity checking.
// Returns true on OK, false if there was an issue. Issues are always logged.
func (db *DB) performIntegrityCheck(ctx context.Context) (ok bool) {
@ -50,26 +84,26 @@ func (db *DB) pragmaIntegrityCheck(ctx context.Context) (ok bool) {
Raw("PRAGMA integrity_check").
Scan(&issues)
if tx.Error != nil {
log.Error().Err(tx.Error).Msg("database error checking integrity")
log.Error().Err(tx.Error).Msg("database: error checking integrity")
return false
}
switch len(issues) {
case 0:
log.Warn().Msg("database integrity check returned nothing, expected explicit 'ok'; treating as an implicit 'ok'")
log.Warn().Msg("database: integrity check returned nothing, expected explicit 'ok'; treating as an implicit 'ok'")
return true
case 1:
if issues[0].Description == "ok" {
log.Debug().Msg("database integrity check ok")
log.Debug().Msg("database: integrity check ok")
return true
}
}
log.Error().Int("num_issues", len(issues)).Msg("database integrity check failed")
log.Error().Int("num_issues", len(issues)).Msg("database: integrity check failed")
for _, issue := range issues {
log.Error().
Str("description", issue.Description).
Msg("database integrity check failure")
Msg("database: integrity check failure")
}
return false
@ -91,23 +125,23 @@ func (db *DB) pragmaForeignKeyCheck(ctx context.Context) (ok bool) {
Raw("PRAGMA foreign_key_check").
Scan(&issues)
if tx.Error != nil {
log.Error().Err(tx.Error).Msg("database error checking foreign keys")
log.Error().Err(tx.Error).Msg("database: error checking foreign keys")
return false
}
if len(issues) == 0 {
log.Debug().Msg("database foreign key check ok")
log.Debug().Msg("database: foreign key check ok")
return true
}
log.Error().Int("num_issues", len(issues)).Msg("database foreign key check failed")
log.Error().Int("num_issues", len(issues)).Msg("database: foreign key check failed")
for _, issue := range issues {
log.Error().
Str("table", issue.Table).
Int("rowid", issue.RowID).
Str("parent", issue.Parent).
Int("fkid", issue.FKID).
Msg("database foreign key relation missing")
Msg("database: foreign key relation missing")
}
return false

4
internal/manager/persistence/jobs.go
View File

@ -15,8 +15,8 @@ import (
"gorm.io/gorm"
"gorm.io/gorm/clause"
"git.blender.org/flamenco/internal/manager/job_compilers"
"git.blender.org/flamenco/pkg/api"
"projects.blender.org/studio/flamenco/internal/manager/job_compilers"
"projects.blender.org/studio/flamenco/pkg/api"
)
type Job struct {

2
internal/manager/persistence/jobs_query.go
View File

@ -5,8 +5,8 @@ import (
"context"
"strings"
"git.blender.org/flamenco/pkg/api"
"github.com/rs/zerolog/log"
"projects.blender.org/studio/flamenco/pkg/api"
)
func (db *DB) QueryJobs(ctx context.Context, apiQ api.JobsQuery) ([]*Job, error) {

6
internal/manager/persistence/jobs_query_test.go
View File

@ -8,9 +8,9 @@ import (
"github.com/stretchr/testify/assert"
"git.blender.org/flamenco/internal/manager/job_compilers"
"git.blender.org/flamenco/internal/uuid"
"git.blender.org/flamenco/pkg/api"
"projects.blender.org/studio/flamenco/internal/manager/job_compilers"
"projects.blender.org/studio/flamenco/internal/uuid"
"projects.blender.org/studio/flamenco/pkg/api"
)
func TestSimpleQuery(t *testing.T) {

6
internal/manager/persistence/jobs_test.go
View File

@ -13,9 +13,9 @@ import (
"github.com/stretchr/testify/require"
"golang.org/x/net/context"
"git.blender.org/flamenco/internal/manager/job_compilers"
"git.blender.org/flamenco/internal/uuid"
"git.blender.org/flamenco/pkg/api"
"projects.blender.org/studio/flamenco/internal/manager/job_compilers"
"projects.blender.org/studio/flamenco/internal/uuid"
"projects.blender.org/studio/flamenco/pkg/api"
)
func TestStoreAuthoredJob(t *testing.T) {

2
internal/manager/persistence/task_scheduler.go
View File

@ -9,7 +9,7 @@ import (
"github.com/rs/zerolog/log"
"gorm.io/gorm"
"git.blender.org/flamenco/pkg/api"
"projects.blender.org/studio/flamenco/pkg/api"
)
var (

6
internal/manager/persistence/task_scheduler_test.go
View File

@ -10,9 +10,9 @@ import (
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"git.blender.org/flamenco/internal/manager/job_compilers"
"git.blender.org/flamenco/internal/uuid"
"git.blender.org/flamenco/pkg/api"
"projects.blender.org/studio/flamenco/internal/manager/job_compilers"
"projects.blender.org/studio/flamenco/internal/uuid"
"projects.blender.org/studio/flamenco/pkg/api"
)
const schedulerTestTimeout = 100 * time.Millisecond

4
internal/manager/persistence/test_support.go
View File

@ -10,13 +10,13 @@ import (
"testing"
"time"
"git.blender.org/flamenco/internal/uuid"
"git.blender.org/flamenco/pkg/api"
"github.com/glebarez/sqlite"
"github.com/rs/zerolog"
"github.com/rs/zerolog/log"
"github.com/stretchr/testify/require"
"gorm.io/gorm"
"projects.blender.org/studio/flamenco/internal/uuid"
"projects.blender.org/studio/flamenco/pkg/api"
)
// Change this to a filename if you want to run a single test and inspect the

2
internal/manager/persistence/timeout.go
View File

@ -6,7 +6,7 @@ import (
"context"
"time"
"git.blender.org/flamenco/pkg/api"
"projects.blender.org/studio/flamenco/pkg/api"
)
// This file contains functions for dealing with task/worker timeouts. Not database timeouts.

2
internal/manager/persistence/timeout_test.go
View File

@ -4,8 +4,8 @@ import (
"testing"
"time"
"git.blender.org/flamenco/pkg/api"
"github.com/stretchr/testify/assert"
"projects.blender.org/studio/flamenco/pkg/api"
)
// SPDX-License-Identifier: GPL-3.0-or-later

4
internal/manager/persistence/worker_sleep_schedule_test.go
View File

@ -6,10 +6,10 @@ import (
"testing"
"time"
"git.blender.org/flamenco/internal/uuid"
"git.blender.org/flamenco/pkg/api"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"projects.blender.org/studio/flamenco/internal/uuid"
"projects.blender.org/studio/flamenco/pkg/api"
)
func TestFetchWorkerSleepSchedule(t *testing.T) {

2
internal/manager/persistence/worker_tag_test.go
View File

@ -6,9 +6,9 @@ import (
"testing"
"time"
"git.blender.org/flamenco/internal/uuid"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"projects.blender.org/studio/flamenco/internal/uuid"
)
func TestCreateFetchTag(t *testing.T) {

2
internal/manager/persistence/workers.go
View File

@ -8,8 +8,8 @@ import (
"strings"
"time"
"git.blender.org/flamenco/pkg/api"
"gorm.io/gorm"
"projects.blender.org/studio/flamenco/pkg/api"
)
type Worker struct {

4
internal/manager/persistence/workers_test.go
View File

@ -10,8 +10,8 @@ import (
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"git.blender.org/flamenco/internal/uuid"
"git.blender.org/flamenco/pkg/api"
"projects.blender.org/studio/flamenco/internal/uuid"
"projects.blender.org/studio/flamenco/pkg/api"
)
func TestCreateFetchWorker(t *testing.T) {

4
internal/manager/sleep_scheduler/calculations.go
View File

@ -6,8 +6,8 @@ import (
"strings"
"time"
"git.blender.org/flamenco/internal/manager/persistence"
"git.blender.org/flamenco/pkg/api"
"projects.blender.org/studio/flamenco/internal/manager/persistence"
"projects.blender.org/studio/flamenco/pkg/api"
)
// scheduledWorkerStatus returns the expected worker status at the given date/time.

4
internal/manager/sleep_scheduler/calculations_test.go
View File

@ -7,8 +7,8 @@ import (
"github.com/stretchr/testify/assert"
"git.blender.org/flamenco/internal/manager/persistence"
"git.blender.org/flamenco/pkg/api"
"projects.blender.org/studio/flamenco/internal/manager/persistence"
"projects.blender.org/studio/flamenco/pkg/api"
)
func TestCalculateNextCheck(t *testing.T) {

8
internal/manager/sleep_scheduler/interfaces.go
View File

@ -5,13 +5,13 @@ package sleep_scheduler
import (
"context"
"git.blender.org/flamenco/internal/manager/persistence"
"git.blender.org/flamenco/internal/manager/webupdates"
"git.blender.org/flamenco/pkg/api"
"projects.blender.org/studio/flamenco/internal/manager/persistence"
"projects.blender.org/studio/flamenco/internal/manager/webupdates"
"projects.blender.org/studio/flamenco/pkg/api"
)
// Generate mock implementations of these interfaces.
//go:generate go run github.com/golang/mock/mockgen -destination mocks/interfaces_mock.gen.go -package mocks git.blender.org/flamenco/internal/manager/sleep_scheduler PersistenceService,ChangeBroadcaster
//go:generate go run github.com/golang/mock/mockgen -destination mocks/interfaces_mock.gen.go -package mocks projects.blender.org/studio/flamenco/internal/manager/sleep_scheduler PersistenceService,ChangeBroadcaster
type PersistenceService interface {
FetchWorkerSleepSchedule(ctx context.Context, workerUUID string) (*persistence.SleepSchedule, error)

6
internal/manager/sleep_scheduler/mocks/interfaces_mock.gen.go generated
View File

@ -1,5 +1,5 @@
// Code generated by MockGen. DO NOT EDIT.
// Source: git.blender.org/flamenco/internal/manager/sleep_scheduler (interfaces: PersistenceService,ChangeBroadcaster)
// Source: projects.blender.org/studio/flamenco/internal/manager/sleep_scheduler (interfaces: PersistenceService,ChangeBroadcaster)
// Package mocks is a generated GoMock package.
package mocks
@ -8,9 +8,9 @@ import (
context "context"
reflect "reflect"
persistence "git.blender.org/flamenco/internal/manager/persistence"
api "git.blender.org/flamenco/pkg/api"
gomock "github.com/golang/mock/gomock"
persistence "projects.blender.org/studio/flamenco/internal/manager/persistence"
api "projects.blender.org/studio/flamenco/pkg/api"
)
// MockPersistenceService is a mock of PersistenceService interface.

26
internal/manager/sleep_scheduler/sleep_scheduler.go
View File

@ -9,10 +9,11 @@ import (
"time"
"github.com/benbjohnson/clock"
"github.com/rs/zerolog"
"github.com/rs/zerolog/log"
"git.blender.org/flamenco/internal/manager/persistence"
"git.blender.org/flamenco/pkg/api"
"projects.blender.org/studio/flamenco/internal/manager/persistence"
"projects.blender.org/studio/flamenco/pkg/api"
)
// Time period for checking the schedule of every worker.
@ -80,6 +81,11 @@ func (ss *SleepScheduler) SetSchedule(ctx context.Context, workerUUID string, sc
return fmt.Errorf("persisting sleep schedule of worker %s: %w", workerUUID, err)
}
logger := addLoggerFields(zerolog.Ctx(ctx), schedule)
logger.Info().
Str("worker", schedule.Worker.Identifier()).
Msg("sleep scheduler: new schedule for worker")
return ss.ApplySleepSchedule(ctx, schedule)
}
@ -239,3 +245,19 @@ func (ss *SleepScheduler) mayUpdateWorker(worker *persistence.Worker) bool {
shouldSkip := skipWorkersInStatus[worker.Status]
return !shouldSkip
}
func addLoggerFields(logger *zerolog.Logger, schedule *persistence.SleepSchedule) zerolog.Logger {
logCtx := logger.With()
if schedule.Worker != nil {
logCtx = logCtx.Str("worker", schedule.Worker.Identifier())
}
logCtx = logCtx.
Bool("isActive", schedule.IsActive).
Str("daysOfWeek", schedule.DaysOfWeek).
Stringer("startTime", schedule.StartTime).
Stringer("endTime", schedule.EndTime)
return logCtx.Logger()
}

6
internal/manager/sleep_scheduler/sleep_scheduler_test.go
View File

@ -11,9 +11,9 @@ import (
"github.com/golang/mock/gomock"
"github.com/stretchr/testify/assert"
"git.blender.org/flamenco/internal/manager/persistence"
"git.blender.org/flamenco/internal/manager/sleep_scheduler/mocks"
"git.blender.org/flamenco/pkg/api"
"projects.blender.org/studio/flamenco/internal/manager/persistence"
"projects.blender.org/studio/flamenco/internal/manager/sleep_scheduler/mocks"
"projects.blender.org/studio/flamenco/pkg/api"
)
func TestFetchSchedule(t *testing.T) {

4
internal/manager/task_logs/mocks/interfaces_mock.gen.go generated
View File

@ -1,5 +1,5 @@
// Code generated by MockGen. DO NOT EDIT.
// Source: git.blender.org/flamenco/internal/manager/task_logs (interfaces: LocalStorage,ChangeBroadcaster)
// Source: projects.blender.org/studio/flamenco/internal/manager/task_logs (interfaces: LocalStorage,ChangeBroadcaster)
// Package mocks is a generated GoMock package.
package mocks
@ -7,8 +7,8 @@ package mocks
import (
reflect "reflect"
api "git.blender.org/flamenco/pkg/api"
gomock "github.com/golang/mock/gomock"
api "projects.blender.org/studio/flamenco/pkg/api"
)
// MockLocalStorage is a mock of LocalStorage interface.

6
internal/manager/task_logs/task_logs.go
View File

@ -11,10 +11,10 @@ import (
"sync"
"time"
"git.blender.org/flamenco/internal/manager/webupdates"
"git.blender.org/flamenco/pkg/api"
"github.com/benbjohnson/clock"
"github.com/rs/zerolog"
"projects.blender.org/studio/flamenco/internal/manager/webupdates"
"projects.blender.org/studio/flamenco/pkg/api"
)
const (
@ -35,7 +35,7 @@ type Storage struct {
}
// Generate mock implementations of these interfaces.
//go:generate go run github.com/golang/mock/mockgen -destination mocks/interfaces_mock.gen.go -package mocks git.blender.org/flamenco/internal/manager/task_logs LocalStorage,ChangeBroadcaster
//go:generate go run github.com/golang/mock/mockgen -destination mocks/interfaces_mock.gen.go -package mocks projects.blender.org/studio/flamenco/internal/manager/task_logs LocalStorage,ChangeBroadcaster
type LocalStorage interface {
// ForJob returns the absolute directory path for storing job-related files.

2
internal/manager/task_logs/task_logs_test.go
View File

@ -14,12 +14,12 @@ import (
"testing"
"time"
"git.blender.org/flamenco/internal/manager/task_logs/mocks"
"github.com/benbjohnson/clock"
"github.com/golang/mock/gomock"
"github.com/rs/zerolog"
"github.com/rs/zerolog/log"
"github.com/stretchr/testify/assert"
"projects.blender.org/studio/flamenco/internal/manager/task_logs/mocks"
)
func TestLogWriting(t *testing.T) {

10
internal/manager/task_state_machine/interfaces.go
View File

@ -5,15 +5,15 @@ package task_state_machine
import (
"context"
"git.blender.org/flamenco/internal/manager/persistence"
"git.blender.org/flamenco/internal/manager/task_logs"
"git.blender.org/flamenco/internal/manager/webupdates"
"git.blender.org/flamenco/pkg/api"
"github.com/rs/zerolog"
"projects.blender.org/studio/flamenco/internal/manager/persistence"
"projects.blender.org/studio/flamenco/internal/manager/task_logs"
"projects.blender.org/studio/flamenco/internal/manager/webupdates"
"projects.blender.org/studio/flamenco/pkg/api"
)
// Generate mock implementations of these interfaces.
//go:generate go run github.com/golang/mock/mockgen -destination mocks/interfaces_mock.gen.go -package mocks git.blender.org/flamenco/internal/manager/task_state_machine PersistenceService,ChangeBroadcaster,LogStorage
//go:generate go run github.com/golang/mock/mockgen -destination mocks/interfaces_mock.gen.go -package mocks projects.blender.org/studio/flamenco/internal/manager/task_state_machine PersistenceService,ChangeBroadcaster,LogStorage
type PersistenceService interface {
SaveTask(ctx context.Context, task *persistence.Task) error

6
internal/manager/task_state_machine/mocks/interfaces_mock.gen.go generated
View File

@ -1,5 +1,5 @@
// Code generated by MockGen. DO NOT EDIT.
// Source: git.blender.org/flamenco/internal/manager/task_state_machine (interfaces: PersistenceService,ChangeBroadcaster,LogStorage)
// Source: projects.blender.org/studio/flamenco/internal/manager/task_state_machine (interfaces: PersistenceService,ChangeBroadcaster,LogStorage)
// Package mocks is a generated GoMock package.
package mocks
@ -8,10 +8,10 @@ import (
context "context"
reflect "reflect"
persistence "git.blender.org/flamenco/internal/manager/persistence"
api "git.blender.org/flamenco/pkg/api"
gomock "github.com/golang/mock/gomock"
zerolog "github.com/rs/zerolog"
persistence "projects.blender.org/studio/flamenco/internal/manager/persistence"
api "projects.blender.org/studio/flamenco/pkg/api"
)
// MockPersistenceService is a mock of PersistenceService interface.

6
internal/manager/task_state_machine/task_state_machine.go
View File

@ -9,9 +9,9 @@ import (
"github.com/rs/zerolog"
"github.com/rs/zerolog/log"
"git.blender.org/flamenco/internal/manager/persistence"
"git.blender.org/flamenco/internal/manager/webupdates"
"git.blender.org/flamenco/pkg/api"
"projects.blender.org/studio/flamenco/internal/manager/persistence"
"projects.blender.org/studio/flamenco/internal/manager/webupdates"
"projects.blender.org/studio/flamenco/pkg/api"
)
// taskFailJobPercentage is the percentage of a job's tasks that need to fail to

6
internal/manager/task_state_machine/task_state_machine_test.go
View File

@ -11,9 +11,9 @@ import (
"github.com/golang/mock/gomock"
"github.com/stretchr/testify/assert"
"git.blender.org/flamenco/internal/manager/persistence"
"git.blender.org/flamenco/internal/manager/task_state_machine/mocks"
"git.blender.org/flamenco/pkg/api"
"projects.blender.org/studio/flamenco/internal/manager/persistence"
"projects.blender.org/studio/flamenco/internal/manager/task_state_machine/mocks"
"projects.blender.org/studio/flamenco/pkg/api"
)
type StateMachineMocks struct {

2
internal/manager/task_state_machine/task_statuses.go
View File

@ -2,7 +2,7 @@ package task_state_machine
// SPDX-License-Identifier: GPL-3.0-or-later
import "git.blender.org/flamenco/pkg/api"
import "projects.blender.org/studio/flamenco/pkg/api"
var (
// Task statuses that always get requeued when the job is requeueing.

4
internal/manager/task_state_machine/worker_requeue.go
View File

@ -5,9 +5,9 @@ package task_state_machine
import (
"context"
"git.blender.org/flamenco/internal/manager/persistence"
"git.blender.org/flamenco/pkg/api"
"github.com/rs/zerolog/log"
"projects.blender.org/studio/flamenco/internal/manager/persistence"
"projects.blender.org/studio/flamenco/pkg/api"
)
// RequeueActiveTasksOfWorker re-queues all active tasks (should be max one) of this worker.

4
internal/manager/task_state_machine/worker_requeue_test.go
View File

@ -5,10 +5,10 @@ package task_state_machine
import (
"testing"
"git.blender.org/flamenco/internal/manager/persistence"
"git.blender.org/flamenco/pkg/api"
"github.com/golang/mock/gomock"
"github.com/stretchr/testify/assert"
"projects.blender.org/studio/flamenco/internal/manager/persistence"
"projects.blender.org/studio/flamenco/pkg/api"
)
func TestRequeueActiveTasksOfWorker(t *testing.T) {

10
internal/manager/timeout_checker/interfaces.go
View File

@ -6,15 +6,15 @@ import (
"context"
"time"
"git.blender.org/flamenco/internal/manager/persistence"
"git.blender.org/flamenco/internal/manager/task_state_machine"
"git.blender.org/flamenco/internal/manager/webupdates"
"git.blender.org/flamenco/pkg/api"
"github.com/rs/zerolog"
"projects.blender.org/studio/flamenco/internal/manager/persistence"
"projects.blender.org/studio/flamenco/internal/manager/task_state_machine"
"projects.blender.org/studio/flamenco/internal/manager/webupdates"
"projects.blender.org/studio/flamenco/pkg/api"
)
// Generate mock implementations of these interfaces.
//go:generate go run github.com/golang/mock/mockgen -destination mocks/interfaces_mock.gen.go -package mocks git.blender.org/flamenco/internal/manager/timeout_checker PersistenceService,TaskStateMachine,LogStorage,ChangeBroadcaster
//go:generate go run github.com/golang/mock/mockgen -destination mocks/interfaces_mock.gen.go -package mocks projects.blender.org/studio/flamenco/internal/manager/timeout_checker PersistenceService,TaskStateMachine,LogStorage,ChangeBroadcaster
type PersistenceService interface {
FetchTimedOutTasks(ctx context.Context, untouchedSince time.Time) ([]*persistence.Task, error)

6
internal/manager/timeout_checker/mocks/interfaces_mock.gen.go generated
View File

@ -1,5 +1,5 @@
// Code generated by MockGen. DO NOT EDIT.
// Source: git.blender.org/flamenco/internal/manager/timeout_checker (interfaces: PersistenceService,TaskStateMachine,LogStorage,ChangeBroadcaster)
// Source: projects.blender.org/studio/flamenco/internal/manager/timeout_checker (interfaces: PersistenceService,TaskStateMachine,LogStorage,ChangeBroadcaster)
// Package mocks is a generated GoMock package.
package mocks
@ -9,10 +9,10 @@ import (
reflect "reflect"
time "time"
persistence "git.blender.org/flamenco/internal/manager/persistence"
api "git.blender.org/flamenco/pkg/api"
gomock "github.com/golang/mock/gomock"
zerolog "github.com/rs/zerolog"
persistence "projects.blender.org/studio/flamenco/internal/manager/persistence"
api "projects.blender.org/studio/flamenco/pkg/api"
)
// MockPersistenceService is a mock of PersistenceService interface.

4
internal/manager/timeout_checker/tasks.go
View File

@ -10,8 +10,8 @@ import (
"github.com/rs/zerolog"
"github.com/rs/zerolog/log"
"git.blender.org/flamenco/internal/manager/persistence"
"git.blender.org/flamenco/pkg/api"
"projects.blender.org/studio/flamenco/internal/manager/persistence"
"projects.blender.org/studio/flamenco/pkg/api"
)
func (ttc *TimeoutChecker) checkTasks(ctx context.Context) {

4
internal/manager/timeout_checker/tasks_test.go
View File

@ -11,8 +11,8 @@ import (
"github.com/golang/mock/gomock"
"github.com/stretchr/testify/assert"
"git.blender.org/flamenco/internal/manager/persistence"
"git.blender.org/flamenco/pkg/api"
"projects.blender.org/studio/flamenco/internal/manager/persistence"
"projects.blender.org/studio/flamenco/pkg/api"
)
const taskTimeout = 20 * time.Minute

2
internal/manager/timeout_checker/timeout_checker_test.go
View File

@ -12,7 +12,7 @@ import (
"github.com/golang/mock/gomock"
"github.com/stretchr/testify/assert"
"git.blender.org/flamenco/internal/manager/timeout_checker/mocks"
"projects.blender.org/studio/flamenco/internal/manager/timeout_checker/mocks"
)
type TimeoutCheckerMocks struct {

4
internal/manager/timeout_checker/workers.go
View File

@ -5,9 +5,9 @@ package timeout_checker
import (
"context"
"git.blender.org/flamenco/internal/manager/persistence"
"git.blender.org/flamenco/pkg/api"
"github.com/rs/zerolog/log"
"projects.blender.org/studio/flamenco/internal/manager/persistence"
"projects.blender.org/studio/flamenco/pkg/api"
)
func (ttc *TimeoutChecker) checkWorkers(ctx context.Context) {

4
internal/manager/timeout_checker/workers_test.go
View File

@ -6,9 +6,9 @@ import (
"testing"
"time"
"git.blender.org/flamenco/internal/manager/persistence"
"git.blender.org/flamenco/pkg/api"
"github.com/golang/mock/gomock"
"projects.blender.org/studio/flamenco/internal/manager/persistence"
"projects.blender.org/studio/flamenco/pkg/api"
)
const workerTimeout = 20 * time.Minute

4
internal/manager/webupdates/job_updates.go
View File

@ -4,8 +4,8 @@ package webupdates
import (
"github.com/rs/zerolog/log"
"git.blender.org/flamenco/internal/manager/persistence"
"git.blender.org/flamenco/pkg/api"
"projects.blender.org/studio/flamenco/internal/manager/persistence"
"projects.blender.org/studio/flamenco/pkg/api"
)
// NewJobUpdate returns a partial SocketIOJobUpdate struct for the given job.

4
internal/manager/webupdates/sio_rooms.go
View File

@ -6,8 +6,8 @@ import (
gosocketio "github.com/graarh/golang-socketio"
"git.blender.org/flamenco/internal/uuid"
"git.blender.org/flamenco/pkg/api"
"projects.blender.org/studio/flamenco/internal/uuid"
"projects.blender.org/studio/flamenco/pkg/api"
)
// Separate type aliases for room names and event types; it's otherwise too easy

4
internal/manager/webupdates/worker_updates.go
View File

@ -4,8 +4,8 @@ package webupdates
import (
"github.com/rs/zerolog/log"
"git.blender.org/flamenco/internal/manager/persistence"
"git.blender.org/flamenco/pkg/api"
"projects.blender.org/studio/flamenco/internal/manager/persistence"
"projects.blender.org/studio/flamenco/pkg/api"
)
// NewWorkerUpdate returns a partial SocketIOWorkerUpdate struct for the given worker.

2
internal/stresser/fake_config.go
View File

@ -5,7 +5,7 @@ package stresser
import (
"github.com/rs/zerolog/log"
"git.blender.org/flamenco/internal/worker"
"projects.blender.org/studio/flamenco/internal/worker"
)
type FakeConfig struct {

4
internal/stresser/fake_worker.go
View File

@ -11,8 +11,8 @@ import (
"github.com/rs/zerolog/log"
"git.blender.org/flamenco/internal/worker"
"git.blender.org/flamenco/pkg/api"
"projects.blender.org/studio/flamenco/internal/worker"
"projects.blender.org/studio/flamenco/pkg/api"
)
const (

4
internal/stresser/stresser.go
View File

@ -9,9 +9,9 @@ import (
"sync"
"time"
"git.blender.org/flamenco/internal/worker"
"git.blender.org/flamenco/pkg/api"
"github.com/rs/zerolog/log"
"projects.blender.org/studio/flamenco/internal/worker"
"projects.blender.org/studio/flamenco/pkg/api"
)
const (

2
internal/upnp_ssdp/server.go
View File

@ -26,9 +26,9 @@ import (
"net/url"
"path"
"git.blender.org/flamenco/internal/appinfo"
"github.com/fromkeith/gossdp"
"github.com/rs/zerolog"
"projects.blender.org/studio/flamenco/internal/appinfo"
)
// Server advertises services via UPnP/SSDP.

4
internal/worker/autodiscovery.go
View File

@ -11,9 +11,9 @@ import (
"sync"
"time"
"git.blender.org/flamenco/internal/upnp_ssdp"
"git.blender.org/flamenco/pkg/api"
"github.com/rs/zerolog/log"
"projects.blender.org/studio/flamenco/internal/upnp_ssdp"
"projects.blender.org/studio/flamenco/pkg/api"
)
// maybeAutodiscoverManager starts Manager auto-discovery if there is no Manager URL configured yet.

Some files were not shown because too many files have changed in this diff Show More