Flamenco Manager configuration should be in fixed directory #102229

New Issue

Ingo Clemens · 2022-11-02T12:32:13+01:00

Ingo Clemens commented

2022-11-02 12:32:13 +01:00

Flamenco Manager's files should be loaded from a fixed directory, just like Flamenco Worker is doing (docs). This will make it easier to upgrade, and places files where people tend to expect them.

Extend this task with information about which files are moved where, exactly. We could make this apply to the files listed below (just the YAML and SQLite database), but it could potentially also apply to the manager-local storage (i.e. task logs, last-rendered images, etc.)
Change default file location to $HOME/.local/share/flamenco/manager (and similar on other platforms).
Add a CLI option -move-config that moves all the Manager-local files to that location, and updates flamenco-manager.yaml to match.

Original report:

Flamenco manager database files are placed in the user's root folder on macOS

The files in question are

  flamenco-manager.sqlite
  flamenco-manager.yaml

and are located in by default in the root folder of the user:

  Volume/Users/xxx/

This is rather unexpected and appears to be an overlook for the macOS platform.
The expected location for these files is

  ~/Library/Application Support/Flamenco

since the database files for the worker are placed there correctly.

Flamenco Manager's files should be loaded from a fixed directory, just like Flamenco Worker is doing ([docs](https://flamenco.blender.org/usage/worker-configuration/)). This will make it easier to upgrade, and places files where people tend to expect them. - [ ] Extend this task with information about which files are moved where, exactly. We could make this apply to the files listed below (just the YAML and SQLite database), but it could potentially also apply to the manager-local storage (i.e. task logs, last-rendered images, etc.) - [ ] Change default file location to `$HOME/.local/share/flamenco/manager` (and similar on other platforms). - [ ] Add a CLI option `-move-config` that moves all the Manager-local files to that location, and updates `flamenco-manager.yaml` to match. ----------------- Original report: ## Flamenco manager database files are placed in the user's root folder on macOS The files in question are ``` flamenco-manager.sqlite flamenco-manager.yaml ``` and are located in by default in the root folder of the user: ``` Volume/Users/xxx/ ``` This is rather unexpected and appears to be an overlook for the macOS platform. The expected location for these files is ``` ~/Library/Application Support/Flamenco ``` since the database files for the worker are placed there correctly.

Ingo Clemens commented

2022-11-02 12:32:13 +01:00

Added subscriber: @iclemens

Anton Raves commented

2022-11-03 18:26:33 +01:00

Added subscriber: @Memento

Sybren A. Stüvel commented

2022-11-11 16:57:16 +01:00

Added subscriber: @dr.sybren

Sybren A. Stüvel commented

2022-11-11 16:57:16 +01:00

This is as designed, it's saving to the current working directory. I want to revisit that design, though. It'll also make upgrading the Manager easier if it uses a fixed location.

Sybren A. Stüvel changed title from ~~Flamenco manager database files are placed in the user's root folder on macOS~~ to Flamenco Manager configuration should be in fixed directory

2023-02-17 11:19:50 +01:00

Sybren A. Stüvel added

and removed

labels 2023-02-17 11:22:23 +01:00

Sybren A. Stüvel added

and removed

labels 2023-02-17 11:26:23 +01:00

Hubble the Wolverine commented

2024-04-24 07:19:21 +02:00

Hello!

I'm in the process of upstreaming a NixOS module for flamenco. It sets up the package (manager, worker or both) as a systemd service. I was trying to define the service and I got really confused when I was trying to set up the paths.

I have two ideas about file management, focusing specifically on the manager, for consideration:

local_manager_storage_path and `getSuitableStorageRoot`

Currently, local_manager_storage_path uses a directory path relative to the executable, discovered by using the getSuitableStorageRoot function.

From what I understand, part of Flamenco's design philosophy is that it could be used as an portable application. Maybe it could be possible to make local_manager_storage_path bypass the getSuitableStorageRoot logic only if the path is an absolute path. This way, default behavior is preserved, while expanding functionality for other uses.

Go provides a function provided with path/filepath, isAbs(), which can tell if a path is an absolute path on multiple platforms, Unix and Windows both included.

Configuration file location

A lot of application I've seen has an config file path argument -config.

Again, following the portability philosophy, default behavior can be that the configuration file is generated and read in the current working directory, but then people can use -config to set an path to an existing configuration file, if they need it.

Currently, the NixOS module deals with these caveats by setting up an temporary working directory with the configuration file inside it, and by adding "../../../../" in front of the local_manager_storage_path, which will effectively set local_manager_storage_path to an absolute path from NixOS' software install directory (in NixOS, the program is stored in the nix store, an read only folder that houses all system files).

These changes would greatly reduce the complexity of hosting flamenco manager on a server, while retaining its ability to run as a portable application.

Hello! I'm in the process of upstreaming a [NixOS module for flamenco](https://github.com/NixOS/nixpkgs/pull/260812). It sets up the package (manager, worker or both) as a systemd service. I was trying to define the service and I got really confused when I was trying to set up the paths. I have two ideas about file management, focusing specifically on the manager, for consideration: ## local_manager_storage_path and `getSuitableStorageRoot` Currently, `local_manager_storage_path` uses a directory path relative to the executable, discovered by using the [`getSuitableStorageRoot`](https://projects.blender.org/studio/flamenco/src/commit/f7aef5bfca068e3885f8000d24226ce47a2f0b38/internal/manager/local_storage/local_storage.go#L108) function. From what I understand, part of Flamenco's design philosophy is that it *could* be used as an portable application. Maybe it could be possible to make `local_manager_storage_path` bypass the `getSuitableStorageRoot` logic *only if* the path is an absolute path. This way, default behavior is preserved, while expanding functionality for other uses. Go provides a function provided with `path/filepath`, [`isAbs()`](https://pkg.go.dev/path/filepath#IsAbs), which can tell if a path is an absolute path on multiple platforms, Unix and Windows both included. ## Configuration file location A lot of application I've seen has an config file path argument `-config`. Again, following the portability philosophy, default behavior can be that the configuration file is generated and read in the current working directory, but then people can use `-config` to set an path to an existing configuration file, if they need it. --- Currently, the NixOS module deals with these caveats by setting up an temporary working directory with the configuration file inside it, and by adding "../../../../" in front of the `local_manager_storage_path`, which will effectively set `local_manager_storage_path` to an absolute path from NixOS' software install directory (in NixOS, the program is stored in the nix store, an *read only* folder that houses all system files). These changes would greatly reduce the complexity of hosting flamenco manager on a server, while retaining its ability to run as a portable application.

Sybren A. Stüvel commented

2024-04-26 16:42:04 +02:00

I'm in the process of upstreaming a NixOS module for flamenco.

Cool, thanks!

From what I understand, part of Flamenco's design philosophy is that it could be used as an portable application.

That's not explicitly part of the design, but I certainly am not opposed to the idea. A clearer separation between these two approaches would be good:

Non-portable mode: config files & writable files are expected to be in a fixed path (like ~/Library/Application Support/Flamenco on macOS, as mentioned in this issue's description).
Portable mode: these files are either expected to be in the current working directory or in some other location specified with a CLI argument.

Currently the way the config files & databases are handled is a bit of a mess, and mix up these two approaches. That's what I want to get rid of.

Maybe it could be possible to make local_manager_storage_path bypass the getSuitableStorageRoot logic only if the path is an absolute path. This way, default behavior is preserved, while expanding functionality for other uses.

That seems reasonable to me.

A lot of application I've seen has an config file path argument -config.

Again, following the portability philosophy, default behavior can be that the configuration file is generated and read in the current working directory, but then people can use -config to set an path to an existing configuration file, if they need it.

Both the Manager and the Worker need multiple files from their "config location".

Manager

flamenco-manager.yaml
flamenco-manager.sqlite

Worker

flamenco-worker.yaml (which may or may not be shared among all workers in a farm)
flamenco-worker-credentials.yaml (which is unique to each worker)
flamenco-worker.sqlite (also unique to that worker)

So one -config option isn't going to cut it. For the Manager it could point to a single directory, but for the Worker there's the potentially-shared config as well as the worker-unique files.

I'm thinking of switching the entire configuration management of both Worker and Manager to Viper. That should make config management a bit easier.

> I'm in the process of upstreaming a [NixOS module for flamenco](https://github.com/NixOS/nixpkgs/pull/260812). Cool, thanks! > From what I understand, part of Flamenco's design philosophy is that it *could* be used as an portable application. That's not explicitly part of the design, but I certainly am not opposed to the idea. A clearer separation between these two approaches would be good: 1. Non-portable mode: config files & writable files are expected to be in a fixed path (like `~/Library/Application Support/Flamenco` on macOS, as mentioned in this issue's description). 2. Portable mode: these files are either expected to be in the current working directory or in some other location specified with a CLI argument. Currently the way the config files & databases are handled is a bit of a mess, and mix up these two approaches. That's what I want to get rid of. > Maybe it could be possible to make `local_manager_storage_path` bypass the `getSuitableStorageRoot` logic *only if* the path is an absolute path. This way, default behavior is preserved, while expanding functionality for other uses. That seems reasonable to me. > A lot of application I've seen has an config file path argument `-config`. > > Again, following the portability philosophy, default behavior can be that the configuration file is generated and read in the current working directory, but then people can use `-config` to set an path to an existing configuration file, if they need it. Both the Manager and the Worker need multiple files from their "config location". Manager : - `flamenco-manager.yaml` - `flamenco-manager.sqlite` Worker : - `flamenco-worker.yaml` (which may or may not be shared among all workers in a farm) - `flamenco-worker-credentials.yaml` (which is unique to each worker) - `flamenco-worker.sqlite` (also unique to that worker) So one `-config` option isn't going to cut it. For the Manager it could point to a single directory, but for the Worker there's the potentially-shared config as well as the worker-unique files. I'm thinking of switching the entire configuration management of both Worker and Manager to [Viper](https://github.com/spf13/viper). That should make config management a bit easier.

Sebastian Parborg commented

2024-05-01 13:20:18 +02:00

@Hubble-the-Wolverine as you are packaging flamenco, perhaps you could use the compile time options to set the paths you want to use?

Again, following the portability philosophy, default behavior can be that the configuration file is generated and read in the current working directory, but then people can use -config to set an path to an existing configuration file, if they need it.

You can already bypass this at compile time by specifying the configFilename variable.
See how these options are used to setup flamenco as a system service on our studio computers:

		www-apps/flamenco/flamenco-9999.ebuild
		Line 76 in d632382b85
	
							"-X projects.blender.org/studio/flamenco/internal/appinfo.customHome=${EPREFIX}/var/lib/flamenco_worker/"

There the manager is setup look for its config in /etc/flamenco_manager/config.yaml (like most other system services)

@Hubble-the-Wolverine as you are packaging flamenco, perhaps you could use the compile time options to set the paths you want to use? > Again, following the portability philosophy, default behavior can be that the configuration file is generated and read in the current working directory, but then people can use -config to set an path to an existing configuration file, if they need it. You can already bypass this at compile time by specifying the `configFilename` variable. See how these options are used to setup flamenco as a system service on our studio computers: https://projects.blender.org/ZedDB/blender-studio-overlay/src/commit/d632382b8566cce91bc3922819f60c2a9e2087a9/www-apps/flamenco/flamenco-9999.ebuild#L76 There the manager is setup look for its config in `/etc/flamenco_manager/config.yaml` (like most other system services)

❤️ 1

Sign in to join this conversation.