Environment variables #104196

New Issue

Sybren A. Stüvel · 2023-03-14T16:37:51+01:00

Sybren A. Stüvel commented

2023-03-14 16:37:51 +01:00

It can be quite useful to set certain environment variables before a command runs. This can influence, for example, the OpenColorIO profiles loaded by Blender.

Rather than adding this as a specific setting to Blender-specific commands, I think it's better to keep it generic and allow any Flamenco command to set environment variables.

Expand the OpenAPI Command definition so that it can contain the desired environment variables.
Add support for setting those in the job compiler scripts. The job compiler can receive the appropriate values via the regular job settings system.
Add support for those to the Manager's persistence layer, so they're saved to the database.
Ensure they're sent to the Worker as part of the commands to execute.
Design a place in the web interface to show them with the task details.
Implement that design.
Ensure the Worker sets those environment variables before executing anything, and in such a way that it is guaranteed to restore its default environment variables when the command is done.

API change

The Command definition in pkg/api/flamenco-openapi.yaml could be expanded to something like this:

    Command:
      type: object
      description: Command represents a single command to execute by the Worker.
      properties:
        name: { type: string }
        parameters: { type: object }
        environment: { $ref: "#/components/schemas/EnvironmentVariables" }
      required: [name, parameters]

    EnvironmentVariables:
      type: object
      description: OS environment variables that will be set before a command executes.
      additionalProperties:
        type: string
      example:
        "BLENDER_SYSTEM_SCRIPTS": "/path/to/some/scripts/dir"

It can be quite useful to set certain environment variables before a command runs. This can influence, for example, the OpenColorIO profiles loaded by Blender. Rather than adding this as a specific setting to Blender-specific commands, I think it's better to keep it generic and allow any Flamenco command to set environment variables. - [ ] Expand the OpenAPI `Command` definition so that it can contain the desired environment variables. - [ ] Add support for setting those in the job compiler scripts. The job compiler can receive the appropriate values via the regular job settings system. - [ ] Add support for those to the Manager's persistence layer, so they're saved to the database. - [ ] Ensure they're sent to the Worker as part of the commands to execute. - [ ] Design a place in the web interface to show them with the task details. - [ ] Implement that design. - [ ] Ensure the Worker sets those environment variables before executing anything, and in such a way that it is guaranteed to restore its default environment variables when the command is done. ### API change The `Command` definition in `pkg/api/flamenco-openapi.yaml` could be expanded to something like this: ```yaml Command: type: object description: Command represents a single command to execute by the Worker. properties: name: { type: string } parameters: { type: object } environment: { $ref: "#/components/schemas/EnvironmentVariables" } required: [name, parameters] EnvironmentVariables: type: object description: OS environment variables that will be set before a command executes. additionalProperties: type: string example: "BLENDER_SYSTEM_SCRIPTS": "/path/to/some/scripts/dir" ```

milanesa commented

2023-03-14 17:57:50 +01:00

It is common for artist in a studio to work out of a shell with environment set automatically based on the task they are working on. These variables can be set in different ways, either already present on the system or set by various working environments launched by artists.

Initially, we could pass current Environment Variables (already present in Blender) to the Workers. We could give the option to set or modify Variables, but most of the time this wont be necessary… unless working on a multi shot environment.

It is common for artist in a studio to work out of a shell with environment set automatically based on the task they are working on. These variables can be set in different ways, either already present on the system or set by various working environments launched by artists. Initially, we could pass current Environment Variables (already present in Blender) to the Workers. We could give the option to set or modify Variables, but most of the time this wont be necessary… unless working on a multi shot environment.

Sybren A. Stüvel commented

2023-03-14 18:17:43 +01:00

It is common for artist in a studio to work out of a shell with environment set automatically based on the task they are working on. These variables can be set in different ways, either already present on the system or set by various working environments launched by artists.

👍

I think e already have a nice picture of the steps from job creation on the Manager to the calling of the right executable (with the right environment vars) on the Worker.

What's left open is mostly how those environment variables are injected into that job.

Option 1: job settings going into specific commands

The easiest way I think is to use the normal JOB_TYPE settings for this. We can extend them to refer to environment variables, like:

{ key: "env_OCIO", type: "string", envvar: "OCIO" },

This can then be used by the JavaScript code to whatever command should actually be using this variable:

author.Command("echo", {
    message: settings.message,
}, {
    OCIO: settings.env_OCIO,
});

This is quite an explicit way to do this, but it does make the handling of these variables more explicit. It's very clear which environment variable will be pulled in by which job type, and where it ends up being used.

This approach also makes it possible to have regular job settings (so artists can make a choice) and have those influence the environment variables of the commands.

Option 2: more global job support going into all tasks

Alternatively, we could go a bit higher level, and expand the job type definition with an environment key, so something like this:

const JOB_TYPE = {
    label: "Job Name Here",
    settings: [
        ...,
    ],
    environment: ['OCIO', 'SHOT', 'SCENE'],
};

This could be implemented in such a way that it's part of each task (of this job) that a Worker receives. The Worker then would put it into the environment of every command that's part of this task.

This makes things work a bit more like 'magic', and it doesn't give you the level of control that the other option gives. Here you can only use environment variables to set the environment variables with the same name on the Worker.

Personally I think I'd prefer option 1, because it's so explicit and provides a lot of flexibility to the people writing the job type javascript files. What do you think @milanesa ?

PS: all this code is just to sketch my thoughts, not necessarily the best way to implement things.

> It is common for artist in a studio to work out of a shell with environment set automatically based on the task they are working on. These variables can be set in different ways, either already present on the system or set by various working environments launched by artists. :+1: I think e already have a nice picture of the steps from job creation on the Manager to the calling of the right executable (with the right environment vars) on the Worker. What's left open is mostly how those environment variables are injected into that job. #### Option 1: job settings going into specific commands The easiest way I think is to use the normal [`JOB_TYPE` settings](https://flamenco.blender.org/usage/job-types/) for this. We can extend them to refer to environment variables, like: ```js { key: "env_OCIO", type: "string", envvar: "OCIO" }, ``` This can then be used by the JavaScript code to whatever command should actually be using this variable: ```js author.Command("echo", { message: settings.message, }, { OCIO: settings.env_OCIO, }); ``` This is quite an explicit way to do this, but it does make the handling of these variables more explicit. It's very clear which environment variable will be pulled in by which job type, and where it ends up being used. This approach also makes it possible to have regular job settings (so artists can make a choice) and have those influence the environment variables of the commands. #### Option 2: more global job support going into all tasks Alternatively, we could go a bit higher level, and expand the job type definition with an `environment` key, so something like this: ``` const JOB_TYPE = { label: "Job Name Here", settings: [ ..., ], environment: ['OCIO', 'SHOT', 'SCENE'], }; ``` This could be implemented in such a way that it's part of each task (of this job) that a Worker receives. The Worker then would put it into the environment of every command that's part of this task. This makes things work a bit more like 'magic', and it doesn't give you the level of control that the other option gives. Here you can only use environment variables to set the environment variables with the same name on the Worker. --------- Personally I think I'd prefer option 1, because it's so explicit and provides a lot of flexibility to the people writing the job type javascript files. What do you think @milanesa ? PS: all this code is just to sketch my thoughts, not necessarily the best way to implement things.

milanesa commented

2023-03-14 19:47:26 +01:00

I like a combination of both options, but with a bit of a different implementation of option 2. I would give the artist the flexibility of option 1. Option 2 can be used to include Evirioments already present in the Blender session. Something like

const JOB_TYPE = {
    label: "Job Name Here",
    settings: [
        ...,
    ],
    use_environment: True,
};

That way artist will use option 1 if they need to set a new environment or override an existing one.

Deadline has a similar approach: Render Environment

I like a combination of both options, but with a bit of a different implementation of option 2. I would give the artist the flexibility of option 1. Option 2 can be used to include Evirioments already present in the Blender session. Something like ``` const JOB_TYPE = { label: "Job Name Here", settings: [ ..., ], use_environment: True, }; ``` That way artist will use option 1 if they need to set a new environment or override an existing one. Deadline has a similar approach: [Render Environment](https://docs.thinkboxsoftware.com/products/deadline/10.1/1_User%20Manual/manual/environment.html#manual-job-submission)

milanesa commented

2023-03-14 21:53:40 +01:00

Re-thinking this a bit. It feels to me option 2 (with the implementation I suggested) should alway be True by default. Option 1 can be there for overrides.

Sybren A. Stüvel commented

2023-03-20 11:25:31 +01:00

I think having explicit functions for setting up the environment variables is a good idea.

Having a broad use_environment: True that picks up all environment variables from Blender, that I would not do. This impacts way too much, as it would also pick up on session- and machine-specific variables like PATH, XDG_DATA_DIRS, DISPLAY, LANG, etc. Trying to exclude specific variables from thing would inevitably lead to unforseen issues where we have to do another release of Flamenco to exclude yet another variable. This is why I proposed to have an explicit list of variables to set.

Additionally, people may use environment variables to adjust the behaviour of Blender for their local machine, and may have already set up values of those variables on each worker machine to make it behave the same (for example when each machine has different paths for something). Using the submittor's environment variables by default would implicitly break such things, leading to hard to debug issues.

Finally, having to deal with only a handful of environment variables makes the web interface a lot easier to design. My system currently has 74 environment variables set, and showing all of those in the web interface in a clear manner will be a considerably bigger challenge. Not only would most of those be irrelevant to the actual Flamenco task, it would also leak private / security-related information.

For this design, please be aware that the Workers do not know much about jobs. All they get is tasks to execute, and the only thing they know about the job is its UUID (because that's included in the task schema). Any information that the Worker needs to know has to be part of the Task schema or the Command schema (because a Task consists of multiple Commands).

I think having explicit functions for setting up the environment variables is a good idea. Having a broad `use_environment: True` that picks up *all* environment variables from Blender, that I would not do. This impacts way too much, as it would also pick up on session- and machine-specific variables like `PATH`, `XDG_DATA_DIRS`, `DISPLAY`, `LANG`, etc. Trying to exclude specific variables from thing would inevitably lead to unforseen issues where we have to do another release of Flamenco to exclude yet another variable. This is why I proposed to have an explicit list of variables to set. Additionally, people may use environment variables to adjust the behaviour of Blender for their local machine, and may have already set up values of those variables on each worker machine to make it behave the same (for example when each machine has different paths for something). Using the submittor's environment variables by default would implicitly break such things, leading to hard to debug issues. Finally, having to deal with only a handful of environment variables makes the web interface a lot easier to design. My system currently has 74 environment variables set, and showing all of those in the web interface in a clear manner will be a considerably bigger challenge. Not only would most of those be irrelevant to the actual Flamenco task, it would also leak private / security-related information. For this design, please be aware that the Workers do not know much about jobs. All they get is tasks to execute, and the only thing they know about the job is its UUID (because that's included in the task schema). Any information that the Worker needs to know has to be part of the `Task` schema or the `Command` schema (because a Task consists of multiple Commands).

milanesa commented

2023-03-20 15:58:22 +01:00

Got it. Yes I agree, Option 1 seems to be a nicer approach.

Sybren A. Stüvel added the

Type

To Do

label 2023-05-01 10:26:01 +02:00

Sign in to join this conversation.