API Design for IDType::foreach_external_file #92315

New Issue

Jeroen Bakker · 2021-10-18T14:58:54+02:00

Jeroen Bakker commented

2021-10-18 14:58:54 +02:00

Goal/Usage

The goal of this change is to be able to 'walk' over external files that are associated with an ID block.
Problem areas:

BKE_bpath.h finding and fixing missing files.
Asset Browser localization of external files.

Requirements

An iterator that reports external files that are directly referenced inside an ID block.
It should be possible to rewrite the external file location for relocating missing files, or relocate libraries.
It should be possible to iterator over files sequence files.
The API should be clear on the parameters and what can be done with the found files.

References

D12818: ID: Foreach External File.: initial implementation we don't want to add to master. Shows limitations we want to solve.
D12423: AssetBrowser: Localize external files.: implementation of the localization of external files. A usecase that would use the foreach_external_file API.

API

From the using side

BKE_idtype.h

IDTypeInfo would gets a foreach_external_file callback.

typedef struct IDTypeInfo {
  ...
  /**
   * Iterate over external files of the given ID.
   */
  IDTypeForeachExternalFileFunction foreach_external_file;
  ...
};

The foreach_external_file has the next signature. (not complete)

NOTE: Should filter options be additive or subtractive. Personally I prefer additive as it shows what you want.

typedef enum eIDTypeForeachExternalFileFilters {
  /* Invoke the callback for filepaths stored in blend file. Will skip packed files. */
  ID_TYPE_EXT_FILE_FILTER_DNA_FILEPATH = (1<<0),

  /* Invoke the callback for packed files. Must be used together with ID_TYPE_EXT_FILE_FILTER_DNA_FILEPATH. */
  ID_TYPE_EXT_FILE_FILTER_PACKED = (1<<1),

  /* Sequences/views file filepaths are not stored with the id. For sequences
    * a single filepath is stored in an IDBlock and the other files are extracted
    * by replacing the frame number with a different one.  For multiview the name
    * of the view (scene data) is added in the end of the filepath.
    */
  ID_TYPE_EXT_FILE_FILTER_SEQUENCES_MULTIVIEW = (1<<2),

  ...

} eIDTypeForeachExternalFileFilters;

typedef void (*IDTypeForeachExternalFileFunction)(
    struct ID *id,
    IDTypeForeachExternalFileFunctionCallback function_callback,
    eIDTypeForeachExternalFileFilters filter,
    void *user_data);

function_callback is called for every found external file of the given ID block that matches the filter.

typedef void (*IDTypeForeachExternalFileFunctionCallback)(
    struct ID *id,
    struct IDTypeExternalFilePath * external_file_path,
    void *user_data);

The structure of IDTypeExternalFilePath isn't accessible. Data retrieval/Operations must be done via predefined functions. This makes the API clear and would eliminate duplicating complexity. We might consider a separate CPP/C API for clarity. The C API is a wrapper around the CPP API.

The struct is able to hide complexity like double pointers, free-ing/reallocating memory

typedef enum eIDTypeExternalFilePathType {
  /* The external file path is a path that is stored directly in a blend file. */
  ID_TYPE_EXT_FILE_PATH_TYPE_DNA,
  /* The external file path is a path that is not stored in a blend file. Examples are sequences or multiviews. */
  ID_TYPE_EXT_FILE_PATH_TYPE_GENERATED,
} eIDTypeExternalFilePathType;

/**
 * Returns the type of the given file paths. 
 */
eIDTypeExternalFilePathType BKE_idtype_external_file_path_type(const struct IDTypeExternalFilePath 
*external_file_path);

/**
 * Get the file path of the given external file path.
 */
const char* BKE_idtype_external_file_path_filepath_get(const struct IDTypeExternalFilePath *external_file_path);

/**
 * Update the external file path with a new path. It only updated the ID block data, but doesn't
 * change anything on file system level.
 * Returns true when the path is updated, false when the path couldn't be updated. 
 */
bool BKE_idtype_external_file_path_filepath_update(struct IDTypeExternalFilePath *external_file_path, const char* new_file_path, const size_t new_file_path_size);

From the implementing side

When implementing the foreach_external_file for an IDType there are additional functions/patterns to follow.

struct IDTypeExternalFilePath *BKE_idtype_external_file_path_create()
void BKE_idtype_external_file_path_free(IDTypeExternalFilePath *external_file_path);

/**
 * Set the data of an external_file_path instance. ExternalFilePath's
 * can be allocated at the start of an `foreach_external_file` and freed at the end.
 * During this time the same instance can be reused to reduce memory allocations.
 */
void BKE_idtype_external_file_path_reinit(IDTypeExternalFilePath *external_file_path, ID *id, char *new_file_path, size_t new_file_path_len, eIDTypeExternalFilePathType type);

Example for bSound

static void sound_foreach_external_file(
    ID *id,
    IDTypeForeachExternalFileFunctionCallback function_callback,
    eIDTypeForeachExternalFileFilters filter,
    void *user_data)
{
  if (filter & ID_TYPE_EXT_FILE_FILTER_DNA_FILEPATH) == 0) {
   return;
  }

  bSound *sound = (bSound *)id;
  if (sound->packedfile != NULL && (filter & ID_TYPE_EXT_FILE_FILTER_PACKED) == 0) {
    return;
  }

  struct IDTypeExternalFilePath *external_file_path = BKE_idtype_external_file_path_create();
  BKE_idtype_external_file_path_reinit(external_file_path, id, sound->filepath, sizeof(sound->filepath), ID_TYPE_EXT_FILE_PATH_TYPE_DNA);
  function_callback(id, external_file_path, user_data);
 BKE_idtype_external_file_path_free(external_file_path);
}

The above example is common for multiple IDTypes and could be hidden inside a centralized implementation.

## Goal/Usage The goal of this change is to be able to 'walk' over external files that are associated with an ID block. Problem areas: * `BKE_bpath.h` finding and fixing missing files. * Asset Browser localization of external files. ### Requirements * An iterator that reports external files that are directly referenced inside an ID block. * It should be possible to rewrite the external file location for relocating missing files, or relocate libraries. * It should be possible to iterator over files sequence files. * The API should be clear on the parameters and what can be done with the found files. ### References * [D12818: ID: Foreach External File.](https://archive.blender.org/developer/D12818): initial implementation we don't want to add to master. Shows limitations we want to solve. * [D12423: AssetBrowser: Localize external files.](https://archive.blender.org/developer/D12423): implementation of the localization of external files. A usecase that would use the foreach_external_file API. ### API **From the using side** **BKE_idtype.h** `IDTypeInfo` would gets a `foreach_external_file` callback. ``` typedef struct IDTypeInfo { ... /** * Iterate over external files of the given ID. */ IDTypeForeachExternalFileFunction foreach_external_file; ... }; ``` The foreach_external_file has the next signature. (not complete) NOTE: Should filter options be additive or subtractive. Personally I prefer additive as it shows what you want. ``` typedef enum eIDTypeForeachExternalFileFilters { /* Invoke the callback for filepaths stored in blend file. Will skip packed files. */ ID_TYPE_EXT_FILE_FILTER_DNA_FILEPATH = (1<<0), /* Invoke the callback for packed files. Must be used together with ID_TYPE_EXT_FILE_FILTER_DNA_FILEPATH. */ ID_TYPE_EXT_FILE_FILTER_PACKED = (1<<1), /* Sequences/views file filepaths are not stored with the id. For sequences * a single filepath is stored in an IDBlock and the other files are extracted * by replacing the frame number with a different one. For multiview the name * of the view (scene data) is added in the end of the filepath. */ ID_TYPE_EXT_FILE_FILTER_SEQUENCES_MULTIVIEW = (1<<2), ... } eIDTypeForeachExternalFileFilters; typedef void (*IDTypeForeachExternalFileFunction)( struct ID *id, IDTypeForeachExternalFileFunctionCallback function_callback, eIDTypeForeachExternalFileFilters filter, void *user_data); ``` `function_callback` is called for every found external file of the given ID block that matches the filter. ``` typedef void (*IDTypeForeachExternalFileFunctionCallback)( struct ID *id, struct IDTypeExternalFilePath * external_file_path, void *user_data); ``` The structure of IDTypeExternalFilePath isn't accessible. Data retrieval/Operations must be done via predefined functions. This makes the API clear and would eliminate duplicating complexity. We might consider a separate CPP/C API for clarity. The C API is a wrapper around the CPP API. The struct is able to hide complexity like double pointers, free-ing/reallocating memory ``` typedef enum eIDTypeExternalFilePathType { /* The external file path is a path that is stored directly in a blend file. */ ID_TYPE_EXT_FILE_PATH_TYPE_DNA, /* The external file path is a path that is not stored in a blend file. Examples are sequences or multiviews. */ ID_TYPE_EXT_FILE_PATH_TYPE_GENERATED, } eIDTypeExternalFilePathType; /** * Returns the type of the given file paths. */ eIDTypeExternalFilePathType BKE_idtype_external_file_path_type(const struct IDTypeExternalFilePath *external_file_path); /** * Get the file path of the given external file path. */ const char* BKE_idtype_external_file_path_filepath_get(const struct IDTypeExternalFilePath *external_file_path); /** * Update the external file path with a new path. It only updated the ID block data, but doesn't * change anything on file system level. * Returns true when the path is updated, false when the path couldn't be updated. */ bool BKE_idtype_external_file_path_filepath_update(struct IDTypeExternalFilePath *external_file_path, const char* new_file_path, const size_t new_file_path_size); ``` **From the implementing side** When implementing the `foreach_external_file` for an IDType there are additional functions/patterns to follow. ``` struct IDTypeExternalFilePath *BKE_idtype_external_file_path_create() void BKE_idtype_external_file_path_free(IDTypeExternalFilePath *external_file_path); /** * Set the data of an external_file_path instance. ExternalFilePath's * can be allocated at the start of an `foreach_external_file` and freed at the end. * During this time the same instance can be reused to reduce memory allocations. */ void BKE_idtype_external_file_path_reinit(IDTypeExternalFilePath *external_file_path, ID *id, char *new_file_path, size_t new_file_path_len, eIDTypeExternalFilePathType type); ``` Example for bSound ``` static void sound_foreach_external_file( ID *id, IDTypeForeachExternalFileFunctionCallback function_callback, eIDTypeForeachExternalFileFilters filter, void *user_data) { if (filter & ID_TYPE_EXT_FILE_FILTER_DNA_FILEPATH) == 0) { return; } bSound *sound = (bSound *)id; if (sound->packedfile != NULL && (filter & ID_TYPE_EXT_FILE_FILTER_PACKED) == 0) { return; } struct IDTypeExternalFilePath *external_file_path = BKE_idtype_external_file_path_create(); BKE_idtype_external_file_path_reinit(external_file_path, id, sound->filepath, sizeof(sound->filepath), ID_TYPE_EXT_FILE_PATH_TYPE_DNA); function_callback(id, external_file_path, user_data); BKE_idtype_external_file_path_free(external_file_path); } ``` The above example is common for multiple IDTypes and could be hidden inside a centralized implementation.

Jeroen Bakker self-assigned this 2021-10-18 14:58:54 +02:00

Jeroen Bakker commented

2021-10-18 14:58:54 +02:00

Added subscriber: @Jeroen-Bakker

Jeroen Bakker commented

2021-10-18 15:47:29 +02:00

Changed status from 'Needs Triage' to: 'Confirmed'

Jeroen Bakker removed their assignment 2021-11-03 10:50:02 +01:00

Sign in to join this conversation.

No Label

Download

What's New

Blender Studio

Manual

Developers Blog

Documentation

Benchmark

Blender Conference

Development Fund

One-time Donations

API Design for IDType::foreach_external_file #92315

Goal/Usage

Requirements

References

API

API Design for `IDType::foreach_external_file` #92315