infrastructure
/
blender-projects-platform
5
1
Issues 20 Projects Activity

User manual Git migration and online editing #51

New Issue
Open
opened 2023-02-27 18:50:38 +01:00 by Brecht Van Lommel · 14 comments
Brecht Van Lommel commented 2023-02-27 18:50:38 +01:00
Owner
Copy Link

Contributing to the user manual is currently quite technical, and requires using subversion. The first step will be moving the manual to a Git repository with LFS. This already will allow some level of online editing.

Git migration:

  • LFS enabled on Gitea (#26)
  • Buildbot integration
    • Nightly builds
    • Incremental builds
    • Commit status reporting
    • Patch builder
    • blender-bot integration
    • Linting, dead link checks
  • Main repo Git migration
    • Basic import
    • Map authors
    • Tags and branches
  • Translation repo Git migration
  • Update contribution docs
  • Document pull request workflow
  • Ensure git-lfs is installed on all buildbot workers
  • Remove old svn folders on buildbot workers

Online editing:

  • RST preview through external renderer
    • External renderer implementation
    • Put Gitea changes in blender-merged
    • Deploy with dedicated sphinx user for isolation on uatest
    • Deploy on projects.blender.org
  • Gitea workflow improvements
    • Default to creating pull request for main repo instead of fork
    • When editing files in main repo, auto create branch in fork
    • Online sync changes from main repo to fork
  • Document online editing workflow
Contributing to the user manual is currently quite technical, and requires using subversion. The first step will be moving the manual to a Git repository with LFS. This already will allow some level of online editing. Git migration: * [x] LFS enabled on Gitea (#26) * [ ] Buildbot integration * [x] Nightly builds * [x] Incremental builds * [x] Commit status reporting * [ ] Patch builder * [ ] blender-bot integration * [ ] Linting, dead link checks * [x] Main repo Git migration * [x] Basic import * [x] Map authors * [x] Tags and branches * [x] Translation repo Git migration * [x] Update contribution docs * [x] Document pull request workflow * [x] Ensure git-lfs is installed on all buildbot workers * [x] Remove old svn folders on buildbot workers Online editing: * [x] RST preview through [external renderer](https://docs.gitea.io/en-us/external-renderers/) * [x] External renderer implementation * [x] Put Gitea changes in `blender-merged` * [x] Deploy with dedicated sphinx user for isolation on uatest * [x] Deploy on projects.blender.org * [ ] Gitea workflow improvements * [x] Default to creating pull request for main repo instead of fork * [ ] When editing files in main repo, auto create branch in fork * [ ] Online sync changes from main repo to fork * [ ] Document online editing workflow
Brecht Van Lommel commented 2023-02-27 18:51:12 +01:00
Author
Owner
Copy Link

CC @blendify

CC @blendify
Brecht Van Lommel added the
buildbot
 label 2023-02-27 18:53:31 +01:00
Aaron Carlisle commented 2023-03-03 03:20:21 +01:00
Copy Link

It's an open question what we should do to improve on that. Is there a good platform that can be used for online editing, or do we keep everything on Gitea?

I think keeping everything on Gitea would be sufficient. Adding more would just complicate the process in my opinion.

Adding the rst preview would be a nice to have though.

> It's an open question what we should do to improve on that. Is there a good platform that can be used for online editing, or do we keep everything on Gitea? I think keeping everything on Gitea would be sufficient. Adding more would just complicate the process in my opinion. Adding the rst preview would be a nice to have though.
Brecht Van Lommel commented 2023-03-05 12:10:44 +01:00
Author
Owner
Copy Link

I also didn't find a good candidate for an alternative online editor, so perhaps it's the best we can do for now.

For rst preview, Gitea needs a command to convert rst to html. The basic pandoc command from the docs generates would have wrong URLs for images, :doc: links are not generated, and probably various other sphinx features are not supported. Pandoc has various customization options so perhaps there is a way to improve this.

But perhaps better, a script could be written that uses sphinx to generate html from a single rst file? Not sure if there is a convenient way to do it. But basic working links and images would be nice even if they can't be verified to be correct.

I also didn't find a good candidate for an alternative online editor, so perhaps it's the best we can do for now. For rst preview, Gitea needs a command to convert rst to html. The basic pandoc command from the docs generates would have wrong URLs for images, `:doc:` links are not generated, and probably various other sphinx features are not supported. Pandoc has various customization options so perhaps there is a way to improve this. But perhaps better, a script could be written that uses sphinx to generate html from a single rst file? Not sure if there is a convenient way to do it. But basic working links and images would be nice even if they can't be verified to be correct.
Brecht Van Lommel commented 2023-03-05 12:32:34 +01:00
Author
Owner
Copy Link

The other approach of course would be to convert the manual to markdown or support both rst and md pages. That would likely help attract contributors and developers, but it's also a complicated change that involves a lot work and trade-offs.

The other approach of course would be to convert the manual to markdown or support both rst and md pages. That would likely help attract contributors and developers, but it's also a complicated change that involves a lot work and trade-offs.
Aaron Carlisle commented 2023-03-06 01:05:11 +01:00
Copy Link

For rst preview, Gitea needs a command to convert rst to html.
But perhaps better, a script could be written that uses sphinx to generate html from a single rst file?

This is possible with the command:

sphinx-build -b html -j auto manual build/html <path_to_rst_file>

The other approach of course would be to convert the manual to markdown or support both rst and md pages. That would likely help attract contributors and developers, but it's also a complicated change that involves a lot work and trade-offs.

I would avoid this, like you said there are many trade offs and rst is much more powerful than markdown and leaves a lot to be desired.

> For rst preview, Gitea needs a command to convert rst to html. >But perhaps better, a script could be written that uses sphinx to generate html from a single rst file? This is possible with the command: `sphinx-build -b html -j auto manual build/html <path_to_rst_file>` ----- > The other approach of course would be to convert the manual to markdown or support both rst and md pages. That would likely help attract contributors and developers, but it's also a complicated change that involves a lot work and trade-offs. > I would avoid this, like you said there are many trade offs and rst is much more powerful than markdown and leaves a lot to be desired.
Brecht Van Lommel commented 2023-03-06 09:46:53 +01:00
Author
Owner
Copy Link

This is possible with the command:

sphinx-build -b html -j auto manual build/html <path_to_rst_file>

The issue is that it gets the contents of a single rst file and nothing else from the repository. That command assumes it has the whole repository available.

Making a checkout of the full repository would be too slow for to do for every preview. Unless perhaps some clever caching mechanism is implemented, but that seems complicated.

For security we'd also need to ensure we're not executing custom Python code written by users in the conf.py in their fork of the repository.

Links and images can still work even without the full repository as long as we set them to the right URL. However sphinx wants to validate that the files exist, which is not helpful in this case and maybe we need some way to generate the links without validation.

> This is possible with the command: > > `sphinx-build -b html -j auto manual build/html <path_to_rst_file>` The issue is that it gets the contents of a single rst file and nothing else from the repository. That command assumes it has the whole repository available. Making a checkout of the full repository would be too slow for to do for every preview. Unless perhaps some clever caching mechanism is implemented, but that seems complicated. For security we'd also need to ensure we're not executing custom Python code written by users in the conf.py in their fork of the repository. Links and images can still work even without the full repository as long as we set them to the right URL. However sphinx wants to validate that the files exist, which is not helpful in this case and maybe we need some way to generate the links without validation.
Brecht Van Lommel commented 2023-03-06 14:54:41 +01:00
Author
Owner
Copy Link

I managed to hack together something to render a single page:
https://projects.blender.org/brecht/gitea-custom/src/branch/sphinx/sphinx
https://github.com/go-gitea/gitea/pull/23333

But this will need changes to ensure security before this can be deployed. Probably running as an isolated user or a web service.

editor_preview_external_renderers.png

And links are missing due to other pages not being available.

I managed to hack together something to render a single page: https://projects.blender.org/brecht/gitea-custom/src/branch/sphinx/sphinx https://github.com/go-gitea/gitea/pull/23333 But this will need changes to ensure security before this can be deployed. Probably running as an isolated user or a web service. ![editor_preview_external_renderers.png](/attachments/6b75dab2-1fea-48b3-a89b-97e5563851bd) And links are missing due to other pages not being available.
editor_preview_external_renderers.png
89 KiB
Brecht Van Lommel commented 2023-03-09 15:55:08 +01:00
Author
Owner
Copy Link

I made :doc: links work.:ref: and :term: can't work based on just a single RST files so those links don't lead anywhere for now.

Security should also be much better now though there are some specifics to be figured out for deployment.

I made `:doc:` links work.`:ref:` and `:term:` can't work based on just a single RST files so those links don't lead anywhere for now. Security should also be much better now though there are some specifics to be figured out for deployment.
Brecht Van Lommel added the
deployment
 label 2023-03-10 01:00:50 +01:00
Brecht Van Lommel commented 2023-04-06 19:48:26 +02:00
Author
Owner
Copy Link

I did a few more fixes to improve the preview, based on testing with the actual manual.

For reference this is how I generated the Git repo.

sudo apt install git-lfs svn2git
pip3 install git-filter-repo

mkdir blender-manual
cd blender-manual
svn2git https://svn.blender.org/svnroot/bf-manual

# Rename default branch
git branch -m master main

# Remove unnecessary top level directory
git filter-repo --force --subdirectory-filter blender_docs/

# LFS filter for image, video, archive, 3D files.
git lfs migrate import --everything --include="*.exr,*.psd,*.pdd,*.psb,*.jp2,*.j2c,*.tx,*.gif,*.jpg,*.jpeg,*.tga,*.tif,*.tiff,*.bmp,*.png,*.dds,*.cin,*.dpx,*.hdr,*.webp,*.ttf,*.TTF,*.otf,*.aif,*.ogg,*.wav,*.aac,*.flac,*.oga,*.wma,*.rns,*.mp3,*.flv,*.mov,*.wmv,*.mpg,*.mpeg,*.avi,*.mp4,*.mkv,*.webm,*.FBX,*.fbx,*.blend,*.rar,*.zip,*.gz,*.7z,*.obj,*.3ds,*.gltf,*.glb,*.ply,*.vdb,*.nvdb,*.usd,*.usdz,*.usdc,*.pdf"
I did a few more fixes to improve the preview, based on testing with the actual manual. For reference this is how I generated the Git repo. ``` sudo apt install git-lfs svn2git pip3 install git-filter-repo mkdir blender-manual cd blender-manual svn2git https://svn.blender.org/svnroot/bf-manual # Rename default branch git branch -m master main # Remove unnecessary top level directory git filter-repo --force --subdirectory-filter blender_docs/ # LFS filter for image, video, archive, 3D files. git lfs migrate import --everything --include="*.exr,*.psd,*.pdd,*.psb,*.jp2,*.j2c,*.tx,*.gif,*.jpg,*.jpeg,*.tga,*.tif,*.tiff,*.bmp,*.png,*.dds,*.cin,*.dpx,*.hdr,*.webp,*.ttf,*.TTF,*.otf,*.aif,*.ogg,*.wav,*.aac,*.flac,*.oga,*.wma,*.rns,*.mp3,*.flv,*.mov,*.wmv,*.mpg,*.mpeg,*.avi,*.mp4,*.mkv,*.webm,*.FBX,*.fbx,*.blend,*.rar,*.zip,*.gz,*.7z,*.obj,*.3ds,*.gltf,*.glb,*.ply,*.vdb,*.nvdb,*.usd,*.usdz,*.usdc,*.pdf" ```
Aaron Carlisle commented 2023-04-09 03:44:18 +02:00
Copy Link

I had to also install git-filter-repo.

I had to also install `git-filter-repo`.
Brecht Van Lommel changed title from User manual online editing to User manual Git migration and online editing 2023-05-08 20:20:22 +02:00
Brecht Van Lommel commented 2023-05-08 20:32:39 +02:00
Author
Owner
Copy Link

@Blendify @Arnd. I've created a test repo here, which also includes branches, tags and commit authors that match user accounts. See the attached scripts and mailmap for how it was done.
https://projects.blender.org/brecht/blender-manual

What I could not get working is nice merge commits, not sure it is even supported by svn2git. It just shows Merge branch 'blender-3.5-release' without any relation to the commits in the branch. I guess this is ok.

The branches have been renamed from e.g. blender-3.5-release to blender-v3.5-release for consistency with the Blender repo.

@Blendify @Arnd. I've created a test repo here, which also includes branches, tags and commit authors that match user accounts. See the attached scripts and mailmap for how it was done. https://projects.blender.org/brecht/blender-manual What I could not get working is nice merge commits, not sure it is even supported by `svn2git`. It just shows `Merge branch 'blender-3.5-release'` without any relation to the commits in the branch. I guess this is ok. The branches have been renamed from e.g. `blender-3.5-release` to `blender-v3.5-release` for consistency with the Blender repo.
mailmap
9.5 KiB
blender-manual-steps.sh
1.7 KiB
Brecht Van Lommel commented 2023-05-09 00:50:21 +02:00
Author
Owner
Copy Link

I've put a translation repo here.
https://projects.blender.org/brecht/blender-manual-translations

This repository is also quite big, but we can't use LFS for it since the translation files are text and treating them as binary files would not give good delta compression.

The author mapping is not as good for this one, since I don't have easy access to the developer.blender.org username to full name mapping. I've attached the mailmap in case anyone wants to try improving it.

There's also just one branch. We could add per release branches and get them used on the buildbot if desired. I'm not sure what is better, translations of unchanged text may be added later, but changed text may also delete translations.

I've put a translation repo here. https://projects.blender.org/brecht/blender-manual-translations This repository is also quite big, but we can't use LFS for it since the translation files are text and treating them as binary files would not give good delta compression. The author mapping is not as good for this one, since I don't have easy access to the developer.blender.org username to full name mapping. I've attached the mailmap in case anyone wants to try improving it. There's also just one branch. We could add per release branches and get them used on the buildbot if desired. I'm not sure what is better, translations of unchanged text may be added later, but changed text may also delete translations.
mailmap-translation
14 KiB
blender-manual-translations-steps.sh
478 B
Brecht Van Lommel commented 2023-05-09 00:56:31 +02:00
Author
Owner
Copy Link

The uatest buildbot is now building from Git. This includes nightly builds, incremental builds (without delivery), and build status reporting of commits. Pull request builds and @blender-bot build are not there yet, that will be for after the initial deployment.

This one was built from Git:
https://docs.blender.org/UATEST/manual/en/dev/

@Blendify what I need from you before we can deploy this:

  • Check if the repositories are ok.
  • Prepare updated manual contribution instructions
  • A devtalk announcement that you can post when this goes live

@Arnd, I need some help from you to deploy the RST preview, particularly regarding user isolation. This is not a blocker but would be great to have available alongside this.
https://projects.blender.org/infrastructure/gitea-custom/src/branch/main/sphinx/

The uatest buildbot is now building from Git. This includes nightly builds, incremental builds (without delivery), and build status reporting of commits. Pull request builds and `@blender-bot build` are not there yet, that will be for after the initial deployment. This one was built from Git: https://docs.blender.org/UATEST/manual/en/dev/ @Blendify what I need from you before we can deploy this: * Check if the repositories are ok. * Prepare updated manual contribution instructions * A devtalk announcement that you can post when this goes live @Arnd, I need some help from you to deploy the RST preview, particularly regarding user isolation. This is not a blocker but would be great to have available alongside this. https://projects.blender.org/infrastructure/gitea-custom/src/branch/main/sphinx/
Brecht Van Lommel commented 2023-05-15 15:24:04 +02:00
Author
Owner
Copy Link

Git repositories are now online and buildbot was updated.
https://code.blender.org/2023/05/sunsetting-subversion/

I'll keep this issue open for the remaining tasks.

Git repositories are now online and buildbot was updated. https://code.blender.org/2023/05/sunsetting-subversion/ I'll keep this issue open for the remaining tasks.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
main
Labels
Clear labels   
blender customization

   
bug

Something is not working

  
buildbot

   
deployment

   
gitea feature request

   
help wanted

Need some help

  
policy
No Label
blender customization
bug
buildbot
deployment
gitea feature request
help wanted
policy
Milestone
Clear milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
No Assignees
2 Participants
Notifications
Due Date
The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: infrastructure/blender-projects-platform#51
No description provided.
Delete Branch "%!s(<nil>)"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?