New Developer Documentation Platform #116055

New Issue

Julian Eisel · 2023-12-11T18:15:05+01:00

Julian Eisel commented

2023-12-11 18:15:05 +01:00

A new developer documentation platform is implemented. This will replace the Wiki.

Before Switch

What needs to be completed before we can make the switch.

General

Blog Post 1: A New Developer Documentation Platform
Basic repository set up: https://projects.blender.org/blender/blender-developer-docs
Set up Git LFS
Copy GPU & Viewport pages
Copy Asset System pages
Add developer.blender.org header
Change buildbot to point to new repository
Ask devs to move personal pages (including weekly reports) to Gitea repositories or other platforms.
Plugins:
- "Edit Page" button
- "Next"/"Previous" page links
- Mermaid diagrams
- Macro plugin for simple commit/report/PR referencing
License page
Contribute Documentation pages
Custom footer (license information)

Transition

Tasks for the actual transition. This effort should be manageable within ~1 week of work.

Improve conversion script to minimize manual changes needed (see https://projects.blender.org/brecht/wiki-to-markdown)
- Convert commit/report/PR references to custom MkDocs macros
Investigate Release Notes porting - How much effort is it to port these? How many releases back to we port?
Define mapping for new navigation structure
Announce transition, ask developers to avoid Wiki edits
developer.blender.org/docs?
Initially port pages using https://projects.blender.org/brecht/wiki-to-markdown
Go over all pages, fix remaining issues (organizational task needed)
Fix pages with switch-templates, e.g. build instructions
Check trash/ files
Check/fix/quiet all warnings from the conversion script and mkdocs
Final pass and approval of navigation structure
Ensure sections without landing page don’t use the first place as landing page instead (index.md)
Prepare redirects configuration for Apache
- Initial generation of file
- Check regexes to use for things like utm
~~Rewrite git repository history to remove images and clean history~~ -- No real reason to do this currently, so rather keep history.
Check on how to handle attribution of past authors (because CC-BY-SA) -- Link to wiki.blender.org & archive.blender.org/wiki from license page.

Day of the Switch

What needs to happen on the day on the switch. Needs coordination with the infra team (Arndt, Sergey and Danny)

Apply new edits on the Wiki to ported pages
Add banner on Wiki explaining it has been archived
Make wiki read-only
Deploy redirects on Apache wiki.blender.org server
Update links on blender.org websites
Update developer.blender.org header to point to new docs platform
Update links within Blender
Write announcement

Soon After

Tasks that can be done on the day of the switch, or in the next few days after.

Release checklist integration (update release notes script?)
Update important scripts generating wiki content
- Release notes
- Weekly reports (https://projects.blender.org/blender/blender/src/branch/main/tools/triage/weekly_report.py)
Add contents to or remove/hide Submit Changes page

Non-Blocking

Non-blocking tasks that can be done after the switch, or before.

~~Blog Post 2: Rethinking the Structure of Developer Documentation~~ -- not planning to work on this now
~~Custom navigation header (Material for MkDocs one doesn't scale enough)~~ -- With the current navigation structure we're fine, so this is a nice-to-have.
Custom landing page (non-urgent)
Consider moving files into main Blender repo, and if not integrating it with make update
Add mkdocs-redirects extension for when pages get moved?
Update wiki file structure script (tools/check_wiki/check_wiki_file_structure.py)
Move old and big image video files to static location (archive.blender.org?), to make checkout smaller

Archive Wiki to Static HTML

Not urgent if keeping wiki.blender.org running for a while is no big burden. Needs discussion with Arndt, Sergey and Danny.

Which URL to move to? (archive.blender.org/wiki already used) -- archive.blender.org/wiki/2024
Move to new URL as readonly (static HTML?)

A new developer documentation platform is implemented. This will replace the Wiki. ## Before Switch What needs to be completed before we can make the switch. ### General - [x] Blog Post 1: A New Developer Documentation Platform - [x] Basic repository set up: https://projects.blender.org/blender/blender-developer-docs - [x] Set up Git LFS - [x] Copy GPU & Viewport pages - [x] Copy Asset System pages - [x] Add developer.blender.org header - [x] Change buildbot to point to new repository - [x] Ask devs to move personal pages (including weekly reports) to Gitea repositories or other platforms. - [x] Plugins: - [x] "Edit Page" button - [x] "Next"/"Previous" page links - [x] Mermaid diagrams - [x] Macro plugin for simple commit/report/PR referencing - [x] License page - [x] Contribute Documentation pages - [x] Custom footer (license information) ### Transition Tasks for the actual transition. This effort should be manageable within ~1 week of work. - [x] Improve conversion script to minimize manual changes needed (see https://projects.blender.org/brecht/wiki-to-markdown) - [x] Convert commit/report/PR references to custom MkDocs macros - [x] Investigate Release Notes porting - How much effort is it to port these? How many releases back to we port? - [x] Define mapping for new navigation structure - [x] Announce transition, ask developers to avoid Wiki edits - [x] `developer.blender.org/docs`? - [x] Initially port pages using https://projects.blender.org/brecht/wiki-to-markdown - [x] Go over all pages, fix remaining issues (organizational task needed) - [x] Fix pages with switch-templates, e.g. [build instructions](https://wiki.blender.org/w/index.php?title=Template:Building_Blender/Linux/intro&action=edit) - [x] Check `trash/` files - [x] Check/fix/quiet all warnings from the conversion script and `mkdocs` - [x] Final pass and approval of navigation structure - [x] Ensure sections without landing page don’t use the first place as landing page instead (`index.md`) - [x] Prepare redirects configuration for Apache - [x] Initial generation of file - [x] Check regexes to use for things like utm - [x] ~~Rewrite git repository history to remove images and clean history~~ *-- No real reason to do this currently, so rather keep history.* - [x] Check on how to handle attribution of past authors (because CC-BY-SA) *-- Link to wiki.blender.org & archive.blender.org/wiki from license page.* ## Day of the Switch What needs to happen on the day on the switch. Needs coordination with the infra team (Arndt, Sergey and Danny) - [x] Apply new edits on the Wiki to ported pages - [x] Add banner on Wiki explaining it has been archived - [x] Make wiki read-only - [x] Deploy redirects on Apache wiki.blender.org server - [x] Update links on blender.org websites - [x] Update developer.blender.org header to point to new docs platform - [x] Update links within Blender - [x] Write announcement ## Soon After Tasks that can be done on the day of the switch, or in the next few days after. - [x] Release checklist integration (update release notes script?) - [x] Update important scripts generating wiki content - [x] Release notes - [x] Weekly reports (https://projects.blender.org/blender/blender/src/branch/main/tools/triage/weekly_report.py) - [x] Add contents to or remove/hide [Submit Changes](https://developer.blender.org/docs/contribute/submit_changes/) page ## Non-Blocking Non-blocking tasks that can be done after the switch, or before. - [ ] ~~Blog Post 2: Rethinking the Structure of Developer Documentation~~ -- not planning to work on this now - [ ] ~~Custom navigation header (Material for MkDocs one doesn't scale enough)~~ -- With the current navigation structure we're fine, so this is a nice-to-have. - [ ] Custom landing page (non-urgent) - [ ] Consider moving files into main Blender repo, and if not integrating it with `make update` - [x] Add mkdocs-redirects extension for when pages get moved? - [x]Update wiki file structure script ([tools/check_wiki/check_wiki_file_structure.py](https://projects.blender.org/blender/blender/src/branch/main/tools/check_wiki/check_wiki_file_structure.py)) - [ ] Move old and big image video files to static location (archive.blender.org?), to make checkout smaller ### Archive Wiki to Static HTML Not urgent if keeping wiki.blender.org running for a while is no big burden. Needs discussion with Arndt, Sergey and Danny. - [x] Which URL to move to? (`archive.blender.org/wiki` already used) -- `archive.blender.org/wiki/2024` - [x] Move to new URL as readonly (static HTML?)

Julian Eisel added the

Type

To Do

label 2023-12-11 18:15:05 +01:00

Julian Eisel added this to the Developer Documentation project 2023-12-11 18:15:07 +01:00

Brecht Van Lommel commented

2024-01-03 16:18:45 +01:00

I reorganized the task list by when we have to do the tasks, to make it more clear what remains to be done before we can switch.

Julian Eisel commented

2024-01-08 19:23:59 +01:00

Prepare redirects configuration for Apache

Initial generation of file

Check regexes to use for things like utm

Regarding UTM:
Apparently the Redirect directive keeps query/GET parameters and appends them to the new URL [1], while RedirectMatch doesn't (see [2] and the equivalent Alias vs. AliasMatch behavior [3]). In the latter case the query can use regex substitution to append the UTM query parameters.

So this should work:

Redirect permanent /wiki/Building_Blender https://developer.blender.org/docs/handbook/building_blender/

Or with regex, something like this:

RedirectMatch permanent ^/wiki/Building_Blender/(.*)$ https://developer.blender.org/docs/handbook/building_blender/$1

I don't see a need for regex & substitution here, just Redirect should be enough.

> - [ ] Prepare redirects configuration for Apache > - [x] Initial generation of file > - [ ] Check regexes to use for things like utm Regarding UTM: Apparently the `Redirect` directive keeps query/GET parameters and appends them to the new URL [[1]](https://httpd.apache.org/docs/2.4/mod/mod_alias.html#redirect), while `RedirectMatch` doesn't (see [[2]](https://httpd.apache.org/docs/2.4/mod/mod_alias.html#redirectmatch) and the equivalent `Alias` vs. `AliasMatch` behavior [[3]](https://httpd.apache.org/docs/2.4/mod/mod_alias.html#aliasmatch)). In the latter case the query can use regex substitution to append the UTM query parameters. So this should work: ```apache Redirect permanent /wiki/Building_Blender https://developer.blender.org/docs/handbook/building_blender/ ``` Or with regex, something like this: ```apache RedirectMatch permanent ^/wiki/Building_Blender/(.*)$ https://developer.blender.org/docs/handbook/building_blender/$1 ``` I don't see a need for regex & substitution here, just `Redirect` should be enough.

Julian Eisel commented

2024-01-12 18:59:11 +01:00

Just FYI, here are the steps I currently see for archiving the Wiki itself, I'll further expand on details so in the end it's mostly just copy & pasting commands/snippets.

Checking with Arnd, it's preferable to archive the Wiki as static HTML right away, which turns out to be easy to do. I have a static HTML version running here: https://julianeisel.github.io/wiki.blender.org-dump/wiki. I checked with Pablo and we want to move the currently archived wiki to archive.blender.org/wiki/2015, and the current Wiki to archive.blender.org/wiki/2024.

@Arnd Move archive.blender.org/wiki to archive.blender.org/wiki/2015
Backup Wiki
@ThomasDinges Announce archival on the chat (E.g. "The Wiki is being archived. It will be moved to archive.blender.org/wiki/2024. The new developer documentation is available at developer.blender.org/docs.")

@JulianEisel Enable wiki.blender.org banner
Create https://wiki.blender.org/wiki/Main_Page/MediaWiki:Sitenotice with content:

<div style="font-size:22px;color:#d0253f;">
<center style="max-width:960px;margin:auto;">
Note: This is an archived version of the Blender Developer Wiki (archived 2024). The current developer documentation is available on [https://developer.blender.org/docs/ developer.blender.org/docs].
</center>
</div>

(Do this first so it's baked into the static HTML.)

@Arnd Put Wiki in read-only mode
In the Wiki's LocalSettings.php, add:

$wgReadOnly = 'This Wiki is archived. The new developer documentation is available at [https://developer.blender.org/docs/ developer.blender.org/docs].';

https://www.mediawiki.org/wiki/Manual:$wgReadOnly

@JulianEisel Run convert.py script to redownload latest wiki files (also downloads creation/modification timestamps and edit count now).

@JulianEisel Download static HTML

nohup wget --recursive --level=inf --page-requisites --adjust-extension --convert-links --no-parent --no-cache --execute robots=off https://wiki.blender.org/wiki/Main_Page

Download/link orpharned pages? https://wiki.blender.org/w/index.php?title=Special:LonelyPages&limit=50&offset=0

@Arnd Update links from

find . -type f -name "*.html" -exec sed -i 's/archive.blender.org\/wiki/archive.blender.org\/wiki\/2015/g' {} \;

@JulianEisel Add redirect to user pages
@Arnd Upload static HTML to archive.blender.org/wiki/2024
@Arnd Deploy redirects (copy into Apache config, restart apache)

@Arnd Update old archive.blender.org.wiki banner
In the archive.blender.org/wiki/2015 directory:

find . -type f -name "*.html" -exec sed -i 's/. The current and active wiki is available on <a href="https:\/\/wiki.blender.org">wiki.blender.org<\/a>/ (archived in 2015). The current developer documentation is available on <a href="http:\/\/developer.blender.org\/docs">developer.blender.org\/docs<\/a>/g' {} \;

Further:

Update archive.blender.org landing page to link to archive.blender.org/wiki/2015 and archive.blender.org/wiki/2024
Replace all wiki.blender.org links in the archived wiki(s)
~~@Arnd Add redirect for archive.blender.org/wiki to archive.blender.org~~ Would make 2015 or 2024 pages impossible to reach, unless we make it an exact match.

Just FYI, here are the steps I currently see for archiving the Wiki itself, I'll further expand on details so in the end it's mostly just copy & pasting commands/snippets. Checking with Arnd, it's preferable to archive the Wiki as static HTML right away, which turns out to be easy to do. I have a static HTML version running here: https://julianeisel.github.io/wiki.blender.org-dump/wiki. I checked with Pablo and we want to move the [currently archived wiki](https://archive.blender.org/wiki/) to `archive.blender.org/wiki/2015`, and the current Wiki to `archive.blender.org/wiki/2024`. 1. [x] @Arnd Move archive.blender.org/wiki to archive.blender.org/wiki/2015 1. [x] Backup Wiki 1. [x] @ThomasDinges Announce archival on the chat (E.g. "The Wiki is being archived. It will be moved to archive.blender.org/wiki/2024. The new developer documentation is available at developer.blender.org/docs.") 2. [x] @JulianEisel Enable wiki.blender.org banner Create https://wiki.blender.org/wiki/Main_Page/MediaWiki:Sitenotice with content: ```html <div style="font-size:22px;color:#d0253f;"> <center style="max-width:960px;margin:auto;"> Note: This is an archived version of the Blender Developer Wiki (archived 2024). The current developer documentation is available on [https://developer.blender.org/docs/ developer.blender.org/docs]. </center> </div> ``` (Do this first so it's baked into the static HTML.) 3. [x] @Arnd Put Wiki in read-only mode In the Wiki's `LocalSettings.php`, add: ```php $wgReadOnly = 'This Wiki is archived. The new developer documentation is available at [https://developer.blender.org/docs/ developer.blender.org/docs].'; ``` https://www.mediawiki.org/wiki/Manual:$wgReadOnly 4. [x] @JulianEisel Run `convert.py` script to redownload latest wiki files (also downloads creation/modification timestamps and edit count now). 6. [x] @JulianEisel Download static HTML ```sh nohup wget --recursive --level=inf --page-requisites --adjust-extension --convert-links --no-parent --no-cache --execute robots=off https://wiki.blender.org/wiki/Main_Page ``` 7. [x] Download/link orpharned pages? https://wiki.blender.org/w/index.php?title=Special:LonelyPages&limit=50&offset=0 7. [x] @Arnd Update links from ```sh find . -type f -name "*.html" -exec sed -i 's/archive.blender.org\/wiki/archive.blender.org\/wiki\/2015/g' {} \; ``` 7. [x] @JulianEisel Add redirect to user pages 8. [x] @Arnd Upload static HTML to archive.blender.org/wiki/2024 9. [x] @Arnd Deploy redirects (copy into Apache config, restart apache) 1. [ ] @Arnd Update old archive.blender.org.wiki banner In the archive.blender.org/wiki/2015 directory: ```sh find . -type f -name "*.html" -exec sed -i 's/. The current and active wiki is available on <a href="https:\/\/wiki.blender.org">wiki.blender.org<\/a>/ (archived in 2015). The current developer documentation is available on <a href="http:\/\/developer.blender.org\/docs">developer.blender.org\/docs<\/a>/g' {} \; ``` Further: - [x] Update archive.blender.org landing page to link to archive.blender.org/wiki/2015 and archive.blender.org/wiki/2024 - [ ] Replace all wiki.blender.org links in the archived wiki(s) - [ ] ~~@Arnd Add redirect for `archive.blender.org/wiki` to `archive.blender.org`~~ Would make 2015 or 2024 pages impossible to reach, unless we make it an exact match.

Julian Eisel commented

2024-01-19 00:38:39 +01:00

One more thing, should we add redirects for user pages to the archived wiki? I guess we should, since they were not ported to the new platform.

Brecht Van Lommel commented

2024-01-19 02:27:31 +01:00

I think so, at least to avoid breaking the weekly reports links.

Maybe it's possible to redirect all wiki.blender.org links to the archive, when they don't already redirect to developer.blender.org/docs?

I think so, at least to avoid breaking the weekly reports links. Maybe it's possible to redirect all wiki.blender.org links to the archive, when they don't already redirect to developer.blender.org/docs?

Arnd Marijnissen commented

2024-01-19 17:48:52 +01:00

Update

The wget mentioned produces a dump that is not complete and doesnt convert all links in the dumped HTML to relative links for the new location of the wiki
This means that Julian's had to take time to figure out how to get a full dump and what data might still need converting/etc.
Also, when the command was changed, the duration of the dumping-process also went up significantly.
Ultimately this has resulted in:

The dump-process still running and needing to be validated when done
The dump being done to Julian's desktop, instead of to the archive-server directly

Next week, monday, we'll investigate how far we are with a coherent/complete/correct dump of the wiki's data and if we can put it on archive.blender.org .
For now, the current archive.blender.org/wiki/2024 dump is 'functional' but not complete and still has links to wiki.blender.org in there.
The wiki.blender.org site is kept up, without further changes ; but does contain a header marking the move of documentation to new location. The content, for now, should be (mostly) identical in any case; but we shouldnt take too long.

Update The wget mentioned produces a dump that is not complete and doesnt convert all links in the dumped HTML to relative links for the new location of the wiki This means that Julian's had to take time to figure out how to get a full dump and what data might still need converting/etc. Also, when the command was changed, the duration of the dumping-process also went up significantly. Ultimately this has resulted in: - The dump-process still running and needing to be validated when done - The dump being done to Julian's desktop, instead of to the archive-server directly Next week, monday, we'll investigate how far we are with a coherent/complete/correct dump of the wiki's data and if we can put it on archive.blender.org . For now, the current archive.blender.org/wiki/2024 dump is 'functional' but not complete and still has links to wiki.blender.org in there. The wiki.blender.org site is kept up, without further changes ; but does contain a header marking the move of documentation to new location. The content, for now, should be (mostly) identical in any case; but we shouldnt take too long.

Julian Eisel referenced this issue

2024-01-25 11:45:55 +01:00

Archived Wiki: Further Processing of Static HTML Dump #117501

Julian Eisel commented

2024-01-25 14:26:20 +01:00

Another update: I had to experiment with the wget download parameters a bit which was hard because downloading was a slow process (had to let one download running over the weekend). Tried its successor wget2 but that kept running into bugs.

Eventually I managed to get a dump that has what we want (plus a bit more, but not too much). We will deploy this now. I've also done some post processing already, to update links. And I've manually downloaded some orphan pages (that are not linked from any page and thus not recursed into by wget). More post processing can follow (e.g. to remove the broken search and any other links to wiki.blender.org), but this is ready for deploying.

We can also now deploy the redirects since we downloaded all relevant data.

Another update: I had to experiment with the wget download parameters a bit which was hard because downloading was a slow process (had to let one download running over the weekend). Tried its successor wget2 but that kept running into bugs. Eventually I managed to get a dump that has what we want (plus a bit more, but not too much). We will deploy this now. I've also done some post processing already, to update links. And I've manually downloaded some orphan pages (that are not linked from any page and thus not recursed into by wget). More post processing can follow (e.g. to remove the broken search and any other links to wiki.blender.org), but this is ready for deploying. We can also now deploy the redirects since we downloaded all relevant data.

🎉 1

Julian Eisel commented

2024-01-25 19:39:19 +01:00

Goodbye Wiki! 👋 🥳

The redirects have been deployed. wiki.blender.org isn't available anymore (brings you to the new developer documentation).
A few fixes to be done still, but it's mostly working as intended.

The old Wiki is archived under https://archive.blender.org/wiki/2024/. Previous user page links will also lead there, e.g. see https://wiki.blender.org/wiki/User:Severin.

## Goodbye Wiki! 👋 🥳 The redirects have been deployed. [wiki.blender.org](https://wiki.blender.org/) isn't available anymore (brings you to the new developer documentation). A few fixes to be done still, but it's mostly working as intended. The old Wiki is archived under https://archive.blender.org/wiki/2024/. Previous user page links will also lead there, e.g. see https://wiki.blender.org/wiki/User:Severin.

Brecht Van Lommel referenced this issue

2024-01-29 16:01:23 +01:00

[WIP] New Developer Documentation Infrastructure: Tasks, Questions, Plan of Action #105349

Brecht Van Lommel referenced this issue

2024-01-29 16:01:39 +01:00

[Draft] New Developer Documentation Infrastructure Proposal (Replacing Wiki?) #105348

Brecht Van Lommel referenced this issue

2024-01-29 16:01:50 +01:00

New Developer Documentation Platform: Navigation #116053

Sign in to join this conversation.