Enable search engine indexing in robots.txt #56

New Issue

Brecht Van Lommel · 2023-03-17T11:03:20+01:00

Brecht Van Lommel commented

2023-03-17 11:03:20 +01:00

The current robots.txt completely disables any indexing.
https://projects.blender.org/robots.txt

Being able to search for at least issues and pull requests through search engines is important, both for users and developers to find things and for making the work we do public.

This should be a good starting point (or something we can use as is):
https://gitea.com/robots.txt

The current robots.txt completely disables any indexing. https://projects.blender.org/robots.txt Being able to search for at least issues and pull requests through search engines is important, both for users and developers to find things and for making the work we do public. This should be a good starting point (or something we can use as is): https://gitea.com/robots.txt

Brecht Van Lommel added the

Type

Deployment

label 2023-03-17 11:03:20 +01:00

Brecht Van Lommel changed title from ~~Search engine indexing of projects.blender.org~~ to Enable search engine indexing in robots.txt

2023-03-20 14:57:37 +01:00

Brecht Van Lommel commented

2023-05-01 16:09:11 +02:00

This was enabled recently. However many issues and pull requests still can't be found,. So keeping this open until this either resolves itself as more crawling happens, or we find something else is needed.