infrastructure/extensions-website
4
8
Fork 10
Code Issues 20 Pull Requests 2 Projects Releases Wiki Activity

Implement basic robots.txt #264

Merged
Dalai Felinto merged 2 commits from robots into main 2024-10-21 16:49:20 +02:00
Conversation 1 Commits 2 Files Changed 3 +15 -1
Dalai Felinto commented 2024-10-21 15:45:28 +02:00
Owner
Copy Link

It basically makes sure that the approval queue does not get crawled.

Example of crawling this patch is trying to avoid (see "Review: Mio3 UV"). This was pointing to the approval queue:

It basically makes sure that the approval queue does not get crawled. --- Example of crawling this patch is trying to avoid (see "Review: Mio3 UV"). This was pointing to the approval queue: <img src="http://projects.blender.org/infrastructure/extensions-website/attachments/eaae13d5-0a73-4baa-a7af-e04ebb3398c6" width="250" />
telegram-cloud-photo-size-1-4951887393397517560-y.jpg
105 KiB
Dalai Felinto added 1 commit 2024-10-21 15:45:29 +02:00 
It basically makes sure that the approval queue does not get crawled.
Dalai Felinto requested review from Francesco Siddi 2024-10-21 15:45:59 +02:00
Dalai Felinto added 1 commit 2024-10-21 15:51:06 +02:00
Dalai Felinto commented 2024-10-21 15:53:00 +02:00
Author
Owner
Copy Link

I wonder if we should make the API not crawlable as well.
So basically:

  • /oauth
  • /admin
  • /api
  • /approval-queue

At the moment only approval-queue is in the robots.txt

I wonder if we should make the API not crawlable as well. So basically: * /oauth * /admin * /api * /approval-queue At the moment only approval-queue is in the robots.txt
Francesco Siddi approved these changes 2024-10-21 16:17:28 +02:00
Francesco Siddi left a comment
Owner
Copy Link

I think this is ok.

I think this is ok.
Dalai Felinto merged commit 41c4ae35e9 into main 2024-10-21 16:49:20 +02:00
Dalai Felinto deleted branch robots 2024-10-21 16:49:21 +02:00
Sign in to join this conversation.
Reviewers
No reviewers
Francesco Siddi
Labels
Clear labels   
Priority
Critical
The priority is critical

  
Priority
High
The priority is high

  
Priority
Low
The priority is low

  
Priority
Normal
The priority is medium

  
Reviewed
Confirmed
Issue has been confirmed

  
Reviewed
Duplicate
This issue or pull request already exists

  
Reviewed
Invalid
Invalid issue

  
Reviewed
Won't Fix
This issue won't be fixed

  
Status
Abandoned
Somebody has started to work on this but abandoned work

  
Status
Blocked
Something is blocking this issue or pull request

  
Status
Need More Info
Feedback is required to reproduce issue or to continue work

  
Type
Breaking
Breaking change that won't be backward compatible

  
Type
Documentation
Documentation changes

  
Type
Enhancement
Improve existing functionality

  
Type
Feature
New functionality

  
Type
Report
Something is not working

  
Type
Security
This is security issue

  
Type
Suggestion
New ideas and proposals

  
Type
Testing
Issue or pull request related to testing

No Label
Priority
Critical
Priority
High
Priority
Low
Priority
Normal
Reviewed
Confirmed
Reviewed
Duplicate
Reviewed
Invalid
Reviewed
Won't Fix
Status
Abandoned
Status
Blocked
Status
Need More Info
Type
Breaking
Type
Documentation
Type
Enhancement
Type
Feature
Type
Report
Type
Security
Type
Suggestion
Type
Testing
Milestone
Clear milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
No Assignees
2 Participants
Notifications
Due Date
The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: infrastructure/extensions-website#264
No description provided.
Delete Branch "robots"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?