Retry GCS on 503 errors #49982

New Issue

Sybren A. Stüvel · 2016-11-10T09:01:44+01:00

Sybren A. Stüvel commented

2016-11-10 09:01:44 +01:00

Google Cloud Service sometimes throws 503 Service Temporarily Unavailable errors at us. We might want to retry once or twice after such a reply, to see if the service has come up. However, we should consider this carefully, given that:

we probably want to pause for a second or two between retries, to give GCS to come up and service us again, and
during this delay the Pillar process is sleeping, blocking others from performing requests on it.

If we do this the wrong way, a hickup at GCS could cause all Pillar processes to hang in a retry loop, DOSsing ourselves.

Probably related to #48956.

Google Cloud Service sometimes throws 503 Service Temporarily Unavailable errors at us. We might want to retry once or twice after such a reply, to see if the service has come up. However, we should consider this carefully, given that: - we probably want to pause for a second or two between retries, to give GCS to come up and service us again, and - during this delay the Pillar process is sleeping, blocking others from performing requests on it. If we do this the wrong way, a hickup at GCS could cause all Pillar processes to hang in a retry loop, DOSsing ourselves. Probably related to #48956.

Sybren A. Stüvel commented

2016-11-10 09:01:44 +01:00

Changed status to: 'Open'