archive/pillar
0
0
Fork 0
Code Issues 42 Pull Requests Projects Releases Wiki Activity

Reduced the GCloud resumable upload threshold 5 MiB → 100 KiB

Browse Source 
Files larger than the threshold are streamed from disk, smaller than the
threshold and they are loaded into memory first.
This commit is contained in:
Sybren A. Stüvel 2016-05-17 09:51:33 +02:00
parent 41a278c4f0
commit 4f509951a7
1 changed files with 4 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

4
pillar/application/modules/file_storage.py
View File

@ -546,6 +546,10 @@ def stream_to_gcs(project_id):
stream_for_gcs = uploaded_file.stream stream_for_gcs = uploaded_file.stream
# Upload the file to GCS. # Upload the file to GCS.
from gcloud.streaming import transfer
# Files larger than this many bytes will be streamed directly from disk, smaller
# ones will be read into memory and then uploaded.
transfer.RESUMABLE_UPLOAD_THRESHOLD = 102400
try: try:
gcs = GoogleCloudStorageBucket(project_id) gcs = GoogleCloudStorageBucket(project_id)
blob = gcs.bucket.blob('_/' + internal_fname, chunk_size=256 * 1024 * 2) blob = gcs.bucket.blob('_/' + internal_fname, chunk_size=256 * 1024 * 2)