Description
Is your feature request related to a problem? Please describe.
I have a use case that necessitates using the Storage/Bucket write method variants with InputStream arguments. The inputs for my use case to store in Cloud Storage are:
- predominantly already compressed,
- have a wide range of file sizes, from a few KB to GB+,
- we don't always know the size in advance.
Describe the solution you'd like
Given that our content is already compressed, I would prefer to avoid spending the CPU time on compressing the content again en route to the Bucket.
Describe alternatives you've considered
We have used the byte[]
variants, with BlobTargetOption.disableGzipContent
and the Compose request. This is suitable but leaves us with a tuning challenge:
- Buffering these streams into memory (
byte[]
chunks) requires additional heap space be available - Compose is limited to only 32 chunks. If the file content we intend to store is far larger than
32 * bufferSize
, we will upload 31 small sized chunks and 1 large chunk that we have to use theInputStream
variant for anyways, and pay the additional overhead of gzip compression.
Additional context
I have done some exploration and it appears that the values on the BlobWriteOption (used on InputStream
variants) and BlobTargetOption (used on byte[]
variants) enums both are translated into StorageRpc.Option
. I have a small contribution to offer in the form of a pull request to follow.