Skip to content

Cloud Storage: provide option to disableGzipContent on InputStream variants #36

Closed
@nblair

Description

@nblair

Is your feature request related to a problem? Please describe.

I have a use case that necessitates using the Storage/Bucket write method variants with InputStream arguments. The inputs for my use case to store in Cloud Storage are:

  • predominantly already compressed,
  • have a wide range of file sizes, from a few KB to GB+,
  • we don't always know the size in advance.

Describe the solution you'd like

Given that our content is already compressed, I would prefer to avoid spending the CPU time on compressing the content again en route to the Bucket.

Describe alternatives you've considered

We have used the byte[] variants, with BlobTargetOption.disableGzipContent and the Compose request. This is suitable but leaves us with a tuning challenge:

  • Buffering these streams into memory (byte[] chunks) requires additional heap space be available
  • Compose is limited to only 32 chunks. If the file content we intend to store is far larger than 32 * bufferSize, we will upload 31 small sized chunks and 1 large chunk that we have to use the InputStream variant for anyways, and pay the additional overhead of gzip compression.

Additional context

I have done some exploration and it appears that the values on the BlobWriteOption (used on InputStream variants) and BlobTargetOption (used on byte[] variants) enums both are translated into StorageRpc.Option. I have a small contribution to offer in the form of a pull request to follow.

Metadata

Metadata

Assignees

Labels

api: storageIssues related to the googleapis/java-storage API.type: feature request‘Nice-to-have’ improvement, new feature or different behavior or design.

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions