StorageClient.DownloadObject fails for gzipped data

[Pagonis]

I am downloading statistic reports from my google play developer console using StorageClient

storageClient.DownloadObject("pubsite_prod_rev_...", objectName, outputFile, null, progress); and I am getting the error message:

Incorrect hash: expected 'CXSlwQ==' (base64), was 'UOn3sg==' (base64)
at Google.Cloud.Storage.V1.StorageClientImpl.DownloadObjectImpl(String baseUri, Stream destination, DownloadObjectOptions options, IProgress1 progress) at Google.Cloud.Storage.V1.StorageClientImpl.DownloadObject(String bucket, String objectName, Stream destination, DownloadObjectOptions options, IProgress1 progress)
at GetGooglePlayData.Program.testAuth() in Program.cs:line 99

The downloaded file is exactly the same as downloaded in gsutil or browser.
I guess the issue is reconciliation of compressed and decompressed file. Google reports the file is 229B compressed and 978B decompressed.

The very same issue was already reported by another guy 6 months ago: https://stackoverflow.com/questions/44693652/downloading-from-google-cloud-storage-always-incorrect-hash

[jskeet]

Thanks for the report - and for linking to the SO question. I hadn't seen that before.

It's very odd to get that error. I'm at a conference right now, but I'll take a look tomorrow.
If I were to create a build with extra diagnostics, would you be able to try that? (I assume that none of your files are public - basically it's tricky until I can reproduce it.)

[Pagonis]

Yes, sure, I can try.
I think the problem is the compressed file (gzip in this case. As I can see). I have tried to download a plain text file and my application works without exceptions.

[jskeet]

Great - I've managed to reproduce the issue now.

I believe the problem is that the .NET library is decompressing the content automatically (due to the Content-Encoding) so it's expecting a hash of the uncompressed text, whereas storage is serving the hash of the compressed data.

Out of interest, were you expecting to get back compressed data or not?

[jskeet]

PR #1467 reproduces the issue, but currently I have no workaround. I'm asking internally about how to handle this.

[Pagonis]

Thanks for looking into this.
I would like to download compressed, because it will be faster and it will take less Bytes. Anyway, I will decompress them locally.

[jskeet]

It would definitely download it in compressed form either way - it's really whether you want to receive compressed or decompressed data in user code. I can see both options being useful, depending on your situation.

I'm digging more deeply into this, and I've found at least two, maybe three problems so far - it's one of those complicated bits with lots of moving parts to consider :(

As a really hacky workaround, you could just catch the exception, check that it's for the hash, and then ignore it. The stream will contain all the data at that point anyway. Obviously this is not something you'd want to do for long, but it may help you temporarily.

[sjkp]

For what it is worth, the work around I ended up using was just to do

var buckets = storageClient.GetBucket(bucketName);
var objects = storageClient.ListObjects(bucketName).ToList();
foreach (var o in objects)
{
  await storageClient.Service.HttpClient.GetAsync(o.MediaLink);
}

[jskeet]

@sjkp: I'm glad you were able to work around it, but that way you're not validating the hash either - I'd like to get to the stage where we can validate the hash correctly. This area is a bit of a tricky one in terms of there being lots of interlocking concerns, but I'm hoping to make progress.