Unable to set blob metadata

[remcohaszing]

I am trying to set custom metadata on a blob. However, this does not work. I'm using the following script:

#!/usr/bin/env python3
import os

from gcloud import storage

client = storage.Client('super secret id')
bucket = client.get_bucket('super secret bucket')

blob = bucket.blob('kirby.png')
blob.metadata = dict(Color='Pink')
with open(os.path.expanduser('~/Pictures/kirby.png'), 'rb') as img_data:
    blob.upload_from_file(img_data)

When retrieving the blob or inspecing it in the developer console, the metadata is still unset.

[tseaver]

Setting the metadata locally does not cause the instance to make the API call to patch it on the server, and we don't send metadata along with the upload_from_file() call. You need to add a call to blob.patch() to persist the changes to blob.metadata.

[remcohaszing]

I modified the script:

#!/usr/bin/env python3
import os

from gcloud import storage

client = storage.Client('dc-app-api')
bucket = client.get_bucket('appsemble-dev')

blob = bucket.blob('kirby.png')
with open(os.path.expanduser('~/Pictures/kirby.png'), 'rb') as img_data:
    blob.upload_from_file(img_data)
blob.metadata = {'Color': 'Pink', 'Spam': 'Cheese'}
blob.metadata['Foo'] = 'Bar'
blob.patch()

This sets the Color and Spam metadata fields, but not the Foo field. There is no warning about this whatsoever.

Also there is no documentation on the Blob class on how to update metadata.

[dhermes]

Thanks for reporting. Looking into it now.

[dhermes]

Ahhh. This is because the Blob.metadata getter returns a copy.deepcopy of the dictionary stored, so doing

blob.metadata['Foo'] = 'Bar'

updates the copy, not the dictionary actually stored on the Blob. We should fix this.

@tseaver WDYT?

[tseaver]

To allow mutation of the metadata in-place (as opposed to assigning to it wholesale), we would need a way to detect those mutations, in order to mark it as changed (for a subsequent call to blob.patch() to pick up).

[dhermes]

Right. Custom dict subclass?

[tseaver]

Right, overriding:

• __delitem__
• __setitem__
• clear
• update
• setdefault
• pop
• popitem

[remcohaszing]

A custom dict class could work for the examples I provided. This would still not make a dict behave as expected tough. Since dicts are mutable, I would expect the following to work:

metadata = {'Holy': 'Grail'}
blob.metadata = metadata
metadata['Ministry'] = 'Silly Walks'
blob.patch()

If the setter of blob.metadata would return this new class, this would no longer refer to the original dict that has been set.

[remcohaszing]

I think this subclass could work if no setter is defined. An empty metadata object could be set in the Blob constructor. This exposes the metadata as if it is a regular dict. This would break with the current behaviour, where setting metadata is allowed.

[dhermes]

The fact that the getter returns a copy is not set in stone. We did it to "protect" the integrity of the actual data own by the class, but didn't think through situations like this.

We don't need a getter, but do need a setter to track changes on the object. Can't have a setter without a getter, so that's just how it's gotta be. (Also, all properties are stored on a non-public attribute called _properties, another reason for a getter.)

[remcohaszing]

I suppose it could look something like this:

class Metadata(dict):
    def __init__(self, blob):
        self._blob = blob

    def __setitem__(self, key, value):
        super(self, Metadata).__setitem__(key, value)
        self._blob._patch_property('metadata', self)

    # XXX Repeat pattern for other modifier functions


class Blob(_PropertyMixin):
    ...

    def __init__(self, ...)
        ...
        self._properties['metadata'] = Metadata(self)

    @property
    def metadata(self):
        return self._properties['metadata']

This marks the metadata as modified if it gets edited and ensures the metadata can't be overridden using a setter.

A side effect is that the metadata property is never None. I personally like that better anyway.

[saxenanurag]

blob.upload_from_file()
blob.metadata = {'luke': 'skywalker'}
blob.patch()

worked for me. I was able to upload a file and apply the meta data after the file upload.

[tseaver]

@saxenanurag I'm in the middle of reviewing how we are using the ex-apitools code (now in gcloud.streaming), and can see how we should be able to send the patched metadata along with the upload request.

[dhermes]

👍 What he said 😀