List subdirectories

[samskillman]

In order to navigate (potentially broad) directory hierarchies in buckets, it would be useful to be able to list the sub-directories in a bucket at a given prefix. Happy to attempt to implement such a method if someone points me in the correct direction. Seems like perhaps a new Iterator?

[dhermes]

@samskillman How about

from gcloud import storage
bucket = storage.Bucket(...)
blob_iter = bucket.list_blobs(max_results=..., prefix=..., delimiter=...)

[samskillman]

I think I wasn't completely clear in the description above. Let's say I have a bucket structure like this:

gs://root_bucket/
    A/blob1
    A/blob2
    B/blob3
    B/blob4

Is there a way to use the list_blobs (or anything else) to return the first level of subdirectory names ["gs://root_bucket/A/", "gs://root_bucket/B/"] rather than all the blobs? This would have the same behavior as gsutil ls gs://root_bucket. Perhaps I don't understand how delimiter and prefix can be used to accomplish this.

[jgeewax]

Being more specific, shouldn't this work:

from gcloud import storage
bucket = storage.Bucket(...)
blob_iter = bucket.list_blobs(delimiter='/')

to get the top level?

And to list subdirectories inside A:

from gcloud import storage
bucket = storage.Bucket(...)
blob_iter = bucket.list_blobs(delimiter='/', prefix='A/')

?

[samskillman]

@jgeewax That is what I had tried originally, but I get empty lists in both cases. From what I can tell in test_bucket.py on line 291, the test actually asserts that the list of blobs is [], so maybe this is an actual bug?

[samskillman]

Huh, but I can do:

iterator = bucket.list_blobs(delimiter="/")
response = iterator.get_next_page_response()
blobs = list(iterator.get_items_from_response(response))
dirs = iterator.prefixes

(Got that hint from regression/storage.py)

[jgeewax]

The docs are a bit clarifying: https://cloud.google.com/storage/docs/json_api/v1/objects/list

If you want "just the stuff immediately in the directory A/", this translates to "Give me the stuff that starts with A/ but doesn't have any other / characters in the non-prefix part of the name, which you do with:

from gcloud import storage
bucket = storage.Bucket(...)
blob_iter = bucket.list_blobs(delimiter='/', prefix='A/')

This will return all stuff that starts with A/ however in the rest of the name, there can't be another /. So A/foo.txt would come back in the list, but A/sub/foo.txt would not come back.

[samskillman]

@jgeewax that would list only the files, right? Then to get the directories I'd have to do what I wrote up above, is that right? How about a bucket.list_prefixes method like this:

def list_prefixes(self, prefix=None):
    iterator = self.list_blobs(delimiter='/', prefix=prefix)
    list(iterator)  # Necessary to populate iterator.prefixes
    for p in iterator.prefixes:
        yield p

[jgeewax]

I think, effectively, what that's doing is going through the list of blobs in the bucket, and collecting all the prefixes. So yes, that should work -- probably just as well as collecting the prefixes from the list of blobs yourself.

If we were going to add a list_prefixes method, it would be tricky. I can't find a way to get "just the top-level list of directories"...

[jgeewax]

@Capstan : What would I do to get the list of all blobs that are "directories" in the root of the bucket?

I thought prefix='/', delimiter='/' would work, but it seems to (true to form) just return the non-directory items (ie, "give me everything starting with / that has no other /s in the name").

What i want is "give me everything starting with / that has exactly one more / in the name" (I think...)

[dhermes]

@samskillman @jgeewax The list(iterator) will page through results and [sets] prefixes every time a new HTTP request is issued. This means that the value iterator.prefixes may not be comprehensive for the entire result set, just for the last request issued. (UPDATE: That's not to say we can't make self.prefixes a set, it just currently isn't implemented that way.)

[Capstan]

@jgeewax @dhermes has it right. FWIW, we don't enforce a preceeding '/' at all, and many objects don't start with that character. If you haven't iterated over the entire list, you won't have the set of top-level directory-likes.

[tseaver]

Given that the separator character is optional, and user defined, we can't really provide a lot of help for this usecase in the library. As @dhermes points out, we could change the iterator such that its prefixes attribute is cumulative across all requested pages: that seems like a reasonable choice.

If we made that change, I imagine that the most reasonable way to accomplish the goal is to write a helper function which "piped the items to /dev/null" and then extracted the relevant prefixes. E.g.:

def extract_prefixes(bucket, prefix=None, delimiter='/'):
    iterator = bucket.list_blobs(prefix=prefix, delimiter=delimiter):
    map(lambda x: None, iterator)
    return iterator.prefixes

[jgeewax]

👍 to that idea, I don't think it belongs in gcloud-python though. However yes, I can see the logic for making prefixes cumulative.

[dhermes]

@tseaver I think it'd be worthwhile to at least have a protected attribute on the Iterator called _current_prefixes just to make the distinction from the set() that is self.prefixes. Worth making it public?