Optimize pk-based hyperlinked relationships.

[Archelyst]

Maybe I'm doing something wrong here. When I ask my application for a list of objects (e.g. /my-objects/) it makes 2 initial database queries (one for the count and one for the list). Then during serialization - using hyperlinked serialization - for each related field on each object another query is fired. So I easily end up with more than 200 database queries for a simple list - that's just unacceptable.

Looking into the code it's clear what causes this: in order to create the hyperlinks to the related objects, those objects must be loaded (often the same object is loaded 100 times). So my assumption is that the described behavior is not caused by misconfiguration but by design.

I created a kind of hacky solution which uses the my_related_object_id property to generate the url for my_related_object without loading the object (of which only the id is used anyway). Is that valid? Shouldn't this be something that's done by drf? It seems to work, haven't had any problems so far.

Even when this is solved the combination with nested routes is a whole new adventure, but that's out of the scope of this project / ticket :)

[kevin-brown]

Let me start off by saying that this is a known issue with Django REST Framework, but it's also very difficult to solve for the general use case.

Shouldn't this be something that's done by drf?

While this sounds like a really easy fix that should work for everyone, not everyone links by the id, and that is why we don't do this by default. That doesn't make it a bad move though, as there really isn't a general use case here.

I have had success working around this issue with combinations of select_related and prefetch_related when getting the queryset in get_queryset. It's not something which DRF can detect and add in automatically, as there are tons of cases to consider.

[lovelydinosaur]

Seconding @kevin-brown, tho it would be great if we could come up with a sensible answer to this. Going to re-review as part of the 3.0 release.

[Archelyst]

Thanks for the explanation, sounds plausible to me. I feel encouraged to go with the solution I have so far - we'll see how far it carries. Also, I'm using select_related for the nested routes because they surely need the related objects in order to create the urls for those. In the end the number of models / serializers is limited and the code is not too hard to write. So there is always the option to modify the queryset per model as you suggested.

I'll close this issue and check again in version 3.0. In case I come up with something extremely intelligent and generic I'll let you know, but that's kinda unlikely because we only link via id.

[carljm]

I haven't checked whether this is fixed in 3.0, but FWIW in 2.x PrimaryKeyRelatedField already has some workarounds to avoid unnecessarily querying related objects when all that's needed is their PK, but HyperlinkedRelatedField has no such workarounds, even though it also needs only the PK. For now I'm using this workaround in a subclass of HyperlinkedRelatedField:

+    def field_to_native(self, obj, field_name):
+        """Avoid querying for related objects when all we need is the pk."""
+        source = self.source or field_name
+        if obj is not None and '.' not in source and not self.many:
+            stub = HyperlinkedRelatedStub(
+                **{self.lookup_field: getattr(obj, source + '_id')})
+            return self.to_native(stub)
+        return super(HyperlinkedRelatedField, self).field_to_native(
+            obj, field_name)

where HyperlinkedRelatedStub is just:

class HyperlinkedRelatedStub(object):
    def __init__(self, **kwargs):
        self.__dict__.update(kwargs)

It's not the prettiest solution, due to the need for that stub object, but the stub avoids the need to change the signature of HyperlinkedRelatedField.to_native or HyperlinkedRelatedField.get_url, and I haven't dug in to see if those might be called from elsewhere.

I'm sure that it would be possible to apply a nicer fix-up to HyperlinkedRelatedField that would avoid needless queries on related objects (presuming it hasn't already been done in 3.0).

[lovelydinosaur]

Not yet done for 3.0, but yes planned, and won't be much work at all. Re opening this to ensure it stays tracked.

[lovelydinosaur]

Any review on the PR appreciated. This:

• No longer makes the lookup on FKs if the hyperlink is pk based. We already have the value.
• Uses values_list('pk', flat=True) for relationship lookups on M2M and reverse FKs if the hyperlink is pk based. Same number of queries, but only the pk is being retrieved.