Read time plugin

[lemon24]

Follow-up to #269.

Currently, the entry read time is computed every time /entries is loaded. It would be really nice if this was done only once, when the entry is updated; now that we have entry tags, we actually have where to put this metadata. A plugin is ideal for this.

Thoughts:

• Should be a stable plugin.
  • Depends on readtime. Time to bring per-plugin extras back? Need to settle naming.
• Two cases:
  • For new/updated entries. Can use after_entry_update_hooks.
  • For old entries (backfill). Harder to do efficiently; likely involves before_/after_feed_update_hooks.
    • If we could filter by entry tags, we could do get_entries(feed=feed, tags=['-.readtime']) in after_feed_update_hooks.
    • Otherwise, how do we know a feed has been backfilled? Using any kind of marker is brittle.
      • Also, I'd like to avoid having a marker tag on each feed.
• What do we take the read time of?
  • All content attributes? (.summary, .content[].value) Sounds like YAGNI; can be added later.
  • Use the web app logic? Should we make it an Entry method?
• It would be nice if we could eventually filter / sort by read time.
  • For now, it means we need to have a relatively simple schema for the tag value.
  • How would a plugin add a new sort key (by a specific tag (sub)value) in a storage-independent way?
• ...

[lemon24]

def after_entry_update_hook(entry):
    set_tag(entry, '.readtime', readtime(entry))
    
# inefficient;
# doesn't do anything for feeds with updates disabled
def after_feed_update_hook(feed):
    for entry in get_entries(feed=feed):
        if has_tag(entry, '.readtime'):
            continue
        set_tag(entry, '.readtime', readtime(entry))

# very efficient, but permanently sets a feed tag;
# doesn't do anything for feeds with updates disabled
def after_feed_update_hook(feed):
    if has_tag(feed, '.readtime.backfill-done'):
        return
    for entry in get_entries(feed=feed):
        if has_tag(entry, '.readtime'):
            continue
        set_tag(entry, '.readtime', readtime(entry))
    set_tag(feed, '.readtime.backfill-done')

# more efficient, needs filter-by-entry-tags feature;
# doesn't do anything for feeds with updates disabled
def after_feed_update_hook(feed):
    for entry in get_entries(feed=feed, tags=['-.readtime']):
        set_tag(entry, '.readtime', readtime(entry))

# efficient, permanently sets just a global tag;
# convoluted; first feed does a lot of extra work, once
def after_feed_update_hook(feed):
    if not has_tag((), '.readtime.backfill-done'):
        for f in get_feeds():
            if f.updates_enabled:
                set_tag(f, '.readtime.backfill-needed')
            else:
                # feeds with disabled updates might never be 
                for entry in get_entries(feed=f):
                    if has_tag(entry, '.readtime'):
                        continue
                    set_tag(entry, '.readtime', readtime(entry))
        set_tag((), '.readtime.backfill-done')
    if not has_tag(feed, '.readtime.backfill-needed'):
        return
    for entry in get_entries(feed=feed):
        if has_tag(entry, '.readtime'):
            continue
        set_tag(entry, '.readtime', readtime(entry))
    delete_tag(feed, '.readtime.backfill-needed')

[lemon24]

Here's a way of backfilling that only requires an extra global tag, and hooks before and after update_feeds():

def before_feeds_update_hook():
    if has_tag((), '.readtime.backfill-done'):
        return
    for feed in get_feeds():
        set_tag(feed, '.readtime.backfill-needed')
    
def after_feed_update_hook(feed):
    if not has_tag(feed, '.readtime.backfill-needed'):
        return
    for entry in get_entries(feed=feed):
        if has_tag(entry, '.readtime'):
            continue
        set_tag(entry, '.readtime', readtime(entry))
    delete_tag(feed, '.readtime.backfill-needed')

def after_feeds_update_hook():
    if has_tag((), '.readtime.backfill-done'):
        return
    for feed in get_feeds(updates_enabled=False):
        for entry in get_entries(feed=feed):
            if has_tag(entry, '.readtime'):
                continue
            set_tag(entry, '.readtime', readtime(entry))
    set_tag((), '.readtime.backfill-done')

[lemon24]

To do:

• before/after_feeds_update_hook
• Entry.get_content() helper (maybe find a better name; remove from webapp)
• don't import plugin modules unless they're requested
• readtime plugin
• setup.cfg extra
• tests
• update web app to use the new read time
• docs
  • docstrings (+versionadded)
  • changelog

[lemon24]

Some stats for the web app after using the readtime tag (60b0459):

	before	after
/entry	0.049	0.009
/?limit=64	0.25	0.19
/?limit=512	1.5	1.0
/?limit=2048	5.7	4.0

Hopefully, this wins most of what was lost when readtime was first added back in 2.6. We could likely get more by making get_entries() also return a set of tags for each entry, instead of doing additional get_tag() calls.

As a bonus, the read time is now also shown for search.