Automatically removing old items from a Plone site

Below is an advanced version for old item date based deletion code which issuitable for huge sites. This snippet is from Products.feedfeeder package. It will look for Feedfeeder items (automatically generated from RSS) which are older than X days and delete them.

It’s based on Zope 3 page registration (sidenote: I noticed that views do not need to be based on BrowserView page class).

  • Transaction thresholds make sure the code runs faster
  • Logging to Plone event log files
  • Number of days to look into past is not hardcoded
  • Manage rights needed to execute the code

You can call this view like:

http://localhost:9999/plonecommunity/@@feed-mega-cleanup?days=90

… hook it to Zope clock server or run as crond job.

Here is the view Python source code:

import logging

import transaction
from zope import interface
from zope import component
import DateTime
import zExceptions

logger = logging.getLogger("feedfeeder")

class MegaClean(object):
    """ Clean-up old feed items by deleting them on the site.

    This is intended to be called from cron weekly.
    """

    def __init__(self, context, request):
        self.context = context
        self.request = request

    def clean(self, days, transaction_threshold=100):
        """ Perform the clean-up by looking old objects and deleting them.

        Commit ZODB transaction for every N objects to that commit buffer does not grow
        too long (timewise, memory wise).

        @param days: if item has been created before than this many days ago it is deleted

        @param transaction_threshold: How often we commit - for every nth item
        """

        logger.info("Beginning feed clean up process")

        context = self.context.aq_inner
        count = 0

        # DateTime deltas are days as floating points
        end = DateTime.DateTime() - days
        start = DateTime.DateTime(2000, 1,1)

        date_range_query = { 'query':(start,end), 'range': 'min:max'}

        items = context.portal_catalog.queryCatalog({"portal_type":"FeedFeederItem",
                                             "created" : date_range_query,
                                             "sort_on" : "created"
                                            })

        items = list(items)

        logger.info("Found %d items to be purged" % len(items))

        for b in items:
            count += 1
            obj = b.getObject()
            logger.info("Deleting:" + obj.absolute_url() + " " + str(obj.created()))
            obj.aq_parent.manage_delObjects([obj.getId()])

            if count % transaction_threshold == 0:
                # Prevent transaction becoming too large (memory buffer)
                # by committing now and then
                logger.info("Committing transaction")
                transaction.commit()

        msg = "Total %d items removed" % count
        logger.info(msg)

        return msg

    def __call__(self):

        days = self.request.form.get("days", None)
        if not days:
            raise zExceptions.InternalError("Bad input. Please give days=60 as HTTP GET query parameter")

        days = int(days)

        return self.clean(days)

Then we have the view ZCML registration:

<page
    name="feed-mega-cleanup"
    for="Products.CMFCore.interfaces.ISiteRoot"
    permission="cmf.ManagePortal"
    class=".feed.MegaClean"
    />

\"\" Subscribe to RSS feed Image Follow me on Twitter Image Follow me on Facebook Image Follow me Google+

Feedburner, Planet Venus and categorized posts

This post gives some insight for blog owners how to tune WordPress & Feedburner when  posting their posts to various open source planets / aggregation services.

1. Preface

Google’s Feedburner is a popular feed subscriber statistics service for blog owners. Planet Venus is popular feed aggregation service software used by many open source projects to create a website of gathered RSS feeds (an example).

WordPress is a popular blogging software and has a plug-in which will automatically enable feedburner statistics for all the feeds using HTTP redirects. For a blogger using the plug-in means painless set-up of Feedburner statistics on his/her blog.

2. Problem

Usually planets (the aggregation services) are only interest blog posts of a certain category (it this example let’s call the category “plone”). WordPress enables categorized RSS 2.0 feeds using URLs of the following syntax

http://opensourcehacker.com/category/plone/feed/

Then this URL is put into the configs of Planet Venus and Planet Venus starts aggregating posts of a certain category from the source blog.

Now, if the blog is feeding out both categorized posts AND is using a feedburner a problem rises. The feedburner redirects at the feedburner.google.com might not properly handle categorized feeds. Instead, you’ll get an error page saying “looks like your computer might be doing too much automated requests” or something along the line.

3. Solution

Luckily there is an easy fix. In the WordPress Feedburner plug-in settings tick the following option:

Settings -> Feedburner ->

This should allow the aggregators to gather the categorized posts and still use Feedburner stats for the users who subscribe the main RSS feed or your blog.

 

\"\" Subscribe to RSS feed Image Follow me on Twitter Image Follow me on Facebook Image Follow me Google+

XHTML mobile profile transformer and cleaner for Python

Mobile phones, and especially mobile site validators, are very picky about the validy of XHTML. It must not be any XHTML, but special mobile profile XHTML. Also, search engines like Google, will punish you in the mobile search results if your site fails to conform to mobile profile.

This is especially troublesome if you display external content (RSS feeds, ATOM feeds) on your mobile site. Incoming HTML cannot be guaranteed to follow any specification.

To solve this problem, we have created gomobile.xhtmlmp Python library which helps you to transform any HTML to content to valid XHTML MP. The library is piloted on plonecommunity.mobi site which  uses aggregated content from varying sources. The library is based on lxml.html.Cleaner. The library is part of GoMobile project which aims to create world class Python mobile web development tools.

1. Highlights

  • Turn any incoming HTML/XHTML to mobile profile compatible
  • Enforce ALT text on images – especially useful for external tracking images (feedburner tracker). ALT texts are required by XHTML MP.
  • Protect against Cross-Site Scripting Attacks (XSS) and other nastiness, as provided by lxml.xhtml.clean
  • Unicode compliant – eats funky characters

As an example we integrated gomobile.xhtmlmp  to Feedfeeder Plone add-on product.

Enjoy.