Tag Archives: lxml

Generating Apache redirects from sitemap.xml for WordPress using Python

Posted on by
3

Sitemap.xml (Sitemaps) is a protocol exposing website pages for the search engines. Beyond hinting crawler bots you can find some creative uses for it.

I was migrating blog posts from one WordPress site to another. I wanted to have the old site still run the front page and few other static pages and I wanted to move the blog posts only. A best practice for content migrations is set HTTP 301 Moved Permantently redirects on the orignal web server so that users and search engines following old links will automatically find the new content.

WordPress sites were and are hosted on Apache web server. Since writing Apache .htaccess redirects by hand is something no living soul wants to do, I wrote a little Python script using Requests (a.k.a. urllib done right) and lxml libraries.

This script will

  • Read sitemap.xml.gz file
  • Extract page URLs out of it
  • (add optional filtering step here)
  • Output Apache .htaccess redirect rules so that old links will point to the new site

The beauty of scraping the information from sitemap.xml: Even though this script was written for WordPress it is valid for any CMS supporting Sitemaps protocol. (hint: Plone, Drupal, Joomla!, you name it.. they all support it)

In theory one could run this script in cron so that all new WordPress content gets automatically redirected continuously.

In additional for setting up .htaccess WordPress provides easy export and import tool for blog posts through its admin interface.

And the script:

# Python stdlib
from StringIO import StringIO
import urlparse

# 3rd party
import requests
from lxml import etree

# Sitemap of the source site
SITEMAP_URL = "http://mobilizejs.com/wp-content/plugins/simple-google-sitemap-xml/sitemap.xml"

# Where posts will be moved
NEW_DOMAIN="http://opensourcehacker.com"

# Apache .htaccess line template
# http://httpd.apache.org/docs/2.0/mod/mod_alias.html#redirectmatch
REDIRECT_TEMPLATE="RedirectMatch permanent %(path)s$ %(new_domain)s%(path)s"

response = requests.get(SITEMAP_URL)

def create_apache_redirect_to_other_domain(url, new_domain):
    """
    Creates .htaccess redirect entry.

    Create HTTP 301 Permanent Redirect.

    Output is something along lines::

        RedirectMatch permanent /tag/cache-control/$ http://opensourcehacker.com/tag/cache-control/

    """
    parts = urlparse.urlparse(url)

    vars = {
        "new_domain" : new_domain,
        "path" : parts.path          
    }    

    # Simple Python string template substituion
    return REDIRECT_TEMPLATE % vars

# Parse sitemap
tree = etree.parse(StringIO(response.content))
root = tree.getroot()

locs = root.xpath("//*[name()='loc']")

# Iterate through each sitemap <loc> entry and creating corresponding redirect
for loc in locs:
    url = loc.text
    redirect_line = create_apache_redirect_to_other_domain(url, NEW_DOMAIN)
    print redirect_line

 

\"\" Subscribe to RSS feed Image Follow me on Twitter Image Follow me on Facebook Image Follow me Google+

XHTML mobile profile transformer and cleaner for Python

Posted on by
Reply

Mobile phones, and especially mobile site validators, are very picky about the validy of XHTML. It must not be any XHTML, but special mobile profile XHTML. Also, search engines like Google, will punish you in the mobile search results if your site fails to conform to mobile profile.

This is especially troublesome if you display external content (RSS feeds, ATOM feeds) on your mobile site. Incoming HTML cannot be guaranteed to follow any specification.

To solve this problem, we have created gomobile.xhtmlmp Python library which helps you to transform any HTML to content to valid XHTML MP. The library is piloted on plonecommunity.mobi site which  uses aggregated content from varying sources. The library is based on lxml.html.Cleaner. The library is part of GoMobile project which aims to create world class Python mobile web development tools.

1. Highlights

  • Turn any incoming HTML/XHTML to mobile profile compatible
  • Enforce ALT text on images – especially useful for external tracking images (feedburner tracker). ALT texts are required by XHTML MP.
  • Protect against Cross-Site Scripting Attacks (XSS) and other nastiness, as provided by lxml.xhtml.clean
  • Unicode compliant – eats funky characters

As an example we integrated gomobile.xhtmlmp  to Feedfeeder Plone add-on product.

Enjoy.