I am speaking at IIUG 2012 about using Python with Informix

The 2012 IIUG (International Informix User Group) conference will be in San Diego, California from April 22 - 25 2012. All three of my talk proposals have been accepted, and one of these is about using Python with Informix. As well as preparing my presentation, I have been working on a number of Python Open Source projects either adding or improving their support for Informix access. So hopefully around the time of the conference I will be able to release a more complete django-informix and support for Informix in Gerald, a database schema comparison tool.

The abstract of my Python talk is as follows:

Add a Python to your Informix programming toolbox

Python is an open-source, dynamically typed, object-oriented programming
language. It offers an easy learning curve and access to a vast array of libraries
including Informix database access. With implementations available for most
operating systems as well as the versions that run under the Java and .NET
virtual machines, you can add Python to your Informix database administration
and application programming toolbox. The presentation will give an overview of
the Python programming language, discuss the Python DBI and the libraries that
allow access to Informix databases.

I am speaking at PyCon AU 2011 about CouchDB

The official schedule for PyCon Australia 2011 has been announced (http://pycon-au.org/2011/conference/schedule/). My talk is the first session after the opening keynote and will be an overview of CouchDB and how you can use it with Python.

"CouchDB  (http://couchdb.apache.org/) is an open source, document-oriented NoSQL Database Management Server.It supports queries via views using MapReduce, and replication. The talk will give an overview of CouchDB followed by how to access and manipulate using Python. There are a number of python libraries for accessing couchdb and these will be quickly discussed followed by  how to use one of these libs with a Python web framework. Also there will be an example of using a Python view server in place of the standard Javascript views provided by couchdb."

PyCon Australia is Australia's only conference dedicated exclusively to the Python programming language, and will be held at the Sydney Masonic Center over the weekend of August 20 and 21.

Python Informix Database Connection Options

I am currently at the International Informix Users Group Conference (http://www.iiug.org/index.php) in Kansas. In the opening keynote by Jerry Keesee, there some discussion about IBM's Open Source Initiatives for Informix. On the accompanying slide, Python and Django were listed. This reminded me that I hadn't taken stock of what the Informix DB connections options were for the Python user lately. A quick "google" produced the following:

Python DB-API (http://www.python.org/dev/peps/pep-0249/) adaptors

1. InformixDB http://informixdb.sourceforge.net/ The original DBAPI compliant database adaptor that wraps the Informix CliSDK. BSD License.
2. ibm-db http://code.google.com/p/ibm-db/ IBM sponsored connector that provides Python, Django and SQLAlchemy support for IBM DB2 and Informix. Connects via the IBM DataServer component. Apache 2.0 License.
3. mxODBC http://www.egenix.com/products/python/mxODBC/ Provides an interface to databases via an ODBC interface so supports Informix via Informix CLiSDK. Commerical License.

I have used both items 1 & 3 and they do work very well. Haven't used ibm-db as our sites normally only have CliSDK installed not IBM DataServer.

My preferred combination for Python Informix database access is using InformixDB in conjunction with SQLAlchemy.  SQLAlchemy allows me to choose how abstracted I want to be with my database, allowing direct SQL thru to utilizing an ORM.

I finally own a smartphone, and it's not an iPhone

After years of being an outcast amongst my colleagues, happy with a simple mobile phone that just made phone calls, I made the decision to upgrade to a smartphone. Why would I, since I was actually happy with my "no frills" Nokia and it's excellent battery life which makes a smartphone look retarded. At work we are working on making our web applications more mobile friendly, so we needed a range of smartphones to get a "real" feel of usability and performance, especially when utilising HTML bells and whistles.

Since a MacBook Pro is my development platform of choice (with VMWare Fusion allowing me to use Linux and Windows whenever I want to), and that I am a happy iPad user, it would be assumed that I would get an iPhone. Since there was already a glut of iPhones and Blackberries amongst my co-workers, and I have been envious of my Google friends and their Android phones. So a "droid" shopping I went. So what did I get - a powerful HTC or Samsung Galaxy, import a Nexus S? No, I purchased an affordable (AUD299), middle of the range Motorola XT5 running Android Eclair 2.1 Why? As I said previously, we wanted to test "real" usability and performance, and the majority of our users who had Andriod phones would not be running 2.2/2.3 or have the latest 1 GHz processors. As developers we need to remember that our users do not always have the luxury of upgrading when the next great piece of hardware and/or software arrives, and we need to ensure our applications perform well on a wide range of platforms. Certainly we should the extra capabilities of the latest devices, but also ensure a good experience for those users without the extras.

To-date I am very happy with performance of my XT5 and Eclair functionality. Now if I can stop writing little python scripts that do things on my phone, I can get back to testing the application performance.

SyPy Meetup - CoffeeScript & Test Driven Development

Last night was the September gathering of the Sydney Python Users Group.

The meeting started with a lightning talk about CoffeeScript (http://jashkenas.github.com/coffee-script/) which is a little language that compiles into Javascript. CoffeeScript borrows some of it's syntax from Ruby, Haml and Python. The javascript it generates uses only the good parts.

Then Vaughan Allan (@vornstar) gave a presentation on TDD with Python. It was a good introduction of what TDD is and the benefits. Slides are here. Dylan Jay followed this with a case study on how he used TDD for Google App Engine project. Q&A became a vibrant discussion about the pros and cons of TDD, best practices etc which proved to me how valuable the monthly meetups are.

Thanks to Google for the venue, food and drink.

MSC Malaysia OSCONF 2009

In Kuala Lumpur, Malaysia from May 31 - June 3 there will be the inaugural MSC Malaysia OSCONF. This conference is designed to bring together Government, Business, Open Source Developer and User communities to showcase and expand Open Source's contribution to the IT industry and socio-economy. The conference program consists of a one day hackathon, followed by the three day main conference. The main conference has three tracks:

• Executive Download - Panels and presentations to introduce Government and Business to Open Source.
  
• Developer - Modelled on OSDC, presentations by Open Source developers for developers.
  
• Community - Presentations for users of Open Source software.
  

The tentative schedule can be viewed here. There is a broad range of international and Malaysian speakers.

As a developer who spends some time each year in Malaysia, and has seen some of the interesting things the Open Source community is doing there, I would recommend this event as a great reason to visit Malaysia.

My OSDC 2007 talks

Today I gave my two talks at OSDC.

This morning I gave a talk on web testing using twill and selenium remote control. A pdf of the slides can be downloaded:

Testing Web Applications with Scripting Languages

And this afternoon I spoke about Moonlight, the Mono implementation of Silverlight.

Moonlight - Shiny, Pretty Things with XML?

I had intended to blog about the various OSDC sessions I have attended, but thanks to the lazy web, Alan Green has already done it for me

mod_wsgi 1.0 Release Candiate Available

Today Graham Dumpleton tagged the SVN and has made available the release candidate of mod_wsgi. The mod_wsgi adapter is an Apache module that provides a WSGI compliant interface for hosting Python 2.3+ Python based web applications under Apache. Graham has written it completely in C code so it has a lower overhead than the mod_python or CGI adapters. It can be used with Apache 1.3, 2.0 or 2.2 and normally runs in 'embedded mode' where the application code is executed within the context of normal Apache child processes. If you run mod_wsgi with Apache 2.x on a Unix/Linux system, it can also run in a 'daemon mode' where one or more distinct processes can be dedicated to one or more WSGI applications. Graham has put together great documentation including instructions on using mod_wsgi with some of the common Python web frameworks and applications including Trac, Django, Pylons, MoinMoin and TurboGears.

Adal Chiriliuc has built Win32 binaries of the release candidate and these can be found here.

I have lucky enough to have been associated with the testing of mod_wsgi since it's inception and want to thank Graham for all his efforts in developing and releasing a WSGI adapter of high quality and functionality. Also thanks to other testers who kept Graham honest and busy. In my humble and most likely biased opinion, I think it will become the WSGI adapter of choice when serving Python applications under Apache.

Sydney traffic chaos disrupts SyPy meeting

Yesterday I hosted the July SyPy meetup at my work. Since the office is North Sydney based, many pythoneers would need to travel to the meeting using Sydney's rail network. It doesn't have the best record for reliability, and yesterday was a classic example. A panel was ripped off a CityRail train as it crossed the Harbour Bridge resulting to commuter chaos for many hours. The good news is that python programmers are a hardy bunch and ten overcame the chaos so we still managed to have a great meeting. Tim Churches gave a talk on the NetEpi Project which is open-source, network-enabled tools for epidemiology and public health practice. It is consists of two components:

1. NetEpi Collection, a Web-based outbreak data management tool, written in Python and using open-source components and infrastructure, including the PostgreSQL database.
2. NetEpi Analysis, a tool for interactive exploratory data analysis of large (10-100 million rows) population health/clinical data sets. It is mainly written in Python, uses Numeric Python (NumPy) and the R statistical environment.

It is good to see that at least one NSW government department is making an investment in open source. Nigel Tao talked about the Gnome Deskbar Applet, which is written in Python and can be extended using python scripts. He also introduced us to YubNub, the commandline for the web. Something else to distract me, thanks Nigel. Dyan Jay, web designer, and entrepreneur did a lightning talk about AGX which is a code-generator for CMF/Plone Products based on the Archetypes framework. The design is done in UML, saved in XMI-Format, and AGX parses the saved model, producing the python code.

Last nights Sydney Python Group Meetup

Last night was the monthly SyPy meetup. Thanks to generosity of Google the venue was the Fraser meeting room at the Sydney Googleplex. It was a great turnout of Sydney Pythoneers with 30+ attending.

First Alec Thomas gave a talk about PyCon 2007 and what he felt were the 10 best talks he attended. Alec's presentation can be found here. Then Andrew Bennetts gave a talk about the Bazaar GPL Distributed Version Control Software. Andrew did a good introduction of what Bazaar can do, then used the rest of the talk time to answer questions. Based on the number of questions, there is certainly alot of interest in this software tool.



After the meetup had finished, a number of us with a thirst for more discussion and beer headed to the cafe/bar next to the Google building where access to happy hour prices had been extended for us. Had some interesting discussions including one with cafe owner about the ultimate ui design for a bar point of sale.

Thanks to Alan Green for his organising skills and Google for providing an excellent venue.

PyCon Envy

I am jealous! I want to be at PyCon 2007 and reading the few blog posts (I think everyone there is having too good a time to blog) makes me want to be there even more. Hopefully the video and audio will be uploaded soon. Guess I will just have to save, convince the "better half" it's for the good of the family and make it to PyCon 2008. Alec Thomas will be giving a debrief on his time at PyCon 2007 for the April SyPy meeting so more envy likely.

mod_wsgi Performance

Graham Dumpleton has published his benchmarks for mod_wsgi.

mod_wsgi - simple and fast apache wsgi serving

There are a many ways to serve a WSGI application via Apache with FastCGI appearing to be the favorite. If you have to support many diverse operating systems running different versions of Apache, serving a WSGI application via CGI is a guaranteed but not fast deployment method. You may be wondering why I haven't suggested the WSGI mod_python adaptor as a solution. I am sure it is a workable alternative but it requires mod_python and I have had lots of issues getting it to compile for some platforms and to work with a specific Python interpreter. Now there is a new solution.

Graham Dumpleton has been working on an Apache module called mod_wsgi. Quoting from his website:

"The mod_wsgi adapter is an Apache module that provides a WSGI compliant interface for hosting Python based web applications within Apache. The adapter is written completely in C code against the Apache C runtime and for hosting WSGI applications within Apache provides significantly better performance than using existing Python based WSGI adapters for mod_python. The module can be compiled for and used with either Apache 1.3, 2.0 or 2.2."

I have been lucky enough to get early access to the code and try it with my WSGI applications. I able to compile, configure and have mod_wsgi running a simple WSGI application under Apache 2.0.55 and Python 2.4.3 in under 5 minutes, Certainly a much better experience than I have had with mod_python. Since I have been testing it on my "gutless" laptop rather listing the results of my benchmarking, I will leave Graham to publish his more comprehensive benchmarking results. But I will make some observations. It appears to run simple WSGI applications at least twice as fast as mod_python and 15-20 times faster than using CGI. I was also able to run a number of my WSGI applications that use either Pylons or combinations of WSGI middleware with no problems. At yesterdays SyPy meeting, Graham showed it running Django. The official release will hopefully be in March.

SyPy Social Meetup

Last night was the second meeting of the Sydney Python Users Group for 2007 and we had a good turnout.

There was a wide range of topics discussed including:

• Unit Testing
• Software Deployment
• Distributed Version Control
• Apache and WSGI - more about this in another post
  

So if you are in the Sydney and are interested these types of topics, why not join us the first Thursday of each month. Details of the next meeting can be found at:

http://groups.google.com/group/sydneypython/web/sypy-home

Hope to see you in March.

IronPython and trace style debugging

When developing with IronPython under Windows, there is excellent debugger support via Visual Studio. But sometimes you just want to do simple trace style debugging, in other words, put some print statements in your code. This works well when developing an application you can run under the console but is problematic if the application is a service or ASP.NET handler. When developing similar applications with CPython under Windows, I used the win32trace.pyd/win32traceutil.py to achieve this.

From the win32 python docs:

These modules allow one Python process to generate output via a "print" statement, and another unrelated Python process to display the data. This works even if the Python process generating the output is running under the context of a service, where more traditional means are not available. The module works on Windows 95 and Windows NT.

To enable this debugging, simply "import win32traceutil" somewhere in your program - thats it! This automatically re-directs the standard Python output streams to the remote collector (If that behaviour is undesirable, you will need to use win32trace directly.) To actually see the output as it is produced, start a DOS prompt, and run the file "win32traceutil.py" (eg, double-click on it from Explorer, or "python.exe win32traceutil.py") This will print all output from all other Python processes (that have imported win32traceutil) until you kill it with Ctrl+Break!)

So I decided you create something similar for IronPython. win32trace uses a memory mapped file for communication between the application and the trace collector. Since the only way I could find to use memory mapped files under CLI, relied on calls to unmanaged win32 api, I decided to use network UDP datagrams instead. Rather than writing something from scratch, I remembered a python script that I had used many years ago, creosote.py by Jeff Bauer. The original link to the script is broken, so I finally tracked a copy down in the Zope CVS. So I wrapped this code in a wrapper traceutil.py.

To use it, you just need to put this script in your IronPython or CPython path, and any output to stdout or stderr will be sent to the collector.

To start the collector:

ipy.exe traceutil.py

If the following code is run as a script by IronPython or CPython:

import traceutil

print "start"
a = "A"
# Create an error
import monty

the following would appear in the collector console running under .NET:

creosote bucket waiting on port: 7739
start
\n
Traceback (most recent call last):\r\n File E:\\IronPython\\IPCE-r4\\test_traceutil.py, line 6, in Initialize\r\n File , line 0, in __import__##4r\nImportErr
or: No module named monty\r\n

and the following would appear in the collector console running under Mono:

creosote bucket waiting on port: 7739
start
\n
Traceback (most recent call last):\n File test, line unknown, in Initialize\nImportError: No module named monty\n

It is important to note that the creosote client code does not block if the collector isn't running which is a good behaviour.

Roundup and WSGI

In a previous post I mentioned that I had an adaptor to run the Roundup Issue Tracker as a WSGI application. I never actually published the location of the code. So if you want to have a play, it can be found here. There are also 2 example ini files for using the adaptor with Paste.Deploy

Currently there is one unresolved issue: when using Roundup's internal authentication, after login the browser is not always re-directed correctly. This is not an issue for me as I use WSGI based authentication middleware.

10 Nov 2006
As of Roundup 1.3, Richard Jones has added a wsgi handler so I suggest you use that one instead.

IronPython and ADO.NET Part 2

This is the second in a series of posts about database access with IronPython and ADO.NET. This post will discuss connecting to the database and executing basic DDL and SQL statements via the Python DB-API instead of directly accessing ADO.Net. So that the examples can run on Windows and non Windows systems, they will support either SQLite3 via the Mono.Data.SQLiteCilent ADO provider or Microsoft Access via the System.Data.Odbc provider.

The Python DB-API is a specification created by the Python Database SIG for a consistent interface to relational databases. For CPython there is at least one DB-API compliant library for most of the relational database engines that are used today. As part of his fepy project, Seo Sanghyeon has created a set of wrappers that provide DB-API support for MySQL, PostgreSql, SQLite, Microsoft SQL Server and ODBC ADO.NET database drivers. Since one of the features of ADO.Net is to also provide a consistent interface to relational databases, you may question why do you need to use another layer for database access with IronPython. The answer is simple, the DB-API allows you to focus on the actual access and manipulation of the data and hides the low-level ADO.Net setup and management code. To show how the DB-API can simpify your IronPython code, the examples from the first post of this series have been modified to use the DB-API.

If you want try the example or use the DB-API with IronPython you will need to install it. You can either download the modules from here and copy to the IronPython Lib directory or build and/or install the IronPython Community Edition which includes the DB-API.

Creating a table example

The first section of code is only required so the examples will work on either *ix or windows platforms. In normal usage, there is no need to import dbapi.py directly, just import the DB-API module for the database you want to use.

import dbapi
try:
  import sqlite3 as db
  dbapi._load_type(db.assembly,db.typename)
  connectstr = 'ip2country.db'
  ip2country_create_table_ddl = '''
    CREATE TABLE ip2country (
      ipfrom          INTEGER,
      ipto            INTEGER,
      countrycode2    CHAR(2),
      countrycode3    CHAR(3),
      countryname     VARCHAR(50),
      PRIMARY KEY (ipfrom,ipto)
    )
'''
except:
  import odbc as db
  dbapi._load_type(db.assembly,db.typename)
  connectstr = 'DSN=ip2country'
  ip2country_create_table_ddl = '''
    CREATE TABLE ip2country (
      ipfrom          DOUBLE,
      ipto            DOUBLE,
      countrycode2    CHAR(2),
      countrycode3    CHAR(3),
      countryname     VARCHAR(50),
      CONSTRAINT ip2country_pk PRIMARY KEY (ipfrom,ipto)
    )
'''

Comparing this code with the ADO.Net example you see that the opening of the database connection, and create a command instance is automatically done by the DB-API module. The Python DB-API PEP specifies implicit transactions that are started automatically and committed or rolled back on demand, so compared to ADO.Net example , a commit is required.

dbcon = db.connect(connectstr)

cursor = dbcon.cursor()

cursor.execute(ip2country_create_table_ddl)

dbcon.commit()

dbcon.close()

Load the data example.

Compared to the ADO.Net example, the DB-API allows the IronPython code to be simpler as it handles the creation of Parameters. (Note to self: MS Access, bulk inserts and transactions == very slow)

dbcon = db.connect(connectstr)

import re
re_csv = re.compile(',(?=(?:[^\"]*\"[^\"]*\")*(?![^\"]*\"))')

cursor = dbcon.cursor()
insert_statement=  '''
INSERT INTO ip2country (
ipfrom, ipto, countrycode2, countrycode3, countryname
) VALUES ( ?,?,?,?,? )
'''

f = open("ip-to-country.csv")
print "Loading..."
for line in f.readlines():
  if line.endswith("\r\n"):
    line = line[:-2] # remove \r\n
  else:
    line = line[:-1] # just remove \n
  print line
  ipf, ipt, cc2, cc3, cn = re_csv.split(line)
  cursor.execute(insert_statement,(ipf[1:-1],ipt[1:-1],cc2[1:-1],cc3[1:-1],cn[1:-1]))
  dbcon.commit()

f.close()
dbcon.close()

Select some data example.

Instead of using the ExecuteReader as in the ADO.Net example, the fetch method of the cursor instance is used to get the query result.

def ip2number(ipaddress):
  '''
  Convert dotted IP address to number
  '''
  A,B,C,D = ipaddress.split(".")
  return (int(A) * 16777216) + (int(B) * 65536) + (int(C) * 256) + int(D)

dbcon = db.connect(connectstr)

cursor = dbcon.cursor()

try:
  ipaddress = sys.argv[1]
  # Convert dotted ip address to number
  ipnumber = ip2number(ipaddress)
except:
  print "Error - An IP Address is required"
  sys.exit(1)

select_statement = '''SELECT * FROM ip2country
WHERE ipfrom <= %s     AND ipto >= %s
''' % (ipnumber, ipnumber)

cursor.execute(select_statement)

row = cursor.fetchone()
print "The location of IP address %s is %s." % (ipaddress, row[4])

dbcon.close()

IronPython Community Edition R3 Released.

Yesterday Seo Sanghyeon announced that the third release of IPCE was available. This release uses the current IronPython stable version 1.0.1 and the modules of the CPython 2.4.4 standard library that are known to work under IronPython. Also Sanghyeon has been busy creating additional CPython-compatible wrappers which are included with this release:

• zlib, using System.IO.Compression.
• hashlib, using System.Security.Cryptography
• sqlite3, using generic DB-API module

I think another important change is the move to a MIT license. There had been some negative discussion about IPCE's previous WTF license, so hopefully this change means more people will consider using IPCE. So if you want a version of IronPython that has "batteries included", give IPCE a go.

You can download it from here or here.

Atom Publishing Server using WSGI

On XML.com "Joe Gregorio's latest Restful Web column implements the Atom Publishing Protocol as a Python web service using WSGI." Only discovered this today, hopefully the second part will be coming soon. Joe also has some other interesting wsgi related posts on his blog

• WSGI Collections
• JSON Server
  

Adding .pth file support to IronPython

In a blog post I read, there was the following statement:

'Damnit. Apparently IronPython doesn’t support .pth files. I’m not sure if I should expect this or not from a “1.0” product, but it’s sure annoying since it seems most libraries use them.'

For me, it is an issue due the fact that many of the libraries I want to use come as Python Eggs, and the easy-install.pth file is required if you want them to work. So I decided to see what was needed to add .pth support to IronPython. In the end it was not too hard as the logic to add the contents of the pth files to the sys.path is included in the CPython site.py. It needed a little modification, but if you add this code to IronPython's site.py, .pth files work.