Python Enum continued

I needed to convert an enum to python so that I could decode some responses from C code. Here is an example of quickly converting a C enum to python.
The C code I am working with has several enum structures, one is a long list of error types:

enum LadybugError
{
   LADYBUG_OK,
   LADYBUG_FAILED,
   LADYBUG_INVALID_ARGUMENT,
   ...
   LADYBUG_INVALID_FRAMERATE,
   LADYBUG_INVALID_OFFSCREEN_BUFFER_SIZE,
   LADYBUG_INVALID_JPEG_IMAGE_STRUCTURE,

   LADYBUG_NUM_LADYBUG_ERRORS,

   LADYBUG_ERROR_FORCE_QUADLET = 0x7FFFFFFF,
}

There are about 70 items in this list.
Here is the conversion using namedtuple:

LadybugError = namedtuple('Enum', '''
   LADYBUG_OK,
   LADYBUG_FAILED,
   LADYBUG_INVALID_ARGUMENT,
   ...
   LADYBUG_INVALID_FRAMERATE,
   LADYBUG_INVALID_OFFSCREEN_BUFFER_SIZE,
   LADYBUG_INVALID_JPEG_IMAGE_STRUCTURE,

   LADYBUG_NUM_LADYBUG_ERRORS''')
LadybugError = LadybugError._make(range(len(LadybugError._fields)))

What is nice about namedtuple is that the keywords can be given as one long string. It parses the string for you by spaces and commas. This saves me the trouble adding quotes around each item, just add quotes around the whole list. I overwrite the original ‘LadybugError’ string with the enumeration because it is no longer needed.
When a program returns an integer representing an error, it can be passed to _fields[] to get the error string.

err = libc.ladybugGetStreamHeader( pContext, streamHeaderInfo, None )
print err, LadybugError._fields[err]

The output from this is:

20 LADYBUG_NOT_INITIALIZED

The value of err is 20 and ‘LADYBUG_NOT_INITIALIZED’ is the 21st item in the LadybugError enumeration.

Because I only need to interpret numerical feedback, I can drop the “_make(range” line and just take advantage of namedtuple’s builtin parsing:

LadybugError = namedtuple('Enum', '''
   LADYBUG_OK,
   LADYBUG_FAILED,
   LADYBUG_INVALID_ARGUMENT,
   ...
   LADYBUG_INVALID_FRAMERATE,
   LADYBUG_INVALID_OFFSCREEN_BUFFER_SIZE,
   LADYBUG_INVALID_JPEG_IMAGE_STRUCTURE,
   LADYBUG_NUM_LADYBUG_ERRORS''')

This helps me easily find out what an error of 12 means:

print error, LadybugError._fields[error]

prints

12 LADYBUG_COULD_NOT_OPEN_FILE

This works smoothly if the enumeration starts with zero. Otherwise, getting the name through _fields[i] will not be correct. Also, remember that exceptions can be added using ._replace(field=new_value). Here’s an example:

LadybugBusSpeed = LadybugBusSpeed._replace(LADYBUG_SPEED_UNKNOWN=-1)
err = -1
print LadybugBusSpeed._fields[LadybugBusSpeed.index(err)]

Python Enum with namedtuple

I saw a question asking how to do enum in Python and because I just learned about namedtuple I thought it should be an easy thing to do.

For an enum starting with 0, you can write it like this:

from collections import namedtuple as ntup

animalnames = 'Dog','Cat','Mouse','Horse','Bunny'

Animal = ntup('Enum', animalnames)._make(range(len(animalnames)))

Then you can

print Animal.Horse

and get 3 returned.
Also, because it is also a tuple, you can access it by the index:

Animal.Horse == Animal[3]

For an enum starting at 1 or higher there are a few ways.
You can use arange and add an offset:

from numpy import arange
animalnames = 'Dog','Cat','Mouse','Horse','Bunny'
Animal = ntup('Enum', animalnames)._make(3+arange(len(animalnames)))

Or stay with range:

animalnames = 'Dog','Cat','Mouse','Horse','Bunny'
Animal = ntup('Enum', animalnames)._make(range(3,3+len(animalnames)))

What is be nice about using this method is that it is still a tuple. Here are few lines and their output:

print Animal
print Animal._fields
print Animal[:]
print list(Animal)
print zip(Animal._fields, Animal)
Enum(Dog=3, Cat=4, Mouse=5, Horse=6, Bunny=7)
('Dog', 'Cat', 'Mouse', 'Horse', 'Bunny')
(3, 4, 5, 6, 7)
[3, 4, 5, 6, 7]
[('Dog', 3), ('Cat', 4), ('Mouse', 5), ('Horse', 6), ('Bunny', 7)]

Data Structures in brief

Making a list of data structures for reference. I probably won’t finish this. I mainly was trying to remember what ‘structured array’ was called and decided to make a list and maybe investigate a couple of them. In the process of searching for it I found ‘namedtuple’ which looks very useful.

List

Stack (list)

Queue (list)

Tuple

Namedtuple

# IMPORT NAMEDTUPLE
from collections import namedtuple
# CREATE A NAMEDTUPLE STRUCTURE
Rock = namedtuple('Rock', 'q w e r t')
# CREATE AN INSTANCE
n2 = Rock(5,4,3,2,1)
# SAME AS
n2 = Rock(t=1, e=3, q=5, r=2, w=4)
# CHANGE A VALUE BY COPYING TO NEW INSTANCE
n2 = n2._replace(w=234, r=[3,4,5])

Deque

Set

Frozenset

Dictionary

Counter (dict)

OrderedDict (dict)

Array

Structured Array

# IMPORT NUMPY ARRAY
from numpy import array
# CREATE A DATA LIST, TUPLE VALUES WILL RECEIVE NAMES
dl = [[(1,2,3,4),(3,2,5,6),(7,6,9,8)],[(1,2,3,4),(3,2,5,6),(7,6,9,8)]]
# CREATE THE DATA TYPE LIST
dt = [('1st','float'),('2nd','int'),('3rd','float'),('4th','int')]
# CREATE A STRUCTURED ARRAY, TUPLES BECOME NP.VOID OBJECTS
d = array(dl, dtype=dt)
# SET ALL VALUES OF A CATEGORY
d['1st'] = 6
# SET ONE VALUE
d['1st'][0,0] = 5
# same as
d[0,0]['1st'] = 5
# GET LISTING OF DTYPE, SIMILAR TO: print dt
print d.dtype
# JUST GET CATEGORY NAMES
print d.dtype.names
# GET A DICTIONARY OF CATEGORY(KEYS) WITH THE DATA TYPE AND BYTE POSITION (I BELIEVE)
print d.dtype.fields

Matrix

Vector

Create a class

Documentation

I decided to explore what is considered to be the proper or best documentation methods. Up til now I just adopted a few things I saw in other scripts or carried over ideas I learned from java programming.

Header

#!/usr/bin/env python
# -*- coding: utf-8 -*-

These should be the top two lines. The first line is necessary for programs to be cross-platform. The second line is optional, where the default encoding is ASCII. The ‘utf-8’ can be replaced with other encoding types.

Next should be a multi-line docstring. It must come next and before all code in order for python to recognize it as the main docstring and attach it to the ‘__doc__’ identifier. Thus,

print objname.__doc__

will print out the contents of this string. The first line should be a short description followed by a blank line. This brief description will show up after the filename (or ‘__name__’ if it is explicitly defined below the docstring but doing this could cause other problems).

In [4]: import SIFT_pack_v3 as SIFT

In [5]: help SIFT
------> help(SIFT)
Help on module SIFT_pack_v3:

NAME
    SIFT_pack_v3 - Created on Sat Dec 24 22:52:12 2011

FILE
    c:\dropbox\my programming\sift_pack_v3.py
...

This shows how the help output is formatted. The first line of my docstring is a creation date statement instead of a description and this is printed out after the file name and dash. The help format follows this up with the file location. After skipping a line from the short description or statement you can write a more verbose description and instructions.

Epydoc seems to be a prominent auto documenting program. If you write you docstrings conforming to their standards, then it can produce nice looking html documentation pages for the code. Details on how to do epydoc formatting are at epydoc.

After the main docstring, it is recommended that you have import statements, beginning with built-in modules and followed by any others.

I’m not completely clear about the difference between using @version in the docstring versus declaring the __version__ variable outside the docstring. They are two different conventions, where @version conforms to the epydoc style, but epydoc will still recognize __version__. You can make up as many of these as you like but only a few are recognized by epydoc. The recognized ones are:

__author__
__authors__
__contact__
__copyright__
__license__
__deprecated__
__date__
__version__

This is described as using module variables as module metadata. There is a way to get epydoc to recognize other metadata (variables) so you can have the best of both worlds. See adding new fields for epydoc.

The next section should have global variables, but it is recommended that globals be avoided. I don’t remember why and I still use them for setting the working directory or testing for the existence of necessary files.

All the functions follow. The first item within the class or function block should also be a docstring. Again, the first line should be a short summary. Then describe the inputs and outputs in the epydoc style,

def example():
    """
    @arg x: This is a description of
        the parameter x to a function.
        Note that the description is
        indented four spaces. Also
        can use @param instead.
    @type x: This is a description of
        x's type.
    @return: This is a description of
        the function's return value.
    @rtype: This is a description of
        the return value's type.
    """

Apart from epydoc formatting, a docstring can be arranged anyway you want. Here is an example found on PEP 257:

def complex(real=0.0, imag=0.0):
    """Form a complex number.

    Keyword arguments:
    real -- the real part (default 0.0)
    imag -- the imaginary part (default 0.0)

    """
    if imag == 0.0 and real == 0.0: return complex_zero
    ...

Finally, the very last item should be:

if __name__ == '__main__':
    main()

This will automatically run ‘main()’ when the program is ran and not when the module is simply imported. This needs to be the last item to ensure that all functions are loaded prior to running the main method regardless of order they are arranged.

On a side note, dir(object) in IPython console will print out a list of all namespaces, functions, and variables(outside of functions) within the object.

Branching river model

This model extends the semi-implicit scheme to include a branching river. It would be easy to extend it to multiple tributaries but I haven’t tried it yet.

In the animation above, the ocean tide enters at the left. The main river is blue and the branching river is cyan. The model is 50km long with a long wave tide (M2) and a smaller oscillation.

River modeling progress

I think I’ve worked out a lot of the planning errors. The code worked before but the result was not very accurate because I didn’t segment the river properly. The current version places height calculations at the center of the cell and the flow at the cell edges. This is using a semi-implicit method.

This video shows a run that includes a slope with inflow from upstream. There is also a tidal and faster oscillation from the ocean side (left boundary).

Animation and the command prompt

Python doesn’t seem to have a simple method for creating animated gifs, but other programs can be executed from python.  Here is what I’m using. It is simple and works.

1) install ImageMagick.  There are other programs that can be used to bundle images into a gif but this is the first and only one I’ve tried so far.  I installed: ImageMagick-6.7.3-10-Q8-windows-dll.exe

2) create a folder with sequentially numbered images.

3) Run the ImageMagick convert.exe program from a command prompt or from a python program.

This can be done from the command prompt:

C:\>convert C:\...path...\*.png C:\...path...\animation.gif

“Convert.exe” is the ImageMagick program that can bundle an image series (such as *.png above) into an animated gif file (such as animation.gif above).
The convert program can be executed from within a python program like this:

from subprocess import call

call(r'\Program Files\ImageMagick-6.7.3-Q8\convert.exe ' +
     r'\Python26\Lib\site-packages\xy\images\*.png ' +
     r'\Python26\Lib\site-packages\xy\images\im55.gif' )

You can also pass a list. The first item is the program, followed by a list of arguments:

from subprocess import call

command = [r'\Program Files\ImageMagick-6.7.3-Q8\convert.exe',
           r'\Python26\Lib\site-packages\xy\images\*.png',
           r'\Python26\Lib\site-packages\xy\images\im23.gif']
call(command)

Or:

from subprocess import call

call([r'\Program Files\ImageMagick-6.7.3-Q8\convert.exe',
      r'\Python26\Lib\site-packages\xy\images\*.png',
      r'\Python26\Lib\site-packages\xy\images\im23.gif'])

NOTE: Spaces are needed when passing the command as one string but not needed in the list format. Also note that the complete path for the program is needed when using call() which is not required if you run convert.exe directly at the command prompt.

4) Delete the series of images (if not needed).

Packages

Python XY with all packages installed

PyEphem

  • Ephemeris program for calculating planet and satellite positions
  • Written in C
  • “import ephem

Pydap

  • Tools for using OPeNDAP
  • “import pydap
  • Installed from the internet using the command prompt:
  1. “easy_install Paste”  (not sure if this is needed, but doesn’t take long)
  2. “easy_install Pydap”

Sync into the clouds

During the time I left my flashdrive in the computer lab and was not able to pick it up for a week, I decided to look for a good way to backup my files. Long story short: Skydrive – doesn’t sync >> Live Mesh –  syncs but hidden sync status >> Dropbox – much better! I’ve been using it for several days now. It is great. I really like having the little status icon on each file showing its sync status. It also saves old versions of all files which Live Mesh does not do. And if files were changed on different computers before syncing then it appears to create a separate file with an extended name showing the computer of origin.

The only advantage for Live Mesh that I can see so far (apart from having extra cloud space if you run out of Dropbox space) is that you can sync any folder. Dropbox creates a single folder where you throw everything you want synced into.

Right now, I’m using Live Mesh to sync the .spyder2 configuration folder with no problem so far. This might possibly be problematic if the configuration file is made for the particular computer it is on. I wanted to keep my template and workspace configurations the same on all three computers that I use for programming.  I use Dropbox for all my programs and everything else.

Live Mesh – sync any folder you can’t conveniently move to the Dropbox.

  • Program folders with configuration files (possibly may cause problems)
  • Sync any folder you want

Dropbox – sync all files you frequently work with on multiple computers.

  • Sync programming code and frequently used data
  • Quickly confirm sync status with file icon (or badge)
  • Stores old versions of all files for recovery