This implements function decorators using the
Java-style syntax described in
http://mail.python.org/pipermail/python-dev/2004-June/045516.html

As the patch changes the grammar, you'll need to
make sure Include/graminit.h and Python/graminit.c
are writable after applying the patch but before
compiling (after a standard CVS checkout they are
read-only, which stops pgen from working).

Changes: 

   - adding @ as a token
   - changing the grammer for funcdef
   - adding com_decorator in compile.c
   - adding test_decorator.py (based on Guido's version)

Logged In: YES 
user_id=29957

After applying this patch, any attempts to use the
newly-built python (I deleted graminit.c and graminit.h, to
make sure they were rebuilt) gives the following on any
'import':

*-s*)  CC='gcc -pthread' LDSHARED='gcc -pthread -shared'
OPT='-DNDEBUG -g -O3 -Wall -Wstrict-prototypes' ./python -E
./setup.py -q build;; \
*)  CC='gcc -pthread' LDSHARED='gcc -pthread -shared'
OPT='-DNDEBUG -g -O3 -Wall -Wstrict-prototypes' ./python -E
./setup.py build;; \
esac
Traceback (most recent call last):
  File "./setup.py", line 6, in ?
    import sys, os, getopt, imp, re
SystemError: compile_node: unexpected node type
make: *** [sharedmods] Error 1

Logged In: YES 
user_id=29957

Hm. Pilot error on my part. I had to build once to get
graminit.h and graminit.c rebuilt, then rebuild again (after
nuking all .o files) to get it to work properly. *sigh*

Logged In: YES 
user_id=29957

I still get a failure in test_transformer after a make
distclean and rebuild and make distclean and rebuild.

FAILED (errors=1)
Traceback (most recent call last):
  File "Lib/test/test_transformer.py", line 37, in ?
    test_main()
  File "Lib/test/test_transformer.py", line 32, in test_main
    test_support.run_unittest(
  File
"/home/anthony/src/py/pyhead/dist/src/Lib/test/test_support.py",
line 290, in run_unittest
    run_suite(suite, testclass)
  File
"/home/anthony/src/py/pyhead/dist/src/Lib/test/test_support.py",
line 275, in run_suite
    raise TestFailed(err)
test.test_support.TestFailed: Traceback (most recent call last):
  File "Lib/test/test_transformer.py", line 16, in
testMultipleLHS
    a = transformer.parse(s)
  File
"/home/anthony/src/py/pyhead/dist/src/Lib/compiler/transformer.py",
line 50, in parse
    return Transformer().parsesuite(buf)
  File
"/home/anthony/src/py/pyhead/dist/src/Lib/compiler/transformer.py",
line 120, in parsesuite
    return self.transform(parser.suite(text))
  File
"/home/anthony/src/py/pyhead/dist/src/Lib/compiler/transformer.py",
line 113, in transform
    return self.compile_node(tree)
  File
"/home/anthony/src/py/pyhead/dist/src/Lib/compiler/transformer.py",
line 149, in compile_node
    return self.file_input(node[1:])
  File
"/home/anthony/src/py/pyhead/dist/src/Lib/compiler/transformer.py",
line 180, in file_input
    self.com_append_stmt(stmts, node)
  File
"/home/anthony/src/py/pyhead/dist/src/Lib/compiler/transformer.py",
line 1025, in com_append_stmt
    result = self.com_node(node)
  File
"/home/anthony/src/py/pyhead/dist/src/Lib/compiler/transformer.py",
line 768, in com_node
    return self._dispatch[node[0]](node[1:])
  File
"/home/anthony/src/py/pyhead/dist/src/Lib/compiler/transformer.py",
line 252, in stmt
    return self.com_stmt(nodelist[0])
  File
"/home/anthony/src/py/pyhead/dist/src/Lib/compiler/transformer.py",
line 1018, in com_stmt
    result = self.lookup_node(node)(node[1:])
  File
"/home/anthony/src/py/pyhead/dist/src/Lib/compiler/transformer.py",
line 312, in expr_stmt
    lval = self.com_augassign(nodelist[0])
  File
"/home/anthony/src/py/pyhead/dist/src/Lib/compiler/transformer.py",
line 922, in com_augassign
    l = self.com_node(node)
  File
"/home/anthony/src/py/pyhead/dist/src/Lib/compiler/transformer.py",
line 768, in com_node
    return self._dispatch[node[0]](node[1:])
KeyError: 268

Logged In: YES 
user_id=29957

Argh. Our Makefile's ability to Do The Right Thing... is not
good. Rebuilding Lib/symbol.py and Lib/token.py makes this
better. The setup.py really should do this automatically.

So - all tests pass. As far as I can see, things still to be
done for this to be in a2:

- Documentation (can wait until a3, I guess)
- More coverage in test_decorator (ditto)

Logged In: YES 
user_id=6656

Well, PEP 306 is the reference for this sort of thing: also
needed (before 2.4 final) are changes to Lib/compiler and
tokenize.py.  Mark, are you willing/able to work on these?

Logged In: YES 
user_id=6380

Before 2.4final I'd like to see some changes to the syntax:

- Instead of @test, it should be somewhat more restricted, 
preferably @ followed by a dotted name followed by an 
optional argument list. @1+2 or @[1,2,3] just doesn't make 
sense IMO.

- I'd like there to always be a newline between the last 
decorator and the 'def' keyword. (This in the sake 
of "greppability" per python-dev discussion.)

Logged In: YES 
user_id=6380

There's some non-standard C: in compile.c line 5641 there's a 
char* declaration that isn't at the start of a block. Easily 
fixed but should definitely be fixed before releasing a2.

Logged In: YES 
user_id=1017234

I've added an updated version of the patch, with the
following changes:

     - Applied against CVS head (as of 2004-07-28)
     - Non-standard C removed (see Guido's comment)
     - Newline before the def is now mandatory
     - Grammer for decorator expressions more restricted
     - tokenize.py and test_tokenize updated to handle @ tokens
     - Lib/compiler/transformer.py minimally updated

Still to do:

     - Complete work on Lib/compiler (i.e. get it to the
point where it
       produces a correct .pyc for code with decorators).
     - Write docs
     - Work through all the steps in PEP 306
     - Add more regression tests to test_decorator

I can put in some time on this in the next few days if this
patch seems likely
to be applied.

The grammar is now:

     decorator: '@' dotted_name [ '(' [arglist] ')' ]
     decorators: decorator ([NEWLINE] decorator)* NEWLINE
     funcdef: [decorators] 'def' NAME parameters ':' suite

This allows

     @foo @bar
     def f(): pass

but not:

     @foo @bar def f(): paqss

Issues:

    - Someone should take a look at the "case decorator:"
code in
      symtable_node() in compile.c.  All the tests pass, but
this
      seems a fragile bit of code to be changing

    - Lib/compiler needs some more work - at present
decorators are
      ignored there.  I'll do some more work on this when I
get time
      (probably this weekend).

    - As noted in the comments here, the build system does
not cope
      automatically with grammar changes.  The steps I used
to apply
      the patch are:

	   cd .../dist/src
           patch -p0 < /path/to/patch-file
	   ./configure
	   make
	   chmod +w Lib/symbol.py Lib/token.py
	   ./python Lib/symbol.py
	   ./python Lib/token.py
	   make clean
	   make
	   make test >& make-test.out

       I'm sure this sequence is non-optimal, but it works
for me, and
       all the tests pass on my Linux system afterwards.	   

Logged In: YES 
user_id=1017234

I've updated the patch again.  See my previous
comment here for the procedure to apply the patch.

Main changes:

     - Added decorator support to Lib/compiler.
       test_decorators.py now passes when compiled
       with Lib/compiler.  The regression tests
       also pass when built with
       Tools/compiler/regrtest.py, apart from
       test_dis.py which I assume is due to
       trivial differences in the bytecode output.
       
     - Added a description of decorators to the
       reference manual.  Changed the docs for
       staticmethod and classmethod in the library
       manual to use decorator syntax (with a pointer
       to the decorator description in the
       reference manual).

     - Added more tests to test_decorator.py,
       including some possibly useful example
       decorators (e.g. memoize)

Other changes (not directly related to decorators):
           
     - I regenerated Lib/compiler/ast.py using
       Tools/compiler/astgen.py.  I'm not sure
       whether astgen.py is still in use -- I had
       to tweak it a bit to make the output match
       ast.py.  I changed astgen.py to emit the
       classes in alphabetical rather than dict
       scan order to make it easier to diff the
       output against ast.py

     - test_pyclbr.py had a broken definition of
       assertEquals - it always returned True if
       called with two arguments.  I renamed this
       to assertEqualsOrIgnored and fixed the bug.
       The working test revealed a test failure in
       pyclbr.py:

     - Changed pyclbr to use the full (dotted)
       module name for pyclbr.Class.module attribute
       
Issues:

     - test_pyclbr.py currently fails because its
       is_method() method fails to spot staticmethod-
       and classmethod- wrapped methods.  See the
       comment on the new test test_decorators().
       I'll have another look at this next week.
       Note that this is a problem with the test,
       not the pyclbr.py module.  For now
       test_decorator could just be deleted.

Logged In: YES 
user_id=1017234

Ignore the "Issues:" paragraph in my previous
comment - I made is_method() test_pyclbr.py a bit
smarter and the test now passes.  I also added
a file Lib/test/pyclbr_input.py to use as a
test module rather than test_decorators.py.

Logged In: YES 
user_id=29957

Checked in on the trunk. I think I got everything. Thanks
for your work on this!

/cvsroot/python/python/dist/src/Doc/lib/asttable.tex,v  <--
 asttable.tex
new revision: 1.2; previous revision: 1.1
/cvsroot/python/python/dist/src/Doc/lib/libfuncs.tex,v  <--
 libfuncs.tex
new revision: 1.168; previous revision: 1.167
/cvsroot/python/python/dist/src/Doc/ref/ref7.tex,v  <-- 
ref7.tex
new revision: 1.39; previous revision: 1.38
/cvsroot/python/python/dist/src/Grammar/Grammar,v  <--  Grammar
new revision: 1.50; previous revision: 1.49
/cvsroot/python/python/dist/src/Include/graminit.h,v  <-- 
graminit.h
new revision: 2.22; previous revision: 2.21
/cvsroot/python/python/dist/src/Include/node.h,v  <--  node.h
new revision: 2.23; previous revision: 2.22
/cvsroot/python/python/dist/src/Include/token.h,v  <--  token.h
new revision: 2.21; previous revision: 2.20
/cvsroot/python/python/dist/src/Lib/pyclbr.py,v  <--  pyclbr.py
new revision: 1.33; previous revision: 1.32
/cvsroot/python/python/dist/src/Lib/symbol.py,v  <--  symbol.py
new revision: 1.17; previous revision: 1.16
/cvsroot/python/python/dist/src/Lib/token.py,v  <--  token.py
new revision: 1.14; previous revision: 1.13
/cvsroot/python/python/dist/src/Lib/tokenize.py,v  <-- 
tokenize.py
new revision: 1.36; previous revision: 1.35
/cvsroot/python/python/dist/src/Lib/compiler/ast.py,v  <-- 
ast.py
new revision: 1.24; previous revision: 1.23
/cvsroot/python/python/dist/src/Lib/compiler/pycodegen.py,v
 <--  pycodegen.py
/cvsroot/python/python/dist/src/Lib/compiler/symbols.py,v 
<--  symbols.py
new revision: 1.16; previous revision: 1.15
/cvsroot/python/python/dist/src/Lib/compiler/transformer.py,v
 <--  transformer.py
new revision: 1.41; previous revision: 1.40
/cvsroot/python/python/dist/src/Lib/test/pyclbr_input.py,v 
<--  pyclbr_input.py
initial revision: 1.1
/cvsroot/python/python/dist/src/Lib/test/test_decorators.py,v
 <--  test_decorators.py
initial revision: 1.1
/cvsroot/python/python/dist/src/Lib/test/test_parser.py,v 
<--  test_parser.py
new revision: 1.19; previous revision: 1.18
/cvsroot/python/python/dist/src/Lib/test/test_pyclbr.py,v 
<--  test_pyclbr.py
new revision: 1.22; previous revision: 1.21
/cvsroot/python/python/dist/src/Lib/test/tokenize_tests.txt,v
 <--  tokenize_tests.txt
new revision: 1.2; previous revision: 1.1
/cvsroot/python/python/dist/src/Lib/test/output/test_tokenize,v
 <--  test_tokenize
new revision: 1.9; previous revision: 1.8
/cvsroot/python/python/dist/src/Modules/parsermodule.c,v 
<--  parsermodule.c
new revision: 2.83; previous revision: 2.82
/cvsroot/python/python/dist/src/Parser/tokenizer.c,v  <-- 
tokenizer.c
new revision: 2.75; previous revision: 2.74
/cvsroot/python/python/dist/src/Python/compile.c,v  <-- 
compile.c
new revision: 2.310; previous revision: 2.309
/cvsroot/python/python/dist/src/Python/graminit.c,v  <-- 
graminit.c
new revision: 2.36; previous revision: 2.35
/cvsroot/python/python/dist/src/Tools/compiler/ast.txt,v 
<--  ast.txt
new revision: 1.6; previous revision: 1.5
/cvsroot/python/python/dist/src/Tools/compiler/astgen.py,v 
<--  astgen.py
new revision: 1.8; previous revision: 1.7
/cvsroot/python/python/dist/src/Tools/compiler/regrtest.py,v
 <--  regrtest.py
new revision: 1.5; previous revision: 1.4