Add RPython implementation

[bencord0]

What is RPython?

"RPython is a restricted subset of Python2 that is amenable to static
analysis" - rpython.readthedocs.io/en/latest/rpython.html

No, you really shouldn't be writing programs in it. It's really
designed for writing language interpreters.

It is distributed as part of the pypy_ project.

.. _pypy: http://pypy.org/

What is this?

An implementation of forest.py, adapted to the RPython toolchain

Changes needed:

• Add entry_point (where the compiled binary starts)
• Add target (where the rpython compiler starts)
• Add compatibility __name__ == __main__ (where python can start)
• No use of the standard library, not even sys. Namedtuples have
  been replaced with tuples (this might be a rule breaker).
• Reduced string formatting options, you basically have str/str
  concat, and C-style formatting (if you need to convert ints).
• RPython has no implementation of any or set.
• RPython does have generators, but they're not very complex.
• Watch out for functions that return multiple types.
  e.g. avoid returning both lists and iterators, as they don't
  share a subclass (and thus are not allowed). It's probably
  more efficient in RPython to just use preallocated lists.

How do I run it?

RPython programs are valid Python programs. You could just run
this under a normal python interpreter. Since the language is
simpler for an interpreter to reason with, you might get a boost
in performance anyway. The tradeoff is that an RPython implementation
is no longer an idiomatic Python implementation.
I'm not even sure if "idiomatic RPython" is a phraise that should exist.

How do I compile it?

You will need a copy of the RPython toolchain. It is implemented
in python, so you don't need much more than to clone the pypy source.

$ hg clone https://bitbucket.org/pypy/pypy ../pypy

The rpython toolchain in in the rpython subdirectory.

$ ../pypy/rpython/bin/rpython ./forest_r.py

This will output forest_r-c.

$ time ./forest_r-c 305 295 300

PR Notes

We need to check that this doesn't further break any rules. Maybe a class based implementation would be the right thing to do here.

Benchmarks

$ time pypy3 forest.py 305 295 300
Initial: Forest(goats=305, wolves=295, lions=300)
Solution: Forest(goats=0, wolves=0, lions=595)

real    0m7.339s
user    0m7.285s
sys     0m0.052s

$ time pypy3 forest_r.py 305 295 300
Initial: 305 295 300
Solution: 305 295 300

real    0m5.633s
user    0m5.579s
sys     0m0.053s

$ time ./forest_r-c 305 295 300
Initial: 305 295 300
Solution: 305 295 300

real    0m1.237s
user    0m1.195s
sys     0m0.040s

[clux]

That's pretty cool. I'm impressed it runs this fast.
Just had a quick look over, as am on my way out. It looks generally fine, but there's a few annoying things;

• plain tuples over namedtuples makes it kind of hard to read (but as you say, idiomatic rpython..)
• you are missing a filter step in mutate
• rpython doesn't even exist in arch packages

[clux]

Hah, I got to try this now. Found rpython on the AUR. That is one trippy compiler. Rainbows everywhere.

The compiled version totally locks up, whereas just running it under pypy produces sensible results.

[bencord0]

I'm tempted to revert back to the tuple method, as it appears that classes are a bit screwy. At least I've fixed the logic when running under the interpreter.

It's a shame there are no good socket libraries that are in rpython. I would have tried to have a bit of fun trying to write a webserver.

Having to unpack complex generators into helper functions, and then it works is an interesting side note. Oh, also, we have to use the .py suffix, even though I would have liked to have used .rpy to distinguish it from the full python implementation.

[clux]

Nice. This is a pretty cool toy now. Actually about as fast as fortran.

I think we can rename it to forest.rpy and just rename the file to forest_r.py during the build phase.
Test extra: append this to debug.diff:

diff --git a/forest_r.py b/forest_r.py
index 7c7acc4..42f5c8c 100755
--- a/forest_r.py
+++ b/forest_r.py
@@ -40,6 +40,8 @@ def solve(forest):
     forests = [forest]
     while any(forests) and not has_stable_forest(forests):
         forests = mutate(forests)
+        for f in forests:
+            print 'Mutation:', f
     return [f for f in forests if is_stable(f)]
 

Adding it to the bench / test scripts would be trivial, but need to actually find a way to add rpython to the Dockerfile sensibly. AUR installs seem to be disabled for root for safety reasons, which is silly when we are in docker.

[clux]

You can add rpython to the build image via AUR with this patch to the Dockerfile:

diff --git a/Dockerfile b/Dockerfile
index 1f86971..5cb4e28 100644
--- a/Dockerfile
+++ b/Dockerfile
@@ -18,3 +18,10 @@ RUN pacman -Syu --noconfirm && pacman --noconfirm -S \
 
 RUN echo "en_US.UTF-8 UTF-8" > /etc/locale.gen && locale-gen
 ENV LANG=en_US.UTF_8
+
+RUN pacman -S --noconfirm sudo git fakeroot && \
+    useradd -ms /bin/bash ci -G wheel && \
+    echo "%wheel ALL=(ALL) NOPASSWD: ALL" >> /etc/sudoers
+USER ci
+RUN cd && git clone https://aur.archlinux.org/rpython.git && \
+    cd rpython && makepkg -si --noconfirm

May need a few tweaks to the build scripts and .travis.yml, but you should be able to run tests on travis with this.

[clux]

Awesome. I'll see how the run there goes and merge it in.

Btw, is the jit helping at all? I'm not seeing any difference locally.

[bencord0]

The jit is not active by default. Actually the toolchain is using the -O2 level. You can specify -O3 or -Ojit, but my benchmarks show a significant increase in compile time for very little gain. We're talking 5ish minutes here.

[clux]

Hah, wow. Well, the results are already super solid.

So is the point of de76b41 to eliminate that weird bug you're having primarily then?

Seems to pass now at any rate, you good with me merging this now?
rpython is pretty chatty now btw, I'll probably end up muting it.

[bencord0]

de76b41 is so that you can run (python2) interpreters on forest.rpy during dev work and for quick unit testing.

I nabbed this from https://morepypy.blogspot.co.uk/2011/04/tutorial-part-2-adding-jit.html example5 https://bitbucket.org/brownan/pypy-tutorial/src/tip/example5.py?fileviewer=file-view-default

I'm good with merging this. Not super happy with the Dockerfile (USER ci, then docker run -u0 seems messy), but meh, it's docker and rpython isn't properly packaged anyway. Maybe just using the pypy source from a fresh hg clone would be less scary? It doesn't really matter.

[clux]

Yeah, I think might probably a better idea. The aur pin package is also pretty outdated upon inspection. I'll have a go at cleaning it up later on.

[clux]

Thanks for the clarification. Not sure how much I'll goof around with rpython after this, but it's a pretty cool implementation to have :)

[clux]

I did a small amount of restructuring it in master now. Ended up taking out the JIT just to not the complexity of it detract from the main problem. But as you said, it didn't affect performance.

There's a small paragraph now about it on the readme as well.