syntax fixes

[dragoncoder047]

]cjump[ expects a number (how many to jump) and the boolean under that. (SED here.) That allows a user to define their own ifff, iffff, etc. without having to edit the Quackery source file or resort to dup ]iff[ ]iff[ hacks, and also to jump backwards with negative jump amounts such as [ false -1 ]cjump[ ] is hang. (Although I haven't encountered anywhere where that would be useful.)

I have not been able to edit the PDF documentation files, you will have to get that yourself. I haven't also checked the Rosetta Code pages to see where this would impact code there that uses]if[, ]iff[ and ]else[, but nonetheless it would be a trivial drop-in substitution - each would simply be replaced by 1 ]cjump[, 2 ]cjump[, and false 1 ]cjump[ respectively - and that's pretty much what I did anyway in the definitions of if, iff, and else.

There are also a couple of minor syntax fixes to make the code more Pythonic (return x instead of return(x), etc.)

[GordonCharlton]

Hi dragoncoder047,

First of all - I see that Phoo is "inspired by Quackery" Wow! That my pet language inspired someone to create their own language has completely made my day. Achievement unlocked! :-)

Sorry you can't edit the TBoQ pdfs - if you have MacOS I can let you have the masters - rather thoughtlessly I used my word processor of preference and managed to lock it into "only the MAcOS version of Pages can edit it" - it's not compatible with the online Pages. D'Oh. The best I have is to suggest that you rustle up an addendum/errata document to accompany it.

]cjump[– I won't be adding that to my version of Quackery for a couple of reasons.

First is that the branching functionality is intended to be limited. I'm enforcing structured programming by design, and ]cjump[ is a relative GO TO. I don't see any need for IFFFF. I'm old enough to remember having to deal with abominations in Lisp like CAADDDR, and consequently I shudder at the thought of IFFFF. IFF gives you IFF item ELSE, IFF item AGAIN, and IF item DONE and IMO that's plenty.

Second is I have a whole load of stuff on Rosettacode that I don't want to plough through looking for meta-words to update. (There's also a small but promising repository of Quackery code at olus2000/ALEx btw.)

On the other hand, I'll definitely be incorporating your Python syntax fixes. Thank you for those. I literally taught myself just enough Python to code Quackery, so if it looks like it was written by a newb, that's because it was!

Yours, Gordon

[dragoncoder047]

Alright, I put it back. I will review the code again for pythonics again.

As to the PDF files, I do not have a Mac. (I have a Raspberry Pi at home and a chromebook at school which I am on right now.) If you're up to the task you could convert them to Markdown, and that way anyone with a text editor could edit them - Python Markdown has a def_list extension that would work GREAT for the documentation of each word.

BTW, Phoo is broken right now. I am working on getting it fixed.

[dragoncoder047]

Syntax looks good now. I also changed some of the error messages. For example you had the same error message for both peek and poke so if that error was encountered it was unclear as to which word it was that caused the error if it went so horribly wrong as to not print the Quackery return stack trace.

I also changed all the error messages in the building code from using sys.exit to raising a relevant error (that doesn't inherit from QuackeryError) so that a) a Python stack trace will be printed, and b) so any other code can catch the error with a simple except:, because the except: is equivalent to except Exception: and SystemExit is not a subclass of Exception and so it won't be caught. Not a big deal and easily overlooked by noobs. (I actually did that several times when I was learning Python too: I thought I could catch KeyboardInterrupt with an except: to do some cleanup and it didn't work. And then I learned about finally:.)

EDIT: typo

EDIT 2: Oops, forgot to mention this. I put some possible future suggestions/improvements in the code too. Search for my username to find them.

[GordonCharlton]

You've been busy! I'll hold off merging your commits until you're confident that no more are imminent.

Some comments.

    `return(isinstance(item, type(lambda: None)))`
    `return isinstance(item, types.FunctionType)`

Thank you for that! I learned something today. I spent too long failing to find a way to not use the type(lambda: None) workaround.

result += '?' # XXX @dragoncoder047 maybe use \uFFFD on platforms that support unicode?

� would be the logical choice. Quackery is a unicode-free-zone. I spent far too long perusing the Unicode specs and decided that if I can't do it right I won't do it at all. I have yet to find a language or OS that gets it right (see: "this message will crash your android/iOS etc.")

maybe simplify to [ dip swap swap ] is rot ?
maybe simplify to [ dip dup swap ] is over ?

The predefined quackery string mentions ROT and OVER before DIP is defined, so you'd have to reorganise that if you migrate ROT and OVER from Python functions to Quackery words. And I suspect that, given how often ROT and OVER are used, it would adversely impact execution speed. I see no advantages.

if isNest(a[0]) and len(a[0]) > 0 and a[0][0] == immovable:

Aha! and short-circuits. How did I miss that?

Error messages within the Python Quackery compiler. Those are only there because I didn't remove them. build() only exists to compile the predefined Quackery string, which is error-free. If the error detection code is anything it is hints for anyone who wants to recode Quackery in a different language. But yes, raising the relevant error is better than using sys.exit.

I'm fine with all the changes to error messages apart from Endless string discovered. I like that. It amuses me. And it's less technical sounding than unterminated. I'm all for avoiding technobabble.

Migrate TBoQ to markdown? Well, "never say never" but I don't see it happening any time soon. That sound like a BIG job.

G

[dragoncoder047]

Great, I don't see any more PEP 20 violations.

As a side note I also wanted to swap out the Python definitions of dup, swap, over, rot, etc. to use the Forth words pick and roll (like I did for Phoo here purely in the name of bootstrapping), but after I saw your comment about ]cjump[ (and remembered the Forth documentation also says something like 'if you find yourself using pick and roll a lot, you need to restructure your code') I decided against it.

I also have some other ideas about how to be able to incorporate external Python libraries (such as turtle) into Quackery instead of resorting to the python word (which uses exec, which is really unsafe). Since they're not really relevant to this PR, I'll put them in a different one once this is merged. (Do you know if, once you merge this PR, will my fork disappear and I will have to re-fork it if I have more ideas? Or will it still exist as some limbo-state "merged fork" repository? This is my first ever PR so I have absolutely no idea. I digress.)

[dragoncoder047]

I'm all for avoiding technobabble.

One other thing before I forget -- would changing from_stack to pop, top_of_stack to peek, to_stack to push, etc., count as technobabble? I don't know whether those terms are well-known enough to not be considered jargon. If not I can add those changes to this PR.

[GordonCharlton]

I don't see pop and push as technobabble. I just don't like the word "pop" in the context of stacks, IMO you pop a balloon, or drink fizzy pop, or pop to the shops. I know that sometimes I'm a bit weird about words. On a more serious note, I don't see any advantage to renaming from_stack() and to_stack() They do what they're called and they make it explicit that it's The Stack as opposed to the Return Stack.

I think your fork will remain after merging. (Definitely not a GitHub expert - I'm only here because a bunch of people said "you should put it on GitHub." Now I'm figuring it out as I go along. Story of my life.)

Looking forward to finding out how to avoid using exec().

Yup, the received wisdom in the Forth community is that PICK and ROLL are to be avoided. I listen to the Forth community because it's full of practical programmers, rather than the theorists of academia.

(How practical? One guy I pay particular attention to, retired now, but developed systems for the most safety critical of all jobs... nuclear reactors. That's a job where you literally put your livelihood on the line. You don't supply programs with the big legal disclaimer that says "hey, take your chances, we guarantee nothing" that comes with every operating system, word processor etc. etc. instead you sign a document that says "I accept personal liability" and no insurance company will insure you against accidentally causing a nuclear catastrophe. So when there's an "event" not only do you get major guilt to live with, but you're bankrupt too. When someone who codes under those conditions says "it's a bad idea" I tend to believe them!)

So instead I added pack, unpack and dip. If you really want pick and roll, they're easy enough:

[ pack swap dip unpack ] is roll

[ pack over dip unpack ] is pick

Not the I'm averse to the academic approach. I found this recently and it blew my mind that rather than "Forth with a sprinkling of Lisp" they went for "Lisp with a sprinkling of Forth" as their starting point. It's a good little paper, amusing and insightful. And they note that all you need is DUP DROP and ROLL. It's a valid opinion.

https://hashingit.com/elements/research-resources/1994-03-ForthStack.pdf

Now that I've seen some Phoo code, I like it. TO name reminds me of Logo, and that's a good thing IMO.

I notice you're using the original versions of of, reverse, and reflect. I don't know Javascript at all, but it turns out that in Python, if you're dealing with very large nests, the old versions really hammer Python's dynamic memory allocation hard. Better to use a binary split approach.

[dragoncoder047]

Back when I had a working version of Phoo (that was pre-me-on-Github) I noticed the strangest thing was Phoo was super fast compared to Quackery (except when I tried to use [ findwith [ over = ] drop ] is find, slow in both cases). Phoo compiled its standard library and opened the REPL almost instantly, but Quackery took about a second. I forget what I did - I think I used a regex to chop off the next word instead of eating character by character like the Python Quackery compiler does.

Right now Quackery is set up so that whenever you call the quackery function, it re-compiles the predefined string every time. Part of my speed idea involves compiling this only once, at module import, and then cloning it (dict_clone = orig_dict.copy()) whenever needed.

And speaking of the "old" versions of of, reverse, and reflect, I think I pulled them from 6a77d67. They still worked, and if Phoo is still as fast as it was, I might not have to fix them. I thing Javascript's garbage collection approach is a little different than Python's and so it doesn't get hung up as much.

Avoiding exec would probably involve using importlib, which is a little daunting even for me, but I'll figure it out. It would probably be a little like the way I have Phoo JS modules set up - they export a name called module which is assumed to be valid. And Python-Markdown extensions also do a similar approach, expecting a function called extendMarkdown to be exported and it is then called with the Markdown instance as the first argument. You could have the loadfile mechanism search for foo.qky like usual but also look for a foo.py if it can't find the Quackery file.

[GordonCharlton]

Yup, rebuilding the predefined words every time is wasteful. If it was as simple as pickling the output of build(predefined) I would have done that. There are plenty of potential ways of speeding up Quackery. The obvious one is to recode traverse in C so that the inner interpreter (as they call it in Forth) runs at top speed.

But… the whole point of Quackery is that you should be able to look at the code for build() and traverse() and say "I could do that", in much the same way that you could read the original fig-Forth installation manual as far as the line "requires 600-800 bytes of machine code", or see Lisp completely defined in the lower half of page 13 of the Lisp 1.5 Programmers manual, and say the same thing. So optimisations that add complexity to the code are not an option for me.

It's a prototype, or a proof-of-concept. Maybe at some point I'll make it run fast, but for the time being switching to PyPy3 as it's host environment has given me sufficient speed boost to make it not painful-to-use, and I want to be an "experienced Quackery programmer" before I consider optimisation. Which is why I've been doing Rosettacode tasks for the last year and some.

(Pypy3 has JIT compilation rather than a byte code interpreter, and incremental mark and sweep garbage collection that's way better than cPython's reference counting.)

[dragoncoder047]

Goodness, no, I don't think you ever are going to need to rewrite traverse in C. And then you'd box yourself in to CPython and block everything else like Jython (Java), IronPython (.NET), Transcrypt (Javascript), etc.

I think the bottleneck is in build. The speed concerns I had were when you first run Quackery it spits out "Welcome to Quackery" and then there is a 2-second delay while it is building before the duck prompt pops up. After that it runs pretty fast enough for me.

The changes I was thinking in terms of speed and nonredundancy were things like moving build, traverse, and all the Python function-words out into the main module instead of indented under def quackery. If anything those changes would be something for a second PR.

Anyway, I think you're safe to merge this PR now. I don't see any more changes.

[GordonCharlton]

" moving build, traverse, and all the Python function-words out into the main module instead of indented"

That would make a difference? Python doesn't compile nested functions until it needs them? Oh my!

[dragoncoder047]

Yep. Python only compiles the defs when it sees them, it doesn't run them. It also doesn't check scoping of variables until runtime (hence I suspect you probably had lots of UnboundLocalErrors until you added the nonlocals). I suspect that you nested them so they would have access to the stacks and state variables, but if you cram those into an object (I have updated my fork with a QuackeryContext - have a look at it) you can just pass the object to the function and refer to the state variables as object properties, which is what I have done.

Also, if you don't mind, I haven't had a chance to test my modifications. They're pretty major.

EDIT: typo

[GordonCharlton]

Not so many UnboundLocalErrorss because I was testing frequently as I was coding, so I figured out the need for nonlocal very early on.

No, I nested all the python function-words because they are local to quackery(). A habit I picked up from Pascal.

I had a quick look at your fork and tried running it. Looks like you have plenty of typos.

My total knowledge of objects, classes etc. is: I bought "Smalltalk-80: The Language and its Implementation" in hardback in 1983 and thoroughly enjoyed reading it. Since then I have forgotten almost everything in it, and the definition of "Object Oriented" has changed substantially since then.

I looked at that aspect of Python and decided I didn't need it. So a lot of your fork whooshed over me like a low pass from a jet fighter.

[dragoncoder047]

Great! Did a lot of debugging today and it should work now. I pasted it into A Pythonanywhere console (CPython 3.9.5) and it ran just fine.

With all the Python Quackery words on the global scope in my fork now, if someone does from quackery import * it will clutter their namespace with everything. A simple __all__ = ['QuackeryState', 'quackery'] will fix that. But seeing as there's nothing yet that embeds Quackery, I don't see the need to do that.

After some fixing of the loadfile system I was thinking of making a web console using Transcrypt and jQuery Terminal (which I am already using for Phoo) and making it available via Github Pages. Do you like that idea?

[GordonCharlton]

Amazing!

I saw not cluttering the namespace as a major benefit of nesting all the functions, and if all it takes is a one-line dunder then I'm in favour of that. Why wait until it's necessary?

I love the idea of making it available in a web console! The easier it is for people to try it out, the better.

[dragoncoder047]

That's the magic of dunders! __all__ only applies on a from module import *, not on from module import someFunction, someOtherFunction, so if people really want to get at the internals of Quackery they could still do so. It's also like Python's name mangling - outside of a class Foo, a property __bar will be mangled into _Foo__bar, but you can still just access that.

In lieu of a web console, the closest substitute I have found is going to https://www.pythonanywhere.com/embedded3/ and pasting in this:

from requests import get
def load(url):
    c = compile(get(url).text, url, 'exec')
    exec(c, globals(), globals())
load('https://raw.githubusercontent.com/GordonCharlton/Quackery/main/quackery.py')

or load('https://raw.githubusercontent.com/dragoncoder047/QuackeryFork/main/quackery.py') for my fork.

For some reason it screws up the Python stack traces when Quackery crashes, but so long as Quackery doesn't crash it will run fine. (Try protected release $ "foo" quackery and then hit ^C.)

[GordonCharlton]

Your fork runs on my 2011 iMac under Python 3.8.1, and on my Mac mini under Python 3.10.2 and PyPy 3.7.12 without any complaints. I can also try it on a Raspberry Pi 400 if you like, but I don't expect it would have any issues. :-)

But… timings suggest that it's running slower. :-(

(the file test.qky is $ "extensions.qky" loadfile)

On the Mac mini,
time python3 quackerydragon.py test.qky returns

1.90s user 0.01s system 99% cpu 1.916 total

time python3 quackery.py test.qky returns

1.31s user 0.01s system 99% cpu 1.316 total

time pypy3 quackerydragon.py test.qky returns

0.75s user 0.03s system 99% cpu 0.792 total

time pypy3 quackery.py test.qky returns

0.42s user 0.04s system 78% cpu 0.592 total

On the iMac,
time python3 quackerydragon.py test.qky returns

real    0m4.350s
user    0m4.299s
sys     0m0.015s

time python3 quackery.py test.qky returns

real    0m2.793s
user    0m2.749s
sys     0m0.013s

[dragoncoder047]

I don't think you'll need to test it on the RPi. I only think the difference is 32/64 bit and RAM amount, so if you try [ 999 999 ** temp put again ] you'll just get a MemoryError or a segfault slightly faster or slower.

Oh well. I suspected as much - calling bound methods of objects has a higher overhead than just accessing variables in an outer scope.

Javascript has a JIT compiler, so the online Transcrypt-compiled Quackery should produce similar numbers as PyPy did (and if my experience with Phoo is still correct, maybe even smaller numbers).

[dragoncoder047]

It would be interesting to see what numbers would come up with a more stressful test (such doing something simple 100000 times) and then compare the numbers. Maybe something like

[ this ] is testsuite.qky
[ ]'[ temp put
  time
  swap times [ temp share do ]
  time swap - 1000 / echo say "ms" ] is test
say "100000 dup drops: "
100000 test [ dup drop ] cr
say "100000 put takes: "
100000 test [ temp put temp take ] cr
say "300000 rots: "
$ "junk" 3 of 100000 test [ rot rot rot ] drop 2drop cr
say "100000 compiles of '1 1 +': "
100000 test [ $ "1 1 +" build ] cr
( ... and so on with the tests )

[GordonCharlton]

It's a bank holiday weekend here, so mostly family time, no opportunity to devise a well thought out test suite, but I found a few minutes here and there to cobble together a quick test by adapting an existing piece of code ("Yellowstone numbers" on rosettacode) to compile and run a thousand times.

(Had to rename the file from .qky to .txt to upload it.)
test.txt

Results are quite interesting. (The system times are negligible so omitted. They changed the terminology from real time to user time and stopped reporting in minutes between the two versions of the OS. Probably because Apple switched the terminal from BASH to ZSH.)

python3 iMac quackery.py           real time 24m15.145s
python3 iMac quackerydragon.py     real time 38m21.770s

pypy3 Mac mini quackery.py         user time 27.99s
pypy3 Mac mini quackerydragon.py   user time 21.98s

python3 Mac mini quackery.py       user time  651.37s
python3 Mac mini quackerydragon.py user time 1020.38s

As I'd expect, JIT is the major factor, but on this test at least, JIT favours your approach.

[dragoncoder047]

I love the idea of making it available in a web console! The easier it is for people to try it out, the better.

I have a testing online Quackery console using Pyodide: https://dragoncoder047.github.io/QuackeryFork/

Right now it appears to have issues because input() is blocking and either throws an OSError because I return a Promise from the stdin callback or just crashes the entire page (i.e. the "This page is not responding" dialog box) if I try to block in a busy loop until input is provided. Browsers like everything to be async but input() is not by design and I don't know how to get around that. There are already issues filed on Pyodide (pyodide/pyodide#1898).

Let me know if it works for you - it hasn't for me yet.

[GordonCharlton]

Not working yet. Gets as far as printing the duckhead prompt /O> then we get

Quackery system damage detected.
Python error: [Errno 29] I/O error

and a bunch of diagnostics.

So I tried this,

Until this problem is resolved, to run Quackery you can go to
https://www.pythonanywhere.com/embedded3/ and paste in this code:
from requests import get
def load(url):
    c = compile(get(url).text, url, 'exec')
    exec(c, globals(), globals())
load('https://raw.githubusercontent.com/GordonCharlton/Quackery/main/quackery.py')

And yes! It works!!!

/O> 10 times [ say "hello world" cr ]
... 
hello world
hello world
hello world
hello world
hello world
hello world
hello world
hello world
hello world
hello world
Stack empty.

"Numfar, do the dance of joy"

[dragoncoder047]

The Pythonanywhere paste hack was just a hack until the Pyodide problem is resolved. I already mentioned it in a previous comment and seeing that it worked I figured I would share it in the error message when the online console didn't work. The disadvantage of that option is that you would need a constant internet connection because the code is actually running on the Pythonanywhere server and you are essentially just SSH'ing in and seeing the output. The online console I have built is running all the code on your browser (and then crashing!) so it would not need an internet connection once it downloads everything and stores it in the cache.

I learned that you can make a Promise synchrounous and block until it is resolved by using Atomics.wait with a timeout of infinity, but to use Atomics.wait you need a SharedArrayBuffer and support for that was largely yanked after the Spectre vulnerability was found. There is another hack that uses a service worker to fake the headers and enable it, and that's what I'm working on now.

EDIT: typo

[dragoncoder047]

Aaack, it doesn't seem to be working as I expected. The serviceworker appears to do absolutely nothing and the input function still crashes it all. I have filed another bug with Pyodide (pyodide/pyodide#2505), which tries to fix a similar issue with their online REPL and input() and hopeully once they can get it fixed Quackery will be fixed too.

[dragoncoder047]

Trying a different approach today... Using the AST module to "rewrite" the Quackery code and make all the functions that need to be async async. Hope it works, still a few bugs.

[dragoncoder047]

Super necropost but what kind of speeds are you getting with the newer CPython versions? The CPython devs are really stepping up the speed game now...

[GordonCharlton]

Just ran a comparison using the code from the Rosetta Code task Blum integer.

python3.9.0  2147.23s
python3.12.0 1117.90s
pypy 7.3.7    114.70s

So yeah, Python 3.12 is about twice as fast as 3.9, but pypy3 still beats it by a country mile.