Background (non-parallel) task/code execution

[minchinweb]

I'm using Windows 7, Python 3.4.2, and Invoke 0.9.0 (installed via pip).

I want to run 'backgound' tasks. In Windows, if you place the start command at the beginning of the command, it will start your task in a new cmd window and then return focus to your original cmd window. Invoke will start the task, but waits for the new cmd window to close before continuing on. I would like Invoke to start the new task, and then continue with its next command, and eventually exit on 'completion.'

My tasks.py file looks like this:

from invoke import run, task
import os

env_deploy_path = 'output'

@task
def develop():
    if os.path.isdir(env_deploy_path):
        run('rm -rf ' + env_deploy_path)
        run('mkdir ' + env_deploy_path)
    run('start pelican -r -s pelicanconf.py')
    run('cd ' + env_deploy_path + ' && start python -m http.server')

[bitprophet]

Actually-parallel tasks fall under #63, FWIW - this is a little different from that even though similar solutions might apply. tl;dr, probably a good use for threading, multiprocessing, greenlets (if we end up supporting them) or maybe asyncio/trollius.

Off the top of my head the most straightforward (maybe only) way to do this is to update run so it can optionally run as a background thread & honor a threading.Event object (or equivalent, if we were to use another methodology) controlled by the caller (in your case, the develop function/task body itself).

So maybe something like this?

@task
def mytask():
    event = threading.Event()
    # Giving non-None to 'event' could implicitly trigger backgrounding.
    # Could also make it explicit though this needs 2 args every time which is bluh
    run('start pelican', event=event) 
    # ... do random other foreground things here
    event.set() # the backgrounded run() notices this & halts itself

We're considering requiring contextualization for all tasks, in which case the event could be an explicit attribute of the shared context, perhaps (again - top of my head here):

@task
def mytask(c):
    c.run('start pelican', background=True)
    # ... other things here ...
    c.event.set() # or a 'nicer' API call like c.stop() or c.finish(); or even bake it into what @task does.

Suspect we'd need both versions of this - folks doing a "I just want to kick this off and have it clean up when I exit" use case like yours might prefer the Task/Context based implicit setup where you just say run(..., background=True). Vs users wanting tighter control or independently-exiting backgrounded tasks (e.g. they would stop at different times prior to task completion) who would want the explicit/manual event management API.

[minchinweb]

Thanks for looking into it. I'm not sure I understand all the options, but yes, I'm looking for a "I just want to kick this off and let it do it's thing". I like the second code example you wrote:

@task
def develop():
    run('pelican -r -s pelicanconf.py', background=True)
    run('cd ' + env_deploy_path + ' && python -m http.server', background=True)

that seems to make intuitive sense to me.

[jaypeedevlin]

EDIT: Have run into some subsequent issues where os.chdir() changes the directory before regenerate() runs, causing it to not be able to find pelicanconf.py. Will see if I can work it out and will update this if/when I do.

EDIT 2: I have updated the code below with a fix.

@minchinweb I think we're trying to solve the same problem using pelican and invoke on Python 3.x

I was able to get this to work by doing the following (Disclaimer: I don't have lots of knowledge on threading, but this works for me)

from invoke import task
import os
import sys
import socketserver
import threading

from pelican.server import ComplexHTTPRequestHandler

ENV_DEPLOY_PATH = 'output'
PORT = 8000

@task
def regenerate(ctx,start_dir=''):
    """Automatically regenerate site upon file modification"""
    ctx.run('pelican -r -v -s {}pelicanconf.py'.format(start_dir))

@task
def serve(ctx):
    """Serve site at http://localhost:8000/"""
    os.chdir(ENV_DEPLOY_PATH)

    class AddressReuseTCPServer(socketserver.TCPServer):
        allow_reuse_address = True

    server = AddressReuseTCPServer(('', PORT), ComplexHTTPRequestHandler)

    sys.stderr.write('Serving on port {0} ...\n'.format(PORT))
    server.serve_forever()

@task
def develop(ctx):
    """`regenerate` and `serve`"""
    start_dir = os.getcwd() + '/'

    t_regenerate = threading.Thread(target=regenerate,args=[ctx,start_dir])
    t_serve = threading.Thread(target=serve,args=[ctx])

    t_regenerate.start()
    t_serve.start()

    t_regenerate.join()
    t_serve.join()

Hopefully this is useful to you while the invoke team work on #63

[minchinweb]

Actually, somewhere along the line, it just started working for me.

Originally, I would call invoke using start, and that worked. (i.e. > start invoke develop). This would start invoke in it's own CMD window and the fact that it never kicked off the subprocess in its own window was never an issue, and I could go merrily on my way.

Somewhere along the line, invoke started working to invoke 'start' commands. My current tasks.py (selection below) works as expected, opening the server into it's own CMD window and returning focus to the original CMD window.

from pathlib import Path
from invoke import run, task

p = Path.cwd()
deploy_path = p.parents[0] / 'blog.minchin.ca-temp'  # used for local testing

@task
def serve(ctx):
    """Serve the local blog output on port 8000."""
    ctx.run('cd {} && start python -m http.server'.format(deploy_path))

# ...

I'm running invoke v0.14.0 on Python 3.6.0 x64 on Windows 10.

[aikchar]

I'm guessing there has been minimal forward progress on this? I'm using v0.22.1 and ctx.run("foo &") waits until foo exits.

[bitprophet]

@aikchar That's gonna happen either way, because & is a shell construct. From the perspective of any process executing a shell (Invoke, subprocess, any other library anywhere) it's not possible to tell that the shell has a backgrounded job. The shell process will still hang around until all of its jobs are done!

That's why this sort of thing needs the backgrounding to be done on the master/control side, thus the notes about threading and similar above.

---

This comes up periodically and while users can (as noted) do their own threading, I definitely want some easy common simple API, kinda similar to ThreadingGroup in Fabric 2. Just at the run level.

[bitprophet]

Also pulling in some possibly not yet linked tickets: #531, #526.

[minchinweb]

Today I discovered Start-Process which allows you to do this from PowerShell directly, and can be included in a tasks.py file. It's slightly unwieldly:

run('Start-Process -FilePath python -ArgumentList "-m http.server {}" -WorkingDirectory {}'.format(port, deploypath))

[bitprophet]

Didn't find this when I searched the other day, but #682 subsumes this. Of note, my assertion about shell backgrounding and blocking actually only applies to some situations like remote sshd-driven shell sessions; local ones, at least on some operating systems, seem to actually work as users expect (but not on Invoke right now, for reasons - see #682).

The notes about Windows' start here are interesting as it may be a slightly different paradigm than a Unix shell + &. If @minchinweb is still around, I think it'd be interesting to have him try out whatever I do in that other ticket, with start or Start-Process to see if the approaches are compatible. Suspect they would be, as the changes I have in mind are not specific to the behavior of & but are largely just "let async happen".