invoke.yaml ignored for a tasks module.

[lukeorland]

If I use a module called tasks, as a directory named tasks/, containing an __init__.py, instead of tasks.py, where should the project-level invoke.yaml configuration file live? It is ignored if it's in the same directory as the tasks/ directory and also ignored if it's in the tasks/ directory.

[bitprophet]

It should be picked up from the same directory of wherever import tasks picked up your code (so, same directory as tasks/ in this case.) So perhaps there's a bug.

Can you post the output of INVOKE_DEBUG=1 inv --list (or any other inv command, but --list will limit how much other extra debuggery shows up)? It will include some info on where it looked for task and config files & what it loaded up.

[lukeorland]

$ tree
.
├── invoke.yaml
└── tasks
    └── __init__.py

$ cat invoke.yaml
tasks:
    auto_dash_names: false

$ cat tasks/__init__.py
from invoke import task


@task
def my_awesome_task(c):
    print("Awesome!")

$ INVOKE_DEBUG=1 inv --list
invoke.config._load_file: Didn't see any /etc/invoke.yaml, skipping.
invoke.config._load_file: Didn't see any /etc/invoke.yml, skipping.
invoke.config._load_file: Didn't see any /etc/invoke.json, skipping.
invoke.config._load_file: Didn't see any /Users/lukeorland/.invoke.yaml, skipping.
invoke.config._load_file: Didn't see any /Users/lukeorland/.invoke.yml, skipping.
invoke.config._load_file: Didn't see any /Users/lukeorland/.invoke.json, skipping.
invoke.config.merge: Merging config sources in order onto new empty _config...
invoke.config.merge: Defaults: {'run': {'shell': '/bin/bash', 'hide': None, 'env': {}, 'replace_env': False, 'in_stream': None, 'encoding': None, 'echo': False, 'warn': False, 'echo_stdin': None, 'watchers': [], 'pty': False, 'out_stream': None, 'err_stream': None, 'fallback': True}, 'tasks': {'search_root': None, 'collection_name': 'tasks', 'dedupe': True, 'auto_dash_names': True}, 'sudo': {'password': None, 'prompt': '[sudo] password: ', 'user': None}, 'runners': {'local': <class 'invoke.runners.Local'>}}
invoke.config.merge: Collection-driven: {}
invoke.config._merge_file: System-wide config file (/etc/invoke.py): {}
invoke.config._merge_file: Per-user config file (/Users/lukeorland/.invoke.py): {}
invoke.config._merge_file: Per-project config file has not been loaded yet, skipping
invoke.config.merge: Environment variable config: {}
invoke.config._merge_file: Runtime config file has not been loaded yet, skipping
invoke.config.merge: Overrides: {}
invoke.config.merge: Modifications: {}
invoke.config.merge: Deletions: {}
invoke.program._parse: argv given to Program.run: None
invoke.program.normalize_argv: argv was None; using sys.argv: ['/Users/lukeorland/workspace/venv/bin/inv', '--list']
invoke.program.parse_core_args: Parsing initial context (core args)
invoke.parser.__init__: Initialized with context: <parser/Context: {u'hide': <Argument: hide>, u'help': <Argument: help (h) ?>, u'config': <Argument: config (f)>, u'write-pyc': <Argument: write-pyc [bool]>, u'list': <Argument: list (l) [bool]>, u'collection': <Argument: collection (c)>, u'echo': <Argument: echo (e) [bool]>, u'version': <Argument: version (V) [bool]>, u'warn-only': <Argument: warn-only (w) [bool]>, u'debug': <Argument: debug (d) [bool]>, u'pty': <Argument: pty (p) [bool]>, u'root': <Argument: root (r)>, u'no-dedupe': <Argument: no-dedupe [bool]>, u'complete': <Argument: complete [bool]>}>
invoke.parser.__init__: Available contexts: {}
invoke.parser.complete_context: Wrapping up context None
invoke.parser.parse_argv: Starting argv: ['--list']
invoke.parser.handle: Handling token: '--list'
invoke.parser.handle: Saw flag '--list'
invoke.parser.switch_to_flag: Moving to flag <Argument: list (l) [bool]>
invoke.parser.switch_to_flag: Marking seen flag <Argument: list (l) [bool]> as True
invoke.parser.changing_state: ParseMachine: 'context' => 'end'
invoke.parser.complete_context: Wrapping up context None
invoke.program.parse_core_args: Core-args parse result: [<parser/Context: {u'hide': <Argument: hide>, u'help': <Argument: help (h) ?>, u'config': <Argument: config (f)>, u'write-pyc': <Argument: write-pyc [bool]>, u'list': <Argument: list (l) [bool]>, u'collection': <Argument: collection (c)>, u'echo': <Argument: echo (e) [bool]>, u'version': <Argument: version (V) [bool]>, u'warn-only': <Argument: warn-only (w) [bool]>, u'debug': <Argument: debug (d) [bool]>, u'pty': <Argument: pty (p) [bool]>, u'root': <Argument: root (r)>, u'no-dedupe': <Argument: no-dedupe [bool]>, u'complete': <Argument: complete [bool]>}>] & unparsed: []
invoke.program._parse: Finished parsing core args
invoke.program._parse: No default namespace provided, trying to load one from disk
invoke.loader.find: FilesystemLoader find starting at '/Users/lukeorland/workspace/invoke-yaml'
invoke.loader.find: Found module: '/Users/lukeorland/workspace/invoke-yaml/tasks'
invoke.parser.__init__: Adding <parser/Context 'my-awesome-task'>
invoke.program.parse_tasks: Parsing tasks against <Collection 'tasks': my-awesome-task>
invoke.parser.__init__: Initialized with context: <parser/Context: {u'hide': <Argument: hide>, u'help': <Argument: help (h) ?>, u'config': <Argument: config (f)>, u'write-pyc': <Argument: write-pyc [bool]>, u'list': <Argument: list (l) [bool]>, u'collection': <Argument: collection (c)>, u'echo': <Argument: echo (e) [bool]>, u'version': <Argument: version (V) [bool]>, u'warn-only': <Argument: warn-only (w) [bool]>, u'debug': <Argument: debug (d) [bool]>, u'pty': <Argument: pty (p) [bool]>, u'root': <Argument: root (r)>, u'no-dedupe': <Argument: no-dedupe [bool]>, u'complete': <Argument: complete [bool]>}>
invoke.parser.__init__: Available contexts: {'my-awesome-task': <parser/Context 'my-awesome-task'>}
invoke.parser.complete_context: Wrapping up context None
invoke.parser.parse_argv: Starting argv: []
invoke.parser.changing_state: ParseMachine: 'context' => 'end'
invoke.parser.complete_context: Wrapping up context None
invoke.program.parse_tasks: Resulting task contexts: []
invoke.program.run: Received a possibly-skippable exception: Exit()
Available tasks:

  my-awesome-task

Notable line: invoke.config._merge_file: Per-project config file has not been loaded yet, skipping

$ pip freeze | grep invoke
invoke==0.20.2

[bitprophet]

Ah, yea. This is because you're not actually getting to the task execution step; project conf files are loaded at task execution time. Note the following from the auto dash docs:

If you’d prefer the underscores to remain instead, you can update your configuration to set tasks.auto_dash_names to False in one of the loaded-by-default config files – typically, either the system or user location.

That said, I found myself surprised in retrospect, and suspect the current setup can be improved. Quick background (mostly rubber-ducking to myself here, but if you grok it, even better):

• Prior to 0.20, all config files were loaded "late", right before executing tasks.
• In 0.20 (due to Allow configuring loader root, collection, etc values #310 and Create dashed aliases automatically #329) we broke that up and tried loading some parts of the config "early", namely the system/user files the docs mention above. However, the rest - including project conf files - continue to be loaded on the later side.
• Project conf files must be loaded at least semi-late - specifically, after the tasks collection is found/loaded. This is why they occur in the later step and not the earlier.
• That later config-update step occurs after the 'parse' step, which is the crux of the problem, since it parses core args, loads collection (in case one used --collection), then parses tasks/task args.
• Your problem is that the third substep there, parsing tasks, is where things currently bail out, expecting my_awesome_task but only seeing my-awesome-task, as project conf isn't loaded yet (as above.)
• The doc snippet I linked technically "fixes" this by encouraging you to put auto_dash_names: false into system or user locations. If there were some other reason that project confs had to come very late in the process, this is where we'd stop, and I'd ask you how to improve visibility of the docs.
• However, I'm realizing now that when I implemented all this, the order-of-operations focus was mostly on Allow configuring loader root, collection, etc values #310, "make sure ppl can configure the task loading stuff in user/system conf files", and I didn't take things to their logical extreme.
• Which in this case is, "can't we make the parse step even more granular, and inject project conf loading as early as possible - namely immediately after the task collection is loaded?"
• And I think that's possible, so I'll see if I can prove it...check back later.

[bitprophet]

Ah yea, there is still a wrinkle - creating the Collection is when we tell it whether or not to auto-dash names. So if we wait until the collection is completely loaded to grab the project config, it's still too late to affect that side of things.

I think we can still shove project loading earlier but it will mean forcing the loader to learn more about the config system (i.e. violating separation of concerns). Not ideal but might still be doable, and I don't think there will be any drawbacks unless one has valid Invoke conf file names lying around between the CWD and the actual tasks file. Even then, it should just be log spam; we can always unload such a config if there turned out to be no collection there.

EDIT: Though this could mean the loader shouldn't be the one making the collection, only returning the info required to make one. That feels better?

[bitprophet]

Yea that feels cleaner now, at the mild expense of making a test module do marginally more work on its end.

Pushing commits now; barring any issues caught by my pre-push hook or Travis, this should get released as 0.20.3 soon. Thanks again & sorry this particular change had so many annoying roadblocks in the way of disabling it!