Expanded configuration mechanisms

[sophacles]

Maintainer edits

Overall TODO generated by huge arse discussion thread.

• Add new concepts doc on configuration, including a clearly marked hierarchy definition, and a big WARNING about how multiple files in same location w/ diff extensions are not merged.
• Make sure to clarify .yaml, not .yml.
• Probably also needs an update to the CLI docs section re: flags and env vars (new subsection) punting on user-driven config updates that aren't env vars; env vars have their own section in config so no need to duplicate
• Re: env vars, need to figure out best way for user-driven config settings to work too, eventually punting
• Brief mention of concepts doc in the tutorial Actually don't see an elegant way to work it in - feels best just as a standalone regular concepts doc for now
• Make sure API in docs gets updated to be consistent & 'modern' - e.g. obj['foo.bar'] should turn into obj['foo']['bar'] or ideally with the option of obj.foo.bar.
• Update concepts/contexts to adapt, possibly paring it way down or removing entirely, tl;dr it's just about explaining how contexts perform shared state.
  • The 'nested collection configs get merged' explanation could still live here but probably lives better in the new config page, or the namespacing page if it exists.
• Vendorize etcaetera
• Use its Config object in place of Context's existing inner dicts, get tests passing again referencing it (it's dict-like).
  • Decide whether the current run/general/etc splits become sub-config objects or just nested namespaces within the primary config object
  • Leaning towards the latter so people can override them everywhere
• Add new CLI flags for setting config options (primarily -f or whatnot for "specific config file")
• Implement hierarchy re: multiple conf files + env vars + cli flags + in-module configs
• Conf file loading of different extensions at the same 'location' is exclusive - no merging of eg /etc/invoke.yaml and /etc/invoke.json.
• Debugging of conf option overrides must be easy
  • either via a no-op CLI flag showing what the tree would be for a given task module (or task invocation?), or
  • via regular debug output (ideally + a no-op option)
  • really, those go to the same place
• If not covered in the previous point: it must be easy to troubleshoot stupid typos in config file paths.
  • e.g. if you accidentally created a /etc/invok.yaml, or fatfingered the arg to invoke -f <path> - it should be clear when running with -d which config files it attempted to load and which ones failed to load.
  • Main problem is that for the first 3-4 levels we have 3x file paths we end up searching for, so it will be somewhat verbose :( but no obvious solution offhand, better to add it to debug output (or some other level?) than try to be clever.
• Update invocations to use this new setup instead of dotted names - they still technically work but are invalid for eg env var translation.
• Context objects (& subclasses like Fabric2's Connection) need a clean API for this configuration stuff - either Context directly grows all the config related methods (perhaps becoming a etcaetera.Config subclass itself?) or it accepts some config object (again perhaps etc.Config) as an instance member.
  • The current driving need here is actually testing related, I need to inject a config value override or two in Connection tests.
• Fill in any remaining skipped tests related to this feature set.

---

Original ticket description from sophacles

It would be nice to be able to pass context variables on the command line to invoke scripts. Something like:

> inv -X var1=val1 -Xvar2=val2 target --target1opt=val

And perhaps even allow targeting variables to specific namespaces, ala:

inv -X ns1.var1=val1 -Xvar2=val2 ns1.target  target2

for those cases where target2 should use the default for var1, but ns.target1 needs the overridden value.

There are of course concerns about how command line set options override things in a call to Collection.configure... take for instance this script:

@ctask
def task1(ctx):
      run("echo {val1}".format(**ctx.config))

@ctask
def task2(ctx):
    run("echo {var1} and {var2}".format(**ctx.config))

ns1 = Collection(task1)
ns1.configure({"var1":"ns1var1"})
ns = Collection(ns, task2)
ns.configure({"var1":"globalvar1","var2":"globalvar2"})

Which is the right output:

option 1:

# inv -X ns1.var1=clival1 -X var2=clivar2 ns1.task1 task2
clival1
globalvar1 and clivar2

(note - because generally it's considered cli options override anything in-program)

option 2:

# inv -X ns1.var1=clival1 -X var2=clivar2 ns1.task1 task2
globalvar1
globalvar1 and clivar2

(note because of the way overrides work in invoke already)

I think everyone would agree that:

# inv -X var1=clival1 ns1.task task.2
clival1
clival1 and globalvar2

is correct.

[bitprophet]

I haven't yet sat down and pondered how I want ye olde config override hierarchy to work here, but this is definitely part of it. Then there's also the possibility of env vars (#22) and config files, possibly multiple layers of those as well (system/global, per user, local to project, etc).

I'm already semi unhappy with how the existing config mechanism can be unintuitive and hard to reason about, so I'd prefer to really hash this out sometime with pros/cons instead of iterating. Would welcome more ideas if you have them, of course.

[sophacles]

Generally speaking.. (leaving aside for a moment the namespace stuff in inv) in unix land the hierarchy goes something like this (lowest priority to highest priority):

• default value
• general environment variable (e.g. $EDITOR)
• configuration file (first global, then local - e.g. ~/.gitconfig is overriden by $project/.git/config)
• program specific environment variables - e.g. $GIT_EDITOR
• command line options.

Of course, some programs swap the config files with one of the environment variables - because strong consistency is not a unixy concept.

So I think in the large, that would be a good idea to go with as a start.

Now, about configs files, and how they override each other, and so on - for an extreme case of what could happen I like to refer to vim's runtimepath. It does (lowest precedence to highest):

• /usr/share/vim (app defaults)
• /etc/vim (system defaults)
• ~/.vim (local settings)

(it also then has "after/" stuff for which it does in the reverse order, with all sorts of other exceptions and other rules (in vim do - :he runtimepath for the gory details). I think that's not the goal here :))

So about invoke... The way collection namespacing and config overriding works now is pretty nice (once the multiple paths to a collection issue is figured out that is). It's basically the same as the ones mentioned above...

• submodule
• importing module
• importing importing module... etc

The common thing about all of these is they basically order precedence on a concept of "locality" or "immediacy". Effectively the, the precedence is given to the option that was most likely to have been changed most recently. In the invoke case this is "i imported already written code, then called configure on a collection that had the imported stuff"... it's most likely I won't import a library that isn't written yet.

All that is a really long-winded way of suggesting this type of "how to set variables" precedence order:

• module level settings
• importing module settings
• importing importing module settings... (etc)
• /etc/invoke.conf
• ~/.invoke set variables
• ${DIRECTORY_CONTAINING_TASKFILE}/config.py (or whatever)[1]
• environment vars
• command line -f path/to/config.py
• command line -X var=val options (with the semantics from above hashed out for namespace overrides)

[1]With the option of having ~/.invoke or /etc/invoke.conf specify a search path for relevant config files. Also note - then of course naming config files conf.py is not a good idea anymore.

Anyway that turned out longer than expected. hah!

[bitprophet]

Hah, no, that's perfect - overall it obviously fits what I or any other longtime Unix user would have come up with if we'd spent a few minutes thinking about it. Thanks!

The only parts I have any quarrel with:

• The 'local to tasks file config.py' - I'm not sure what this gives users over setting the equivalent values within their tasks.py instead? If it were a YAML or JSON file, maybe, but even then I wonder if it is worth the conceptual/implementation overhead.
• The config.py's vs invoke.conf's - shouldn't those all be the same type/format? (and curious if you've thoughts on what format - personally I strongly prefer YAML despite its shortcomings, but barring that I'd use JSON; ini and configobj are both terribad at nested dicts.)

[matleh]

+1 for this - to be able to override configuration settings and for the use of JSON and/or (JSON-subset style) YAML for configuration files.

[sophacles]

Not sure why I put config.py instead of invoke.conf - I'm just going to blame "Friday afternoonism". I agree that they should all be the same format. (And no preference on what the actual format is).

As for the use case of task local invoke.conf vs having it built into the tasks.py: for me it comes down to separation of concerns - I just like having config details different than the code that executes them. It's not a strong technical argument, just a preference - in the end either way would be fine.

[bitprophet]

Overriding this ticket to be a general config overhaul/expansion one. I need to start poking at this to support Fabric 2 development, where things like "storing the default SSH port and local username, then allow overriding in various ways" come into play.

I skimmed tickets on Fabric's tracker and most PRs or issues mentioning the topic, and specifically JSON/YAML support (which I think is absolutely required) just tie such file loads into Fab 1's single level env dict.

I googled for Python libs dealing with this to see if we can avoid reinventing the wheel (tho as usual I worry that we'd have the usual fork/collaborate/fuckit problem trying to hone the inevitable 10% of functionality that we need to change).

I'll edit this post to add more links, but so far I've found:

• https://github.com/oleiade/etcaetera#etcaetera
  • young but reasonably fleshed out
  • seems flexible about what order things load in (i.e. would probably let us implement @sophacles' ideas above)
  • does JSON, YAML, Python modules, etc etc.
  • handles nested configs somewhat (unclear how this meshes with YAML/JSON tho)
• https://github.com/duckworthd/configurati#configurati
  • reasonably similar to Etcaetera - JSON, YAML, etc all supported
  • has additional tools for making assertions about configuration groupings
  • seems flexible about nested structures
  • even younger, no official tagged releases
• https://github.com/dnephin/PyStaticConfiguration
  • also reasonably powerful/flexible, also supports many formats
  • the intro tour makes it feel like it may be a bit over-engineered (proxies, facades, etc)
  • may actually be the most mature so far (though that still doesn't say much)
• http://configman.readthedocs.org/en/latest/tutorial.html
  • Needs more investigation, may also be too CLI focused

[sophacles]

+1 for etcaetera but all of them look pretty good. (note: my vote is based only on random details in my first impression and is totally subjective. I have not arguments for or against any of them)

[bitprophet]

Cool, thanks @sophacles. Digging deeper into etcaetera now (tho partly just as it's first). For any of these, our specific needs seem to be:

• Ability to handle arbitrarily nested config to match nested namespaces (unless we can make dotted names work well enough or whatnot)
• At least one level of nesting to handle different areas of config (general, run() related, etc)
• Strong control over hierarchy, as per earlier comments about flags vs env vs multiple config locations - we want it to work our way and not be foisted on us.
• JSON support
• YAML
• environment variables
• some way of manually injecting values from our CLI parser at a specific hierarchy level
• (ideally/optionally) Python file support)

[bitprophet]

Notes from etcaetera:

• Typing the name is a pain :)

• It interprets a single nested config file (eg YAML) as nested subdicts, which is good.

• You can add additional config objects (each with their own set of adapters) as sub-configs; these do not get "merged" but are instead available as sub-objects via attribute. E.g.:

  config['foobar']
  config['foobar']['bizbaz']
  config.subconfig['whee'] # This comes from some other source, eg another conf file
  # NOT VALID
  config['subconfig]['whee']
  

  • This may or may not be useful for us depending how much we directly expose Etcaetera objects to our users.

So far it fits all of my previous bullet point needs except possibly the "inject our own CLI flag values" - I need to see if we can stick those in the middle or if they can only be added via the 'defaults' adapter. Optimistic tho.

EDIT: looks like the only builtin way to do that is the Overrides adapter which always comes last in the hierarchy, but that may be appropriate for CLI flags anyway. Gotta doublecheck against the above notes. We could also probably make a custom adapter, have to look into that.

[bitprophet]

There are no explicit docs on custom adapters, but it's pretty basic, the AdapterSet accepts any number of adapter objects between the defaults (always forced first in the deque) and overrides (always last) and as long as one implements a few basic "here's my settings" methods, should be easy to subclass. And again that's if Overrides don't work for us.

[bitprophet]

Taking the above hierarchy list, here's how it could work with an etcaetera based Config object + adapters. Note: it might replace the existing namespace driven hierarchy, or it might augment it, I need to revisit the corner cases of that design to see if it works ok in what etc gives us.

• module level settings - anything defined in Invoke itself in the default context/collection objects; could be an etc Defaults adapter obj
• importing module settings - 'importing module' means a sub-collection, so that falls under the 'note' above. Could be we just take our existing "stuff from the namespace" and end up putting it all in the initial Defaults once we've internally collapsed it.
• importing importing module settings... (etc) - ditto
• /etc/invoke.conf - setting aside default format & multiple formats for now (tho see below) this would simply be the next adapter in the set, a File.
• ~/.invoke set variables - again setting aside specifics, this would be another File adapter
• ${DIRECTORY_CONTAINING_TASKFILE}/config.py (or whatever) - yet another File adapter
• environment vars - an Env adapter
• command line -f path/to/config.py - another File adapter
• command line -X var=val options - this would be an Overrides adapter (or a custom adapter) generated out of our existing parser stuff. Presently, these values are (like the Collection namespace configs) self-merged into the core Context object, so this is another spot where we need to figure out how much we delegate to etcaetera.

[bitprophet]

Re: file formats, random thoughts:

• Use of etc or the other libs lets us deal with Python module files, YAML and JSON roughly equally/transparently (etc's File adapter selects how to load data based on file extension, for example)
• The easiest-implementation option is to say "we only support one type, we will seek out only config.<that type>, deal with it" and then there's no ambiguity about ordering.
• That's not too friendly and etc makes it easy to load more than one, so we could instead add adapters for each possible extension - i.e. the /etc/invoke.conf "item" becomes 2 or 3 adapters, i.e. File('/etc/invoke.yml'), File('/etc/invoke.json'), File('/etc/invoke.py').
  • HOWEVER the problem is if some poor sod ends up with >1 of those on the FS - due to etc's implementation (really - any implementation) there has to be some sort of order and there's no clear one right intuitive answer.
  • Feel it's best to go for practicality over purity (hi Zen of Python) so off the cuff I'd just go with yaml -> json -> py (if we offer py).
• At first I thought etcaetera's Python-module support was single-level only (module level constants) but no, it happily consumes dict values. So Python ends up being just as flexible as YAML/JSON in terms of nesting; we might as well support all 3.