Allow configuration file to be specified by an environment variable

[kevinjqiu]

Similar to how kubectl loads configuration files, if KUBECONFIG environment variable is set, it will load that config file as opposed to $HOME/.kube/config.

We have a few CLI scripts built on top of invoke, and we'd like the ability to specify config files during runtime.

e.g., suppose our CLI needs to talk to different servers for different environments,

inv -f dev.yaml <command>
inv -f prd.yaml <command>

we'd like to have the ability to do:

export INV_CONFIG=$(pwd)/dev.yaml
inv <command>

@bitprophet if you think this is a worthy feature, I wouldn't mind contributing a PR.

Thanks for pyinvoke!

[bitprophet]

I'd like this to be possible, sure! Most env var use so far has focused on the "env vars as a late-binding config layer" and this definitely can't work via that avenue because the runtime conf file wants to load before the env var layer.

So it'll need to be "regular" env var lookup, sort of like the early use of INVOKE_DEBUG. (Which is then also, later, shoved into the config itself as an override of the top level debug setting – not something you need to worry about for this feature, I don't think.)

I was all set to give you a quick checklist including "write tests for this" but I went and looked and we're not testing INVOKE_DEBUG either. So I'm gonna just go try and stab this directly real quick. Famous last words.

[bitprophet]

All set, is now the INVOKE_RUNTIME_CONFIG env var, let me know how it looks if you want to download the master branch! Will go out in the 1.1 release otherwise. Kinda want to tackle at least one more real feature before I put that on PyPI. Hopefully soonish.

[kevinjqiu]

@bitprophet Thanks for the quick turnaround. One request: can we config the env var that invoke looks for? We customize invoke to build our CLI with a different name, and it'd be nice to have the environment variable reflect our CLI's name.

e.g., if our CLI is awesome, the env var we'd like awesome to look for is AWESOME_CONFIG.

Thanks!

[bitprophet]

You can! See http://docs.pyinvoke.org/en/1.0/concepts/library.html#changing-filenames-and-or-env-var-prefix

[kevinjqiu]

But as you said, the INVOKE_RUNTIME_CONFIG is a "regular env var" unlike the late-binding env vars. I tried the tester program on master with the latest RUNTIME_CONFIG feature, setting TESTER_RUNTIME_CONFIG (after using a specialized Config class with prefix set to tester) didn't work:

$ tree
.
|____setup.py
|____tester
| |____tasks.py
| |____main.py
|____config.yaml

$ cat tester/main.py 
from invoke import Program, Collection, Config
from tester import tasks

class TesterConfig(Config):
    prefix = 'tester'


program = Program(version='0.1.0',
                  namespace=Collection.from_module(tasks),
                  config_class=TesterConfig)

$ cat tester/tasks.py 
from invoke import task


@task
def hello(ctx):
    print('Hello world: {}'.format(ctx.config.foo))

$ cat config.yaml 
foo: bar

$ tester -f config.yaml hello
Hello world: bar
$ export INVOKE_RUNTIME_CONFIG=$(pwd)/config.yaml
$ tester hello
Hello world: bar
$ unset INVOKE_RUNTIME_CONFIG
$ export TESTER_RUNTIME_CONFIG=$(pwd)/config.yaml
$ tester hello
Traceback (most recent call last):
  File "/home/kevin/src/invoke/invoke/config.py", line 112, in __getattr__
    return self._get(key)
  File "/home/kevin/src/invoke/invoke/config.py", line 177, in _get
    value = self._config[key]
KeyError: 'foo'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/kevin/src/invoke/.env/bin/tester", line 11, in <module>
    load_entry_point('tester', 'console_scripts', 'tester')()
  File "/home/kevin/src/invoke/invoke/program.py", line 335, in run
    self.execute()
  File "/home/kevin/src/invoke/invoke/program.py", line 491, in execute
    executor.execute(*self.tasks)
  File "/home/kevin/src/invoke/invoke/executor.py", line 134, in execute
    result = call.task(*args, **call.kwargs)
  File "/home/kevin/src/invoke/invoke/tasks.py", line 128, in __call__
    result = self.body(*args, **kwargs)
  File "/home/kevin/src/invoke/tmp/tester/tasks.py", line 6, in hello
    print('Hello world: {}'.format(ctx.config.foo))
  File "/home/kevin/src/invoke/invoke/config.py", line 124, in __getattr__
    raise AttributeError(err)
AttributeError: No attribute or config key found for 'foo'

Valid keys: ['run', 'runners', 'sudo', 'tasks']

Valid real attributes: ['clear', 'clone', 'env_prefix', 'file_prefix', 'from_data', 'global_defaults', 'load_base_conf_files', 'load_collection', 'load_defaults', 'load_overrides', 'load_project', 'load_runtime', 'load_shell_env', 'load_system', 'load_user', 'merge', 'pop', 'popitem', 'prefix', 'set_project_location', 'set_runtime_path', 'setdefault', 'update']

[bitprophet]

Sorry, for some reason I'd interpreted what you said as a generic question instead of a specific followup – you're right, that new env var we just added is hardcoded, I'll make a new ticket for making it configurable. (Smells a little off -- think it may want tighter tie-in with how Program and Config interact re: env vars and prefixes -- but still something to put down as a nice-to-have.)

[bitprophet]

See #550 for that!