Handling application settings hierarchy more easily with Python’s ChainMap

Say you have an application that, as almost every more complex application, exposes some settings that affect how that application behaves. These settings can be configured in several ways, such as specifying them in a configuration file, or by using the system’s environment variables. They can also be provided as command line arguments when starting the application.

If the same setting is provided at more than a single configuration layer, the value from the layer with the highest precedence is taken. This could mean that, for example, configuration file settings override those from environment variables, but command line arguments have precedence over both. If a particular setting is not specified in any of these three layers, its default value is used.

Ordered by priority (highest first), our hierarchy of settings layers looks as follows:

• command line arguments
• configuration file
• environment variables
• application defaults

A natural way of representing settings is by using a dictionary, one for each layer. An application might thus contain the following:

defaults = {
    'items_per_page': 100,
    'log_level': 'WARNING',
    'max_retries': 5,
}

env_vars = {
    'max_retries': 10
}

config_file = {
    'log_level': 'INFO'
}

cmd_args = {
    'max_retries': 2
}

Putting them all together while taking the precedence levels into account, here’s how the application settings would look like:

>>> settings  # all settings put together
{
    'items_per_page': 100,  # a default value
    'log_level': 'INFO',  # from the config file
    'max_retries': 2,  # from a command line argument
}

There is a point in the code where a decision needs to be made on what setting value to consider. One way of of determining that is by examining the dictionaries one by one until the setting is found:¹

setting_name = 'log_level'

for settings in (cmd_args, config_file, env_vars, defaults):
    if setting_name in settings:
        value = settings[setting_name]
        break
else:
    raise ValueError('Setting not found: {}'.format(setting_name))

# do something with value...

A somewhat verbose approach, but at least better than a series of if-elif statements.

An alternative approach is to merge all settings into a single dictionary before using any of their values:

settings = defaults.copy()
for d in (env_vars, config_file, cmd_args):
    settings.update(d)

Mind that here the order of applying the settings layers must be reversed, i.e. the highest priority layers get applied last, so that no lower-level layers can override them. This works quite well, with a possibly only downside that if any of the underlying settings dictionaries get updated, the “main” settings dictionary must be rebuilt, because the changes are not reflected in it automatically:

>>> config_file['items_per_page'] = 25
>>> settings['items_per_page']
100  # change not propagated

Now, I probably wouldn’t be writing about all this, if there didn’t already exist an elegant solution in the standard library. Python 3.3² brought us collections.ChainMap, a handy class that can transparently group multiple dicts together. We just need to pass it all our settings layers (higher-priority ones first), and ChainMap takes care of the rest:

>>> from collections import ChainMap
>>> settings = ChainMap(cmd_args, config_file, env_vars, defaults)
>>> settings['items_per_page']
100
>>> env_vars['items_per_page'] = 25
>>> settings['items_per_page']
25  # yup, automatically propagated

Pretty handy, isn’t it?