Configuration with Environment Variables

Managing application settings from local development to production

In my previous post, one of the points I touched on in the Deployment section was on application configuration management. Specifically, that it's best to use environment variables to manage configuration settings that vary among different deployments of your application.

The Debate About Env Vars

This is a concept borrowed from the 12 Factor App, and while the idea seems like a natural fit theoretically, it's still difficult for many people to imagine how it would work in real-world development scenarios. I, myself, was skeptical at first, for a variety of different reasons, many of them nicely summarized on a Gist by telent:

  1. Environment variables can 'leak', possibly exposing your secrets.
  2. Environment variables are no more secure than files.
  3. You still need to populate your variables from some source.
  4. Reloading or reconfiguring in-place might be difficult.
  5. There is no single place to look for what variables are required.

Initially I found these to be convincing, but after looking a little deeper, I realized that some of these arguments are actually re-stating (or misinterpreting) the guideline as "don't use files for anything", which isn't the case at all. Kristian Glass provides a solid counter-argument explaining that:

12factor says your applications should read their config from the environment; it has very little to say about how you populate the environment – use whatever works for you.

In addition to this, he outlines a number of reasons why environment variables are such a good solution to this problem. But the debate rages on, and many developers are still reluctant to adopt them as the mechanism for configuring applications on different deployments.

It's Not About Security

One important thing to realize is that the goal of this particular 12factor guideline has nothing to do with security or protecting your application's "secrets". Many people are conflating "settings that vary among application deployments" with "secret keys", and then re-framing the argument to be about best practices in secrets management. This is missing the point.

The reasoning behind separating the settings that vary among deploys is to ensure that your application bundle can be deployed anywhere without modification, as long as the appropriate environment settings are laid down in parallel. It's about separating your code from the ecosystem in which it runs, and factoring out the parts that only deal with the environment.

This can include secrets like database passwords or API access keys, but beyond that it includes hostname settings, endpoints for third-party resources, email addresses and log level thresholds, and even scaling and cluster settings so that local development doesn't require a full production-scale stack.

If you are worried about secrets being exposed via environment variables, then you can use a dedicated secrets management service like Ansible's Vault or Conjur, or lay down secrets using Puppet or Chef, or even build your own on top of a service like Amazon KMS. But in any of those solutions, the fact remains that your secrets need to be made available at some point to your application in unencrypted form, and you need to ask yourself if reading from a file poses any greater or lesser risk than reading from environment variables.

Whatever approach you choose, the key takeaway is: separate your config from your app.

A Real-World Example

The biggest question developers seem to have regarding the use of environment variables for config is: how? How do you store them, and how do you populate them for deployments, and how do you read them in your application? To help answer those questions, here are a few examples of how we here at Touchstone use environment variables in our deployment process:

Application Code

A good way to think about structuring your code to facilitate a clean separation of environment configuration is to treat your application as though it were a function: A function has a specific signature, and the execution and result of the function is dependent upon the parameters you pass in to it. The environment-specific configuration settings represent the signature of your application. The app can be deployed on any environment, and invoked using the proper configuration values.

This way of thinking also stops your from using named categories to encapsulate groups of environment settings (e.g., "dev", "test", "prod"). It's much better to treat them all as independent variables, as you would when passing values to a function. Generally speaking, you wouldn't write a function to accept a single "name" parameter and then hard-code a bunch of values in the code for an invocation using "dev".

A good way to document this 'application signature' is to define, depending on your language, a class or file with key-value pairs representing each required environment variable.

class environment {
    public static $my_secret_key;

    public static $smtp_host = "default.example.com";
    public static $smtp_user;
    public static $smtp_pass;

    // additional environment variables ...
}

At runtime, the app can examine the file and load variables from the environment based on the key, and set the appropriate value. Rather than coding getenv() style calls throughout your app, the rest of the code simply looks up the values using this class.

// check for the file defining the required variables
if(file_exists($environment = "/path/to/config/environment.php")) {
  include_once $environment;

  // ensure the file defines an environment class
  if(class_exists('environment', false)) {
    $class = new ReflectionClass('environment');
    $props = $class->getStaticProperties();

    // using reflection, get each property name, uppercase it,
    // use getenv() to retrieve the value and then set it on the class.
    foreach($props as $name => $value) {
      if($val = getenv(strtoupper($name)))
        $class->setStaticPropertyValue($name, $val);

      else if($value === null)
        trigger_error("missing environment value: " . strtoupper($name));
    }
  }
}

Since this approach only sets the values if they are found in the environment, you can define sensible fallbacks in the original class, which will be used if no corresponding environment variable is found.

$mail = new PHPMailer(true);
$mail->isSMTP();

// use the environment variables from the class directly
$mail->Host = environment::$smtp_host;
$mail->Username = environment::$smtp_user;
$mail->Password = environment::$smtp_pass;

// set additional email message properties ...

// send the message
$smtp->send();

Here we reference the environment variables directly, without worrying about where specifically they are coming from. By ensuring that all code depending on environment variables uses this single 'source of truth', the variables can be altered at any time without having to modify any of the runtime code.

Local Development

Okay, so your app is structured in a way that lets you easily configure it using environment variables, but how do you actually run it locally? It would be far too tedious to have to manually specify all the variables every time you want to run the app, so you're obviously going to need to store them somewhere and load them on demand.

An option I wouldn't recommend is using your own ~/.profile file or similar. You don't want to mix up your local shell environment with application-specific values, both for security reasons and so that you can possibly run multiple copies of the app with different settings. You'll instead want to store them in their own file and load them on-demand when your run your app, and the best way to do that is to use a task runner.

At Touchstone, we use Gulp as our preferred task runner, and the dotenv module to source an environment file prior to launching the app. One key difference is that I always keep my application environment files under my home directory in a ~/.env/ folder, and not at the root of the repository. While it's tempting, I prefer to eliminate the possibility of accidentally checking it in should the file contain sensitive data.

Here's a snippet from one of our gulpfiles to show how we use dotenv, along with gulp-connect-php and browsersync, to set the environment variables before launching the process:


var gulp = require('gulp'),
    env = require('dotenv'),
    php = require('gulp-connect-php'),
    sync = require('browser-sync').create();

gulp.task('env', function() {
    env.load({path: process.env.HOME + '/.env/myapp'})
});

gulp.task('serve', ['env'], function() {
    php.server({
        // php server config ...
    });

    sync.init({
        // browsersync config ...
    });

    // gulp watch tasks ...
});

Since the environment variables are added to the path within the gulp process, they will be passed on to the php server process and will be available via getenv(). This also prevents them from 'leaking' into a higher level process or your own shell.

Remote Deployment

For remote deployments, you'll need to solve the problem a different way, but it's generally pretty easy to handle. If you're using a PaaS provider like Heroku or Elastic Beanstalk, the problem almost takes care of itself: Heroku offers an easy way to add and remove config variables using the CLI. Or, you can use a configuration management system like Ansible, Puppet, or Chef to set environment variables directly on the host system. Most of these also have plugins available to interface with secrets management systems, so you won't have to store your secrets in plaintext within your config management repository.

For smaller applications, particularly smaller websites, we've found that using Elastic Beanstalk is ideal, and setting environment variables is very easy. For the initial deployment, you can create a simple .ebextensions config file that defines the values using empty-string placeholders:

--- # .ebextensions/01-environment.config
option_settings:
  - option_name: RAILS_SKIP_MIGRATIONS
    value: ""
  - option_name: SECRET_TOKEN
    value: ""
  - option_name: CUSTOM_ENV
    value: ""

When you use the EB CLI to create the environment, these variables will be defined and you can edit them in the AWS Console:

On future deployments, the console-defined variables will not be overwritten by the ones defined in the .ebextensions config file.

Additionally, you can save a copy of the entire configuration for an Elastic Beanstalk environment, including the environment variable values. This makes it trivially easy to spin-up and spin-down environments on-demand, since you don't have to reconfigure the environment each time. The EB CLI makes this easy via command-line flags:

eb create my_new_env --cfg my_saved_env_config

Wrap-Up

These are just a couple of small, specific examples of how you can move toward using environment variables throughout the entire development process. There are plenty of other ways to do it, but given how painless it was for me personally to make the transition, I think it's a worthwhile endeavor.

There are downsides, however, and even if you don't decide to go with using environment variables, you should always strive to separate your environment-specific configuration from your code.

Add Comment

Used only for Gravatar, and will remain private.