Once upon a time there was a web application stack built atop of a particular CI (continue integration) pattern with particular security constraints which I had the pleasure of configuring. The application needed certain secrets (third-party API keys, passwords, trade-secrets, etc) in order to run properly but that data needed to be very well guarded (and definitely not shipped with the codebase). The following is how I tackled the assignment, I think it works well but I could be mistaken. Please feel free to comment with any questions, suggestions, or other feedback.
First, I decide to put all configuration information in environment-based YAML files except for the sensitive info. I decided to store sensitive data in shell environment variables in-memory having and referencing them in the codebase where needed. This particular application uses Puppet for CM (configuration management), the provisioning and continuous deployment process is similar to the following...
A fresh server is booted-up using an image which executes a bootstrapping script which installs a few essentials and prompts for some user input. One of the expected inputs is a password which is used to access the contents of a password-protected 7zip archive which gets downloaded from a private file hosting server. The 7zip archive contains a GPG (GNU Privacy Guard) public and private keyring which is used to decrypt the environment config file which gets downloaded from a private GitHub repository which only hosts the various app configurations. The bash-based environment configuration file is then sourced in-place (e.g. eval
gpg -d /tmp/envfile) having the commands within the file executed and certain secrets exposed as environment variables. Certain bits are also copied to disk in-case of failure and reboot (each location is secured).
Having the secrets exposed and a process and workflow which seems to be somewhat secure, I just needed a better way of getting at the complex environment variable names which held the sensitive data. So I decided to write Config::Environment, a module which give you access to app-specific environment variables (with some additional goodies as well).