Make sure that users' changes to the config files are preserved. Fixes #4440.
Rather than hardcoding a config which we always forget to update, generate it from the default config.