How do I prevent puppet interfering with manual server changes
A lot of our server configuration is managed via puppet. Puppet runs every 15 minutes via cron and applies configuration changes.
If a server has problems I sometimes need to change configuration manually on the server (e.g try different settings, enable more logging). The problem is that puppet then overwrites my manual changes whenever the cron job runs.
What would you recomment to handle this problem?
Maybe deactivate puppet while working on the servers? Could be the risk that I forget to reenable it and then have one server out of sync.
What I normally do is the following
- Manually stop puppet on that machine
- Have a nagios or monitoring tool of your choice alert that will warn me when puppet hasn’t been run for more than 2 hours
- If you want to be extra cautious also make a pupet watchdog, one that checks that the /var/run file is in place, if you stop puppet properly it’ll erase it, if the file is there but there’s no process it died so it’ll start up again
With #1 and #2 you’ll cover 99% of your case scenarios, and if you want to maybe #3 will help you when puppet misbehaves (sometimes it did on me)
Good luck!
Check more discussion of this question.
Related posts:
- Puppet causes endless restarts of CUPS (how does one prevent this)
- Puppet – Any way to copy predefined custom configuration files for software on clients from the puppet master (host)?
- I’ve broken my puppet, clients are failing reporting “ Could not run Puppet configuration client: Invalid parameter stage”
- Is it possible to use the Puppet inventory for puppet modules and other systems?
- Custom fact to copy bar file from all Puppet clients to main Puppet server