I’ve recently started a very interesting project with puppet and foreman to implement CIS benchmark document for hardening server infrastructure. Some of the items in the document can seriously cripple your machines (Disable X11 Forwarding… rpm -v *…) so instead of enforcing everything or creating complicated “if-then” code, we applied a mechanism in Foreman UI to alert when certain node is not in compliance with CIS.
At first, we used “Notify” resources in puppet for every alert that needs to be thrown when a machine is not in compliance. This way we can have puppet reports show CIS warnings in Foreman with colorful messages. The problem with the “Notify” resource is that Puppet treats these events as an active change to the machine. This defeats the purpose of a monitoring system because every time puppet agent runs, it sends its report to Foreman and Foreman shows all nodes not in compliance as being “Active”. This is misleading because nothing really changed, it’s just the Notify message causing Puppet to think something changed on the machine.
To remedy this issue, I tried to make some modifications in Foreman source code to make Foreman ignore the Notify events, but that didn’t turned out so well because I had to enable “noop” attribute for every single Notify piece of code (200+ CIS items = 200+ Notify events in “noop” generates even more noise in Foreman dashboard)
Fortunately, someone at StackOverFlow was kind enough to point out that I should use a custom resource type that is available in Puppet Forge called “Echo“. It does exactly the same thing as the Notify resource, without having Puppet report to Foreman that the node has changed. Problem solved!
We now have a fairly good indication when servers in production are not in compliance with CIS benchmarks, using Foreman and Puppet.