Use Bare Git Repo To bypass Corporate Firewall

We started using Git to track changes and manage Puppet code. At first we had only one Puppet master and all was nice and simple.

Once we gained trust and everyone was happy, we added more puppet masters in different isolated networks, one of which we had to find a way to punch a hole through the company Firewall. Sometimes it’s just not worth it to fight through bureaucracy and security regulations so we had to come up with a work-around how to keep Puppet code repository synced with all the other repos.

We couldn’t get direct network access from the isolated puppet master to the git server, but we had 2 other servers, one in each side that were open for communication on port 22. We thought we can use these two servers as some sort of a git bridge to keep all puppet masters in synced with the latest code.

We did this using a bare git repo to act as a bridge between the puppet master and the git server. A bare git repo is a repo that does not have a working tree attached to it. You can  initialize it like this:

$ git init --bare

We had to do this twice: one on a server located outside of the sealed off network and second on a server inside the secured network, who can talk with the puppet master.

We then created a post-receive git hook to push data automagically whenever new code is pushed to the git server. You have to setup SSH keys for this to work. A post-receive hook is a simple script you drop in the HOOKS directory of any git repo, containing commands you want to be executed by Git, whenever the repo is being updated by pushing code to it.

The end result was this:

1. You push from workstation to git server

2.git hook pushes from git server to a git bare repo A.

3. Bare git repo A is using another git hook to push to bare repo B which is located at the other more secured network.

4. It’s now possible to pull updated code from git to puppet.


What’s next? Create more git hooks to automatically push/pull code whenever a git commit is done by a developer so that we wouldn’t have to manually update the puppet masters with new code. This is a bit dangerous since we are working with only one git branch (master) and you don’t want untested code to get into production. So we’ll have to create dev/staging/prod branches before we jump into more automation tasks.

2016-07-02 12_30_17-Git + GitLab introduction - Google Slides



Using Foreman & Puppet to monitor CIS Benchmarks

cislogoweb + foreman_medium + pl_logo_vertical_rgb_sm

I’ve recently started a very interesting project with puppet and foreman to implement CIS benchmark document for hardening server infrastructure. Some of the items in the document can seriously cripple your machines (Disable X11 Forwarding… rpm -v *…) so instead of enforcing everything or creating complicated “if-then” code, we applied a mechanism in Foreman UI to alert when certain node is not in compliance with CIS.

At first, we used “Notify” resources in puppet for every alert that needs to be thrown when a machine is not in compliance. This way we can have puppet reports show CIS warnings in Foreman with colorful messages. The problem with the “Notify” resource is that Puppet treats these events as an active change to the machine. This defeats the purpose of a monitoring system because every time puppet agent runs, it sends its report to Foreman and Foreman shows all nodes not in compliance as being “Active”. This is misleading because nothing really changed, it’s just the Notify message causing Puppet to think something changed on the machine.

To remedy this issue, I tried to make some modifications in Foreman source code to make Foreman ignore the Notify events, but that didn’t turned out so well because I had to enable “noop” attribute for every single Notify piece of code (200+ CIS items = 200+ Notify events in “noop” generates even more noise in Foreman dashboard)


Fortunately, someone at StackOverFlow was kind enough to point out that I should use a custom resource type that is available in Puppet Forge called “Echo“. It does exactly the same thing as the Notify resource, without having Puppet report to Foreman that the node has changed. Problem solved!

We now have a fairly good indication when servers in production are not in compliance with CIS benchmarks, using Foreman and Puppet.