Where did everything go in Puppet 4.x?

This is a summary of [1] at docs.puppet.com.

• New All-in-One puppet-agent package - On managed *nix systems, you’ll now install puppet-agent instead of puppet. (This package also provides puppet apply, suitable for standalone Puppet systems.) It includes tools (private versions) like Facter, Hiera, and Ruby; also MCollective.

• *nix executables are in /opt/puppetlabs/bin/ - On *nix platforms, the main executables moved to /opt/puppetlabs/bin. This means Puppet and related tools aren’t included in your PATH by default. You have to add it to your PATH or use full path when running puppet commands. 

1. Private bin directories - The executables in /opt/puppetlabs/bin are just the “public” applications that make up Puppet. Private supporting commands like ruby and gem are in /opt/puppetlabs/puppet/bin

• *nix confdir is Now /etc/puppetlabs/puppet - Puppet’s system confdir (used by root and the puppet user) is now /etc/puppetlabs/puppet, instead of /etc/puppet. Open source Puppet now uses the same confdir as Puppet Enterprise.

1. ssldir is inside confdir - The default location is in the $confdir/ssl on all platforms.
2. Other stuff in /etc/puppetlabs - Other configs are in /etc/puppetlabs directory. Puppet Server now uses /etc/puppetlabs/puppetserver, and MCollective uses /etc/puppetlabs/mcollective.

• New codedir holds all modules(1), manifests(1) and data(2) - The default codedir location is /etc/puppetlabs/code on *nix. It has environments, modules directories and hiera.yaml config file.

1. Directory environments are always on - The default environmentpath is $codedir/environments and a directory is created at install for the default production environment. Modules are in $codedir/environments/production/modules and main manifest is in $codedir/environments/production/manifests. However you can still use global modules in $codedir/modules and a global manifest.
2. Hiera data goes in environments by default - Hiera’s default settings now use an environment-specific datadir for the YAML and JSON backends. So the production environment’s default Hiera data directory would be /etc/puppetlabs/code/environments/production/hieradata

• Some other directories have moved - The system vardir for puppet agent has moved, and is now separate from Puppet Server’s vardir. For *nix: /opt/puppetlabs/puppet/cache. The rundir, where the service PID files are kept, (on *nix) has moved to /var/run/puppetlabs. (Puppet Server has a puppetserver directory in this directory.)

Reference:
[1] https://docs.puppet.com/puppet/latest/reference/whered_it_go.html

Using wso2/puppet-modules to deploy WSO2 Products

This article will provide an insight to setup a standalone product using wso2/puppet-modules [1] for the given product. It will provide a sequence for you to follow in testing your WSO2 product of interest for a standalone deployment setup in a given puppet agent instance. 

This can be considered as a guide to setup your own puppet master/agent environment for testing given that you are going to manually setup the server instances.

3 Tasks

1. Setup two instances
2. Setup puppet master and puppet agent
3. Perform a catalog run to setup product instance

Task 1 - Setup two instances

This can be either two EC2 instances, two WSO2 IaaS instances, or two VM instances.

e.g.: 192.168.19.33 - puppet master

        192.168.19.35 - puppet agent

Task 2 - Setup puppet master and puppet agent

Steps to install and configure puppet master and agent are as follows.

Installing puppet

Puppet master

• Login to puppet master as the super user
• Issue the following commands.

> ntpdate pool.ntp.org ; apt-get update && sudo apt-get -y install ntp ; service ntp restart

> cd /tmp

> wget https://apt.puppetlabs.com/puppetlabs-release-trusty.deb

> dpkg -i puppetlabs-release-trusty.deb

> apt-get update

> apt-get install puppetmaster

• Check the puppet version as: puppet -V

Puppet version Should be 3.8.3 or higher

• Edit etc/hostname file and set property hostname

puppetmaster

• Edit your puppet master hosts file /etc/hosts as follows and set puppet master hostname

127.0.0.1 localhost

127.0.0.1 puppetmaster

Puppet agent

• Login to puppet agent as the super user
• Issue the following commands.

> apt-get update

> apt-get install puppet

• Edit etc/hostname file and set property hostname

e.g.: qaa-node-1

• Edit /etc/hosts file of your puppet agent

127.0.0.1 localhost

192.168.19.XXX puppet # puppet master IP address

Configuring puppet

Puppet master

• Login to puppet master as the super user
• Create a directory /etc/puppet/environment/production/
• Modify /etc/puppet/puppet.conf file of puppet master appending the following to [main] and [master] sections accordingly.

[main]

dns_alt_names=puppetmaster,puppet

environmentpath = $confdir/environments

[master]

autosign=true

• Before restarting the puppet master, clean all certificates, including puppet master’s certificate which is having its old DNS alt names.

> puppet cert clean --all

• Restart the puppet master for new configuration changes to take effect and to regenerate the certificate with the new dns_alt_names

> service puppetmaster restart

• Download git repo files

git clone wso2/puppet-modules [1] to a temp folder

• Prepare puppet modules
• Copy all modules in ../puppet-modules/modules to /etc/puppet/environment/production/modules/
• Install java module from puppet forge [2].

> puppet module install 7terminals-java

• Install stdlib module from puppet forge [3].

> puppet module install puppetlabs-stdlib

• Prepare hieradata
• Copy ../puppet-modules/hiera.yaml to /etc/puppet
• Copy ../puppet-modules/hieradata folder to /etc/puppet/
• Rename /etc/puppet/hieradata/dev -> /etc/puppet/hieradata/production
• Prepare site.pp manifest
• Copy ../puppet-modules/manifests/site.pp to /etc/puppet/environment/production/manifests/
• Prepare files/packs
• Copy the pack file (e.g.: wso2am-1.9.1.zip) to ../modules/wso2am/files/
• Create folder ../modules/wso2base/files
• Copy jdk installation file (e.g.: jdk-7u79-linux-x64.gz) to ../modules/wso2base/files
• Add jdk version (home) and filename information to /etc/puppet/hieradata/production/common.yaml

Puppet agent

• Login to puppet agent as the super user
• Modify /etc/puppet/puppet.conf file of puppet agent appending the following to the [main] section.

[main]

server = puppet

Task 3 - Set Facter variables and perform a puppet agent run.

Following sample files attached are for APIM 1.9.1. Modify the deployment.conf file according to your product.

• Login to puppet agent as super user.
• Copy the file deployment.conf [4] -> /opt/deployment.conf
• Copy the file setup.sh [5] -> /opt/setup.sh

> chmod 755 setup.sh

• Run following command to setup product instance.

> ./setup.sh

An instance of a standard WSO2 product of your choice will be installed.

e.g.: 

CARBON_HOME ->  /mnt/10.0.2.89/wso2am-1.9.1

For APIM 1.9.1 the complete deployment time (deployment + server startup) will be approximately 1 minute.

If you are planning to go ahead with Vagrant setup, i.e. to spawn VM instances please refer [6].

[1] https://github.com/wso2/puppet-modules

[2] https://forge.puppetlabs.com/7terminals/java

[3] https://forge.puppetlabs.com/puppetlabs/stdlib

[4] Attachment: deployment.conf

product_name=wso2am

product_version=1.9.1

product_profile=default

vm_type=openstack

[5] Attachment: setup.sh

#!/bin/bash

echo "#####################################################"

echo "                   Starting cleanup "

echo "#####################################################"

ps aux | grep -i wso2 | awk {'print $2'} | xargs kill -9

#rm -rf /mnt/*

sed -i '/environment/d' /etc/puppet/puppet.conf

echo "#####################################################"

echo "               Setting up environment "

echo "#####################################################"

rm -f /etc/facter/facts.d/deployment_pattern.txt

mkdir -p /etc/facter/facts.d

while read -r line; do declare  $line; done < deployment.conf

echo product_name=$product_name >> /etc/facter/facts.d/deployment_pattern.txt

echo product_version=$product_version >> /etc/facter/facts.d/deployment_pattern.txt

echo product_profile=$product_profile >> /etc/facter/facts.d/deployment_pattern.txt

echo vm_type=$vm_type >> /etc/facter/facts.d/deployment_pattern.txt

echo "#####################################################"

echo "                    Installing "

echo "#####################################################"

puppet agent --enable

puppet agent -vt

puppet agent --disable

[6] https://github.com/wso2/puppet-modules/wiki/Setting-up-the-Development-Environment

Puppet create a multi-level directory structure

Lets assume that you currently have your only puppet agent and the puppet master.
In this case you can write your code segment in /etc/puppet/manifests/site.pp file itself under default node definition (This will be applied to all nodes if NO matching node definition is found for the given puppet agent hostname).

node default {

        # create a directory

        $dirpath = '/tmp/dbscripts/mysql'

        exec { "create_dir_structure":

                path        => '/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin',

                command => "mkdir -p ${dirpath}",

        }

        # create the file

        file {  "/tmp/dbscripts/mysql/hellopuppet.txt":

                ensure  => file,

                content => 'master says hello to puppet agent',

        }

}

This will successfully create the directory structure even if parent directory does not exist.

Note:
If you attempt to create the directory structure as follows,
# create a directory
file { "/tmp/dbscripts/mysql/":
                ensure => "directory",
      }

You will get the following error.
Error: Cannot create /tmp/dbscripts/mysql; parent directory /tmp/dbscripts does not exist

There has been a suggested feature [1] logged in Puppet Labs yet NOT implemented as of puppet version 3.4.3 master/agent. May be there are smart alternatives as above rather than adding an additional feature... :)

[1] http://projects.puppetlabs.com/issues/86
[2] https://github.com/ghoneycutt/puppet-module-common/blob/master/manifests/mkdir_p.pp

Puppet fill template using a central configuration file inside a module

Sometimes we need to keep all the configuration information in a central configuration file. It is easy to maintain and gives a clear idea of the scenario.

This article provides guidance on how to keep this kind of a central file and fill template values during a puppet catalog run in agent node.

Arrays of data which will be looped inside template filling process is also demonstrated in this example.

In the current tested scenario, the hostname of the puppet agent node is appserver-wkr

Configuration file is params.pp which resides inside module's manifests folder.

This file is inherited by init.pp of the module which is having a class with the same name as the module's (appserver).

In order to push a configuration in init.pp file we simply include the module in site.pp folder.

When a puppet agent find the master and requests for a catalog run, master will map the hostname with the relevant node configuration in site.pp file and apply it to the agent node (if no match found apply the default node configuration).

To have more clear idea, see the graphical illustration below followed by code content of each relevant file.

Following are the /etc/puppet tree structure in puppet master node and its file contents.

├── manifests

│   └── site.pp

└── modules

   └── appserver

       ├── manifests

       │   ├── init.pp

       │   └── params.pp

       └── templates

           └── polite-file.erb

1. site.pp

import 'appserver'

node 'appserver-wkr' {

       include appserver

}

node default {

}

2. init.pp

class appserver inherits appserver::params{

       file {  "/tmp/$myname":

               ensure  => file,

               content => template('appserver/polite-file.erb'),

       }

}

3. params.pp

class appserver::params ($say_hello_to = 'guys and gals') {

       #$say_hello_to = 'guys and gals'

       $myname = 'welcome file.xml'

       $serverOptions     = '-DworkerNode=true'

       $subDomain         = 'worker'

       $members           = ['10.0.1.196', '10.0.1.198', '10.0.1.200', '10.0.1.202']

}

4. polite-file.erb

<% if @say_hello_to -%>

Hello <%= @say_hello_to %>,

<% end -%>

I'm <%= @myname %>, on a <%= @operatingsystem %> system, nice to meet you.

<members>

         <%- if @members -%>

# loop hostnames

         <%- @members.each do |hostname| -%>

         <member>

          <hostName><%= hostname %></hostName>

            <port>4100</port>

         </member>

         <%- end -%>

         <%- end -%>

</members>

When all the files are in place simple perform a puppet catalog run in the agent node.

> puppet agent -t

Following is the created file (i.e. /tmp/welcome file.xml) contents in agent node.

Hello guys and gals,

I'm welcome file.xml, on a Ubuntu system, nice to meet you.

<members>

<member>

           <hostName>10.0.1.196</hostName>

           <port>4100</port>

</member>

<member>

           <hostName>10.0.1.198</hostName>

           <port>4100</port>

</member>

<member>

           <hostName>10.0.1.200</hostName>

           <port>4100</port>

</member>

<member>

           <hostName>10.0.1.202</hostName>

           <port>4100</port>

</member>

</members>

What if you need to parameterize the port value also?
Here is how to do it...

Re-define members as follows in params.pp file (data in hash form [2])
$members = { '192.168.1.156' => '4100',

             '192.168.1.157' => '4000' }

template file (polite-file.erb) members section is to be modified as follows.
<members>
   <%- if @members -%>
   <%- @members.each_pair do |hostname,port| -%>
      <member>
           <hostName><%= hostname %></hostName>
           <port><%= port %></port>
      </member>
   <%- end -%>
   <%- end -%>

</members>

Created new file contents will be as follows.

Hello guys and gals,

I'm welcome file.xml, on a Ubuntu system, nice to meet you.

        <members>
                <member>
                        <hostName>192.168.1.156</hostName>
                        <port>4100</port>
                </member>
                <member>
                        <hostName>192.168.1.157</hostName>
                        <port>4000</port>
                </member>

        </members>

Related stackoverflow questions.

[1]http://stackoverflow.com/questions/27183036/puppet-fill-templates-with-ip-addresses-of-newly-spawned-instances

[2]http://stackoverflow.com/questions/27222189/puppet-iteration-with-2-dimensional-array-to-fill-template

[3]http://stackoverflow.com/questions/27125371/puppet-catalog-run-error-error-400-on-server-not-authorized-to-call-find-on-p

Installing and configuring puppet master and agent

Don't know what puppet is?

Go through my previous blog article: Introduction to Puppet

You can always try with 2 physical computers, one for puppet master and the other for puppet agent.

But why don't you try with VMware or VirtualBox virtual machines? It is easy and convenient. You don't have to install puppet in your host OS. Take two ubuntu images (say ubuntu 12.04) and put up two instances (say 1 GB RAM each). One will be used for puppet master and the other for puppet agent.

This tutorial goes on assuming you have ubuntu instances prepared for master and agent installation.

1. Installing and configuring puppet master

For the ease of tracking your progress, I have simply broken the master installation process into few simple steps.

Steps to be followed are as follows.

Step 1 - Install puppet master

$ sudo apt-get update

$ sudo apt-get install puppetmaster

Step 2 - Configure puppet.conf file

Add following lines to the appropriate sections in puppet.conf file as indicated below.

[main]

dns_alt_names=puppet,puppet.example.com,puppetmaster01,puppetmaster01.example.com

[master]

autosign=true

dns_alt_names are mentioned so that when creating ssl certificates for the master itself, the names will be embedded to the certificate itself which is easy for the agent to find out that agent is connecting to the intended puppet master.

autosign=true is used to automatically sign puppet agent join requests for the time being. So that you can easily learn puppet and later comment out the said line to manually sign agent certificates.

Step 3 - Modify hosts file

Modify the /etc/hosts file content as follows.

127.0.0.1       localhost

127.0.1.1       puppet.example.com

Step 4 - Modify hostname content

puppet.example.com

2. Installing and configuring puppet agent

For the ease of tracking your progress, I have simply broken the agent installation process also as follows.

Step 1 - Install puppet agent

$ sudo apt-get update

$ sudo apt-get install puppet

Step 2 - Configure puppet.conf file

Add following lines to the appropriate sections in puppet.conf file as indicated below.

[main]

server=puppet.example.com

Step 3 - Modify hosts file

Modify the /etc/hosts file content as follows.

127.0.0.1       localhost

127.0.1.1       agent1

192.168.92.2    puppet.example.com puppet

Note that the 192.168.92.2 is the ip assigned for the puppet master.

Step 4 - Modify hostname content

agent 1

First reboot master instance and then reboot agent instance as well.

3. Test master-agent connection by performing a simple catalog run.

Step 1 - Edit site.pp file

Go to master node terminal as root and open /etc/puppet/manifests/site.pp file for editing.

Insert a small code segment as follows.

node default {

}

This code segment does nothing at the moment but will help us in confirming that the connection between master and agent is successfully established.

Step 2 - Perform a catalog run in agent

Now go to the agent node instance terminal as root. 

Perform a catalog run as follows.

root@agent1:~# puppet agent --test

info: Caching catalog for agent1.domain.name

info: Applying configuration version '1416123976'

notice: Finished catalog run in 0.01 seconds

root@agent1:~# 

Congratulations! You have successfully established the master-agent connection.

But things always don't happen as we expect.

When you are trying to install master-agent you may get puppet master-agent communication errors.

See this blog article which may help to solve your common problems: Puppet master-agent communication errors

Want to go further with puppet?

See this blog article on how to create files from templates and change its content on the go: Puppet how to create a file and change its content using templates