Puppet master-agent communication errors

Don't know what puppet is?
Go through my previous blog article: Introduction to Puppet

Setup: two VMs inside Oracle VirtualBox; one for puppet-master and one for puppet-agent.
Puppet master is running on ip 192.168.92.2 and listening to 8140 port.

suhan@suhan-VirtualBox:~$ ps -ef | grep puppetmaster
suhan     2229  1802  0 10:11 pts/2    00:00:00 grep --color=auto puppetmaster
suhan@suhan-VirtualBox:~$ ps -ef | grep puppet
puppet    2224     1  0 10:11 ?        00:00:00 /usr/bin/ruby1.8 /usr/bin/puppet master --masterport=8140
suhan     2233  1802  0 10:11 pts/2    00:00:00 grep --color=auto puppet

suhan@suhan-VirtualBox:~$ netstat -tupln | grep 8140

(Not all processes could be identified, non-owned process info

 will not be shown, you would have to be root to see it all.)

tcp        0      0 0.0.0.0:8140            0.0.0.0:*               LISTEN  

Agent also up and running on ip 192.168.92.3

suhan@suhan-VirtualBox:~$ ps -ef | grep puppet
root       907     1  0 10:07 ?        00:00:00 /usr/bin/ruby1.8 /usr/bin/puppet agent
suhan     2204  1776  0 10:36 pts/0    00:00:00 grep --color=auto puppet

Some how when my agent tried to communicate to master following errors occurred.

Error 1:
suhan@suhan-VirtualBox:~$ sudo puppet agent --test --server=puppet.example.com
err: Could not request certificate: No route to host - connect(2)
Exiting; failed to retrieve certificate and waitforcert is disabled

This was due to master node was not accessible to agent node by the said server identification.
This can be due to reasons like invalid server alias, network inaccessibility.
Even though I have created two host-only adaptors in VirtualBox and attached them to each virtual machine, communication between each machine was not possible.
Therefore I modified each /etc/network/interfaces file in both nodes and created eth0 interface with respective ip addresses.

e.g.: in puppet master,
auto eth0
iface eth0 inet static
address 192.168.92.2
network 192.168.92.0
netmask 255.255.255.0
broadcast 192.168.92.255

Then I configured each machine's virtual box settings for network interfaces as bridged network, bridge0.

Then got the following error since previously generated certificates are mismatched.

Error 2:
suhan@suhan-VirtualBox:~$ sudo puppet agent --test --server=puppet.example.com
info: Caching certificate for ca
info: Caching certificate for suhan-virtualbox
err: Could not request certificate: The certificate retrieved from the master does not match the agent's private key.
Certificate fingerprint: 97:48:44:EC:F7:E0:24:B0:C4:AB:48:F4:5C:F5:26:2D
To fix this, remove the certificate from both the master and the agent and then start a puppet run, which will automatically regenerate a certficate.
On the master:
  puppet cert clean suhan-virtualbox
On the agent:
  rm -f /var/lib/puppet/ssl/certs/suhan-virtualbox.pem
  puppet agent -t

Exiting; failed to retrieve certificate and waitforcert is disabled

As per the instruction given in the error message itself I followed the steps.

On master,
suhan@suhan-VirtualBox:~$ sudo puppet cert clean suhan-virtualbox
[sudo] password for suhan: 
notice: Revoked certificate with serial 3
notice: Removing file Puppet::SSL::Certificate suhan-virtualbox at '/var/lib/puppet/ssl/ca/signed/suhan-virtualbox.pem'
notice: Removing file Puppet::SSL::Certificate suhan-virtualbox at '/var/lib/puppet/ssl/certs/suhan-virtualbox.pem'
notice: Removing file Puppet::SSL::Key suhan-virtualbox at '/var/lib/puppet/ssl/private_keys/suhan-virtualbox.pem'

On agent,
suhan@suhan-VirtualBox:~$ sudo rm -f /var/lib/puppet/ssl/certs/suhan-virtualbox.pem
[sudo] password for suhan: 

suhan@suhan-VirtualBox:~$ puppet agent -t
info: Creating a new SSL key for suhan-virtualbox
info: Caching certificate for ca
info: Creating a new SSL certificate request for suhan-virtualbox
info: Certificate Request fingerprint (md5): B7:AE:21:4C:B8:03:2A:BC:8A:D5:A6:92:4E:A3:F1:84
Exiting; no certificate found and waitforcert is disabled

Issue was resolved. :)

But why wait for master to manually sign the certificate since this is an experimental setting?

Therefore I modified the /etc/puppet/puppet.conf file's [master] section to auto sign certificates for agents as follows,

[master]
# These are needed when the puppetmaster is run by passenger
# and can safely be removed if webrick is used.
# ssl_client_header = SSL_CLIENT_S_DN
# ssl_client_verify_header = SSL_CLIENT_VERIFY
autosign=true

Then the certificate was auto signed upon request by puppet master and can continue to work further.

Error 3:
root@base:~# puppet agent -t
err: Could not retrieve catalog from remote server: Error 400 on SERVER: Could not find default node or by name with 'base.openstacklocal, base' on node base.openstacklocal
warning: Not using cache on failed catalog
err: Could not retrieve catalog; skipping run

This was due to absence of a node definition in puppet master's /etc/puppet/manifests/site.pp for the node who is trying to contact puppet master.

I have added the following node to my site.pp file (hostname of my agent is 'appserver' in /etc/hostname and /etc/hosts).

node 'appserver' {

}

Then the catalog run was run successfully.

root@appserver:~# puppet agent -t

info: Caching catalog for appserver.openstacklocal

info: Applying configuration version '1415768619'

notice: Finished catalog run in 0.01 seconds

Error 4:
root@suhan-keymanager:/home/ubuntu# puppet agent -t
Warning: Unable to fetch my node definition, but the agent run will continue:
Warning: SSL_connect returned=1 errno=0 state=SSLv3 read server certificate B: certificate verify failed: [self signed certificate in certificate chain for /CN=Puppet CA: qaa-puppet-master.openstacklocal]
Info: Retrieving plugin
Error: /File[/var/lib/puppet/lib]: Failed to generate additional resources using 'eval_generate': SSL_connect returned=1 errno=0 state=SSLv3 read server certificate B: certificate verify failed: [self signed certificate in certificate chain for /CN=Puppet CA: qaa-puppet-master.openstacklocal]
Error: /File[/var/lib/puppet/lib]: Could not evaluate: SSL_connect returned=1 errno=0 state=SSLv3 read server certificate B: certificate verify failed: [self signed certificate in certificate chain for /CN=Puppet CA: qaa-puppet-master.openstacklocal] Could not retrieve file metadata for puppet://puppet/plugins: SSL_connect returned=1 errno=0 state=SSLv3 read server certificate B: certificate verify failed: [self signed certificate in certificate chain for /CN=Puppet CA: qaa-puppet-master.openstacklocal]
Error: Could not retrieve catalog from remote server: SSL_connect returned=1 errno=0 state=SSLv3 read server certificate B: certificate verify failed: [self signed certificate in certificate chain for /CN=Puppet CA: qaa-puppet-master.openstacklocal]
Warning: Not using cache on failed catalog
Error: Could not retrieve catalog; skipping run
Error: Could not send report: SSL_connect returned=1 errno=0 state=SSLv3 read server certificate B: certificate verify failed: [self signed certificate in certificate chain for /CN=Puppet CA: qaa-puppet-master.openstacklocal]

This occurred when the puppet servers were changed and the CA certs changed, or when rebuilt a host with the same hostname. Please find the following fix.

find /var/lib/puppet/ssl -name '*.pem' -exec rm {} \;

This should make the agent re-download the puppet CA certificates, and re-apply for its own certificate.

Error 5:

root@qaa-puppet-apim-store-pub-1:/home/ubuntu# puppet agent -t --noop

Warning: Unable to fetch my node definition, but the agent run will continue:

Warning: SSL_connect returned=1 errno=0 state=SSLv3 read server session ticket A: sslv3 alert certificate revoked

Info: Retrieving pluginfacts

Error: /File[/var/lib/puppet/facts.d]: Failed to generate additional resources using 'eval_generate': SSL_connect returned=1 errno=0 state=SSLv3 read server session ticket A: sslv3 alert certificate revoked

Error: /File[/var/lib/puppet/facts.d]: Could not evaluate: Could not retrieve file metadata for puppet://puppet/pluginfacts: SSL_connect returned=1 errno=0 state=SSLv3 read server session ticket A: sslv3 alert certificate revoked

Info: Retrieving plugin

Error: /File[/var/lib/puppet/lib]: Failed to generate additional resources using 'eval_generate': SSL_connect returned=1 errno=0 state=SSLv3 read server session ticket A: sslv3 alert certificate revoked

Error: /File[/var/lib/puppet/lib]: Could not evaluate: Could not retrieve file metadata for puppet://puppet/plugins: SSL_connect returned=1 errno=0 state=SSLv3 read server session ticket A: sslv3 alert certificate revoked

Info: Loading facts

Error: Could not retrieve catalog from remote server: SSL_connect returned=1 errno=0 state=SSLv3 read server session ticket A: sslv3 alert certificate revoked

Warning: Not using cache on failed catalog

Error: Could not retrieve catalog; skipping run

Error: Could not send report: SSL_connect returned=1 errno=0 state=SSLv3 read server session ticket A: sslv3 alert certificate revoked

This occurs when the puppet agent certificate is revoked from puppet master.

To clean puppet agent certificate in puppet agent and regenerate, issue the following command.

root@qaa-puppet-apim-store-pub-1:/home/ubuntu# puppet cert clean

Error: The certificate retrieved from the master does not match the agent's private key.

Certificate fingerprint: A9:6C:3E:2E:99:03:9F:95:9C:A8:94:BF:E9:75:C3:7D:EB:67:6A:85:09:32:A3:0B:D5:96:79:AF:9F:0D:93:EB

To fix this, remove the certificate from both the master and the agent and then start a puppet run, which will automatically regenerate a certficate.

On the master:

  puppet cert clean apim-publisher-1

On the agent:

  1a. On most platforms: find /var/lib/puppet/ssl -name apim-publisher-1.pem -delete

  1b. On Windows: del "/var/lib/puppet/ssl/apim-publisher-1.pem" /f

  2. puppet agent -t

Instructions will be given in the error itself.

As an example if you are using Ubuntu 14.04:

In puppet master,

> puppet cert clean apim-publisher-1

In puppet agent,

> find /var/lib/puppet/ssl -name apim-publisher-1.pem -delete

> puppet agent -t --noop

OR

> find /var/lib/puppet/ssl -name qaa-puppet-apim-key-manager-1.openstacklocal.pem -delete

> puppet agent -t --noop

root@qaa-puppet-apim-store-pub-1:/home/ubuntu# puppet agent -t --noop

Info: Creating a new SSL key for apim-publisher-1

Info: csr_attributes file loading from /etc/puppet/csr_attributes.yaml

Info: Creating a new SSL certificate request for apim-publisher-1

Info: Certificate Request fingerprint (SHA256): 09:FA:92:18:0E:C7:31:9B:7E:F7:D6:31:78:63:85:52:DF:4F:A7:7D:B8:D6:B1:A2:B7:23:66:63:07:C2:21:F7

Info: Caching certificate for apim-publisher-1

Info: Caching certificate for apim-publisher-1

Info: Retrieving pluginfacts

Info: Retrieving plugin

Info: Loading facts

...

Enough with communication errors... :)

How about for a change we learn how to create files from templates and change its content on the go. Believe me its fun! Go through my blog article:Puppet how to create a file and change its content using templates

Introduction to Puppet

What is puppet?
Puppet is an open-source IT automation tool [1].
"The Puppet Domain Specific Language (DSL) is a Ruby-based coding language that provides a precise and adaptable way to describe a desired state for each machine in your infrastructure. Once you've described a desired state, Puppet does the work to bring your systems in line and keeping them there" - Puppet Labs

Why not write a simple shell script instead of using puppet?
For small number of nodes and simple shell commands, writing shell scripts may appear appealing.
But what if you are having large number of nodes/machines or having different operating systems?
How you are going to manage all of your large number of scripts?

This is where Puppet comes in to play.
Puppet allows you to define details of a configuration in a way that takes off the burden of operating system specifics.
It lets you easily maintain, audit and understand your whole IT infrastructure than having large number of complicated shell scripts.

How to check puppet version?
>puppet -V
Note that it must be the capital letter 'V'.

You will get a response as below if you have installed puppet in your system.
>3.6.2 (Puppet Enterprise 3.3.2)

Puppet not installed?

Manually install and configure puppet master and agent
Read this blog article: Installing and configuring puppet master and agent

Lazy to install puppet?
How about a preconfigured VM image to learn puppet :)
Given that you have VMware or Oracle VirtualBox installed on your system,
you can follow the link [1] and download a preconfigured VM to follow the puppet tutorial from puppet labs.

When you are trying to install master-agent you may get puppet master-agent communication errors.
See this blog article which may help to solve your common problems: Puppet master-agent communication errors

Want to go further with puppet?
See this blog article on how to create files from templates and change its content on the go: Puppet how to create a file and change its content using templates

Reference:
[1] http://puppetlabs.com/download-learning-vm

Building your own Python docker cartridge for Apache Stratos

How about building your own python-cartridge?

It’s easy, lets follow the steps below.

1. Download github python-cartridge file contents - Download.

1. Copy the github files to a shared location where your host machine and the VM can access. [e.g.: mine: users/suhanr/Documents in VirtualBox Settings -> Shared Folders]

This will be easy when you want to edit configuration files and modify file contents.

Important files and folders.

• Dockerfile - Facilitate automated docker image building process.
• packs folder contains our stratos cartridge agent, ActiveMQ libraries required and java 1.7 package.
• init.sh
• run_scripts.sh

2. Open a terminal session in the VM and find the shared folder as follows. Go inside that folder.

Say for example your shared folder name is like “VMShared”, then it will appear inside /media/ folder as sf_VMShared.

3. Simply issue the docker build command to start Docker image automated building process.

docker build -t python-cartridge .

When the build is successful the output will be similar as shown in the below image.

4. Now tag your root repository as follows

docker tag python-cartridge 192.168.57.30:5042/python-cartridge

If you try to push the root repository to docker you will get the below error.

Impossible to push a "root" repository. Please rename your repository in <user>/<repo> (ex: <user>/python-cartridge)

5. It is time to push your newly built docker image with the newly created tag to the glance repository

docker push 192.168.57.30:5042/python-cartridge

When the image is successfully pushed you will get an output similar to this.

6. Now go to glance repository and get the image id of your newly pushed docker image. [This will be needed when you configure your newly built cartridge in Apache Stratos]

> cd /home/stratos/workspace/devstack/

> . openrc admin admin

> glance image-list

sample image ID: 7f179943-0b6b-4d54-af46-c334f65b76f3

7. Now it is time to configure your newly built cartridge in Apache Stratos

You can find the python configuration files here - Download

Open your web browser in VM and go to the following link.

https://dev.stratos.org:9445/console/

Since partition and auto scale policies are already defined configure Deployment policy and cartridge configuration.

Sample Deployment policy:

{

   "id": "python-openstack",

   "partitionGroup": {

       "id": "openstack",

       "partitionAlgo": "one-after-another",

       "partition": [

       {

           "id": "P1",

           "partitionMax": "2",

           "partitionMin": "1"

       }

       ]

   }

}

Sample Cartridge configuration:

{

     "type": "python",

     "provider": "python",

     "host": "dev.stratos.org",

     "defaultAutoscalingPolicy": "economyPolicy",

     "defaultDeploymentPolicy": "python-openstack",

     "displayName": "Python",

     "description": "Python Cartridge",

     "version": "2.7",

     "multiTenant": "false",

     "portMapping": [

        {

           "protocol": "http",

           "port": "80"

        }

      ],

     "deployment": {},

      "iaasProvider": [

       {

         "type": "openstack",

         "imageId": "RegionOne/7f179943-0b6b-4d54-af46-c334f65b76f3",

         "maxInstanceLimit": "1",

         "property": [

           {

            "name": "instanceType",

            "value": "RegionOne/2"

           },

    {

           "name": "keyPair",

           "value": "demo"

           },

  {

           "name": "securityGroups",

           "value": "open-all"

          }

         ]

       }

     ],

  "property":[

       {

           "name":"payload_parameter.MB_IP",

           "value":"192.168.57.30"

       },

       {

           "name":"payload_parameter.MB_PORT",

           "value":"61616"

       },

       {

           "name":"payload_parameter.CEP_IP",

           "value":"172.17.42.1"

       },

       {

           "name":"payload_parameter.CEP_PORT",

           "value":"7613"

       }

   ]

}

Stratos terminal output will be as follows.

8. Now subscribe to your newly built cartridge

Once you subscribe to the cartridge it will take some time to activate the instance.

You can find the python application on docker instances public ip address, in this scenario 192.168.92.1

When using python pip: read operation timeouts and distributions not found

What is Python pip?
Python pip is a package management system used in installing and managing python packages [1].

Why did I use pip?
In my Python docker cartridge for Apache Stratos project, I have created some python web applications and hosted them in github [suhan at github].
I installed the following dependencies prior using pip in my ubuntu 12.04.
apt-get install -y build-essential python-dev python-pip python-virtualenv libpq-dev

Prior running the python web applications, I have to install their python dependencies. Usually I put them inside a requirements.txt file as follows,
flask
cherrypy

Then I try to install the requirements via pip as follows,
pip install -r requirements.txt

Errors
Somewhere in the middle during installation I get these errors.
---- Installing git python web application project requirements ----
Downloading/unpacking flask
  Running setup.py egg_info for package flask
    
    warning: no files found matching '*' under directory 'tests'
    warning: no previously-included files matching '*.pyc' found under directory 'docs'
    warning: no previously-included files matching '*.pyo' found under directory 'docs'
    warning: no previously-included files matching '*.pyc' found under directory 'tests'
    warning: no previously-included files matching '*.pyo' found under directory 'tests'
    warning: no previously-included files matching '*.pyc' found under directory 'examples'
    warning: no previously-included files matching '*.pyo' found under directory 'examples'
    no previously-included directories found matching 'docs/_build'
    no previously-included directories found matching 'docs/_themes/.git'
Downloading/unpacking cherrypy
  Running setup.py egg_info for package cherrypy
    
Downloading/unpacking Werkzeug>=0.7 (from flask)
  Running setup.py egg_info for package Werkzeug
    
    warning: no files found matching '*' under directory 'werkzeug/debug/templates'
    warning: no files found matching '*' under directory 'tests'
    warning: no previously-included files matching '*.pyc' found under directory 'docs'
    warning: no previously-included files matching '*.pyo' found under directory 'docs'
    warning: no previously-included files matching '*.pyc' found under directory 'tests'
    warning: no previously-included files matching '*.pyo' found under directory 'tests'
    warning: no previously-included files matching '*.pyc' found under directory 'examples'
    warning: no previously-included files matching '*.pyo' found under directory 'examples'
    no previously-included directories found matching 'docs/_build'
Downloading/unpacking Jinja2>=2.4 (from flask)
  Running setup.py egg_info for package Jinja2
    
    warning: no files found matching '*' under directory 'custom_fixers'
    warning: no previously-included files matching '*' found under directory 'docs/_build'
    warning: no previously-included files matching '*.pyc' found under directory 'jinja2'
    warning: no previously-included files matching '*.pyc' found under directory 'docs'
    warning: no previously-included files matching '*.pyo' found under directory 'jinja2'
    warning: no previously-included files matching '*.pyo' found under directory 'docs'
Downloading/unpacking itsdangerous>=0.21 (from flask)
  Could not find any downloads that satisfy the requirement itsdangerous>=0.21 (from flask)
No distributions at all found for itsdangerous>=0.21 (from flask)
Storing complete log in /root/.pip/pip.log

CherryPy server fails to start since dependencies are not properly installed.

---- Starting CherryPy Server ----

Traceback (most recent call last):

 File "/var/www/www/server.py", line 5, in <module>

   from app import app

 File "/var/www/www/app.py", line 1, in <module>

   from flask import Flask

ImportError: No module named flask

Sometimes I get the following error.

Exception:
Traceback (most recent call last):
  File "/vagrant/venv/local/lib/python2.7/site-packages/pip/basecommand.py", line 134, in main
    status = self.run(options, args)
  File "/vagrant/venv/local/lib/python2.7/site-packages/pip/commands/install.py", line 236, in run
    requirement_set.prepare_files(finder, force_root_egg_info=self.bundle, bundle=self.bundle)
  File "/vagrant/venv/local/lib/python2.7/site-packages/pip/req.py", line 1092, in prepare_files
    self.unpack_url(url, location, self.is_download)
  File "/vagrant/venv/local/lib/python2.7/site-packages/pip/req.py", line 1238, in unpack_url
    retval = unpack_http_url(link, location, self.download_cache, self.download_dir)
  File "/vagrant/venv/local/lib/python2.7/site-packages/pip/download.py", line 622, in unpack_http_url
    download_hash = _download_url(resp, link, temp_location)
  File "/vagrant/venv/local/lib/python2.7/site-packages/pip/download.py", line 495, in _download_url
    chunk = resp.read(4096)
  File "/usr/lib/python2.7/socket.py", line 380, in read
    data = self._sock.recv(left)
  File "/usr/lib/python2.7/httplib.py", line 561, in read
    s = self.fp.read(amt)
  File "/usr/lib/python2.7/socket.py", line 380, in read
    data = self._sock.recv(left)
  File "/usr/lib/python2.7/ssl.py", line 241, in recv
    return self.read(buflen)
  File "/usr/lib/python2.7/ssl.py", line 160, in read
    return self._sslobj.read(len)
SSLError: The read operation timed out

These errors are due to internet connectivity problems, slow internet speed, etc...
Since I'm using a VM to build my docker images these issues/errors are further amplified.


As a temporary solution I did retry several times and finally it was successfully installed.
retry: pip install -r requirements.txt

Then my CherryPy server started successfully.
---- Starting CherryPy Server ----
ENGINE Bus STARTING
ENGINE Started monitor thread 'Autoreloader'.
ENGINE Started monitor thread '_TimeoutMonitor'.
ENGINE Serving on http://0.0.0.0
ENGINE Bus STARTED

However according to stackoverflow [2] if you can use the following command the problem will be resolved.
pip --default-timeout=100 install django

Reference:
[1]http://en.wikipedia.org/wiki/Pip_(package_manager)
[2]http://stackoverflow.com/questions/18958508/sslerror-the-read-operation-timed-out-when-using-pip/19334551#19334551