Monday, October 27, 2014

Puppet master-agent communication errors

Don't know what puppet is?
Go through my previous blog article: Introduction to Puppet

Setup: two VMs inside Oracle VirtualBox; one for puppet-master and one for puppet-agent.
Puppet master is running on ip 192.168.92.2 and listening to 8140 port.

suhan@suhan-VirtualBox:~$ ps -ef | grep puppetmaster
suhan     2229  1802  0 10:11 pts/2    00:00:00 grep --color=auto puppetmaster
suhan@suhan-VirtualBox:~$ ps -ef | grep puppet
puppet    2224     1  0 10:11 ?        00:00:00 /usr/bin/ruby1.8 /usr/bin/puppet master --masterport=8140
suhan     2233  1802  0 10:11 pts/2    00:00:00 grep --color=auto puppet

suhan@suhan-VirtualBox:~$ netstat -tupln | grep 8140
(Not all processes could be identified, non-owned process info
 will not be shown, you would have to be root to see it all.)
tcp        0      0 0.0.0.0:8140            0.0.0.0:*               LISTEN  

Agent also up and running on ip 192.168.92.3

suhan@suhan-VirtualBox:~$ ps -ef | grep puppet
root       907     1  0 10:07 ?        00:00:00 /usr/bin/ruby1.8 /usr/bin/puppet agent
suhan     2204  1776  0 10:36 pts/0    00:00:00 grep --color=auto puppet

Some how when my agent tried to communicate to master following errors occurred.

Error 1:
suhan@suhan-VirtualBox:~$ sudo puppet agent --test --server=puppet.example.com
err: Could not request certificate: No route to host - connect(2)
Exiting; failed to retrieve certificate and waitforcert is disabled

This was due to master node was not accessible to agent node by the said server identification.
This can be due to reasons like invalid server alias, network inaccessibility.
Even though I have created two host-only adaptors in VirtualBox and attached them to each virtual machine, communication between each machine was not possible.
Therefore I modified each /etc/network/interfaces file in both nodes and created eth0 interface with respective ip addresses.

e.g.: in puppet master,
auto eth0
iface eth0 inet static
address 192.168.92.2
network 192.168.92.0
netmask 255.255.255.0
broadcast 192.168.92.255

Then I configured each machine's virtual box settings for network interfaces as bridged network, bridge0.

Then got the following error since previously generated certificates are mismatched.

Error 2:
suhan@suhan-VirtualBox:~$ sudo puppet agent --test --server=puppet.example.com
info: Caching certificate for ca
info: Caching certificate for suhan-virtualbox
err: Could not request certificate: The certificate retrieved from the master does not match the agent's private key.
Certificate fingerprint: 97:48:44:EC:F7:E0:24:B0:C4:AB:48:F4:5C:F5:26:2D
To fix this, remove the certificate from both the master and the agent and then start a puppet run, which will automatically regenerate a certficate.
On the master:
  puppet cert clean suhan-virtualbox
On the agent:
  rm -f /var/lib/puppet/ssl/certs/suhan-virtualbox.pem
  puppet agent -t

Exiting; failed to retrieve certificate and waitforcert is disabled

As per the instruction given in the error message itself I followed the steps.

On master,
suhan@suhan-VirtualBox:~$ sudo puppet cert clean suhan-virtualbox
[sudo] password for suhan: 
notice: Revoked certificate with serial 3
notice: Removing file Puppet::SSL::Certificate suhan-virtualbox at '/var/lib/puppet/ssl/ca/signed/suhan-virtualbox.pem'
notice: Removing file Puppet::SSL::Certificate suhan-virtualbox at '/var/lib/puppet/ssl/certs/suhan-virtualbox.pem'
notice: Removing file Puppet::SSL::Key suhan-virtualbox at '/var/lib/puppet/ssl/private_keys/suhan-virtualbox.pem'

On agent,
suhan@suhan-VirtualBox:~$ sudo rm -f /var/lib/puppet/ssl/certs/suhan-virtualbox.pem
[sudo] password for suhan: 

suhan@suhan-VirtualBox:~$ puppet agent -t
info: Creating a new SSL key for suhan-virtualbox
info: Caching certificate for ca
info: Creating a new SSL certificate request for suhan-virtualbox
info: Certificate Request fingerprint (md5): B7:AE:21:4C:B8:03:2A:BC:8A:D5:A6:92:4E:A3:F1:84
Exiting; no certificate found and waitforcert is disabled

Issue was resolved. :)

But why wait for master to manually sign the certificate since this is an experimental setting?

Therefore I modified the /etc/puppet/puppet.conf file's [master] section to auto sign certificates for agents as follows,

[master]
# These are needed when the puppetmaster is run by passenger
# and can safely be removed if webrick is used.
# ssl_client_header = SSL_CLIENT_S_DN
# ssl_client_verify_header = SSL_CLIENT_VERIFY
autosign=true

Then the certificate was auto signed upon request by puppet master and can continue to work further.

Error 3:
root@base:~# puppet agent -t
err: Could not retrieve catalog from remote server: Error 400 on SERVER: Could not find default node or by name with 'base.openstacklocal, base' on node base.openstacklocal
warning: Not using cache on failed catalog
err: Could not retrieve catalog; skipping run

This was due to absence of a node definition in puppet master's /etc/puppet/manifests/site.pp for the node who is trying to contact puppet master.

I have added the following node to my site.pp file (hostname of my agent is 'appserver' in /etc/hostname and /etc/hosts).

node 'appserver' {
}

Then the catalog run was run successfully.

root@appserver:~# puppet agent -t
info: Caching catalog for appserver.openstacklocal
info: Applying configuration version '1415768619'
notice: Finished catalog run in 0.01 seconds

Error 4:
root@suhan-keymanager:/home/ubuntu# puppet agent -t
Warning: Unable to fetch my node definition, but the agent run will continue:
Warning: SSL_connect returned=1 errno=0 state=SSLv3 read server certificate B: certificate verify failed: [self signed certificate in certificate chain for /CN=Puppet CA: qaa-puppet-master.openstacklocal]
Info: Retrieving plugin
Error: /File[/var/lib/puppet/lib]: Failed to generate additional resources using 'eval_generate': SSL_connect returned=1 errno=0 state=SSLv3 read server certificate B: certificate verify failed: [self signed certificate in certificate chain for /CN=Puppet CA: qaa-puppet-master.openstacklocal]
Error: /File[/var/lib/puppet/lib]: Could not evaluate: SSL_connect returned=1 errno=0 state=SSLv3 read server certificate B: certificate verify failed: [self signed certificate in certificate chain for /CN=Puppet CA: qaa-puppet-master.openstacklocal] Could not retrieve file metadata for puppet://puppet/plugins: SSL_connect returned=1 errno=0 state=SSLv3 read server certificate B: certificate verify failed: [self signed certificate in certificate chain for /CN=Puppet CA: qaa-puppet-master.openstacklocal]
Error: Could not retrieve catalog from remote server: SSL_connect returned=1 errno=0 state=SSLv3 read server certificate B: certificate verify failed: [self signed certificate in certificate chain for /CN=Puppet CA: qaa-puppet-master.openstacklocal]
Warning: Not using cache on failed catalog
Error: Could not retrieve catalog; skipping run
Error: Could not send report: SSL_connect returned=1 errno=0 state=SSLv3 read server certificate B: certificate verify failed: [self signed certificate in certificate chain for /CN=Puppet CA: qaa-puppet-master.openstacklocal]

This occurred when the puppet servers were changed and the CA certs changed, or when rebuilt a host with the same hostname. Please find the following fix.


find /var/lib/puppet/ssl -name '*.pem' -exec rm {} \;

This should make the agent re-download the puppet CA certificates, and re-apply for its own certificate.

Error 5:
root@qaa-puppet-apim-store-pub-1:/home/ubuntu# puppet agent -t --noop
Warning: Unable to fetch my node definition, but the agent run will continue:
Warning: SSL_connect returned=1 errno=0 state=SSLv3 read server session ticket A: sslv3 alert certificate revoked
Info: Retrieving pluginfacts
Error: /File[/var/lib/puppet/facts.d]: Failed to generate additional resources using 'eval_generate': SSL_connect returned=1 errno=0 state=SSLv3 read server session ticket A: sslv3 alert certificate revoked
Error: /File[/var/lib/puppet/facts.d]: Could not evaluate: Could not retrieve file metadata for puppet://puppet/pluginfacts: SSL_connect returned=1 errno=0 state=SSLv3 read server session ticket A: sslv3 alert certificate revoked
Info: Retrieving plugin
Error: /File[/var/lib/puppet/lib]: Failed to generate additional resources using 'eval_generate': SSL_connect returned=1 errno=0 state=SSLv3 read server session ticket A: sslv3 alert certificate revoked
Error: /File[/var/lib/puppet/lib]: Could not evaluate: Could not retrieve file metadata for puppet://puppet/plugins: SSL_connect returned=1 errno=0 state=SSLv3 read server session ticket A: sslv3 alert certificate revoked
Info: Loading facts
Error: Could not retrieve catalog from remote server: SSL_connect returned=1 errno=0 state=SSLv3 read server session ticket A: sslv3 alert certificate revoked
Warning: Not using cache on failed catalog
Error: Could not retrieve catalog; skipping run

Error: Could not send report: SSL_connect returned=1 errno=0 state=SSLv3 read server session ticket A: sslv3 alert certificate revoked

This occurs when the puppet agent certificate is revoked from puppet master.
To clean puppet agent certificate in puppet agent and regenerate, issue the following command.

root@qaa-puppet-apim-store-pub-1:/home/ubuntu# puppet cert clean
Error: The certificate retrieved from the master does not match the agent's private key.
Certificate fingerprint: A9:6C:3E:2E:99:03:9F:95:9C:A8:94:BF:E9:75:C3:7D:EB:67:6A:85:09:32:A3:0B:D5:96:79:AF:9F:0D:93:EB
To fix this, remove the certificate from both the master and the agent and then start a puppet run, which will automatically regenerate a certficate.
On the master:
  puppet cert clean apim-publisher-1
On the agent:
  1a. On most platforms: find /var/lib/puppet/ssl -name apim-publisher-1.pem -delete
  1b. On Windows: del "/var/lib/puppet/ssl/apim-publisher-1.pem" /f
  2. puppet agent -t

Instructions will be given in the error itself.
As an example if you are using Ubuntu 14.04:

In puppet master,
puppet cert clean apim-publisher-1

In puppet agent,
> find /var/lib/puppet/ssl -name apim-publisher-1.pem -delete
> puppet agent -t --noop

OR


> find /var/lib/puppet/ssl -name qaa-puppet-apim-key-manager-1.openstacklocal.pem -delete
> puppet agent -t --noop

root@qaa-puppet-apim-store-pub-1:/home/ubuntu# puppet agent -t --noop
Info: Creating a new SSL key for apim-publisher-1
Info: csr_attributes file loading from /etc/puppet/csr_attributes.yaml
Info: Creating a new SSL certificate request for apim-publisher-1
Info: Certificate Request fingerprint (SHA256): 09:FA:92:18:0E:C7:31:9B:7E:F7:D6:31:78:63:85:52:DF:4F:A7:7D:B8:D6:B1:A2:B7:23:66:63:07:C2:21:F7
Info: Caching certificate for apim-publisher-1
Info: Caching certificate for apim-publisher-1
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Loading facts
...

Enough with communication errors... :)

How about for a change we learn how to create files from templates and change its content on the go. Believe me its fun! Go through my blog article:Puppet how to create a file and change its content using templates