Python load configurations from a file

Say you have a file deployment.cfg with configurations which are nicely grouped in sections.

[environment]

env = openstack

[envconfig]

image = suhan-daf-agentv4-ubuntu14.04

flavor = m2.small

network = qaa-net

instancePassword = suhan

keyPair = mycloudkey

[nodes]

appserver-mgr2

appserver-wkr2

This is how you should read this configuration file in python.

import ConfigParser
import collections

# Global variables
config = ConfigParser.RawConfigParser(allow_no_value=True)

# Get environment
def get_environment():
config.read('deployment.cfg')
return config.get('environment', 'env')

# Load environment configuration
def get_openstack_image():
config.read('deployment.cfg')
return config.get('envconfig', 'image')

.
.
.

# Nova credentials are loaded from OS environmental variables
def load_server_config():
serverList = []
config.read('deployment.cfg')
orderedDic = collections.OrderedDict(config.items('nodes'))
for node, ip in orderedDic.iteritems():
    serverList.append(node)
print serverList
return serverList

Reference:
[1] http://stackoverflow.com/questions/27483096/python-config-parser-array-sort-according-to-file-content-sequence

Puppet fill template using a central configuration file inside a module

Sometimes we need to keep all the configuration information in a central configuration file. It is easy to maintain and gives a clear idea of the scenario.

This article provides guidance on how to keep this kind of a central file and fill template values during a puppet catalog run in agent node.

Arrays of data which will be looped inside template filling process is also demonstrated in this example.

In the current tested scenario, the hostname of the puppet agent node is appserver-wkr

Configuration file is params.pp which resides inside module's manifests folder.

This file is inherited by init.pp of the module which is having a class with the same name as the module's (appserver).

In order to push a configuration in init.pp file we simply include the module in site.pp folder.

When a puppet agent find the master and requests for a catalog run, master will map the hostname with the relevant node configuration in site.pp file and apply it to the agent node (if no match found apply the default node configuration).

To have more clear idea, see the graphical illustration below followed by code content of each relevant file.

Following are the /etc/puppet tree structure in puppet master node and its file contents.

├── manifests

│   └── site.pp

└── modules

   └── appserver

       ├── manifests

       │   ├── init.pp

       │   └── params.pp

       └── templates

           └── polite-file.erb

1. site.pp

import 'appserver'

node 'appserver-wkr' {

       include appserver

}

node default {

}

2. init.pp

class appserver inherits appserver::params{

       file {  "/tmp/$myname":

               ensure  => file,

               content => template('appserver/polite-file.erb'),

       }

}

3. params.pp

class appserver::params ($say_hello_to = 'guys and gals') {

       #$say_hello_to = 'guys and gals'

       $myname = 'welcome file.xml'

       $serverOptions     = '-DworkerNode=true'

       $subDomain         = 'worker'

       $members           = ['10.0.1.196', '10.0.1.198', '10.0.1.200', '10.0.1.202']

}

4. polite-file.erb

<% if @say_hello_to -%>

Hello <%= @say_hello_to %>,

<% end -%>

I'm <%= @myname %>, on a <%= @operatingsystem %> system, nice to meet you.

<members>

         <%- if @members -%>

# loop hostnames

         <%- @members.each do |hostname| -%>

         <member>

          <hostName><%= hostname %></hostName>

            <port>4100</port>

         </member>

         <%- end -%>

         <%- end -%>

</members>

When all the files are in place simple perform a puppet catalog run in the agent node.

> puppet agent -t

Following is the created file (i.e. /tmp/welcome file.xml) contents in agent node.

Hello guys and gals,

I'm welcome file.xml, on a Ubuntu system, nice to meet you.

<members>

<member>

           <hostName>10.0.1.196</hostName>

           <port>4100</port>

</member>

<member>

           <hostName>10.0.1.198</hostName>

           <port>4100</port>

</member>

<member>

           <hostName>10.0.1.200</hostName>

           <port>4100</port>

</member>

<member>

           <hostName>10.0.1.202</hostName>

           <port>4100</port>

</member>

</members>

What if you need to parameterize the port value also?
Here is how to do it...

Re-define members as follows in params.pp file (data in hash form [2])
$members = { '192.168.1.156' => '4100',

             '192.168.1.157' => '4000' }

template file (polite-file.erb) members section is to be modified as follows.
<members>
   <%- if @members -%>
   <%- @members.each_pair do |hostname,port| -%>
      <member>
           <hostName><%= hostname %></hostName>
           <port><%= port %></port>
      </member>
   <%- end -%>
   <%- end -%>

</members>

Created new file contents will be as follows.

Hello guys and gals,

I'm welcome file.xml, on a Ubuntu system, nice to meet you.

        <members>
                <member>
                        <hostName>192.168.1.156</hostName>
                        <port>4100</port>
                </member>
                <member>
                        <hostName>192.168.1.157</hostName>
                        <port>4000</port>
                </member>

        </members>

Related stackoverflow questions.

[1]http://stackoverflow.com/questions/27183036/puppet-fill-templates-with-ip-addresses-of-newly-spawned-instances

[2]http://stackoverflow.com/questions/27222189/puppet-iteration-with-2-dimensional-array-to-fill-template

[3]http://stackoverflow.com/questions/27125371/puppet-catalog-run-error-error-400-on-server-not-authorized-to-call-find-on-p

Installing and configuring puppet master and agent

Don't know what puppet is?

Go through my previous blog article: Introduction to Puppet

You can always try with 2 physical computers, one for puppet master and the other for puppet agent.

But why don't you try with VMware or VirtualBox virtual machines? It is easy and convenient. You don't have to install puppet in your host OS. Take two ubuntu images (say ubuntu 12.04) and put up two instances (say 1 GB RAM each). One will be used for puppet master and the other for puppet agent.

This tutorial goes on assuming you have ubuntu instances prepared for master and agent installation.

1. Installing and configuring puppet master

For the ease of tracking your progress, I have simply broken the master installation process into few simple steps.

Steps to be followed are as follows.

Step 1 - Install puppet master

$ sudo apt-get update

$ sudo apt-get install puppetmaster

Step 2 - Configure puppet.conf file

Add following lines to the appropriate sections in puppet.conf file as indicated below.

[main]

dns_alt_names=puppet,puppet.example.com,puppetmaster01,puppetmaster01.example.com

[master]

autosign=true

dns_alt_names are mentioned so that when creating ssl certificates for the master itself, the names will be embedded to the certificate itself which is easy for the agent to find out that agent is connecting to the intended puppet master.

autosign=true is used to automatically sign puppet agent join requests for the time being. So that you can easily learn puppet and later comment out the said line to manually sign agent certificates.

Step 3 - Modify hosts file

Modify the /etc/hosts file content as follows.

127.0.0.1       localhost

127.0.1.1       puppet.example.com

Step 4 - Modify hostname content

puppet.example.com

2. Installing and configuring puppet agent

For the ease of tracking your progress, I have simply broken the agent installation process also as follows.

Step 1 - Install puppet agent

$ sudo apt-get update

$ sudo apt-get install puppet

Step 2 - Configure puppet.conf file

Add following lines to the appropriate sections in puppet.conf file as indicated below.

[main]

server=puppet.example.com

Step 3 - Modify hosts file

Modify the /etc/hosts file content as follows.

127.0.0.1       localhost

127.0.1.1       agent1

192.168.92.2    puppet.example.com puppet

Note that the 192.168.92.2 is the ip assigned for the puppet master.

Step 4 - Modify hostname content

agent 1

First reboot master instance and then reboot agent instance as well.

3. Test master-agent connection by performing a simple catalog run.

Step 1 - Edit site.pp file

Go to master node terminal as root and open /etc/puppet/manifests/site.pp file for editing.

Insert a small code segment as follows.

node default {

}

This code segment does nothing at the moment but will help us in confirming that the connection between master and agent is successfully established.

Step 2 - Perform a catalog run in agent

Now go to the agent node instance terminal as root. 

Perform a catalog run as follows.

root@agent1:~# puppet agent --test

info: Caching catalog for agent1.domain.name

info: Applying configuration version '1416123976'

notice: Finished catalog run in 0.01 seconds

root@agent1:~# 

Congratulations! You have successfully established the master-agent connection.

But things always don't happen as we expect.

When you are trying to install master-agent you may get puppet master-agent communication errors.

See this blog article which may help to solve your common problems: Puppet master-agent communication errors

Want to go further with puppet?

See this blog article on how to create files from templates and change its content on the go: Puppet how to create a file and change its content using templates

NovaClient and Floating IP related errors

Error 1:
ERROR (NotFound): FloatingIpPoolNotFound: Floating ip pool not found. (HTTP 404)

root@puppet:/opt/python-script# nova floating-ip-create
ERROR (NotFound): FloatingIpPoolNotFound: Floating ip pool not found. (HTTP 404) (Request-ID: req-11559652-fb3d-40a2-b260-34a2af1eb9a6)

This error occur when the default nova floating ip pool is not present.

Default floating ip pool is 'nova'.

In my case it was mentioned as 'external'

After specifying the external pool in floating ip create command, ip was successfully created.

root@puppet:/opt/python-script# nova floating-ip-create external
+----------------+-----------+----------+----------+
| Ip             | Server Id | Fixed Ip | Pool     |
+----------------+-----------+----------+----------+
| 192.168.17.156 | -         | -        | external |
+----------------+-----------+----------+----------+

Error 2:
No nw_info cache associated with instance (HTTP 400) (Request-ID: req-ba8c1adc-cc41-4592-8ac2-9d01cd66f6e8)

When trying to add a floating ip address (which is already created) to a newly spawned instance in OpenStack cloud, I was getting the following error.

<FloatingIP fixed_ip=None, id=7e438ee9-4e5e-4b6d-b43b-246d18039890, instance_id=None, ip=192.168.17.159, pool=external>
exception is  No nw_info cache associated with instance (HTTP 400) (Request-ID: req-ba8c1adc-cc41-4592-8ac2-9d01cd66f6e8)

This was due to instance was not properly started its networking operations and we were trying to assign the floating ip address to it.

After introducing a small delay between spawning the instance and assigning the newly created floating ip address to it, the operation was successful.

root@puppet:/opt/python-script# python test.py 

Generating nova credentials...

Retrieving image, flavor and network information...

<Image: suhan-daf-agent-ubuntu12.04>

<Network: network-routable>

7bb05f0b-44b4-4c6d-a36f-7194dc43bd42

ACTIVE

<FloatingIP fixed_ip=None, id=d7938b77-ead2-40ce-b992-65770544304b, instance_id=None, ip=192.168.17.162, pool=external>

{u'network-routable': [{u'OS-EXT-IPS-MAC:mac_addr': u'fa:16:3e:74:b4:80', u'version': 4, u'addr': u'192.168.30.174', u'OS-EXT-IPS:type': u'fixed'}]}

192.168.30.174

940c3175-f066-4062-ba77-9fa3b74904c4

ACTIVE

Puppet - How to create a file and change its contents using templates

Don't know what puppet is?
Go through my previous blog article: Introduction to Puppet

If you come across difficulties in establishing connections between puppet master and agent, go through this blog article: Puppet master-agent communication errors
It may help to solve your common communication problems.

In this quick tutorial you can create a file in puppet agent using a predefined template and modify its contents.

Before starting this tutorial you need to have two instances of puppet master and puppet agent separately installed and configured (say 1 GB RAM, ubuntu VM instances).
Say puppet agent's hostname is 'appserver-wkr'.

You can follow these 3 simple quick steps.
1. Create a template file
2. Configure node definition in site.pp or class file for the agent node
3. Perform a catalog run in puppet agent

Step 1: Creating the template file.
In puppet master create the template file as follows.
Location: /etc/puppet/modules/appserver/templates/
Filename: welcome-template-file.erb
Content:
<% if @say_hello_to -%>
Hello <%= @say_hello_to %>,
<% end -%>

I'm <%= @myname %>, on a <%= @operatingsystem %> system, nice to meet you.

Step 2: Configuring site.pp file
node 'appserver-wkr' {

        $say_hello_to = 'guys and gals'
        $myname = 'welcome file'

        file {  "/tmp/$myname":
                ensure  => file,
                content => template('appserver/welcome-template-file.erb'),
        }
}

Note that the path is given ignoring 'templates' folder as appserver/<template-name.erb>

Step 3: Perform a catalog run in puppet agent

root@appserver-wkr:/opt# puppet agent --test

dnsdomainname: Name or service not known

dnsdomainname: Name or service not known

info: Caching catalog for appserver-wkr.openstacklocal

info: Applying configuration version '1415862309'

notice: /Stage[main]//Node[appserver-wkr]/File[/tmp/welcome file]/ensure: defined content as '{md5}3210b176441c528b9291ef4a61511f72'

notice: Finished catalog run in 0.16 seconds

Verify content.

root@appserver-wkr:/opt# ls /tmp/

file03  hsperfdata_root  welcome file

root@appserver-wkr:/opt# vi /tmp/welcome\ file

Hello guys and gals,

I'm welcome file, on a Ubuntu system, nice to meet you.

Wondering how value came for @operatingsystem?

Apart from custom variables defined, puppet can use variables predefined by Factor.

What is the significance of this content modifications?

Well, when you are deploying instances in large scale (Physical computer server clusters, VM clusters) these templates could come in handy where you can modify xml configuration files contents specific to each node.

As for an example say you have a WSO2 Application Server cluster with Worker/Manager Clustering pattern.
You don't have to keep separate installation product packs for each manager and worker. Can keep only one pack and modify the configuration xml files according to the node role (manager or worker).

Define templates and save them in your product pack's templates folder in puppet master.
Say, /etc/puppet/modules/appserver/templates/
Sit back and relax, let the puppet do the work...

Slight modification to this code could come in handy, to solve your daily deployment needs... :)

How about modifying an existing line in your /etc/hosts file or add a line to it if the particular line is already not there?

host { 'puppet.daf.wso2.com':
    ensure => 'present',
    target => '/etc/hosts',
    ip => '192.168.30.86',
}

NovaClient connection errors - simple mistakes to avoid to save your time

1. SSH Permission denied (publickey) - even the key was generated and mentioned.

I have spawned an instance in the OpenStack remote cloud. 

When I tried to connect to the newly spawned instance I got the following error.

root@suhan-VirtualBox:~# ssh -i cloud.key ubuntu@192.168.30.88

Warning: Identity file cloud.key not accessible: No such file or directory.

Permission denied (publickey).

Then I realised there is nothing wrong with the key, but the cloud.key file is not in the location from which I was trying to connect to.

I retried the connection from the location where the cloud.key was present and it was successful.

root@suhan-VirtualBox:/opt/internal-cloud# ls

backup cloud.key  conf.xml  deploy_script.py  printValuesIP.py  test.py

root@suhan-VirtualBox:/opt/internal-cloud# 

root@suhan-VirtualBox:/opt/internal-cloud# ssh -i cloud.key ubuntu@192.168.30.88

Welcome to Ubuntu 12.04.3 LTS (GNU/Linux 3.2.0-57-virtual x86_64)

 * Documentation:  https://help.ubuntu.com/

  System information as of Tue Nov 11 07:39:33 UTC 2014

  System load:  0.0               Processes:           67

  Usage of /:   3.6% of 39.37GB   Users logged in:     0

  Memory usage: 2%                IP address for eth0: 192.168.30.88

  Swap usage:   0%

  Graph this data and manage this system at https://landscape.canonical.com/

  Get cloud support with Ubuntu Advantage Cloud Guest:

    http://www.ubuntu.com/business/services/cloud

109 packages can be updated.

63 updates are security updates.

New release '14.04.1 LTS' available.

Run 'do-release-upgrade' to upgrade to it.

Last login: Tue Nov 11 07:09:44 2014 from 10.100.5.175

ubuntu@test-to-delete:~$ 

2. You must provide a username via either --os_username or env[OS_USERNAME]

I tried to ssh the newly created instance via a remote host using the key I created.
Nova credentials are to be embedded to the environmental variables via .bash_profile.
But when I tried to retrieve the nova list got the following error.

ubuntu@test-to-delete:~$ nova list
ERROR: You must provide a username via either --os_username or env[OS_USERNAME]

Environmental variables are also set.

Then I realised I was not logged in as root.

After retrying as root, I got the results I wanted... :)

ubuntu@test-to-delete:~$ sudo -i

root@test-to-delete:~# nova list

+--------------------------------------+------------------------------+--------+-------------------------------------------------------+

|                  ID                  |             Name             | Status |                             Networks                            |

+--------------------------------------+------------------------------+--------+-------------------------------------------------------+

| 200845a1-739e-41cf-bb05-744cec1398c1 | test-to-delete               | ACTIVE | network-routable=192.168.30.88                        |

| 2eeba7a3-f7f7-46d4-baff-7c7ab1f433b9 | lasindudef-ddc               | ACTIVE | network-non-routable=10.0.0.135                       |

| 3a4f8d7e-753d-456b-b26d-058548bc7994 | manula1                      | ACTIVE | network-routable=192.168.30.14, 192.168.30.15         |

| 5902f2bc-2d9d-4822-a3ea-580941717b23 | defaultlas-fb2               | ACTIVE | network-non-routable=10.0.0.108                       |

| 6175c44e-c1bb-45da-af3e-f48e01194729 | ch-stratos                   | ACTIVE | network-non-routable=10.0.0.130, 192.168.17.55        |

| 811d7d88-8e00-4354-b52c-3bb6e05aa80c | centos                       | ACTIVE | network-non-routable=10.0.0.24, 192.168.17.12         |

| 9a215704-0aff-44d4-993d-90fe70c46243 | Lasindu-Stratos              | ACTIVE | network-non-routable=10.0.0.5, 192.168.17.4           |

| aea68060-db94-434e-915f-31e9e72f152a | ch-puppetmaster              | ACTIVE | network-non-routable=10.0.0.88, 192.168.17.33         |

| e3e2558f-1c2d-43ff-a2b8-86315176de45 | CentOSBAM                    | ACTIVE | network-non-routable=10.0.0.25, 192.168.17.13         |

| eae27af4-a500-437a-878a-1a184fee5f7b | S4Test                       | ACTIVE | network-non-routable=10.0.0.56, 192.168.17.58         |

| ee8190bc-db8d-4542-b8d0-ff9a38373e56 | cisco-dev-service            | ACTIVE | network-routable=192.168.30.87                        |

| f47cf558-3435-4f6d-8fd4-739da746ce63 | suhan-daf-master-ubuntu12.04 | ACTIVE | network-routable=192.168.30.86                        |

+--------------------------------------+------------------------------+--------+-------------------------------------------------------+

3. Could not find any suitable endpoint. Correct region?
4. exception is  'Client' object has no attribute 'networks'

This means the python-novaclient is not properly installed.
Try following commands to update python package manager and python-novaclient.

>apt-get update
>apt-get install python-setuptools
>apt-get install python-pip
>pip install --upgrade python-novaclient

Bluetooth files from Android Samsung mobile to MacBook Pro OS X : Failed Request Rejected

I have successfully Bluetooth paired my Samsung S Advance mobile (visible to other devices) and my MacBook Pro laptop (OS X Yosemite).

I sent a test transfer from my Mac (a simple text file) and it was successful.
File transfer from my laptop to mobile is working as expected.

Then I tried to transfer a file from my Samsung mobile to my MacBook Pro laptop.

It failed saying Request Rejected by Mac (shown in bluetooth file transfer log in my mobile).

After Googling I found the answer.

You have to Enable Bluetooth sharing from sharing settings at System Preferences

Follow the steps as shown below via snapshots.

System Preferences -> Sharing -> turn on Bluetooth sharing

After retrying the file transfer was successful.

Hope this will save you from trouble and save your time... :)

How to create an instance in WSO2 Internal IaaS (OpenStack) and establish a ssh session

Host OS: Mac OSX Yosemite
Instance OS: ubuntu 12.04

Login to WSO2 IaaS.

Step 1 - Generate key pair

Go to
Project -> Compute -> Access & Security -> Key Pairs.

Click on import key pair button.

As per the description shown above in the screenshot itself, generate a key pair by login in to the terminal of the host machine.

Suhans-MacBook-Pro:~ suhanr$ cd .ssh/
> ssh-keygen -t rsa -f cloud.key

Your cloud key pair will be generated.
You can view them on your ssh certificates folder.

Suhans-MacBook-Pro:~ suhanr$ cd .ssh/

Suhans-MacBook-Pro:.ssh suhanr$ ls

cloud.key cloud.key.pub github_rsa github_rsa.pub known_hosts

Copy the public key content.

> vi cloud.key.pub

ssh-rsa AAAAB3NzaC1yc2 <.....................> j6BVmklren3DauAOGFLsql8gcXGLmCJbnz+b83MnWIMHF343lIvpfeHentJbS60tCipY4fEcRv1aSYpfJ8VswI8MJgRmdP+2bJdKGr1gDvwIvI/q6F6uXoT0F suhanr@Suhans-MacBook-Pro.local

Paste it on the Public Key field (refer image above).

Give a name for the Key Pair Name at your convenience.

Press Import Key Pair button.

Now you can see your new key in the Key Pairs tab as shown below.

Step 2 - Launch instance by using the generated key pair

Go to the Instances tab.

Click on launch instance button.

Fill the basic information as follows.

In the Access & Security tab give the key pair you generated in Step 1.

Click Launch button.

It will take nearly 10 seconds to put the instance to the running state.

Step 3 - Connect to the instance via ssh

From the Instances tab find the ip of the created new instance (refer image above).

Go to terminal of the host machine.

Suhans-MacBook-Pro:~ suhanr$ cd .ssh/

Connect to the instance via following command.

> ssh -i cloud.key  ubuntu@192.168.57.75

Suhans-MacBook-Pro:.ssh suhanr$ ssh -i cloud.key  ubuntu@192.168.57.75

Welcome to Ubuntu 12.04.5 LTS (GNU/Linux 3.2.0-69-virtual x86_64)

 * Documentation:  https://help.ubuntu.com/

  System information as of Wed Nov  5 04:20:00 UTC 2014

  System load:  0.0               Processes:           59

  Usage of /:   20.7% of 3.94GB   Users logged in:     0

  Memory usage: 12%               IP address for eth0: 192.168.57.75

  Swap usage:   0%

  Graph this data and manage this system at:

    https://landscape.canonical.com/

  Get cloud support with Ubuntu Advantage Cloud Guest:

    http://www.ubuntu.com/business/services/cloud

New release '14.04.1 LTS' available.

Run 'do-release-upgrade' to upgrade to it.

Last login: Tue Nov  4 11:15:56 2014 from 10.100.5.175

ubuntu@suhan-test:~$ 

Let the party begin :)