Showing posts with label Apache Stratos. Show all posts
Showing posts with label Apache Stratos. Show all posts

Tuesday, October 14, 2014

Building your own Python docker cartridge for Apache Stratos

How about building your own python-cartridge?
It’s easy, lets follow the steps below.
  1. Download github python-cartridge file contents - Download.
  1. Copy the github files to a shared location where your host machine and the VM can access. [e.g.: mine: users/suhanr/Documents in VirtualBox Settings -> Shared Folders]
This will be easy when you want to edit configuration files and modify file contents.
Important files and folders.
  • Dockerfile - Facilitate automated docker image building process.
  • packs folder contains our stratos cartridge agent, ActiveMQ libraries required and java 1.7 package.
  • init.sh
  • run_scripts.sh
  1. Open a terminal session in the VM and find the shared folder as follows. Go inside that folder.
Say for example your shared folder name is like “VMShared”, then it will appear inside /media/ folder as sf_VMShared.

  1. Simply issue the docker build command to start Docker image automated building process.
docker build -t python-cartridge .
When the build is successful the output will be similar as shown in the below image.

  1. Now tag your root repository as follows
docker tag python-cartridge 192.168.57.30:5042/python-cartridge

If you try to push the root repository to docker you will get the below error.
Impossible to push a "root" repository. Please rename your repository in <user>/<repo> (ex: <user>/python-cartridge)

  1. It is time to push your newly built docker image with the newly created tag to the glance repository
docker push 192.168.57.30:5042/python-cartridge
When the image is successfully pushed you will get an output similar to this.

  1. Now go to glance repository and get the image id of your newly pushed docker image. [This will be needed when you configure your newly built cartridge in Apache Stratos]
> cd /home/stratos/workspace/devstack/
> . openrc admin admin
> glance image-list

sample image ID: 7f179943-0b6b-4d54-af46-c334f65b76f3

  1. Now it is time to configure your newly built cartridge in Apache Stratos

You can find the python configuration files here - Download

Open your web browser in VM and go to the following link.
https://dev.stratos.org:9445/console/
Since partition and auto scale policies are already defined configure Deployment policy and cartridge configuration.

Sample Deployment policy:
{
   "id": "python-openstack",
   "partitionGroup": {
       "id": "openstack",
       "partitionAlgo": "one-after-another",
       "partition": [
       {
           "id": "P1",
           "partitionMax": "2",
           "partitionMin": "1"
       }
       ]
   }
}

Sample Cartridge configuration:
{
     "type": "python",
     "provider": "python",
     "host": "dev.stratos.org",
     "defaultAutoscalingPolicy": "economyPolicy",
     "defaultDeploymentPolicy": "python-openstack",
     "displayName": "Python",
     "description": "Python Cartridge",
     "version": "2.7",
     "multiTenant": "false",
     "portMapping": [
        {
           "protocol": "http",
           "port": "80"
        }
      ],
     "deployment": {},
      "iaasProvider": [
       {
         "type": "openstack",
         "imageId": "RegionOne/7f179943-0b6b-4d54-af46-c334f65b76f3",
         "maxInstanceLimit": "1",
         "property": [
           {
            "name": "instanceType",
            "value": "RegionOne/2"
           },
    {
           "name": "keyPair",
           "value": "demo"
           },
  {
           "name": "securityGroups",
           "value": "open-all"
          }
         ]
       }
     ],
  "property":[
       {
           "name":"payload_parameter.MB_IP",
           "value":"192.168.57.30"
       },
       {
           "name":"payload_parameter.MB_PORT",
           "value":"61616"
       },
       {
           "name":"payload_parameter.CEP_IP",
           "value":"172.17.42.1"
       },
       {
           "name":"payload_parameter.CEP_PORT",
           "value":"7613"
       }
   ]
}



Stratos terminal output will be as follows.

  1. Now subscribe to your newly built cartridge

Once you subscribe to the cartridge it will take some time to activate the instance.



You can find the python application on docker instances public ip address, in this scenario 192.168.92.1

Monday, October 13, 2014

When using python pip: read operation timeouts and distributions not found

What is Python pip?
Python pip is a package management system used in installing and managing python packages [1].

Why did I use pip?
In my Python docker cartridge for Apache Stratos project, I have created some python web applications and hosted them in github [suhan at github].
I installed the following dependencies prior using pip in my ubuntu 12.04.
apt-get install -y build-essential python-dev python-pip python-virtualenv libpq-dev

Prior running the python web applications, I have to install their python dependencies. Usually I put them inside a requirements.txt file as follows,
flask
cherrypy

Then I try to install the requirements via pip as follows,
pip install -r requirements.txt

Errors
Somewhere in the middle during installation I get these errors.
---- Installing git python web application project requirements ----
Downloading/unpacking flask
  Running setup.py egg_info for package flask
    
    warning: no files found matching '*' under directory 'tests'
    warning: no previously-included files matching '*.pyc' found under directory 'docs'
    warning: no previously-included files matching '*.pyo' found under directory 'docs'
    warning: no previously-included files matching '*.pyc' found under directory 'tests'
    warning: no previously-included files matching '*.pyo' found under directory 'tests'
    warning: no previously-included files matching '*.pyc' found under directory 'examples'
    warning: no previously-included files matching '*.pyo' found under directory 'examples'
    no previously-included directories found matching 'docs/_build'
    no previously-included directories found matching 'docs/_themes/.git'
Downloading/unpacking cherrypy
  Running setup.py egg_info for package cherrypy
    
Downloading/unpacking Werkzeug>=0.7 (from flask)
  Running setup.py egg_info for package Werkzeug
    
    warning: no files found matching '*' under directory 'werkzeug/debug/templates'
    warning: no files found matching '*' under directory 'tests'
    warning: no previously-included files matching '*.pyc' found under directory 'docs'
    warning: no previously-included files matching '*.pyo' found under directory 'docs'
    warning: no previously-included files matching '*.pyc' found under directory 'tests'
    warning: no previously-included files matching '*.pyo' found under directory 'tests'
    warning: no previously-included files matching '*.pyc' found under directory 'examples'
    warning: no previously-included files matching '*.pyo' found under directory 'examples'
    no previously-included directories found matching 'docs/_build'
Downloading/unpacking Jinja2>=2.4 (from flask)
  Running setup.py egg_info for package Jinja2
    
    warning: no files found matching '*' under directory 'custom_fixers'
    warning: no previously-included files matching '*' found under directory 'docs/_build'
    warning: no previously-included files matching '*.pyc' found under directory 'jinja2'
    warning: no previously-included files matching '*.pyc' found under directory 'docs'
    warning: no previously-included files matching '*.pyo' found under directory 'jinja2'
    warning: no previously-included files matching '*.pyo' found under directory 'docs'
Downloading/unpacking itsdangerous>=0.21 (from flask)
  Could not find any downloads that satisfy the requirement itsdangerous>=0.21 (from flask)
No distributions at all found for itsdangerous>=0.21 (from flask)
Storing complete log in /root/.pip/pip.log

CherryPy server fails to start since dependencies are not properly installed.

---- Starting CherryPy Server ----
Traceback (most recent call last):
 File "/var/www/www/server.py", line 5, in <module>
   from app import app
 File "/var/www/www/app.py", line 1, in <module>
   from flask import Flask

ImportError: No module named flask

Sometimes I get the following error.

Exception:
Traceback (most recent call last):
  File "/vagrant/venv/local/lib/python2.7/site-packages/pip/basecommand.py", line 134, in main
    status = self.run(options, args)
  File "/vagrant/venv/local/lib/python2.7/site-packages/pip/commands/install.py", line 236, in run
    requirement_set.prepare_files(finder, force_root_egg_info=self.bundle, bundle=self.bundle)
  File "/vagrant/venv/local/lib/python2.7/site-packages/pip/req.py", line 1092, in prepare_files
    self.unpack_url(url, location, self.is_download)
  File "/vagrant/venv/local/lib/python2.7/site-packages/pip/req.py", line 1238, in unpack_url
    retval = unpack_http_url(link, location, self.download_cache, self.download_dir)
  File "/vagrant/venv/local/lib/python2.7/site-packages/pip/download.py", line 622, in unpack_http_url
    download_hash = _download_url(resp, link, temp_location)
  File "/vagrant/venv/local/lib/python2.7/site-packages/pip/download.py", line 495, in _download_url
    chunk = resp.read(4096)
  File "/usr/lib/python2.7/socket.py", line 380, in read
    data = self._sock.recv(left)
  File "/usr/lib/python2.7/httplib.py", line 561, in read
    s = self.fp.read(amt)
  File "/usr/lib/python2.7/socket.py", line 380, in read
    data = self._sock.recv(left)
  File "/usr/lib/python2.7/ssl.py", line 241, in recv
    return self.read(buflen)
  File "/usr/lib/python2.7/ssl.py", line 160, in read
    return self._sslobj.read(len)
SSLError: The read operation timed out

These errors are due to internet connectivity problems, slow internet speed, etc...
Since I'm using a VM to build my docker images these issues/errors are further amplified.


As a temporary solution I did retry several times and finally it was successfully installed.
retry: pip install -r requirements.txt

Then my CherryPy server started successfully.
---- Starting CherryPy Server ----
ENGINE Bus STARTING
ENGINE Started monitor thread 'Autoreloader'.
ENGINE Started monitor thread '_TimeoutMonitor'.
ENGINE Serving on http://0.0.0.0
ENGINE Bus STARTED

However according to stackoverflow [2] if you can use the following command the problem will be resolved.
pip --default-timeout=100 install django

Reference:
[1]http://en.wikipedia.org/wiki/Pip_(package_manager)
[2]http://stackoverflow.com/questions/18958508/sslerror-the-read-operation-timed-out-when-using-pip/19334551#19334551