Wednesday, October 19, 2016

Where did everything go in Puppet 4.x?

This is a summary of [1] at docs.puppet.com.

  • New All-in-One puppet-agent package - On managed *nix systems, you’ll now install puppet-agent instead of puppet. (This package also provides puppet apply, suitable for standalone Puppet systems.) It includes tools (private versions) like Facter, Hiera, and Ruby; also MCollective.
  • *nix executables are in /opt/puppetlabs/bin/ - On *nix platforms, the main executables moved to /opt/puppetlabs/bin. This means Puppet and related tools aren’t included in your PATH by default. You have to add it to your PATH or use full path when running puppet commands. 
    1. Private bin directories - The executables in /opt/puppetlabs/bin are just the “public” applications that make up Puppet. Private supporting commands like ruby and gem are in /opt/puppetlabs/puppet/bin
  • *nix confdir is Now /etc/puppetlabs/puppet - Puppet’s system confdir (used by root and the puppet user) is now /etc/puppetlabs/puppet, instead of /etc/puppet. Open source Puppet now uses the same confdir as Puppet Enterprise.
    1. ssldir is inside confdir - The default location is in the $confdir/ssl on all platforms.
    2. Other stuff in /etc/puppetlabs - Other configs are in /etc/puppetlabs directory. Puppet Server now uses /etc/puppetlabs/puppetserver, and MCollective uses /etc/puppetlabs/mcollective.
  • New codedir holds all modules(1), manifests(1) and data(2) - The default codedir location is /etc/puppetlabs/code on *nix. It has environments, modules directories and hiera.yaml config file.
    1. Directory environments are always on - The default environmentpath is $codedir/environments and a directory is created at install for the default production environment. Modules are in $codedir/environments/production/modules and main manifest is in $codedir/environments/production/manifests. However you can still use global modules in $codedir/modules and a global manifest.
    2. Hiera data goes in environments by default - Hiera’s default settings now use an environment-specific datadir for the YAML and JSON backends. So the production environment’s default Hiera data directory would be /etc/puppetlabs/code/environments/production/hieradata
  • Some other directories have moved - The system vardir for puppet agent has moved, and is now separate from Puppet Server’s vardir. For *nix: /opt/puppetlabs/puppet/cache. The rundir, where the service PID files are kept, (on *nix) has moved to /var/run/puppetlabs. (Puppet Server has a puppetserver directory in this directory.)

Reference:
[1] https://docs.puppet.com/puppet/latest/reference/whered_it_go.html

Wednesday, October 12, 2016

Accessing Google APIs using OAuth 2.0

Here is my use case.
I recently wanted to use WSO2 ESB GoogleSpreadsheet Connector. Before attempting any other spreadsheet operation in connector, I have to call the googlespreadsheet.init element first. This configuration authenticates with Google Spreadsheet by configuring the user credentials using OAuth2 authentication for accessing the Google account that contains the spreadsheets.

I couldn't initially get any simple answers/instructions in www on how to set up the following four elements from my Google Account. I have shared the process for your benefit hoping this will save loads of your valuable time.
1. Client ID
2. Client Secret
3. Refresh Token
4. Access Token

Steps
1. Go to https://console.developers.google.com
2. Under Credentials -> Create Credentials -> OAuth client ID

3. Select Application Type: Web Application
    Name: <appName>
   Authorized redirect URIs: https://developers.google.com/oauthplayground

4. Click Create.
Google will immediately show your Client ID and Client Secret as follows (you can get these details later also).

Following is how the edit view shows you the required information.

6. Click the settings icon.
7. Select the Use your own OAuth credentials check box
8. Fill two fields OAuth Client ID and OAuth Client secret with the values you retrieved in step 4.

9. Now in Step 1 select the Google Sheets APIv4 all 4 scopes as shown below.
10. Click on Authorize APIs
11. Once ask approve permission to access the selected scopes.


 12. In Step 2 Click on Exchange authorization code for tokens button.
13. These are your Refresh and Access Tokens

Now all four fields required are retrieved.
Following is how you will be using googlespreadsheet.init element inside your proxy/sequence configuration.

<googlespreadsheet.init>
    <accessToken>{$ctx:accessToken}</accessToken>
    <clientId>{$ctx:clientId}</clientId>
    <clientSecret>{$ctx:clientSecret}</clientSecret>
    <refreshToken>{$ctx:refreshToken}</refreshToken>
    <accessTokenRegistryPath>{$ctx:accessTokenRegistryPath}</accessTokenRegistryPath>
    <apiUrl>{$ctx:apiUrl}</apiUrl>
</googlespreadsheet.init>

Following is a sample JSON payload to invoke googlespreadsheet.init method in ESB google spreadsheet connector. Note that the registry path is optional here.

{  
  "googlespreadsheetAccessToken": "xxNN.XXXXXXXXXXXXXXXXXXXXXXXXXXXX XXXXXXXXXXXXXXXXXXXXX",
  "googlespreadsheetRefreshToken": "1/KSKKDKDK_Isdf_dsalkfdffafdskfdsafdksfj_sdfSWE",
  "googlespreadsheetClientId": "721330057745-aq810qmciljtvlablt1hjlnh6vjjer7j.apps.googleusercontent.com",
  "googlespreadsheetClientSecret": "0GeUJ4KK3Flk8C9S0Ua51jZS",
  "googlespreadsheetApiUrl": "https://sheets.googleapis.com"
}

Note: As of today WSO2 ESB googlespreadsheet connector is compatible with Google Spreadsheets API v3(Legacy). But for authentication purposes there will be no much of a difference. I have written my own connector for Google Spreadsheets API v4 to use v4 specific methods like spreadsheets.create
Refer [1] for more information on writing your own connector.

Reference:
[1] https://docs.wso2.com/display/ESBCONNECTORS/Writing+a+Connector

Tuesday, October 11, 2016

Building/Uploading/Using the Sample ESB Connector in few minutes

I have used WSO2 ESB 5.0.0 in this article.
Go to an empty folder and run the following command to generate the maven project sample connector code.

mvn archetype:generate -DarchetypeGroupId=org.wso2.carbon.extension.archetype -DarchetypeArtifactId=org.wso2.carbon.extension.esb.connector-archetype -DarchetypeVersion=2.0.0 -DgroupId=org.wso2.carbon.connector -DartifactId=org.wso2.carbon.connector.sample -Dversion=1.0.0 -DarchetypeRepository=http://maven.wso2.org/nexus/content/repositories/wso2-public/

You will be asked to specify a name for the connector.
Define value for property 'connector_name': : Sample
After asking to confirm the properties type Y and enter to confirm.

Issue the following command to build the connector zip file.

mvn clean install -Dmaven.test.skip=true

Download/Start WSO2 ESB 4.9.0 or ESB 5.0.0 Server.

Login to the management console (default username/password: admin/admin) and upload the created 
connector zip file.
(zip file will be at ../org.wso2.carbon.connector.sample/target/sample-connector-1.0.0.zip)


Add the following proxy service. (Home -> Manage -> Services -> Add -> Proxy Service)
Select custom proxy service -> view source -> paste following code -> Save

<?xml version="1.0" encoding="UTF-8"?>
<proxy xmlns="http://ws.apache.org/ns/synapse"
       name="sampleConnectorProxy"
       startOnLoad="true"
       statistics="disable"
       trace="disable"
       transports="https,http">
   <target>
      <inSequence>
         <log level="custom">
            <property name="**********" value="Inside sampleConnectorProxy **********"/>
         </log>
         <Sample.sample_template>
            <generated_param>Hi</generated_param>
         </Sample.sample_template>
      </inSequence>
   </target>
   <description/>
</proxy>

Following will be printed in the server log file

TID: [-1234] [] [2016-10-11 13:33:36,755]  INFO {org.wso2.carbon.mediation.dependency.mgt.DependencyTracker} -  Proxy service : sampleConnectorProxy was added to the Synapse configuration successfully {org.wso2.carbon.mediation.dependency.mgt.DependencyTracker}
TID: [-1234] [] [2016-10-11 13:33:36,755]  INFO {org.apache.synapse.core.axis2.ProxyService} -  Building Axis service for Proxy service : sampleConnectorProxy {org.apache.synapse.core.axis2.ProxyService}
TID: [-1234] [] [2016-10-11 13:33:36,757]  INFO {org.apache.synapse.core.axis2.ProxyService} -  Adding service sampleConnectorProxy to the Axis2 configuration {org.apache.synapse.core.axis2.ProxyService}
TID: [-1234] [] [2016-10-11 13:33:36,758]  INFO {org.wso2.carbon.core.deployment.DeploymentInterceptor} -  Deploying Axis2 service: sampleConnectorProxy {super-tenant} {org.wso2.carbon.core.deployment.DeploymentInterceptor}
TID: [-1234] [] [2016-10-11 13:33:36,759]  INFO {org.apache.synapse.core.axis2.ProxyService} -  Successfully created the Axis2 service for Proxy service : sampleConnectorProxy {org.apache.synapse.core.axis2.ProxyService}
                                
Go to services -> Try this service -> Send

Following will be printed in the server log file

TID: [-1234] [] [2016-10-11 13:33:47,314]  INFO {org.apache.synapse.mediators.builtin.LogMediator} -  ********** = Inside sampleConnectorProxy ********** {org.apache.synapse.mediators.builtin.LogMediator}
TID: [-1234] [] [2016-10-11 13:33:47,316]  INFO {org.apache.synapse.mediators.builtin.LogMediator} -  To: /services/sampleConnectorProxy.sampleConnectorProxyHttpSoap12Endpoint, WSAction: urn:mediate, SOAPAction: urn:mediate, MessageID: urn:uuid:e3b5f31c-37b8-4ca8-b483-9bbc4a6daf04, Direction: request, *******template_param******** = Hi, Envelope: <?xml version='1.0' encoding='utf-8'?><soapenv:Envelope xmlns:soapenv="http://www.w3.org/2003/05/soap-envelope"><soapenv:Body/></soapenv:Envelope> {org.apache.synapse.mediators.builtin.LogMediator}
TID: [-1234] [] [2016-10-11 13:33:47,316]  INFO {org.wso2.carbon.connector.sampleConnector} -  sample sample connector received message :Hi {org.wso2.carbon.connector.sampleConnector}

To do modifications to the sample connector and experiment, you can set up your development environment. I'm using Intellij IDEA Community Edition. Refer to the following command.

mvn clean install idea:idea

Learn more about the components inside the connector.
https://docs.wso2.com/display/ESBCONNECTORS/Writing+a+Connector

Creating a third party connector and publishing it in WSO2 store.
https://docs.wso2.com/display/ESBCONNECTORS/Creating+a+Third+Party+Connector+and+Publishing+in+WSO2+Store

Wednesday, August 24, 2016

WSO2 DSS, Secure Data Service using Basic Auth

Hope you have read this article: Using WSO2 DSS retrieve data from multiple databases in a single Query. Lets use the same data service and secure it using Basic Auth. Then lets try to invoke the secured data service with SOAP UI or from a WSO2 ESB.

Securing the data service
1. If not already created, add a new collection to /_system/config as security, i.e., /_system/config/security
2. Then add this policy file[1] (if you want you can rename the policy file, say policy1.xml) to /_system/config/security/ using Add Resource | Upload content from file | choose file




3. Next you have to add this policy to your data service. Go to data service xml editor view.
Add following to Data Service configuration at the end just before </data> closing tag.

   <policy key="conf:security/policy1.xml"/>
   <enableSec/>


Wait for few seconds and go to Deployed Services page and view the available data services. Now you can see the updated data service is as secured.


When you are testing the data service from the SOAP UI, you need to send the Authorization header.
Go to request level of your SOAP UI project created in the previous article and add Header as follows.

Header Name : Authorization
Value : Basic YWRtaW46YWRtaW4=

We are using admin:admin default credentials here.
YWRtaW46YWRtaW4= is the base64 encoded value of admin:admin
You can calculate the base64 encoded value online [2].

If a WSO2 ESB is invoking the service simply add following configuration in the synapse config
before sending message to the endpoint.

<property xmlns:ns="http://org.apache.synapse/xsd"
 name="Authorization"
 expression="fn:concat('Basic ', base64Encode('username:password'))"
 scope="transport"/>

If you are using jaggery to call DSS endpoints you can send the headers with the request as follows.

var POST_HEADERS = { "Content-Type": "application/json", "Authorization": "Basic YWRtaW46YWRtaW4="};

var resp = put(dssLDAPUserDSURL + "/employee/status", stringify(dataPost), POST_HEADERS);

You can find the sample data service, policy file, SOAP UI project and the relevant MySQL database scripts here [3].

Reference:
[1] https://svn.wso2.org/repos/wso2/people/suhan/BasicAuthSecuredBackendService/UT_policy.xml
[2] https://www.base64encode.org/
[3] https://svn.wso2.org/repos/wso2/people/suhan/BasicAuthSecuredBackendService/


Tuesday, August 23, 2016

Using WSO2 DSS retrieve data from multiple databases in a single Query

I came across an issue of having same table in multiple databases and having to write data multiple times when syncing these data with external systems. Therefore I thought of transferring these duplicated tables into one single common database. Then this common database will to be used by several of our internal systems. However I recently came across an issue that I needed to retrieve data from multiple databases using a single query. Since I'm already using WSO2 DSS, I found a solution to tackle this problem.

Here's how to do it. For the ease of understanding I have divided the steps into two parts.

Part 1 - Configure MySQL Database

Login to your local MySQL db and create two databases.
Then create two tables and insert some test data as follows.

mysql> create database db1;
mysql> create database db2;

mysql> use db1; 
mysql> CREATE TABLE employees ( EmployeeID int(11) NOT NULL AUTO_INCREMENT, FirstName varchar(255) DEFAULT NULL, LastName varchar(255) DEFAULT NULL, Team varchar(255) DEFAULT NULL, PRIMARY KEY (EmployeeID));
mysql> insert into employees (FirstName, LastName, Team) values('Suhan', 'Dharmasuriya', 'InternalIT');
mysql> insert into employees (FirstName, LastName, Team) values('Thilina', 'Perera', 'Finance');
mysql> select * from employees;
+------------+-----------+--------------+------------+
| EmployeeID | FirstName | LastName     | Team       |
+------------+-----------+--------------+------------+
|          1 | Suhan     | Dharmasuriya | InternalIT |
|          2 | Thilina   | Perera       | Finance    |
+------------+-----------+--------------+------------+
2 rows in set (0.00 sec)

mysql> use db2;
mysql> CREATE TABLE engagements ( EngagementID int(11) NOT NULL AUTO_INCREMENT, EmployeeID int(11), PRIMARY KEY (EngagementID));
mysql> insert into engagements (EmployeeID) values(1);
mysql> select * from engagements;
+--------------+------------+
| EngagementID | EmployeeID |
+--------------+------------+
|            1 |          1 |
+--------------+------------+
1 row in set (0.00 sec)

Now lets test the following MySQL query and get results. Here I have used a LEFT OUTER JOIN.
Result will be obtained from the two databases. We will save this query in WSO2 DSS. 

mysql> SELECT A.EmployeeID, A.FirstName, A.LastName, A.Team, B.EngagementID from db1.employees AS A LEFT OUTER JOIN db2.engagements AS B ON A.EmployeeID=B.EmployeeID;
+------------+-----------+--------------+------------+--------------+
| EmployeeID | FirstName | LastName     | Team       | EngagementID |
+------------+-----------+--------------+------------+--------------+
|          1 | Suhan     | Dharmasuriya | InternalIT |            1 |
|          2 | Thilina   | Perera       | Finance    |         NULL |
+------------+-----------+--------------+------------+--------------+
2 rows in set (0.00 sec)

Part 2 - Configure WSO2 DSS

  1. Download WSO2 DSS 3.5.0 from here (http://wso2.com/products/data-services-server/). If you already have the product zip file (wso2dss-3.5.0.zip) continue with the next step.
  2. Unzip the product to a path containing no spaces in the path name. This is your <DSS_HOME>
  3. Download the MySQL connector jar here (http://dev.mysql.com/downloads/connector/j/) and copy it to <DSS_HOME>/repository/components/lib/
  4. Start the WSO2 DSS server.
To start the server, your have to run the script wso2server.bat (on Windows) or
wso2server.sh (on Linux/Solaris) from the <DSS_HOME>/bin folder.
  1. Log in to DSS by using the default credentials (username: admin/ password: admin).

Creating the data service

  1. Create a new data service 'sampleDS'
  2. Home -> Manage -> Services -> Add -> Data Service -> Create
  3. Create a datasource as follows by referring to the above local MySQL database.
  4. Add new Query; query ID: 'getEngagementsPerEmployee', query: above tested query, and press generate response link
  5. No operation is defined in this example, therefore press next
  6. Add new Resource; Resource Path: '/engagements' by selecting Resource Method: 'GET' and selecting Query ID: 'getEngagementsPerEmployee'
  7. Press Finish to create the data service

Refer following screenshots for more information.










Now you can test the datasource as follows.

Open a new tab in your browser and type the following URL: http://localhost:9763/services/sampleDS/engagements

You will get the following response:
<Entries xmlns="http://ws.wso2.org/dataservice">
<Entry>
<EmployeeID>1</EmployeeID>
<FirstName>Suhan</FirstName>
<LastName>Dharmasuriya</LastName>
<Team>InternalIT</Team>
<EngagementID>1</EngagementID>
</Entry>
<Entry>
<EmployeeID>2</EmployeeID>
<FirstName>Thilina</FirstName>
<LastName>Perera</LastName>
<Team>Finance</Team>
<EngagementID xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:nil="true"/>
</Entry>
</Entries>

Refer following Data Service synapse configuration for more information.
<data name="sampleDS" transports="http https local">
   <config enableOData="false" id="datasource">
      <property name="driverClassName">com.mysql.jdbc.Driver</property>
      <property name="url">jdbc:mysql://localhost:3306</property>
      <property name="username">root</property>
      <property name="password">root</property>
   </config>
   <query id="getEngagementsPerEmployee" useConfig="datasource">
      <sql>SELECT A.EmployeeID, A.FirstName, A.LastName, A.Team, B.EngagementID from db1.employees AS A LEFT OUTER JOIN db2.engagements AS B ON A.EmployeeID=B.EmployeeID;</sql>
      <result element="Entries" rowName="Entry">
         <element column="EmployeeID" name="EmployeeID" xsdType="string"/>
         <element column="FirstName" name="FirstName" xsdType="string"/>
         <element column="LastName" name="LastName" xsdType="string"/>
         <element column="Team" name="Team" xsdType="string"/>
         <element column="EngagementID" name="EngagementID" xsdType="string"/>
      </result>
   </query>
   <resource method="GET" path="/engagements">
      <call-query href="getEngagementsPerEmployee"/>
   </resource>
</data>


Wednesday, February 3, 2016

Using wso2/puppet-modules to deploy WSO2 Products

This article will provide an insight to setup a standalone product using wso2/puppet-modules [1] for the given productIt will provide a sequence for you to follow in testing your WSO2 product of interest for a standalone deployment setup in a given puppet agent instance. 
This can be considered as a guide to setup your own puppet master/agent environment for testing given that you are going to manually setup the server instances.

3 Tasks
  1. Setup two instances
  2. Setup puppet master and puppet agent
  3. Perform a catalog run to setup product instance

Task 1 - Setup two instances
This can be either two EC2 instances, two WSO2 IaaS instances, or two VM instances.
e.g.: 192.168.19.33 - puppet master
        192.168.19.35 - puppet agent

Task 2 - Setup puppet master and puppet agent
Steps to install and configure puppet master and agent are as follows.
Installing puppet
Puppet master
  • Login to puppet master as the super user
  • Issue the following commands.
> ntpdate pool.ntp.org ; apt-get update && sudo apt-get -y install ntp ; service ntp restart
> cd /tmp
> dpkg -i puppetlabs-release-trusty.deb
> apt-get update
> apt-get install puppetmaster
  • Check the puppet version as: puppet -V
Puppet version Should be 3.8.3 or higher
  • Edit etc/hostname file and set property hostname
puppetmaster
  • Edit your puppet master hosts file /etc/hosts as follows and set puppet master hostname
127.0.0.1 localhost
127.0.0.1 puppetmaster

Puppet agent
  • Login to puppet agent as the super user
  • Issue the following commands.
> apt-get update
> apt-get install puppet
  • Edit etc/hostname file and set property hostname
e.g.: qaa-node-1
  • Edit /etc/hosts file of your puppet agent
127.0.0.1 localhost
192.168.19.XXX puppet # puppet master IP address


Configuring puppet
Puppet master
  • Login to puppet master as the super user
  • Create a directory /etc/puppet/environment/production/
  • Modify /etc/puppet/puppet.conf file of puppet master appending the following to [main] and [master] sections accordingly.
[main]
dns_alt_names=puppetmaster,puppet
environmentpath = $confdir/environments
[master]
autosign=true
  • Before restarting the puppet master, clean all certificates, including puppet master’s certificate which is having its old DNS alt names.
> puppet cert clean --all
  • Restart the puppet master for new configuration changes to take effect and to regenerate the certificate with the new dns_alt_names
> service puppetmaster restart

  • Download git repo files
git clone wso2/puppet-modules [1] to a temp folder
  • Prepare puppet modules
    • Copy all modules in ../puppet-modules/modules to /etc/puppet/environment/production/modules/
    • Install java module from puppet forge [2].
> puppet module install 7terminals-java
    • Install stdlib module from puppet forge [3].
> puppet module install puppetlabs-stdlib
  • Prepare hieradata
    • Copy ../puppet-modules/hiera.yaml to /etc/puppet
    • Copy ../puppet-modules/hieradata folder to /etc/puppet/
    • Rename /etc/puppet/hieradata/dev -> /etc/puppet/hieradata/production
  • Prepare site.pp manifest
    • Copy ../puppet-modules/manifests/site.pp to /etc/puppet/environment/production/manifests/
  • Prepare files/packs
    • Copy the pack file (e.g.: wso2am-1.9.1.zip) to ../modules/wso2am/files/
    • Create folder ../modules/wso2base/files
    • Copy jdk installation file (e.g.: jdk-7u79-linux-x64.gz) to ../modules/wso2base/files
    • Add jdk version (home) and filename information to /etc/puppet/hieradata/production/common.yaml

Puppet agent
  • Login to puppet agent as the super user
  • Modify /etc/puppet/puppet.conf file of puppet agent appending the following to the [main] section.
[main]
server = puppet

Task 3 - Set Facter variables and perform a puppet agent run.

Following sample files attached are for APIM 1.9.1. Modify the deployment.conf file according to your product.
  • Login to puppet agent as super user.
  • Copy the file deployment.conf [4] -> /opt/deployment.conf
  • Copy the file setup.sh [5] -> /opt/setup.sh
> chmod 755 setup.sh
  • Run following command to setup product instance.
> ./setup.sh

An instance of a standard WSO2 product of your choice will be installed.
e.g.: 
CARBON_HOME ->  /mnt/10.0.2.89/wso2am-1.9.1
For APIM 1.9.1 the complete deployment time (deployment + server startup) will be approximately 1 minute.

If you are planning to go ahead with Vagrant setup, i.e. to spawn VM instances please refer [6].


[4] Attachment: deployment.conf
product_name=wso2am
product_version=1.9.1
product_profile=default
vm_type=openstack

[5] Attachment: setup.sh
#!/bin/bash
echo "#####################################################"
echo "                   Starting cleanup "
echo "#####################################################"
ps aux | grep -i wso2 | awk {'print $2'} | xargs kill -9
#rm -rf /mnt/*
sed -i '/environment/d' /etc/puppet/puppet.conf
echo "#####################################################"
echo "               Setting up environment "
echo "#####################################################"
rm -f /etc/facter/facts.d/deployment_pattern.txt
mkdir -p /etc/facter/facts.d

while read -r line; do declare  $line; done < deployment.conf

echo product_name=$product_name >> /etc/facter/facts.d/deployment_pattern.txt
echo product_version=$product_version >> /etc/facter/facts.d/deployment_pattern.txt
echo product_profile=$product_profile >> /etc/facter/facts.d/deployment_pattern.txt
echo vm_type=$vm_type >> /etc/facter/facts.d/deployment_pattern.txt

echo "#####################################################"
echo "                    Installing "
echo "#####################################################"

puppet agent --enable
puppet agent -vt
puppet agent --disable