Where did everything go in Puppet 4.x?

This is a summary of [1] at docs.puppet.com.

• New All-in-One puppet-agent package - On managed *nix systems, you’ll now install puppet-agent instead of puppet. (This package also provides puppet apply, suitable for standalone Puppet systems.) It includes tools (private versions) like Facter, Hiera, and Ruby; also MCollective.

• *nix executables are in /opt/puppetlabs/bin/ - On *nix platforms, the main executables moved to /opt/puppetlabs/bin. This means Puppet and related tools aren’t included in your PATH by default. You have to add it to your PATH or use full path when running puppet commands. 

1. Private bin directories - The executables in /opt/puppetlabs/bin are just the “public” applications that make up Puppet. Private supporting commands like ruby and gem are in /opt/puppetlabs/puppet/bin

• *nix confdir is Now /etc/puppetlabs/puppet - Puppet’s system confdir (used by root and the puppet user) is now /etc/puppetlabs/puppet, instead of /etc/puppet. Open source Puppet now uses the same confdir as Puppet Enterprise.

1. ssldir is inside confdir - The default location is in the $confdir/ssl on all platforms.
2. Other stuff in /etc/puppetlabs - Other configs are in /etc/puppetlabs directory. Puppet Server now uses /etc/puppetlabs/puppetserver, and MCollective uses /etc/puppetlabs/mcollective.

• New codedir holds all modules(1), manifests(1) and data(2) - The default codedir location is /etc/puppetlabs/code on *nix. It has environments, modules directories and hiera.yaml config file.

1. Directory environments are always on - The default environmentpath is $codedir/environments and a directory is created at install for the default production environment. Modules are in $codedir/environments/production/modules and main manifest is in $codedir/environments/production/manifests. However you can still use global modules in $codedir/modules and a global manifest.
2. Hiera data goes in environments by default - Hiera’s default settings now use an environment-specific datadir for the YAML and JSON backends. So the production environment’s default Hiera data directory would be /etc/puppetlabs/code/environments/production/hieradata

• Some other directories have moved - The system vardir for puppet agent has moved, and is now separate from Puppet Server’s vardir. For *nix: /opt/puppetlabs/puppet/cache. The rundir, where the service PID files are kept, (on *nix) has moved to /var/run/puppetlabs. (Puppet Server has a puppetserver directory in this directory.)

Reference:
[1] https://docs.puppet.com/puppet/latest/reference/whered_it_go.html

Accessing Google APIs using OAuth 2.0

Here is my use case.
I recently wanted to use WSO2 ESB GoogleSpreadsheet Connector. Before attempting any other spreadsheet operation in connector, I have to call the googlespreadsheet.init element first. This configuration authenticates with Google Spreadsheet by configuring the user credentials using OAuth2 authentication for accessing the Google account that contains the spreadsheets.

I couldn't initially get any simple answers/instructions in www on how to set up the following four elements from my Google Account. I have shared the process for your benefit hoping this will save loads of your valuable time.
1. Client ID
2. Client Secret
3. Refresh Token
4. Access Token

Steps
1. Go to https://console.developers.google.com
2. Under Credentials -> Create Credentials -> OAuth client ID

3. Select Application Type: Web Application

    Name: <appName>

   Authorized redirect URIs: https://developers.google.com/oauthplayground

4. Click Create.

Google will immediately show your Client ID and Client Secret as follows (you can get these details later also).

Following is how the edit view shows you the required information.

5. Go to https://developers.google.com/oauthplayground

6. Click the settings icon.

7. Select the Use your own OAuth credentials check box

8. Fill two fields OAuth Client ID and OAuth Client secret with the values you retrieved in step 4.

9. Now in Step 1 select the Google Sheets APIv4 all 4 scopes as shown below.

10. Click on Authorize APIs

11. Once ask approve permission to access the selected scopes.

 12. In Step 2 Click on Exchange authorization code for tokens button.

13. These are your Refresh and Access Tokens

Now all four fields required are retrieved.
Following is how you will be using googlespreadsheet.init element inside your proxy/sequence configuration.

<googlespreadsheet.init>
    <accessToken>{$ctx:accessToken}</accessToken>
    <clientId>{$ctx:clientId}</clientId>
    <clientSecret>{$ctx:clientSecret}</clientSecret>
    <refreshToken>{$ctx:refreshToken}</refreshToken>
    <accessTokenRegistryPath>{$ctx:accessTokenRegistryPath}</accessTokenRegistryPath>
    <apiUrl>{$ctx:apiUrl}</apiUrl>
</googlespreadsheet.init>

Following is a sample JSON payload to invoke googlespreadsheet.init method in ESB google spreadsheet connector. Note that the registry path is optional here.

{  
  "googlespreadsheetAccessToken": "xxNN.XXXXXXXXXXXXXXXXXXXXXXXXXXXX XXXXXXXXXXXXXXXXXXXXX",
  "googlespreadsheetRefreshToken": "1/KSKKDKDK_Isdf_dsalkfdffafdskfdsafdksfj_sdfSWE",
  "googlespreadsheetClientId": "721330057745-aq810qmciljtvlablt1hjlnh6vjjer7j.apps.googleusercontent.com",
  "googlespreadsheetClientSecret": "0GeUJ4KK3Flk8C9S0Ua51jZS",
  "googlespreadsheetApiUrl": "https://sheets.googleapis.com"
}

Note: As of today WSO2 ESB googlespreadsheet connector is compatible with Google Spreadsheets API v3(Legacy). But for authentication purposes there will be no much of a difference. I have written my own connector for Google Spreadsheets API v4 to use v4 specific methods like spreadsheets.create. 
Refer [1] for more information on writing your own connector.

Reference:
[1] https://docs.wso2.com/display/ESBCONNECTORS/Writing+a+Connector

Building/Uploading/Using the Sample ESB Connector in few minutes

I have used WSO2 ESB 5.0.0 in this article.
Go to an empty folder and run the following command to generate the maven project sample connector code.

mvn archetype:generate -DarchetypeGroupId=org.wso2.carbon.extension.archetype -DarchetypeArtifactId=org.wso2.carbon.extension.esb.connector-archetype -DarchetypeVersion=2.0.0 -DgroupId=org.wso2.carbon.connector -DartifactId=org.wso2.carbon.connector.sample -Dversion=1.0.0 -DarchetypeRepository=http://maven.wso2.org/nexus/content/repositories/wso2-public/

You will be asked to specify a name for the connector.
Define value for property 'connector_name': : Sample
After asking to confirm the properties type Y and enter to confirm.

Issue the following command to build the connector zip file.

mvn clean install -Dmaven.test.skip=true

Download/Start WSO2 ESB 4.9.0 or ESB 5.0.0 Server.

Login to the management console (default username/password: admin/admin) and upload the created 

connector zip file.

(zip file will be at ../org.wso2.carbon.connector.sample/target/sample-connector-1.0.0.zip)

Add the following proxy service. (Home -> Manage -> Services -> Add -> Proxy Service)

Select custom proxy service -> view source -> paste following code -> Save

<?xml version="1.0" encoding="UTF-8"?>

<proxy xmlns="http://ws.apache.org/ns/synapse"

       name="sampleConnectorProxy"

       startOnLoad="true"

       statistics="disable"

       trace="disable"

       transports="https,http">

   <target>

      <inSequence>

         <log level="custom">

            <property name="**********" value="Inside sampleConnectorProxy **********"/>

         </log>

         <Sample.sample_template>

            <generated_param>Hi</generated_param>

         </Sample.sample_template>

      </inSequence>

   </target>

   <description/>

</proxy>

Following will be printed in the server log file

TID: [-1234] [] [2016-10-11 13:33:36,755]  INFO {org.wso2.carbon.mediation.dependency.mgt.DependencyTracker} -  Proxy service : sampleConnectorProxy was added to the Synapse configuration successfully {org.wso2.carbon.mediation.dependency.mgt.DependencyTracker}

TID: [-1234] [] [2016-10-11 13:33:36,755]  INFO {org.apache.synapse.core.axis2.ProxyService} -  Building Axis service for Proxy service : sampleConnectorProxy {org.apache.synapse.core.axis2.ProxyService}

TID: [-1234] [] [2016-10-11 13:33:36,757]  INFO {org.apache.synapse.core.axis2.ProxyService} -  Adding service sampleConnectorProxy to the Axis2 configuration {org.apache.synapse.core.axis2.ProxyService}

TID: [-1234] [] [2016-10-11 13:33:36,758]  INFO {org.wso2.carbon.core.deployment.DeploymentInterceptor} -  Deploying Axis2 service: sampleConnectorProxy {super-tenant} {org.wso2.carbon.core.deployment.DeploymentInterceptor}

TID: [-1234] [] [2016-10-11 13:33:36,759]  INFO {org.apache.synapse.core.axis2.ProxyService} -  Successfully created the Axis2 service for Proxy service : sampleConnectorProxy {org.apache.synapse.core.axis2.ProxyService}

                                

Go to services -> Try this service -> Send

Following will be printed in the server log file

TID: [-1234] [] [2016-10-11 13:33:47,314]  INFO {org.apache.synapse.mediators.builtin.LogMediator} -  ********** = Inside sampleConnectorProxy ********** {org.apache.synapse.mediators.builtin.LogMediator}

TID: [-1234] [] [2016-10-11 13:33:47,316]  INFO {org.apache.synapse.mediators.builtin.LogMediator} -  To: /services/sampleConnectorProxy.sampleConnectorProxyHttpSoap12Endpoint, WSAction: urn:mediate, SOAPAction: urn:mediate, MessageID: urn:uuid:e3b5f31c-37b8-4ca8-b483-9bbc4a6daf04, Direction: request, *******template_param******** = Hi, Envelope: <?xml version='1.0' encoding='utf-8'?><soapenv:Envelope xmlns:soapenv="http://www.w3.org/2003/05/soap-envelope"><soapenv:Body/></soapenv:Envelope> {org.apache.synapse.mediators.builtin.LogMediator}

TID: [-1234] [] [2016-10-11 13:33:47,316]  INFO {org.wso2.carbon.connector.sampleConnector} -  sample sample connector received message :Hi {org.wso2.carbon.connector.sampleConnector}

To do modifications to the sample connector and experiment, you can set up your development environment. I'm using Intellij IDEA Community Edition. Refer to the following command.

mvn clean install idea:idea

Learn more about the components inside the connector.

https://docs.wso2.com/display/ESBCONNECTORS/Writing+a+Connector

Creating a third party connector and publishing it in WSO2 store.

https://docs.wso2.com/display/ESBCONNECTORS/Creating+a+Third+Party+Connector+and+Publishing+in+WSO2+Store