Friday, March 20, 2015

Using file system as transport medium with WSO2 ESB VFS Transport

In this article I'm going to show you how to use the file system as transport medium using WSO2 ESB 4.8.1 VFS Transport.

Brief Overview of the scenario

By using sample SimpleStockQuoteService service we are going to test this scenario. We will send a SOAP request to the SimpleStockQuoteService and get the response back from it. Here both the request and the response will be saved in separate files.

I'm going to create a directory structure as follows.
Suhans-MacBook-Pro:test suhanr$ tree
.
├── in
├── original
└── out

i.e.,
  1. /Users/suhanr/Desktop/test/in - Location to put SOAP request xml files to be processed.
  2. /Users/suhanr/Desktop/test/original - Location to put processed xml files (which were in ../in folder before processing).
  3. /Users/suhanr/Desktop/test/out - Location to store the responses received from service as xml files.
Our proxy will automatically poll for files in the given location (i.e. ../in location above). I have set the transport.PollInterval to 15 seconds in this scenario which is the default interval. You don't have to know about this parameter to followup this article.

If a file is found in ../in location, it will be processed and will be moved to ../original folder then the response is saved in a file in ../out folder.

Steps to follow

1. Get a fresh WSO2 ESB 4.8.1 pack. You can download it from here [1].

2. Extract it to any directory convenient to you (without containing spaces to the path of your extracted folder). Lets call this path ESB_HOME.
e.g.: my ESB_HOME > /WSO2/ESB/demo/simplevfs/wso2esb-4.8.1

3. Prepare the SimpleStockQuoteService service [2].

i) Go to ESB_HOME/samples/axis2Server/src/SimpleStockQuoteService
you can see the files and folders as follows.
build.xml conf      src

 ii) Issue an ant command to build the SimpleStockQuoteService.aar
Suhans-MacBook-Pro:SimpleStockQuoteService suhanr$ ant
Buildfile: /WSO2/ESB/demo/simplevfs/wso2esb-4.8.1/samples/axis2Server/src/SimpleStockQuoteService/build.xml

clean:

init:
    [mkdir] Created dir: /WSO2/ESB/demo/simplevfs/wso2esb-4.8.1/samples/axis2Server/src/SimpleStockQuoteService/temp
    [mkdir] Created dir: /WSO2/ESB/demo/simplevfs/wso2esb-4.8.1/samples/axis2Server/src/SimpleStockQuoteService/temp/classes
    [mkdir] Created dir: /WSO2/ESB/demo/simplevfs/wso2esb-4.8.1/samples/axis2Server/repository/services
compile-all:
    [javac] /WSO2/ESB/demo/simplevfs/wso2esb-4.8.1/samples/axis2Server/src/SimpleStockQuoteService/build.xml:47: warning: 'includeantruntime' was not set, defaulting to build.sysclasspath=last; set to false for repeatable builds
    [javac] Compiling 9 source files to /WSO2/ESB/demo/simplevfs/wso2esb-4.8.1/samples/axis2Server/src/SimpleStockQuoteService/temp/classes
build-service:
    [mkdir] Created dir: /WSO2/ESB/demo/simplevfs/wso2esb-4.8.1/samples/axis2Server/src/SimpleStockQuoteService/temp/SimpleStockQuote
    [mkdir] Created dir: /WSO2/ESB/demo/simplevfs/wso2esb-4.8.1/samples/axis2Server/src/SimpleStockQuoteService/temp/SimpleStockQuote/META-INF
     [copy] Copying 1 file to /WSO2/ESB/demo/simplevfs/wso2esb-4.8.1/samples/axis2Server/src/SimpleStockQuoteService/temp/SimpleStockQuote/META-INF
     [copy] Copying 9 files to /WSO2/ESB/demo/simplevfs/wso2esb-4.8.1/samples/axis2Server/src/SimpleStockQuoteService/temp/SimpleStockQuote
      [jar] Building jar: /WSO2/ESB/demo/simplevfs/wso2esb-4.8.1/samples/axis2Server/repository/services/SimpleStockQuoteService.aar
BUILD SUCCESSFUL
Total time: 1 second

Now your SimpleStockQuoteService.aar jar is copied to services.

4. Start axis2 server

i) Go to ESB_HOME/samples/axis2Server.

ii) Start server by issuing the following command.
Suhans-MacBook-Pro:axis2Server suhanr$ sh axis2server.sh 
 Using JAVA_HOME:   /Library/Java/JavaVirtualMachines/jdk1.7.0_67.jdk/Contents/Home
 Using AXIS2 Repository :   /WSO2/ESB/demo/simplevfs/wso2esb-4.8.1/samples/axis2Server/repository
 Using AXIS2 Configuration :   /WSO2/ESB/demo/simplevfs/wso2esb-4.8.1/samples/axis2Server/repository/conf/axis2.xml
15/03/20 10:41:31 INFO util.SampleAxis2ServerManager: [SimpleAxisServer] Starting
[SimpleAxisServer] Using the Axis2 Repository : /WSO2/ESB/demo/simplevfs/wso2esb-4.8.1/samples/axis2Server/repository
[SimpleAxisServer] Using the Axis2 Configuration File : /WSO2/ESB/demo/simplevfs/wso2esb-4.8.1/samples/axis2Server/repository/conf/axis2.xml
15/03/20 10:41:31 INFO deployment.ModuleDeployer: Deploying module: addressing - file:/WSO2/ESB/demo/simplevfs/wso2esb-4.8.1/samples/axis2Server/repository/modules/addressing.mar
15/03/20 10:41:31 INFO deployment.ModuleDeployer: Deploying module: rampart - file:/WSO2/ESB/demo/simplevfs/wso2esb-4.8.1/samples/axis2Server/repository/modules/rampart.mar
15/03/20 10:41:32 INFO deployment.ModuleDeployer: Deploying module: sandesha2 - file:/WSO2/ESB/demo/simplevfs/wso2esb-4.8.1/samples/axis2Server/repository/modules/sandesha2.mar
15/03/20 10:41:32 INFO deployment.ModuleDeployer: Deploying module: addressing - file:/WSO2/ESB/demo/simplevfs/wso2esb-4.8.1/repository/components/plugins/org.wso2.carbon.addressing_4.2.0.jar
15/03/20 10:41:32 INFO deployment.ModuleDeployer: Deploying module: wso2caching - file:/WSO2/ESB/demo/simplevfs/wso2esb-4.8.1/repository/components/plugins/org.wso2.carbon.caching.core_4.2.0.jar
15/03/20 10:41:32 INFO deployment.ModuleDeployer: Deploying module: ComponentMgtModule - file:/WSO2/ESB/demo/simplevfs/wso2esb-4.8.1/repository/components/plugins/org.wso2.carbon.feature.mgt.services_4.2.0.jar
15/03/20 10:41:32 INFO deployment.ModuleDeployer: Deploying module: wso2mex - file:/WSO2/ESB/demo/simplevfs/wso2esb-4.8.1/repository/components/plugins/org.wso2.carbon.mex_4.2.0.jar
15/03/20 10:41:32 INFO deployment.ModuleDeployer: Deploying module: pagination - file:/WSO2/ESB/demo/simplevfs/wso2esb-4.8.1/repository/components/plugins/org.wso2.carbon.registry.server_4.2.0.jar
15/03/20 10:41:32 INFO deployment.ModuleDeployer: Deploying module: relay - file:/WSO2/ESB/demo/simplevfs/wso2esb-4.8.1/repository/components/plugins/org.wso2.carbon.relay.module_4.2.0.jar
15/03/20 10:41:32 INFO deployment.ModuleDeployer: Deploying module: sandesha2 - file:/WSO2/ESB/demo/simplevfs/wso2esb-4.8.1/repository/components/plugins/org.wso2.carbon.rm_4.2.0.jar
15/03/20 10:41:32 INFO deployment.ModuleDeployer: Deploying module: POXSecurityModule - file:/WSO2/ESB/demo/simplevfs/wso2esb-4.8.1/repository/components/plugins/org.wso2.carbon.security.mgt_4.2.2.jar
15/03/20 10:41:32 INFO deployment.ModuleDeployer: Deploying module: ServerAdminModule - file:/WSO2/ESB/demo/simplevfs/wso2esb-4.8.1/repository/components/plugins/org.wso2.carbon.server.admin_4.2.0.jar
15/03/20 10:41:32 INFO deployment.ModuleDeployer: Deploying module: wso2statistics - file:/WSO2/ESB/demo/simplevfs/wso2esb-4.8.1/repository/components/plugins/org.wso2.carbon.statistics_4.2.2.jar
15/03/20 10:41:32 INFO deployment.ModuleDeployer: Deploying module: wso2throttle - file:/WSO2/ESB/demo/simplevfs/wso2esb-4.8.1/repository/components/plugins/org.wso2.carbon.throttle.core_4.2.0.jar
15/03/20 10:41:32 INFO deployment.ModuleDeployer: Deploying module: usagethrottling - file:/WSO2/ESB/demo/simplevfs/wso2esb-4.8.1/repository/components/plugins/org.wso2.carbon.throttling.agent_2.2.0.jar
15/03/20 10:41:32 INFO deployment.ModuleDeployer: Deploying module: wso2tracer - file:/WSO2/ESB/demo/simplevfs/wso2esb-4.8.1/repository/components/plugins/org.wso2.carbon.tracer_4.2.0.jar
15/03/20 10:41:32 INFO deployment.ModuleDeployer: Deploying module: metering - file:/WSO2/ESB/demo/simplevfs/wso2esb-4.8.1/repository/components/plugins/org.wso2.carbon.usage.agent_2.2.0.jar
15/03/20 10:41:32 INFO deployment.ModuleDeployer: Deploying module: wso2xfer - file:/WSO2/ESB/demo/simplevfs/wso2esb-4.8.1/repository/components/plugins/org.wso2.carbon.xfer_4.2.0.jar
15/03/20 10:41:32 INFO deployment.ModuleDeployer: Deploying module: rampart - file:/WSO2/ESB/demo/simplevfs/wso2esb-4.8.1/repository/components/plugins/rampart-core_1.6.1.wso2v12.jar
15/03/20 10:41:32 INFO deployment.ModuleDeployer: Deploying module: rahas - file:/WSO2/ESB/demo/simplevfs/wso2esb-4.8.1/repository/components/plugins/rampart-trust_1.6.1.wso2v12.jar
15/03/20 10:41:32 ERROR sandesha2.SandeshaModule: Could not load module policies. Using default values.
15/03/20 10:41:32 INFO config.ClientConnFactoryBuilder: HTTPS Loading Identity Keystore from : ../../repository/resources/security/wso2carbon.jks
15/03/20 10:41:32 INFO config.ClientConnFactoryBuilder: HTTPS Loading Trust Keystore from : ../../repository/resources/security/client-truststore.jks
15/03/20 10:41:32 INFO nhttp.HttpCoreNIOSender: HTTPS Sender starting
15/03/20 10:41:32 INFO nhttp.HttpCoreNIOSender: HTTP Sender starting
15/03/20 10:41:32 INFO jms.JMSSender: JMS Sender started
15/03/20 10:41:32 INFO jms.JMSSender: JMS Transport Sender initialized...
15/03/20 10:41:32 INFO deployment.DeploymentEngine: Deploying Web service: SimpleStockQuoteService.aar - file:/WSO2/ESB/demo/simplevfs/wso2esb-4.8.1/samples/axis2Server/repository/services/SimpleStockQuoteService.aar
15/03/20 10:41:33 INFO nhttp.HttpCoreNIOListener: HTTPS Listener started on 0:0:0:0:0:0:0:0:9002
15/03/20 10:41:33 INFO nhttp.HttpCoreNIOListener: HTTP Listener started on 0:0:0:0:0:0:0:0:9000
15/03/20 10:41:33 INFO util.SampleAxis2ServerManager: [SimpleAxisServer] Started

5. Configure VFS Transport
It is time to configure your VFS Transport in axis2.xml file [3]

i) Go to ESB_HOME/repository/conf/axis2

ii) Open axis2.xml file to edit

Suhans-MacBook-Pro:wso2esb-4.8.1 suhanr$ vi repository/conf/axis2/axis2.xml

iii) Uncomment the VFS listener and VFS sender as follows.

<transportreceiver name="vfs" class="org.apache.synapse.transport.vfs.VFSTransportListener"/>
...
<transportSender name="vfs" class="org.apache.synapse.transport.vfs.VFSTransportSender"/>

iv) Save the changes

6. Create the SOAP request input xml file

i) Create the folder structure as already shown in the overview section at the top of this article. You can rename these folders as you like and can create anywhere which is accessible by the server.
i.e. 3 folders namely, in, original and out.

 ii) Go to ../in folder that you have created and create a file named test.xml and copy the following content to it.
<?xml version='1.0' encoding='UTF-8'?>
<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:wsa="http://www.w3.org/2005/08/addressing">
    <soapenv:Body>
        <m0:getQuote xmlns:m0="http://services.samples">
            <m0:request>
                <m0:symbol>WSO2</m0:symbol>
            </m0:request>
        </m0:getQuote>
    </soapenv:Body>
</soapenv:Envelope>

7. After adding the StockQuoteProxy proxy configuration, your ESB synapse configuration will look like as follows.
<?xml version="1.0" encoding="UTF-8"?>
<definitions xmlns="http://ws.apache.org/ns/synapse">
   <registry provider="org.wso2.carbon.mediation.registry.WSO2Registry">
      <parameter name="cachableDuration">15000</parameter>
   </registry>
   <proxy name="StockQuoteProxy" transports="vfs" startOnLoad="true">
      <target>
         <endpoint>
            <address uri="http://localhost:9000/services/SimpleStockQuoteService"
                     format="soap12"/>
         </endpoint>
         <outSequence>
            <property name="transport.vfs.ReplyFileName"
                      expression="fn:concat(fn:substring-after(get-property('MessageID'), 'urn:uuid:'), '.xml')"
                      scope="transport"/>
            <property name="OUT_ONLY" value="true"/>
            <send>
               <endpoint>
                  <address uri="vfs:file:///Users/suhanr/Desktop/test/out"/>
               </endpoint>
            </send>
         </outSequence>
      </target>
      <publishWSDL uri="file:repository/samples/resources/proxy/sample_proxy_1.wsdl"/>
      <parameter name="transport.PollInterval">15</parameter>
      <parameter name="transport.vfs.ActionAfterProcess">MOVE</parameter>
      <parameter name="transport.vfs.FileURI">file:///Users/suhanr/Desktop/test/in</parameter>
      <parameter name="transport.vfs.MoveAfterProcess">file:///Users/suhanr/Desktop/test/original</parameter>
      <parameter name="transport.vfs.MoveAfterFailure">file:///Users/suhanr/Desktop/test/original</parameter>
      <parameter name="transport.vfs.FileNamePattern">.*\.xml</parameter>
      <parameter name="transport.vfs.ContentType">text/xml</parameter>
      <parameter name="transport.vfs.ActionAfterFailure">MOVE</parameter>
   </proxy>
   <sequence name="fault">
      <log level="full">
         <property name="MESSAGE" value="Executing default 'fault' sequence"/>
         <property name="ERROR_CODE" expression="get-property('ERROR_CODE')"/>
         <property name="ERROR_MESSAGE" expression="get-property('ERROR_MESSAGE')"/>
      </log>
      <drop/>
   </sequence>
   <sequence name="main">
      <in>
         <log level="full"/>
         <filter source="get-property('To')" regex="http://localhost:9000.*">
            <send/>
         </filter>
      </in>
      <out>
         <send/>
      </out>
      <description>The main sequence for the message mediation</description>
   </sequence>
</definitions>
You can copy this configuration easily by clicking on the "view plain" link at the top left corner of the code view.

Once you save the configuration, within 15 seconds time your file will be processed. Now lets verify the results.

Analysing the output

Once you save your configuration, ESB console output will be as follows.
[2015-03-20 11:32:53,581]  INFO - SynapseTaskManager Shutting down the task manager
[2015-03-20 11:32:53,583]  INFO - XMLConfigurationBuilder Generating the Synapse configuration model by parsing the XML configuration
[2015-03-20 11:32:53,589]  INFO - ProxyService Building Axis service for Proxy service : StockQuoteProxy
[2015-03-20 11:32:53,610]  INFO - ProxyService Adding service StockQuoteProxy to the Axis2 configuration
[2015-03-20 11:32:53,622]  INFO - DeploymentInterceptor Deploying Axis2 service: StockQuoteProxy {super-tenant}
[2015-03-20 11:32:53,695]  INFO - ProxyService Successfully created the Axis2 service for Proxy service : StockQuoteProxy

When the file is processed, you can see this output in your simple Axis2 server console.

Fri Mar 20 11:33:08 IST 2015 samples.services.SimpleStockQuoteService :: Generating quote for : WSO2

Check your folder tree structure as follows.
Suhans-MacBook-Pro:test suhanr$ tree
.
├── in
├── original
│   ├── test.xml
└── out
    └── a9b795da-a668-4b6a-ac8a-90db42d7311d.xml

As you can see the test.xml file is processed and moved to the ../original folder.
The response received from the StockQuoteProxy is placed in ../out folder as a9b795da-a668-4b6a-ac8a-90db42d7311d.xml

Response file contents is as follows.

   
      
         
            -2.378108963314817
            12.594090499041867
            -79.46865111676226
            79.76869958422132
            Fri Mar 20 11:33:08 IST 2015
            -79.2594140983345
            -7611066.458300805
            WSO2 Company
            81.77710215829231
            24.900435994896995
            3.054782179111802
            -77.84872452039339
            WSO2
            7308
Improvements

You can store the endpoint in governance registry or configuration registry [4].
e.g.: Governance registry endpoint: "StockQuoteEP"
<endpoint xmlns="http://ws.apache.org/ns/synapse">
   <address uri="http://localhost:9000/services/SimpleStockQuoteService" format="soap12">
      <suspendOnFailure>
         <progressionFactor>1.0</progressionFactor>
      </suspendOnFailure>
      <markForSuspension>
         <retriesBeforeSuspension>0</retriesBeforeSuspension>
         <retryDelay>0</retryDelay>
      </markForSuspension>
   </address>
</endpoint>
Modified proxy service will be as follows. 
<?xml version="1.0" encoding="UTF-8"?>
<definitions xmlns="http://ws.apache.org/ns/synapse">
   <registry provider="org.wso2.carbon.mediation.registry.WSO2Registry">
      <parameter name="cachableDuration">15000</parameter>
   </registry>
   <proxy name="StockQuoteProxy" transports="vfs" startOnLoad="true">
      <target>
         <endpoint key="gov:/StockQuoteEP"/>
         <outSequence>
            <property name="transport.vfs.ReplyFileName"
                      expression="fn:concat(fn:substring-after(get-property('MessageID'), 'urn:uuid:'), '.xml')"
                      scope="transport"/>
            <property name="OUT_ONLY" value="true"/>
            <send>
               <endpoint>
                  <address uri="vfs:file:///Users/suhanr/Desktop/test/out"/>
               </endpoint>
            </send>
         </outSequence>
      </target>
      <publishWSDL uri="file:repository/samples/resources/proxy/sample_proxy_1.wsdl"/>
      <parameter name="transport.PollInterval">15</parameter>
      <parameter name="transport.vfs.ActionAfterProcess">MOVE</parameter>
      <parameter name="transport.vfs.FileURI">file:///Users/suhanr/Desktop/test/in</parameter>
      <parameter name="transport.vfs.MoveAfterProcess">file:///Users/suhanr/Desktop/test/original</parameter>
      <parameter name="transport.vfs.MoveAfterFailure">file:///Users/suhanr/Desktop/test/original</parameter>
      <parameter name="transport.vfs.FileNamePattern">.*\.xml</parameter>
      <parameter name="transport.vfs.ContentType">text/xml</parameter>
      <parameter name="transport.vfs.ActionAfterFailure">MOVE</parameter>
   </proxy>
   <sequence name="fault">
      <log level="full">
         <property name="MESSAGE" value="Executing default 'fault' sequence"/>
         <property name="ERROR_CODE" expression="get-property('ERROR_CODE')"/>
         <property name="ERROR_MESSAGE" expression="get-property('ERROR_MESSAGE')"/>
      </log>
      <drop/>
   </sequence>
   <sequence name="main">
      <in>
         <log level="full"/>
         <filter source="get-property('To')" regex="http://localhost:9000.*">
            <send/>
         </filter>
      </in>
      <out>
         <send/>
      </out>
      <description>The main sequence for the message mediation</description>
   </sequence>
</definitions>
Future Improvements

ESB 4.8.1 DOES NOT support fetching transport.vfs.FileURI from registry (Currently ESB does not support fetching service parameters from registry).
However in the next release i.e. ESB 4.9.0 we will be able to cater this requirement through inboundEndpoint [5].

Sample configuration is given below as per [5] (Applicable to ESB 4.9.0 and later releases only).
Please note that the inboundEndpoint configuration shown below is subjected to change since ESB 4.9.0 is not released at the time of writing this article.

<inboundEndpoint xmlns="http://ws.apache.org/ns/synapse";  name="wso2File"
 sequence="request"  onError="fault"  protocol="file"
 suspend="false">
   <parameters>
      <parameter name="interval">1000</parameter>
      <parameter name="transport.vfs.ActionAfterErrors">NONE</parameter>
      <parameter name="transport.vfs.LockReleaseSameNode">false</parameter>
      <parameter name="transport.vfs.AutoLockRelease">false</parameter>
      <parameter name="transport.vfs.ActionAfterFailure">NONE</parameter>
      <parameter name="transport.vfs.ActionAfterProcess">NONE</parameter>
      <parameter name="sequential">false</parameter>
      <parameter name="transport.vfs.FileURI"
*key="conf:/repository/esb/esb-configurations/test"*/>
      <parameter name="transport.vfs.DistributedLock">false</parameter>
      <parameter name="transport.vfs.Locking">enable</parameter>
   </parameters>
</inboundEndpoint

Summary

In this article we have looked at using the file system as a transport medium using WSO2 ESB 4.8.1 VFS Transport. For this use case we have used the Sample SimpleStockQuoteService service. SOAP request is stored in an xml file and the response received from the SimpleStockQuoteService is also stored in an xml file.

Links

Posted by at
Labels: , , , , ,

No comments:

Post a Comment

Subscribe to: Post Comments (Atom)