Wednesday, November 16, 2016

Distributed Transaction (XATransaction) For JMS Trasnport- WSO2 ESB 5.0.0


Distributed Transaction (XATransaction) For JMS Trasnport- WSO2 ESB 5.0.0

Distributed transactions plays a major role in today's business world. It adds a very important functionality to your system, which you will only see in a system crash. Having good transactional support in your system will definitely save your business in a system crash.

Distributed JMS transaction

An external transaction manager manages the coordination of the transaction. Designing and using JMS distributed transactions is more complex than using local JMS transactions.

The transaction manager is the primary component of the distributed transaction infrastructure and distributed JMS transactions are managed by the XAResource enabled transaction manager in the J2EE application server. Also note that you need to check if your message broker supports XA transactions.

XA two-phase commit process

The two-phase commit consists of the following steps:
  • Immediately after the transaction begins, the transaction manager invokes start() on the JMS XA resource, which indicates that the resource should initialize a new transaction. The JMS XA resource now generates a new transaction ID and sends it over the network to the remote broker.
  • The JMS XA resource now forwards all of the operations that arise during a JMS session (for example, messages, acknowledgments, and so on) to the remote broker.
  • On the broker side, the received operations are not performed immediately. Because the operations are happening in a transaction context and the transaction is not yet committed, the broker buffers all of the operations in a transaction store (held in memory, initially). Messages held in the transaction store are not forwarded to JMS consumers.
  • In a two-phase commit process, the first phase of completing the transaction is where the transaction manager invokes prepare() on all of the participating XA resources. At this stage, the JMS XA resource sends the prepare() operation to the remote broker.
  • On the broker side, when the transaction store receives the prepare() operation, it writes all of the buffered operations to disk. Hence, after the prepare phase, there is no longer any risk of losing data associated with this transaction branch.
  • The second phase of completing the transaction is where the transaction manager invokes commit() on all of the participating XA resources. The JMS XA resource sends the commit() operation to the remote broker.
  • On the broker side, the transaction store marks this transaction as complete. The pending operations are now executed and any pending messages can now be forwarded to JMS consumers.
Please follow documentation to configure WSO2 esb with activemq browser.
XA transaction example.

ESB listen to message queue and send that message to multiple queues.  if something went wrong in sending the message to one of those queues, the original message should be rollbacked to Listening queue and none of the queues should receive the message. The entire transaction should be rollbacked.



The following example code shows the configuration of ESB which listen to a JMS queue and consume messages as well as to send messages to multiple JMS queues in transactional manner.

<proxy xmlns="http://ws.apache.org/ns/synapse"
       name="JMSListenerProxy"
       transports="https http jms"
       startOnLoad="true">
   <description/>
   <target>
      <inSequence>
         <property name="OUT_ONLY" value="true"/>
         <log level="custom">
            <property name="MESSAGE_ID_A" expression="get-property('MessageID')"/>
         </log>
         <log level="custom">
            <property name="BEFORE" expression="$body"/>
         </log>
         <property name="MESSAGE_ID_B"
                   expression="get-property('MessageID')"
                   scope="operation"
                   type="STRING"/>
         <property name="failureResultProperty"
                   scope="default"
                   description="FailureResultProperty">
            <result xmlns="">failure</result>
         </property>
         <enrich>
            <source clone="true" xpath="$ctx:failureResultProperty"/>
            <target type="body"/>
         </enrich>
         <log level="custom">
            <property name="AFTER" expression="$body"/>
         </log>
         <property name="BEFORE1" value="ABCD" scope="axis2" type="STRING"/>
         <callout serviceURL="jms:/ActiveMQPublisher1?transport.jms.ConnectionFactoryJNDIName=XAConnectionFactory&amp;java.naming.factory.initial=org.apache.activemq.jndi.ActiveMQInitialContextFactory&amp;java.naming.provider.url=tcp://localhost:61616&amp;transport.jms.DestinationType=queue;transport.jms.TransactionCommand=begin">
            <source type="envelope"/>
            <target xmlns:s12="http://www.w3.org/2003/05/soap-envelope"
                    xmlns:s11="http://schemas.xmlsoap.org/soap/envelope/"
                    xpath="s11:Body/child::*[fn:position()=1] | s12:Body/child::*[fn:position()=1]"/>
         </callout>
         <callout serviceURL="jms:/ActiveMQPublisher2?transport.jms.ConnectionFactoryJNDIName=XAConnectionFactory&amp;java.naming.factory.initial=org.apache.activemq.jndi.ActiveMQInitialContextFactory&amp;java.naming.provider.url=tcp://localhost:61616&amp;transport.jms.DestinationType=queue">
            <source type="envelope"/>
            <target xmlns:s12="http://www.w3.org/2003/05/soap-envelope"
                    xmlns:s11="http://schemas.xmlsoap.org/soap/envelope/"
                    xpath="s11:Body/child::*[fn:position()=1] | s12:Body/child::*[fn:position()=1]"/>
         </callout>
         <callout serviceURL="jms:/ActiveMQPublisher3?transport.jms.ConnectionFactoryJNDIName=XAConnectionFactory&amp;java.naming.factory.initial=org.apache.activemq.jndi.ActiveMQInitialContextFactory&amp;java.naming.provider.url=tcp://localhost:61616&amp;transport.jms.DestinationType=queue;transport.jms.TransactionCommand=end">
            <source type="envelope"/>
            <target xmlns:s12="http://www.w3.org/2003/05/soap-envelope"
                    xmlns:s11="http://schemas.xmlsoap.org/soap/envelope/"
                    xpath="s11:Body/child::*[fn:position()=1] | s12:Body/child::*[fn:position()=1]"/>
         </callout>
         <drop/>
      </inSequence>
      <faultSequence>
         <log level="custom">
            <property name="Transaction Action" value="Rollbacked"/>
         </log>
         <callout serviceURL="jms:/ActiveMQPublisherFault?transport.jms.ConnectionFactoryJNDIName=XAConnectionFactory&amp;java.naming.factory.initial=org.apache.activemq.jndi.ActiveMQInitialContextFactory&amp;java.naming.provider.url=tcp://localhost:61616&amp;transport.jms.DestinationType=queue;transport.jms.TransactionCommand=rollback">
            <source type="envelope"/>
            <target xmlns:s12="http://www.w3.org/2003/05/soap-envelope"
                    xmlns:s11="http://schemas.xmlsoap.org/soap/envelope/"
                    xpath="s11:Body/child::*[fn:position()=1] | s12:Body/child::*[fn:position()=1]"/>
         </callout>
      </faultSequence>
   </target>
   <parameter name="transport.jms.ContentType">
      <rules>
         <jmsProperty>contentType</jmsProperty>
         <default>application/xml</default>
      </rules>
   </parameter>
   <parameter name="transport.jms.Destination">MyJMSQueue</parameter>
</proxy>

To place a message into a JMS queue, execute following command from /samples/axis2Client directory.

ant stockquote -Dmode=placeorder -Dtrpurl="jms:/MyJMSQueue?transport.jms.ConnectionFactoryJNDIName=QueueConnectionFactory&java.naming.factory.initial=org.apache.activemq.jndi.ActiveMQInitialContextFactory&java.naming.provider.url=tcp://localhost:61616&transport.jms.ContentTypeProperty=Content-Type&transport.jms.DestinationType=queue"

You should be able to observe ESB consumes messages from MYJMSQueue and sends to multiple queues. To check rollback functionality provide unreachable host name to any destination queue and save the configurations. You should be able to observe ESB fault sequence getting invoked and failed message delivered to configured destination within fault sequence
Anyway,
  • The problem with XA is it can be a bit slow; as the XA protocol requires multiple syncs to disk to ensure it can always recover properly under every possible failure scenario. This adds significant cost (in terms of latency, performance, resources and complexity). Also quite a few EJB servers and databases don't actually properly support XA
  • So a good optimisation is to use regular JMS transactions - with no XA - and just perform some duplicate message detection in your code to check you have not already processed the message. (ref ActiveMQ)

Friday, October 14, 2016

Amazon API GW integration with WSO2 IS for OAuth 2.0 token validation

In this blog thought of discussing another interesting topic which is relate token validation and how you can register a custom Authorizers via lambda functions to the Amazon API Gateway and use WSO2 Identity Server as authorization server.

  So, Following steps will guide you on how to use WSO2 IS as a custom authorize for Amazon APIs.

Setup WSO2 Identity Server
  • We will be generating tokens in WSO2 IS and using them to call APIs configured in Amazon API Gateway Preparing WSO2 IS And Generating tokens 
  • When testing WSO2 IS for Amazon API Gateway, we used OAuth 2.0 Token Introspection API. You need host  (introspect.war) ​ that you need to deploy in IS. 
  • The source for this is found in [1]
    • Extract the downloaded zip file to a convenient file location. This distributed folder (by default the folder name is wso2is-5.2.0​ ) will be referred to as throughout the document.
    • Copy introspect.war file to /repository/deployment/server/webapps location. In order to generate an access token we need to create an Oauth2 application and get client credentials from there
    • Start the server and login to the management console with admin:admin credentials. Steps to start the product are in [2]. Also you can find the complete installation guide in [3] Note : In order to be accessed by Amazon Gateway Lambda function, this up and running Identity Server should be hosted with a public URL 4. In the main menu in management console, click “Add” under Service Provider
  • Give a Service Provider name and click Register. You will see the Service Provider
    Configuration page.
  • Expand Inbound Authentication Configuration​ panel.
  • Expand the OAuth/OpenID Connect Configuration​ and click Configure.
  • Fill in the form that appears. For the Allowed Grant Types​ you can disable the ones you do not require or wish to block.
  • Use following request to generate tokens with password grant type
 Replace and values with admin. Replace with Base64 encoded values

curl -k -d
"grant_type=password&username=<USERNAME>&password=<PASSWORD>&scope=<SCOPE>"​ ​-H
"Authorization: Basic <ENCODED_KEY>"​ ​-H "Content-Type:
application/x-www-form-urlencoded"
https://<IS_HOST>t:<IS_HTTPS_PORT>/oauth2/token
will have a response similar to,
curl -k -d "grant_type=password&username=admin&password=admin&scope=default "​ ​-H
"Authorization: Basic
X1ZpREJUMWJUSHF5eXFfR1Y0UWJoc0V6X1IwYTpLdzIxV1JPRmYyeTc4RGViMXY0UGpoRkdydGhq"​ ​-H
"Content-Type: application/x-www-form-urlencoded"
https://localhost:9443/oauth2/token
This will have a response similar to,
{
 "scope": "default"
 "token_type": "Bearer"
 "expires_in": 956
 "refresh_token": "23444-dff-32b0-b4cf-7dc7d8fd8205"
 "access_token": "455533ed-d222-3b0e-9512-fec0b94c1592"
}

  • We will use this access_token value when invoking APIs in AWS side.

Configuring Amazon API Gateway

  • Goto Lambda console in Amazon API Gateway and create new Lambda function.

  • This function should call Token Introspection API of WSO2 IS Server. I have attached a sample lambda function (Lambda.js) which is calling this API. You can copy the source of this file and
console.log('starting lambda');
var http = require('http');
exports.handler = function(event, context) {
    var tkn = event.authorizationToken;
 var postData = 'token='+tkn;
 console.log(postData);

 var options = {
   hostname: 'ip.address',
   port: 9763,
   path: '/introspect',
   method: 'POST',
   headers: {
     'Content-Type': 'application/x-www-form-urlencoded'
   }
 };

 var req = http.request(options, (res) => {
   console.log(`STATUS: ${res.statusCode}`);
   console.log(`HEADERS: ${JSON.stringify(res.headers)}`);
   res.setEncoding('utf8');
   res.on('data', (chunk) => {
     console.log(`BODY: ${JSON.stringify(chunk)}`);
  validationReq(chunk,event,context);
   });
   res.on('end', () => {
     console.log('No more data in response.');
   });
 });

 req.on('error', (e) => {
   console.log(`problem with request: ${e.message}`);
 });

 // write data to request body
 req.write(postData);
 req.end();

}

var validationReq = function (obj,evt,ctx) {
    console.log(obj);
    obj = JSON.parse(obj);

    var bool = obj['active'];
 if(bool) {
     console.log('Token verified');
     ctx.succeed(generatePolicy('user', 'Allow', evt.methodArn));
 } else {
     ctx.fail("Unauthorized");
 }
 
}


var generatePolicy = function(principalId, effect, resource) {
    var authResponse = {};
    console.log(resource);
    authResponse.principalId = principalId;
    if (effect && resource) {
        var policyDocument = {};
        policyDocument.Version = '2012-10-17'; // default version
        policyDocument.Statement = [];
        var statementOne = {};
        statementOne.Action = 'execute-api:Invoke'; // default action
        statementOne.Effect = effect;
        statementOne.Resource = resource;
        policyDocument.Statement[0] = statementOne;
        authResponse.policyDocument = policyDocument;
    }
    return authResponse;
}






  • Copy it as the Lambda function code. Note that you have to replace <host_name of publiclyhosted wso2 is> and <port of publicly hosted wso2 is> values with actual host and port before using it.
  •  Save the Lambda function.
  • Follow Configure Custom Authorizer Using the API Gateway Console ​in [1] When creating the custom authorizer, make sure you give Lamba Region and Lambda Function details according to the function you created in Step 1.







  • After above configurations are done, you should have an API deployed in Amazon API
  • Gateway which is configured to use the custom authorizer which calls WSO2 IS to authorize. 
  • Invoke this API giving the access_token obtained from WSO2 IS as the Identity token source header value you configured when above step.
 Note:- How Please refer OAuth 2.0 Token Introspection API for WSO2 Identity Server to understand how introspect API calls would validate the access tokens and responses
Empty token
curl -k -H 'Content-Type: application/x-www-form-urlencoded' -X POST --data 'token=' https://localhost:9443/introspect
Response: {"active":false} 
Invalid token: 
curl -k -H 'Content-Type: application/x-www-form-urlencoded' -X POST --data 'token=Bjhk98792k9hkjhk' https://localhost:9443/introspect 
Response: {"active":false,"token_type":"bearer"} 
Get a valid token: 
curl -v -X POST --basic -u client_id:client_secret -H "Content-Type: application/x-www-form-urlencoded;charset=UTF-8" -k -d "grant_type=client_credentials" https://localhost:9443/oauth2/token Validate the token:
curl -k -H 'Content-Type: application/x-www-form-urlencoded' -X POST --data 'token=99f0a7092c71a6e772cbcf77addd39ea' https://localhost:9443/introspect 
   Response: 
   { "username":"admin@carbon.super", 
     "nbf":3272, 
     "active":true, 
     "token_type":"bearer", 
     "client_id":"LUG28MI5yjL5dATxQWdYGhDLSywa" 
   } 

[1] https://github.com/facilelogin/aratuwa/tree/master/api-security/org.wso2.carbon.identity.oauth.introspection
[2] https://docs.wso2.com/display/IS520/Running+the+Product
[3] https://docs.wso2.com/display/IS520/Installation+Guide
[4] http://docs.aws.amazon.com/apigateway/latest/developerguide/use-custom-authorizer.html

Saturday, June 11, 2016

IoT Use case WSO2

In this blog, I thought of discussing kind of an interesting use case which involves communicating and managing alerts though the concept IoT (Internet Of Things)

The use case which I'm describing involve WSO2 middleware stack, before that lets try to understand use case first.

Story
 A organization called 'Amber Shopping complex' who is well reputed organization doing clothing business. Then recently their management decided to enhance the client experience and planning to bring more user friendly environment when a buyer enters in to the shop complex. So the concept is 'Amber' publish advertisement on TV, News Papers asking to download newly developed mobile application. Since 'Amber' is so  popular, all most all its buyer have download the mobile application.
     Then, once users movies different parts of the shopping complex, the mobile device sending the notification to the 'Amber Shopping IT genius', so what will be happening after that you will be amazed.
  • Client going to buy shirts (which he has already search through mobile device), he received alters from to the mobile application asking his choices, and new arrivals to the store.
  • Client walking pass TV display guess what, the  TV display suddenly personalizing and start to show what he has preferred in the past and choices available and locations (Digital Signage)
  • Client go to different section "let say" kid toys section, client received alerts to his mobile devices, the new toys arrived and prices
 ... and many more other cool stuff...
 Let assume how we can achieve above use case, to accomplish this use case we are using
WSO2 Iot Server , WSO2 CEP, WSO2 MB, WSO2 ESB,WSO2 APIM,WSO2 DAS

I know most of you may already aware others products other than WSO2 Iot server (which of cause yet to be released officially by WSO2), so let me give you a small introduction what WSO2 Iot Server (which is still at Alpha state), if anyone interest they could join to the beta program.

What is IoT server ?

"WSO2 IoT Server (IoTS) is the core of WSO2's Open IoT Platform. It provides the capabilities required by device manufacturers and enterprises to build a complete IoT architecture around their connected device offering. Device manufactures are able to manage the device architecture from the server-side using WSO2 IoTS, extend the IoTS to any device type, and support other protocols and data formats as it runs on top of WSO2 Connected Device Management Framework (WSO2 CDMF).
WSO2 IoTS offers customizable and advanced IoT device analytics via WSO2 Data Analytics Server that include predictive analytics using machine learning capabilities. It also supports edge computing devices, policy-based edge analytics, and pre-built instant visualization for sensor readings using live data streams gathered from devices.
Take a look at the WSO2 IoT Server presentation on Your foundation for the Internet of things to expand your knowledge on IoTS. This is a great place for you to start if you are new to WSO2 IoTS."




If you need more information please visit [1]

Now lets try to see how we could overcome the given use case, please see the below diagram which show complete interaction or solution architecture, that can be implemented to overcome the given use case.

Solution Architectural View




Following diagram illustrating the complete interaction from device registration and alerting model would work

Flow



What we need to understand here is the role which WSO2 Iot server plays, i.e how it interacts with different modules to complete the user interaction, specially we have to isolate use experience for that, it is must that have to maintain a device in which client use along side with some other information.

On the other hand, the for intercommunication,such as mobile devices to message broker we thought of using mqtt protocal (pub/sub)

Why we need MQTT? ref [2]
MQTT was developed by Andy Stanford-Clark (IBM) and Arlen Nipper (Eurotech; now Cirrus Link) in 1999 for the monitoring of an oil pipeline through the desert. The goals were to have a protocol, which is bandwidth-efficient and uses little battery power, because the devices were connected via satellite link and this was extremely expensive at that time.
The protocol uses a publish/subscribe architecture in contrast to HTTP with its request/response paradigm.Publish/Subscribe is event-driven and enables messages to be pushed to clients. The central communication point is the MQTT broker, it is in charge of dispatching all messages between the senders and the rightful receivers. Each client that publishes a message to the broker, includes a topic into the message. The topic is the routing information for the broker. Each client that wants to receive messages subscribes to a certain topic and the broker delivers all messages with the matching topic to the client. Therefore the clients don’t have to know each other, they only communicate over the topic. This architecture enables highly scalable solutions without dependencies between the data producers and the data consumers.
CEP role
CEP (Complex event processor) plays equally important role in this design with its real time analysis capability while alerting other sub modules such as if client move one place to another, mobile device publishes GPS detail and CEP engine keep analyzing them and sends alerts to ESB, where it interact with other sub-modules. (please read [3] for more information)

Finally
If you need to write such requirement from the sketch, it would require a thousand lines of codes may be much more complicated interaction between system. But with above of cause with minimal effort you can simply build the integration layer.




[1]https://docs.wso2.com/display/IoTS100/WSO2+IoT+Server
[2] http://www.hivemq.com/blog/how-to-get-started-with-mqtt 
[3] http://wso2.com/library/blog-post/2013/11/understanding-complex-event-processing-(cep)-operators-with-wso2-cep-(siddhi)/

Monday, May 30, 2016

Troubleshoot and understanding Puppet deployment

Following blog, I will try to describe the steps that you may need to follow up in order to setup a WSO2 product cluster quick time using Puppet scripts.
  Basically, thought this is straight forward you may have to spent few minutes or  some time few hours to get firm to your self. Because if you just start to work these environments thing gonna be bit difficult. In this blog I try to target people who just started to work with puppet and of cause they may be face similar issues.

For the ease of understanding, we have divided the instructions into 3 Tasks.

    Step 1. Setup two instances
    Step 2. Setup Puppet Master and puppet agent
    Step 3. Perform a catalog run to configure product instance

First of all lets try to understand puppet directory structure [1],basically, hiera structure defines the variable and parameters per environment per platform per product such as how it should populate while under module, the templates of each products defined with .erp

If you refer below structure, you will notice something called 'hiera', the term we are using With Hiera, you can externalize your systems' configuration data and easily understand how those values are assigned to your servers. With that data separated from your Puppet code, you then can encrypt sensitive values, such as passwords and keys [2]

Once you navigate, please try to understand the project structure, in this case we are trying to prepare 'dev' environment, so understanding hiera model is important.

 lets see how hiera.yaml definition looks

:hierarchy:
    - "node/%{::clientcert}"
    - "wso2/%{::product_name}/%{::product_version}/%{::platform}/%{::product_profile}"
    - "wso2/%{::product_name}/%{::product_version}/%{::platform}/default"
    - "osfamily/%{::osfamily}"
    - "vm_type/%{::vm_type}"
    - "platform/%{::platform}"
    - wso2/common
    - common
:backends:
    - yaml
:yaml:
    :datadir: "/etc/puppet/hieradata/dev"


In above structure, it would explain how hiera would travel through, then extract variables to prepare the platform that you are running.

Let assume I need to run api-store profile (APIM having store,publisher,gateway,key manager), then once I initiate the product profile from puppet agent (which I will explain later), when puppet agent communicates with puppet master, it would go through above mentioned hierarchy to resolve variables i.e
api-store.yaml --> default.yaml--> common.yaml --> (root common) -->common.yaml

However, first of all we should install puppet do refer [3], follow the steps given

Sometimes you will not succeeded straightforward (As I was struggle for a while to establish connection between puppet agent and master, therefore if you have any trouble please do refer[4])

From my personal  experience,the most common issue you will face

err: Could not request certificate: No route to host - connect(2)
Exiting; failed to retrieve certificate and waitforcert is disabled 


I would say, if you need to resolve this, you don't need to fiddle around much 


There are few steps you need to verify,

Step-1
Please make sure /etc/hosts files of both nodes as given below, make sure you gave puppet master host name correctly

Puppet master
/etc/hosts
127.0.0.1 localhost
127.0.0.1 puppet puppetmaster

/etc/hostname
puppetmaster


Puppet Agent
/etc/hosts
127.0.0.1      localhost

[IP-address of puppetmaster]     puppetmaster
[IP-address of agent]      [Agent-hostage]


Step-2

Verify at puppet agent the puppet.conf file under /etc/puppet/,  the only thing you need to focus [agent] tag and verify the server mapping as highlighted below

[main]
logdir=/var/log/puppet
vardir=/var/lib/puppet
ssldir=/var/lib/puppet/ssl
rundir=/var/run/puppet
factpath=$vardir/lib/facter
templatedir=$confdir/templates
prerun_command=/etc/puppet/etckeeper-commit-pre
postrun_command=/etc/puppet/etckeeper-commit-post

[master]
# These are needed when the puppetmaster is run by passenger
# and can safely be removed if webrick is used.
ssl_client_header = SSL_CLIENT_S_DN
ssl_client_verify_header = SSL_CLIENT_VERIFY

[agent]
server=puppetmaster


Step3
Then at puppet master, /puppet/hieradata/dev/wso2/common.yaml, please check following entry

 # Host mapping to be made in etc/hosts
wso2::hosts_mapping:
  localhost:
    ip_address: 127.0.0.1
    hostname: localhost
  puppetmaster :
    ip_address : [puppet-master-ip]
    hostname : puppetmaster
  pub.am.wso2.com :
    ip_address : [agent-ip]
    hostname : [agent-hostname]

** Make to give correct information under host name, as these gonna replace /etc/hosts  mapping during run time, else thing gonna break up while scripts running

Step4
 Check puppet.conf under puppet master "puppet/puppet.conf", please make sure it has "autosign=true" which required for certification signing for proper master-agent communication

[main]
logdir=/var/log/puppet
vardir=/var/lib/puppet
ssldir=/var/lib/puppet/ssl
rundir=/var/run/puppet
#factpath=$vardir/lib/facter
templatedir=$confdir/templates
dns_alt_names=puppetmaster,puppet
hiera_config = /etc/puppet/hiera.yaml

[master]
# These are needed when the puppetmaster is run by passenger
# and can safely be removed if webrick is used.
ssl_client_header = SSL_CLIENT_S_DN
ssl_client_verify_header = SSL_CLIENT_VERIFY
autosign=true

If you verify above given steps carefully, you should execute following command;

 Puppet Master
  1. puppet cert clean —all
  2.  service puppetmaster restart
  3. ** If you need to see whats going inside puppet master, then you can start "puppet master --no-daemonize --debug"
Puppet Agent
  • "find /var/lib/puppet/ssl -name '*.pem' -exec rm {} \;"  (to clean up invalid certificates)
  • puppet agent -t —debug  (to establish master-agent connection)
Puppet Agent run time
Once verify above, then just run setup.sh, which should start to deploy agent profile while communicating with master

deployment.cofig

product_name=wso2am   // defines the product type
product_version=1.10.0  //define the product version
product_profile=default  //defines the profiles
environment=dev  // the environment
vm_type=openstack
platform=default

setup.sh

#!/bin/bash
echo "#####################################################"
echo "                   Starting cleanup "
echo "#####################################################"
ps aux | grep -i wso2 | awk {'print $2'} | xargs kill -9
#rm -rf /mnt/*
sed -i '/environment/d' /etc/puppet/puppet.conf
echo "#####################################################"
echo "               Setting up environment "
echo "#####################################################"
rm -f /etc/facter/facts.d/deployment_pattern.txt
mkdir -p /etc/facter/facts.d

while read -r line; do declare  $line; done < deployment.conf
echo "reading...."
echo product_name=$product_name product_version=$product_version product_profile=$product_profile
echo product_name=$product_name >> /etc/facter/facts.d/deployment_pattern.txt
echo product_version=$product_version >> /etc/facter/facts.d/deployment_pattern.txt
echo product_profile=$product_profile >> /etc/facter/facts.d/deployment_pattern.txt
echo vm_type=$vm_type >> /etc/facter/facts.d/deployment_pattern.txt
echo environment=$environment >> /etc/facter/facts.d/deployment_pattern.txt
echo platform=$platform >> /etc/facter/facts.d/deployment_pattern.txt

echo "reading end"

echo "#####################################################"
echo "                    Installing "
echo "#####################################################"

puppet agent --enable
puppet agent -vt
puppet agent --disable

Please use [5] to download the puppet-master module which I have used to config, but there you have to copy JDK, products etc

[1]
├── LICENSE
├── README.md
├── hiera.yaml
├── hieradata
│   └── dev
│       ├── common.yaml
│       ├── platform
│       │   └── kubernetes.yaml
│       ├── vm_type
│       │   └── docker.yaml
│       └── wso2
│           ├── common.yaml
│           ├── wso2am
│           │   ├── 1.10.0
│           │   │   ├── default
│           │   │   │   ├── api-key-manager.yaml
│           │   │   │   ├── api-publisher.yaml
│           │   │   │   ├── api-store.yaml
│           │   │   │   ├── default.yaml
│           │   │   │   ├── gateway-manager.yaml
│           │   │   │   └── gateway-worker.yaml
│           │   │   └── kubernetes
│           │   │       ├── api-key-manager.yaml
│           │   │       ├── api-publisher.yaml
│           │   │       ├── api-store.yaml
│           │   │       ├── default.yaml
│           │   │       ├── gateway-manager.yaml
│           │   │       └── gateway-worker.yaml
├── manifests
│   └── site.pp
├── modules
│   ├── java
│   │   ├── LICENSE
│   │   ├── Modulefile
│   │   ├── README.markdown
│   │   ├── manifests
│   │   │   ├── init.pp
│   │   │   └── setup.pp
│   │   ├── metadata.json
│   │   ├── spec
│   │   │   ├── spec.opts
│   │   │   └── spec_helper.rb
│   │   └── tests
│   │       └── init.pp
│   ├── wso2am
│   │   ├── README.md
│   │   ├── files
│   │   │   ├── configs
│   │   │   │   └── repository
│   │   │   │       └── components
│   │   │   │           ├── dropins
│   │   │   │           └── lib
│   │   │   ├── patches
│   │   │   │   └── repository
│   │   │   │       └── components
│   │   │   │           └── patches
│   │   │   ├── system
│   │   │   └── wso2am-1.10.0.zip
│   │   ├── manifests
│   │   │   └── init.pp
│   │   ├── metadata.json
│   │   └── templates
│   │       ├── 1.10.0
│   │       │   ├── bin
│   │       │   │   ├── ciphertool.sh.erb
│   │       │   │   └── wso2server.sh.erb
│   │       │   ├── password-tmp.erb
│   │       │   └── repository
│   │       │       └── conf
│   │       │           ├── api-manager.xml.erb
│   │       │           ├── axis2
│   │       │           │   └── axis2.xml.erb
│   │       │           ├── carbon.xml.erb
│   │       │           ├── datasources
│   │       │           │   ├── am-datasources.xml.erb
│   │       │           │   └── master-datasources.xml.erb
│   │       │           ├── identity
│   │       │           │   └── identity.xml.erb
│   │       │           ├── registry.xml.erb
│   │       │           ├── security
│   │       │           │   ├── cipher-text.properties.erb
│   │       │           │   └── cipher-tool.properties.erb
│   │       │           ├── tomcat
│   │       │           │   └── catalina-server.xml.erb
│   │       │           └── user-mgt.xml.erb



[2] http://www.linuxjournal.com/content/using-hiera-puppet
[3] https://github.com/wso2/puppet-modules/wiki/Use-WSO2-Puppet-Modules-in-puppet-master-agent-Environment
[4] http://suhan-opensource.blogspot.com/2014/10/puppet-master-agent-communication-errors.html
[5] https://www.dropbox.com/s/6sljlbj1bop46lx/puppet.zip?dl=0

Friday, May 27, 2016

WSO2 ESB Accessing application/x-www-form-urlencoded Data


Problem:

I want to pass application/x-www-form-urlencoded data to the ESB:
curl -v -X POST 'http://myhost/myapi/stuff' -d 'myvar=hello' -H "Content-Type: application/x-www-form-urlencoded"
...and then be able to receive each form variable in my ESB API .
To get a given variable, I'm doing this:
<resource methods="POST" uri-template="/stuff">
      <inSequence>
         <property name="myvar" expression="//xformValues//myvar/text()"></property>
...
Then I'm storing that in an argument:
<args>
   <arg evaluator="xml" expression="$ctx:myvar"></arg>
</args>
Then I'm later trying to stuff it in a payload with $1.
I know that the issue is with the property line:
<property name="myvar" expression="//xformValues//myvar/text()"></property>
...because if I store a literal value in the line, it will work.
I do have the messageReceiver configured in the axis.xml:
<messageFormatter contentType="application/x-www-form-urlencoded"
    class="org.apache.axis2.transport.http.XFormURLEncodedFormatter"/>
<messageFormatter contentType="multipart/form-data"
And, I have the messageBuilder configured in the axis.xml file:
<messageBuilder contentType="application/x-www-form-urlencoded"
    class="org.apache.synapse.commons.builders.XFormURLEncodedBuilder"/>
ANSWER : Just use
`<property name="myvar" expression="//myvar/text()"/>`
e.g
wire logs
[2016-05-27 12:59:22,160] DEBUG - wire >> "POST /foo123 HTTP/1.1[\r][\n]"
[2016-05-27 12:59:22,161] DEBUG - wire >> "User-Agent: curl/7.37.1[\r][\n]"
[2016-05-27 12:59:22,161] DEBUG - wire >> "Host: 192.168.56.1:8280[\r][\n]"
[2016-05-27 12:59:22,161] DEBUG - wire >> "Accept: */*[\r][\n]"
[2016-05-27 12:59:22,161] DEBUG - wire >> "Authorization: Basic QU93ZmJQRW5YcXVDVnhIS1ZZT2ZlVFQxT01NYTpEeGpBY3dodFB5Wm5JTU9CZmJEakRHc21FQjBh[\r][\n]"
[2016-05-27 12:59:22,161] DEBUG - wire >> "Content-Type: application/x-www-form-urlencoded;charset=UTF-8[\r][\n]"
[2016-05-27 12:59:22,161] DEBUG - wire >> "Content-Length: 135[\r][\n]"
[2016-05-27 12:59:22,161] DEBUG - wire >> "[\r][\n]"
[2016-05-27 12:59:22,161] DEBUG - wire >> "grant_type=authorization_code&code=291a24141f1e3332baf9799c4c378033&redirect_uri=https://identity.qa.example.com/playground2/oauth2.jsp"
[2016-05-27 12:59:22,163] DEBUG - LogMediator Start : Log mediator
[2016-05-27 12:59:22,163]  INFO - LogMediator To: /foo123, MessageID: urn:uuid:c20f24d4-668d-447e-9de0-dbe5885b6d16, Direction: request
[2016-05-27 12:59:22,164] DEBUG - LogMediator End : Log mediator
[2016-05-27 12:59:22,164] DEBUG - LogMediator Start : Log mediator
[2016-05-27 12:59:22,164]  INFO - LogMediator payload = <mediate><code>291a24141f1e3332baf9799c4c378033</code><redirect_uri>https://identity.qa.example.com/playground2/oauth2.jsp</redirect_uri><grant_type>authorization_code</grant_type></mediate>,code = 291a24141f1e3332baf9799c4c378033
[2016-05-27 12:59:22,164] DEBUG - LogMediator End : Log mediator
[2016-05-27 12:59:22,165] DEBUG - wire << "HTTP/1.1 200 OK[\r][\n]"
[2016-05-27 12:59:22,165] DEBUG - wire << "Authorization: Basic QU93ZmJQRW5YcXVDVnhIS1ZZT2ZlVFQxT01NYTpEeGpBY3dodFB5Wm5JTU9CZmJEakRHc21FQjBh[\r][\n]"
[2016-05-27 12:59:22,166] DEBUG - wire << "Host: 192.168.56.1:8280[\r][\n]"
[2016-05-27 12:59:22,166] DEBUG - wire << "Content-Type: application/x-www-form-urlencoded;charset=UTF-8[\r][\n]"
[2016-05-27 12:59:22,166] DEBUG - wire << "Accept: */*[\r][\n]"
[2016-05-27 12:59:22,166] DEBUG - wire << "Date: Fri, 27 May 2016 19:59:22 GMT[\r][\n]"
[2016-05-27 12:59:22,166] DEBUG - wire << "Server: WSO2-PassThrough-HTTP[\r][\n]"
[2016-05-27 12:59:22,166] DEBUG - wire << "Transfer-Encoding: chunked[\r][\n]"
[2016-05-27 12:59:22,166] DEBUG - wire << "[\r][\n]"
[2016-05-27 12:59:22,166] DEBUG - wire << "91[\r][\n]"
[2016-05-27 12:59:22,166] DEBUG - wire << "code=291a24141f1e3332baf9799c4c378033&redirect_uri=https%3A%2F%2Fidentity.qa.example.com%2Fplayground2%2Foauth2.jsp&grant_type=authorization_code[\r][\n]"
[2016-05-27 12:59:22,166] DEBUG - wire << "0[\r][\n]"
[2016-05-27 12:59:22,167] DEBUG - wire << "[\r][\n]"
API
<api xmlns="http://ws.apache.org/ns/synapse" name="foo123" context="/foo123">
   <resource methods="POST">
      <inSequence>
         <log></log>
         <property name="payload" expression="//mediate"></property>
         <property name="code" expression="//mediate/code"></property>
         <log level="custom" separator=",">
            <property name="payload" expression="$ctx:payload"></property>
            <property name="code" expression="$ctx:code"></property>
         </log>
         <respond></respond>
      </inSequence>
   </resource>
</api>
 

Wednesday, May 11, 2016

WSO2 IS SSO with .NET application

In this blog, I would like to discuss another common case which, I couldn't find many entries in explaining how to integrate WSO2 Identity Server with .NET web application.

 There are different types of 3rd party components available but out of that thought of demonstrating the demo with Component Space [1] which provides a comprehensive framework to implement SSO solutions and the demonstration was prepared with using those references.

I will use some steps (with images) to demonstrate how easily could integrate, the sample code can be download [3] and you could easily open up .NET IDE so, you can customize it according to your preference.


  • First, you should register .NET web application you have written as a service provider, you can customize SP parameters such as assertion encryption, SLO settings according to your preference. (I would assume at this point you may have covered fundamentals of WSO2 Identity server, such as registering Sevice Provide (SP), Identity Provider (IP) and other basic fundamentals, if not recommendation is to go through [2]




  • Then, you have to customize the SSO web app settings under web.config
<configuration>
  <appSettings>
    <add key="idpssoURL" value="https://localhost:9443/samlsso" />
    <add key="idpArtifactResponderURL" value="http://localhost:51394/SAML2IdentityProvider/SAML/ArtifactResponder.aspx" />
    <add key="idpLogoutURL" value="https://localhost:9443/samlsso" />
 </appSettings>
<connectionStrings/>




  • Once you do all the required modifications at SSO web app, then when deploying to IIS server, during initial startup it would ask the way SSOSAMLRequest should have generated, this is customizable, which means you don't need to expose to the actual client but for development perspective, you can play around with different options available.




  •  The next step would be, when you click "continue" button, it would direct you to IS SSO login page as depicted below.




  • Then once you log in successfully, it would show the welcome page as given below.






[1] https://docs.wso2.com/display/IS500/Samples
[2] http://www.componentspace.com/SAMLv20.aspx
[3] https://www.dropbox.com/s/jfik1852xdhf342/SAML2ServiceProvider.zip?dl=0