Monday, May 30, 2016

Troubleshoot and understanding Puppet deployment

Following blog, I will try to describe the steps that you may need to follow up in order to setup a WSO2 product cluster quick time using Puppet scripts.
  Basically, thought this is straight forward you may have to spent few minutes or  some time few hours to get firm to your self. Because if you just start to work these environments thing gonna be bit difficult. In this blog I try to target people who just started to work with puppet and of cause they may be face similar issues.

For the ease of understanding, we have divided the instructions into 3 Tasks.

    Step 1. Setup two instances
    Step 2. Setup Puppet Master and puppet agent
    Step 3. Perform a catalog run to configure product instance

First of all lets try to understand puppet directory structure [1],basically, hiera structure defines the variable and parameters per environment per platform per product such as how it should populate while under module, the templates of each products defined with .erp

If you refer below structure, you will notice something called 'hiera', the term we are using With Hiera, you can externalize your systems' configuration data and easily understand how those values are assigned to your servers. With that data separated from your Puppet code, you then can encrypt sensitive values, such as passwords and keys [2]

Once you navigate, please try to understand the project structure, in this case we are trying to prepare 'dev' environment, so understanding hiera model is important.

 lets see how hiera.yaml definition looks

:hierarchy:
    - "node/%{::clientcert}"
    - "wso2/%{::product_name}/%{::product_version}/%{::platform}/%{::product_profile}"
    - "wso2/%{::product_name}/%{::product_version}/%{::platform}/default"
    - "osfamily/%{::osfamily}"
    - "vm_type/%{::vm_type}"
    - "platform/%{::platform}"
    - wso2/common
    - common
:backends:
    - yaml
:yaml:
    :datadir: "/etc/puppet/hieradata/dev"


In above structure, it would explain how hiera would travel through, then extract variables to prepare the platform that you are running.

Let assume I need to run api-store profile (APIM having store,publisher,gateway,key manager), then once I initiate the product profile from puppet agent (which I will explain later), when puppet agent communicates with puppet master, it would go through above mentioned hierarchy to resolve variables i.e
api-store.yaml --> default.yaml--> common.yaml --> (root common) -->common.yaml

However, first of all we should install puppet do refer [3], follow the steps given

Sometimes you will not succeeded straightforward (As I was struggle for a while to establish connection between puppet agent and master, therefore if you have any trouble please do refer[4])

From my personal  experience,the most common issue you will face

err: Could not request certificate: No route to host - connect(2)
Exiting; failed to retrieve certificate and waitforcert is disabled 


I would say, if you need to resolve this, you don't need to fiddle around much 


There are few steps you need to verify,

Step-1
Please make sure /etc/hosts files of both nodes as given below, make sure you gave puppet master host name correctly

Puppet master
/etc/hosts
127.0.0.1 localhost
127.0.0.1 puppet puppetmaster

/etc/hostname
puppetmaster


Puppet Agent
/etc/hosts
127.0.0.1      localhost

[IP-address of puppetmaster]     puppetmaster
[IP-address of agent]      [Agent-hostage]


Step-2

Verify at puppet agent the puppet.conf file under /etc/puppet/,  the only thing you need to focus [agent] tag and verify the server mapping as highlighted below

[main]
logdir=/var/log/puppet
vardir=/var/lib/puppet
ssldir=/var/lib/puppet/ssl
rundir=/var/run/puppet
factpath=$vardir/lib/facter
templatedir=$confdir/templates
prerun_command=/etc/puppet/etckeeper-commit-pre
postrun_command=/etc/puppet/etckeeper-commit-post

[master]
# These are needed when the puppetmaster is run by passenger
# and can safely be removed if webrick is used.
ssl_client_header = SSL_CLIENT_S_DN
ssl_client_verify_header = SSL_CLIENT_VERIFY

[agent]
server=puppetmaster


Step3
Then at puppet master, /puppet/hieradata/dev/wso2/common.yaml, please check following entry

 # Host mapping to be made in etc/hosts
wso2::hosts_mapping:
  localhost:
    ip_address: 127.0.0.1
    hostname: localhost
  puppetmaster :
    ip_address : [puppet-master-ip]
    hostname : puppetmaster
  pub.am.wso2.com :
    ip_address : [agent-ip]
    hostname : [agent-hostname]

** Make to give correct information under host name, as these gonna replace /etc/hosts  mapping during run time, else thing gonna break up while scripts running

Step4
 Check puppet.conf under puppet master "puppet/puppet.conf", please make sure it has "autosign=true" which required for certification signing for proper master-agent communication

[main]
logdir=/var/log/puppet
vardir=/var/lib/puppet
ssldir=/var/lib/puppet/ssl
rundir=/var/run/puppet
#factpath=$vardir/lib/facter
templatedir=$confdir/templates
dns_alt_names=puppetmaster,puppet
hiera_config = /etc/puppet/hiera.yaml

[master]
# These are needed when the puppetmaster is run by passenger
# and can safely be removed if webrick is used.
ssl_client_header = SSL_CLIENT_S_DN
ssl_client_verify_header = SSL_CLIENT_VERIFY
autosign=true

If you verify above given steps carefully, you should execute following command;

 Puppet Master
  1. puppet cert clean —all
  2.  service puppetmaster restart
  3. ** If you need to see whats going inside puppet master, then you can start "puppet master --no-daemonize --debug"
Puppet Agent
  • "find /var/lib/puppet/ssl -name '*.pem' -exec rm {} \;"  (to clean up invalid certificates)
  • puppet agent -t —debug  (to establish master-agent connection)
Puppet Agent run time
Once verify above, then just run setup.sh, which should start to deploy agent profile while communicating with master

deployment.cofig

product_name=wso2am   // defines the product type
product_version=1.10.0  //define the product version
product_profile=default  //defines the profiles
environment=dev  // the environment
vm_type=openstack
platform=default

setup.sh

#!/bin/bash
echo "#####################################################"
echo "                   Starting cleanup "
echo "#####################################################"
ps aux | grep -i wso2 | awk {'print $2'} | xargs kill -9
#rm -rf /mnt/*
sed -i '/environment/d' /etc/puppet/puppet.conf
echo "#####################################################"
echo "               Setting up environment "
echo "#####################################################"
rm -f /etc/facter/facts.d/deployment_pattern.txt
mkdir -p /etc/facter/facts.d

while read -r line; do declare  $line; done < deployment.conf
echo "reading...."
echo product_name=$product_name product_version=$product_version product_profile=$product_profile
echo product_name=$product_name >> /etc/facter/facts.d/deployment_pattern.txt
echo product_version=$product_version >> /etc/facter/facts.d/deployment_pattern.txt
echo product_profile=$product_profile >> /etc/facter/facts.d/deployment_pattern.txt
echo vm_type=$vm_type >> /etc/facter/facts.d/deployment_pattern.txt
echo environment=$environment >> /etc/facter/facts.d/deployment_pattern.txt
echo platform=$platform >> /etc/facter/facts.d/deployment_pattern.txt

echo "reading end"

echo "#####################################################"
echo "                    Installing "
echo "#####################################################"

puppet agent --enable
puppet agent -vt
puppet agent --disable

Please use [5] to download the puppet-master module which I have used to config, but there you have to copy JDK, products etc

[1]
├── LICENSE
├── README.md
├── hiera.yaml
├── hieradata
│   └── dev
│       ├── common.yaml
│       ├── platform
│       │   └── kubernetes.yaml
│       ├── vm_type
│       │   └── docker.yaml
│       └── wso2
│           ├── common.yaml
│           ├── wso2am
│           │   ├── 1.10.0
│           │   │   ├── default
│           │   │   │   ├── api-key-manager.yaml
│           │   │   │   ├── api-publisher.yaml
│           │   │   │   ├── api-store.yaml
│           │   │   │   ├── default.yaml
│           │   │   │   ├── gateway-manager.yaml
│           │   │   │   └── gateway-worker.yaml
│           │   │   └── kubernetes
│           │   │       ├── api-key-manager.yaml
│           │   │       ├── api-publisher.yaml
│           │   │       ├── api-store.yaml
│           │   │       ├── default.yaml
│           │   │       ├── gateway-manager.yaml
│           │   │       └── gateway-worker.yaml
├── manifests
│   └── site.pp
├── modules
│   ├── java
│   │   ├── LICENSE
│   │   ├── Modulefile
│   │   ├── README.markdown
│   │   ├── manifests
│   │   │   ├── init.pp
│   │   │   └── setup.pp
│   │   ├── metadata.json
│   │   ├── spec
│   │   │   ├── spec.opts
│   │   │   └── spec_helper.rb
│   │   └── tests
│   │       └── init.pp
│   ├── wso2am
│   │   ├── README.md
│   │   ├── files
│   │   │   ├── configs
│   │   │   │   └── repository
│   │   │   │       └── components
│   │   │   │           ├── dropins
│   │   │   │           └── lib
│   │   │   ├── patches
│   │   │   │   └── repository
│   │   │   │       └── components
│   │   │   │           └── patches
│   │   │   ├── system
│   │   │   └── wso2am-1.10.0.zip
│   │   ├── manifests
│   │   │   └── init.pp
│   │   ├── metadata.json
│   │   └── templates
│   │       ├── 1.10.0
│   │       │   ├── bin
│   │       │   │   ├── ciphertool.sh.erb
│   │       │   │   └── wso2server.sh.erb
│   │       │   ├── password-tmp.erb
│   │       │   └── repository
│   │       │       └── conf
│   │       │           ├── api-manager.xml.erb
│   │       │           ├── axis2
│   │       │           │   └── axis2.xml.erb
│   │       │           ├── carbon.xml.erb
│   │       │           ├── datasources
│   │       │           │   ├── am-datasources.xml.erb
│   │       │           │   └── master-datasources.xml.erb
│   │       │           ├── identity
│   │       │           │   └── identity.xml.erb
│   │       │           ├── registry.xml.erb
│   │       │           ├── security
│   │       │           │   ├── cipher-text.properties.erb
│   │       │           │   └── cipher-tool.properties.erb
│   │       │           ├── tomcat
│   │       │           │   └── catalina-server.xml.erb
│   │       │           └── user-mgt.xml.erb



[2] http://www.linuxjournal.com/content/using-hiera-puppet
[3] https://github.com/wso2/puppet-modules/wiki/Use-WSO2-Puppet-Modules-in-puppet-master-agent-Environment
[4] http://suhan-opensource.blogspot.com/2014/10/puppet-master-agent-communication-errors.html
[5] https://www.dropbox.com/s/6sljlbj1bop46lx/puppet.zip?dl=0

Friday, May 27, 2016

WSO2 ESB Accessing application/x-www-form-urlencoded Data


Problem:

I want to pass application/x-www-form-urlencoded data to the ESB:
curl -v -X POST 'http://myhost/myapi/stuff' -d 'myvar=hello' -H "Content-Type: application/x-www-form-urlencoded"
...and then be able to receive each form variable in my ESB API .
To get a given variable, I'm doing this:
<resource methods="POST" uri-template="/stuff">
      <inSequence>
         <property name="myvar" expression="//xformValues//myvar/text()"></property>
...
Then I'm storing that in an argument:
<args>
   <arg evaluator="xml" expression="$ctx:myvar"></arg>
</args>
Then I'm later trying to stuff it in a payload with $1.
I know that the issue is with the property line:
<property name="myvar" expression="//xformValues//myvar/text()"></property>
...because if I store a literal value in the line, it will work.
I do have the messageReceiver configured in the axis.xml:
<messageFormatter contentType="application/x-www-form-urlencoded"
    class="org.apache.axis2.transport.http.XFormURLEncodedFormatter"/>
<messageFormatter contentType="multipart/form-data"
And, I have the messageBuilder configured in the axis.xml file:
<messageBuilder contentType="application/x-www-form-urlencoded"
    class="org.apache.synapse.commons.builders.XFormURLEncodedBuilder"/>
ANSWER : Just use
`<property name="myvar" expression="//myvar/text()"/>`
e.g
wire logs
[2016-05-27 12:59:22,160] DEBUG - wire >> "POST /foo123 HTTP/1.1[\r][\n]"
[2016-05-27 12:59:22,161] DEBUG - wire >> "User-Agent: curl/7.37.1[\r][\n]"
[2016-05-27 12:59:22,161] DEBUG - wire >> "Host: 192.168.56.1:8280[\r][\n]"
[2016-05-27 12:59:22,161] DEBUG - wire >> "Accept: */*[\r][\n]"
[2016-05-27 12:59:22,161] DEBUG - wire >> "Authorization: Basic QU93ZmJQRW5YcXVDVnhIS1ZZT2ZlVFQxT01NYTpEeGpBY3dodFB5Wm5JTU9CZmJEakRHc21FQjBh[\r][\n]"
[2016-05-27 12:59:22,161] DEBUG - wire >> "Content-Type: application/x-www-form-urlencoded;charset=UTF-8[\r][\n]"
[2016-05-27 12:59:22,161] DEBUG - wire >> "Content-Length: 135[\r][\n]"
[2016-05-27 12:59:22,161] DEBUG - wire >> "[\r][\n]"
[2016-05-27 12:59:22,161] DEBUG - wire >> "grant_type=authorization_code&code=291a24141f1e3332baf9799c4c378033&redirect_uri=https://identity.qa.example.com/playground2/oauth2.jsp"
[2016-05-27 12:59:22,163] DEBUG - LogMediator Start : Log mediator
[2016-05-27 12:59:22,163]  INFO - LogMediator To: /foo123, MessageID: urn:uuid:c20f24d4-668d-447e-9de0-dbe5885b6d16, Direction: request
[2016-05-27 12:59:22,164] DEBUG - LogMediator End : Log mediator
[2016-05-27 12:59:22,164] DEBUG - LogMediator Start : Log mediator
[2016-05-27 12:59:22,164]  INFO - LogMediator payload = <mediate><code>291a24141f1e3332baf9799c4c378033</code><redirect_uri>https://identity.qa.example.com/playground2/oauth2.jsp</redirect_uri><grant_type>authorization_code</grant_type></mediate>,code = 291a24141f1e3332baf9799c4c378033
[2016-05-27 12:59:22,164] DEBUG - LogMediator End : Log mediator
[2016-05-27 12:59:22,165] DEBUG - wire << "HTTP/1.1 200 OK[\r][\n]"
[2016-05-27 12:59:22,165] DEBUG - wire << "Authorization: Basic QU93ZmJQRW5YcXVDVnhIS1ZZT2ZlVFQxT01NYTpEeGpBY3dodFB5Wm5JTU9CZmJEakRHc21FQjBh[\r][\n]"
[2016-05-27 12:59:22,166] DEBUG - wire << "Host: 192.168.56.1:8280[\r][\n]"
[2016-05-27 12:59:22,166] DEBUG - wire << "Content-Type: application/x-www-form-urlencoded;charset=UTF-8[\r][\n]"
[2016-05-27 12:59:22,166] DEBUG - wire << "Accept: */*[\r][\n]"
[2016-05-27 12:59:22,166] DEBUG - wire << "Date: Fri, 27 May 2016 19:59:22 GMT[\r][\n]"
[2016-05-27 12:59:22,166] DEBUG - wire << "Server: WSO2-PassThrough-HTTP[\r][\n]"
[2016-05-27 12:59:22,166] DEBUG - wire << "Transfer-Encoding: chunked[\r][\n]"
[2016-05-27 12:59:22,166] DEBUG - wire << "[\r][\n]"
[2016-05-27 12:59:22,166] DEBUG - wire << "91[\r][\n]"
[2016-05-27 12:59:22,166] DEBUG - wire << "code=291a24141f1e3332baf9799c4c378033&redirect_uri=https%3A%2F%2Fidentity.qa.example.com%2Fplayground2%2Foauth2.jsp&grant_type=authorization_code[\r][\n]"
[2016-05-27 12:59:22,166] DEBUG - wire << "0[\r][\n]"
[2016-05-27 12:59:22,167] DEBUG - wire << "[\r][\n]"
API
<api xmlns="http://ws.apache.org/ns/synapse" name="foo123" context="/foo123">
   <resource methods="POST">
      <inSequence>
         <log></log>
         <property name="payload" expression="//mediate"></property>
         <property name="code" expression="//mediate/code"></property>
         <log level="custom" separator=",">
            <property name="payload" expression="$ctx:payload"></property>
            <property name="code" expression="$ctx:code"></property>
         </log>
         <respond></respond>
      </inSequence>
   </resource>
</api>
 

Wednesday, May 11, 2016

WSO2 IS SSO with .NET application

In this blog, I would like to discuss another common case which, I couldn't find many entries in explaining how to integrate WSO2 Identity Server with .NET web application.

 There are different types of 3rd party components available but out of that thought of demonstrating the demo with Component Space [1] which provides a comprehensive framework to implement SSO solutions and the demonstration was prepared with using those references.

I will use some steps (with images) to demonstrate how easily could integrate, the sample code can be download [3] and you could easily open up .NET IDE so, you can customize it according to your preference.


  • First, you should register .NET web application you have written as a service provider, you can customize SP parameters such as assertion encryption, SLO settings according to your preference. (I would assume at this point you may have covered fundamentals of WSO2 Identity server, such as registering Sevice Provide (SP), Identity Provider (IP) and other basic fundamentals, if not recommendation is to go through [2]




  • Then, you have to customize the SSO web app settings under web.config
<configuration>
  <appSettings>
    <add key="idpssoURL" value="https://localhost:9443/samlsso" />
    <add key="idpArtifactResponderURL" value="http://localhost:51394/SAML2IdentityProvider/SAML/ArtifactResponder.aspx" />
    <add key="idpLogoutURL" value="https://localhost:9443/samlsso" />
 </appSettings>
<connectionStrings/>




  • Once you do all the required modifications at SSO web app, then when deploying to IIS server, during initial startup it would ask the way SSOSAMLRequest should have generated, this is customizable, which means you don't need to expose to the actual client but for development perspective, you can play around with different options available.




  •  The next step would be, when you click "continue" button, it would direct you to IS SSO login page as depicted below.




  • Then once you log in successfully, it would show the welcome page as given below.






[1] https://docs.wso2.com/display/IS500/Samples
[2] http://www.componentspace.com/SAMLv20.aspx
[3] https://www.dropbox.com/s/jfik1852xdhf342/SAML2ServiceProvider.zip?dl=0