Monday, May 30, 2016

Troubleshoot and understanding Puppet deployment

Following blog, I will try to describe the steps that you may need to follow up in order to setup a WSO2 product cluster quick time using Puppet scripts.
  Basically, thought this is straight forward you may have to spent few minutes or  some time few hours to get firm to your self. Because if you just start to work these environments thing gonna be bit difficult. In this blog I try to target people who just started to work with puppet and of cause they may be face similar issues.

For the ease of understanding, we have divided the instructions into 3 Tasks.

    Step 1. Setup two instances
    Step 2. Setup Puppet Master and puppet agent
    Step 3. Perform a catalog run to configure product instance

First of all lets try to understand puppet directory structure [1],basically, hiera structure defines the variable and parameters per environment per platform per product such as how it should populate while under module, the templates of each products defined with .erp

If you refer below structure, you will notice something called 'hiera', the term we are using With Hiera, you can externalize your systems' configuration data and easily understand how those values are assigned to your servers. With that data separated from your Puppet code, you then can encrypt sensitive values, such as passwords and keys [2]

Once you navigate, please try to understand the project structure, in this case we are trying to prepare 'dev' environment, so understanding hiera model is important.

 lets see how hiera.yaml definition looks

:hierarchy:
    - "node/%{::clientcert}"
    - "wso2/%{::product_name}/%{::product_version}/%{::platform}/%{::product_profile}"
    - "wso2/%{::product_name}/%{::product_version}/%{::platform}/default"
    - "osfamily/%{::osfamily}"
    - "vm_type/%{::vm_type}"
    - "platform/%{::platform}"
    - wso2/common
    - common
:backends:
    - yaml
:yaml:
    :datadir: "/etc/puppet/hieradata/dev"


In above structure, it would explain how hiera would travel through, then extract variables to prepare the platform that you are running.

Let assume I need to run api-store profile (APIM having store,publisher,gateway,key manager), then once I initiate the product profile from puppet agent (which I will explain later), when puppet agent communicates with puppet master, it would go through above mentioned hierarchy to resolve variables i.e
api-store.yaml --> default.yaml--> common.yaml --> (root common) -->common.yaml

However, first of all we should install puppet do refer [3], follow the steps given

Sometimes you will not succeeded straightforward (As I was struggle for a while to establish connection between puppet agent and master, therefore if you have any trouble please do refer[4])

From my personal  experience,the most common issue you will face

err: Could not request certificate: No route to host - connect(2)
Exiting; failed to retrieve certificate and waitforcert is disabled 


I would say, if you need to resolve this, you don't need to fiddle around much 


There are few steps you need to verify,

Step-1
Please make sure /etc/hosts files of both nodes as given below, make sure you gave puppet master host name correctly

Puppet master
/etc/hosts
127.0.0.1 localhost
127.0.0.1 puppet puppetmaster

/etc/hostname
puppetmaster


Puppet Agent
/etc/hosts
127.0.0.1      localhost

[IP-address of puppetmaster]     puppetmaster
[IP-address of agent]      [Agent-hostage]


Step-2

Verify at puppet agent the puppet.conf file under /etc/puppet/,  the only thing you need to focus [agent] tag and verify the server mapping as highlighted below

[main]
logdir=/var/log/puppet
vardir=/var/lib/puppet
ssldir=/var/lib/puppet/ssl
rundir=/var/run/puppet
factpath=$vardir/lib/facter
templatedir=$confdir/templates
prerun_command=/etc/puppet/etckeeper-commit-pre
postrun_command=/etc/puppet/etckeeper-commit-post

[master]
# These are needed when the puppetmaster is run by passenger
# and can safely be removed if webrick is used.
ssl_client_header = SSL_CLIENT_S_DN
ssl_client_verify_header = SSL_CLIENT_VERIFY

[agent]
server=puppetmaster


Step3
Then at puppet master, /puppet/hieradata/dev/wso2/common.yaml, please check following entry

 # Host mapping to be made in etc/hosts
wso2::hosts_mapping:
  localhost:
    ip_address: 127.0.0.1
    hostname: localhost
  puppetmaster :
    ip_address : [puppet-master-ip]
    hostname : puppetmaster
  pub.am.wso2.com :
    ip_address : [agent-ip]
    hostname : [agent-hostname]

** Make to give correct information under host name, as these gonna replace /etc/hosts  mapping during run time, else thing gonna break up while scripts running

Step4
 Check puppet.conf under puppet master "puppet/puppet.conf", please make sure it has "autosign=true" which required for certification signing for proper master-agent communication

[main]
logdir=/var/log/puppet
vardir=/var/lib/puppet
ssldir=/var/lib/puppet/ssl
rundir=/var/run/puppet
#factpath=$vardir/lib/facter
templatedir=$confdir/templates
dns_alt_names=puppetmaster,puppet
hiera_config = /etc/puppet/hiera.yaml

[master]
# These are needed when the puppetmaster is run by passenger
# and can safely be removed if webrick is used.
ssl_client_header = SSL_CLIENT_S_DN
ssl_client_verify_header = SSL_CLIENT_VERIFY
autosign=true

If you verify above given steps carefully, you should execute following command;

 Puppet Master
  1. puppet cert clean —all
  2.  service puppetmaster restart
  3. ** If you need to see whats going inside puppet master, then you can start "puppet master --no-daemonize --debug"
Puppet Agent
  • "find /var/lib/puppet/ssl -name '*.pem' -exec rm {} \;"  (to clean up invalid certificates)
  • puppet agent -t —debug  (to establish master-agent connection)
Puppet Agent run time
Once verify above, then just run setup.sh, which should start to deploy agent profile while communicating with master

deployment.cofig

product_name=wso2am   // defines the product type
product_version=1.10.0  //define the product version
product_profile=default  //defines the profiles
environment=dev  // the environment
vm_type=openstack
platform=default

setup.sh

#!/bin/bash
echo "#####################################################"
echo "                   Starting cleanup "
echo "#####################################################"
ps aux | grep -i wso2 | awk {'print $2'} | xargs kill -9
#rm -rf /mnt/*
sed -i '/environment/d' /etc/puppet/puppet.conf
echo "#####################################################"
echo "               Setting up environment "
echo "#####################################################"
rm -f /etc/facter/facts.d/deployment_pattern.txt
mkdir -p /etc/facter/facts.d

while read -r line; do declare  $line; done < deployment.conf
echo "reading...."
echo product_name=$product_name product_version=$product_version product_profile=$product_profile
echo product_name=$product_name >> /etc/facter/facts.d/deployment_pattern.txt
echo product_version=$product_version >> /etc/facter/facts.d/deployment_pattern.txt
echo product_profile=$product_profile >> /etc/facter/facts.d/deployment_pattern.txt
echo vm_type=$vm_type >> /etc/facter/facts.d/deployment_pattern.txt
echo environment=$environment >> /etc/facter/facts.d/deployment_pattern.txt
echo platform=$platform >> /etc/facter/facts.d/deployment_pattern.txt

echo "reading end"

echo "#####################################################"
echo "                    Installing "
echo "#####################################################"

puppet agent --enable
puppet agent -vt
puppet agent --disable

Please use [5] to download the puppet-master module which I have used to config, but there you have to copy JDK, products etc

[1]
├── LICENSE
├── README.md
├── hiera.yaml
├── hieradata
│   └── dev
│       ├── common.yaml
│       ├── platform
│       │   └── kubernetes.yaml
│       ├── vm_type
│       │   └── docker.yaml
│       └── wso2
│           ├── common.yaml
│           ├── wso2am
│           │   ├── 1.10.0
│           │   │   ├── default
│           │   │   │   ├── api-key-manager.yaml
│           │   │   │   ├── api-publisher.yaml
│           │   │   │   ├── api-store.yaml
│           │   │   │   ├── default.yaml
│           │   │   │   ├── gateway-manager.yaml
│           │   │   │   └── gateway-worker.yaml
│           │   │   └── kubernetes
│           │   │       ├── api-key-manager.yaml
│           │   │       ├── api-publisher.yaml
│           │   │       ├── api-store.yaml
│           │   │       ├── default.yaml
│           │   │       ├── gateway-manager.yaml
│           │   │       └── gateway-worker.yaml
├── manifests
│   └── site.pp
├── modules
│   ├── java
│   │   ├── LICENSE
│   │   ├── Modulefile
│   │   ├── README.markdown
│   │   ├── manifests
│   │   │   ├── init.pp
│   │   │   └── setup.pp
│   │   ├── metadata.json
│   │   ├── spec
│   │   │   ├── spec.opts
│   │   │   └── spec_helper.rb
│   │   └── tests
│   │       └── init.pp
│   ├── wso2am
│   │   ├── README.md
│   │   ├── files
│   │   │   ├── configs
│   │   │   │   └── repository
│   │   │   │       └── components
│   │   │   │           ├── dropins
│   │   │   │           └── lib
│   │   │   ├── patches
│   │   │   │   └── repository
│   │   │   │       └── components
│   │   │   │           └── patches
│   │   │   ├── system
│   │   │   └── wso2am-1.10.0.zip
│   │   ├── manifests
│   │   │   └── init.pp
│   │   ├── metadata.json
│   │   └── templates
│   │       ├── 1.10.0
│   │       │   ├── bin
│   │       │   │   ├── ciphertool.sh.erb
│   │       │   │   └── wso2server.sh.erb
│   │       │   ├── password-tmp.erb
│   │       │   └── repository
│   │       │       └── conf
│   │       │           ├── api-manager.xml.erb
│   │       │           ├── axis2
│   │       │           │   └── axis2.xml.erb
│   │       │           ├── carbon.xml.erb
│   │       │           ├── datasources
│   │       │           │   ├── am-datasources.xml.erb
│   │       │           │   └── master-datasources.xml.erb
│   │       │           ├── identity
│   │       │           │   └── identity.xml.erb
│   │       │           ├── registry.xml.erb
│   │       │           ├── security
│   │       │           │   ├── cipher-text.properties.erb
│   │       │           │   └── cipher-tool.properties.erb
│   │       │           ├── tomcat
│   │       │           │   └── catalina-server.xml.erb
│   │       │           └── user-mgt.xml.erb



[2] http://www.linuxjournal.com/content/using-hiera-puppet
[3] https://github.com/wso2/puppet-modules/wiki/Use-WSO2-Puppet-Modules-in-puppet-master-agent-Environment
[4] http://suhan-opensource.blogspot.com/2014/10/puppet-master-agent-communication-errors.html
[5] https://www.dropbox.com/s/6sljlbj1bop46lx/puppet.zip?dl=0

No comments:

Post a Comment