Tuesday 15 November 2016

EC2 “On demand” instances start/Stop Automation:


1.       Create an IAM user and grant full access to the EC2 instances.

2.       Download the Access id and Secret key for the IAM user.

3.       Shell script to perform the start and stop

Start:

#!/bin/sh
ec2-start-instances i-“instance-id” i-“instance-id” -O “access-id” -W “secret-key”

Stop:

#!/bin/sh
ec2-stop-instances i-“instance-id” i-“instance-id” -O “access-id” -W “secret-key”

In the above script, you can add more instances-id’s to be restarted. The access id and secret key is of the IAM user performing the restart.

4.       Create a cronjob to run this on Friday evening to stop the instances and start it on Monday morning, so that you save lot of money on EC2 instances which are not used on weekends.


5.       The cron job should be run on an EC2 instance which is on the same subnet as all the instances which are to be restarted.

NAGIOS Monitoring solution for JBoss 7.1

NAGIOS Monitoring solution for JBoss  :


1.       Performed the below on server where Nagios is installed:

Defined the host, jhostip and contact group in host.cfg : under /usr/local/nagios/etc/objects


define host{
        use                     linux-server            ; Name of host template to use
                                                        ; This host definition will inherit all variables that are defined
                                                        ; in (or inherited by) the linux-server host template definition.
        host_name               Prod***-APP1
        alias                   Prod***-APP1
        contact_groups          jboss-admin
        notifications_enabled   1
        address                 10.1.*.***
        }
define host{
        use                     linux-server            ; Name of host template to use
                                                        ; This host definition will inherit all variables that are defined
                                                        ; in (or inherited by) the linux-server host template definition.
        host_name               Prod***-APP2
        alias                   Prod***-APP2
        contact_groups          jboss-admin
        notifications_enabled   1
        address                 10.1.*.***
        }

2.       In the services.cfg, defined the services to be monitored like ping, 8080 port, load :

Define a service to check HTTP on the local machine.
# Disable notifications for this service by default, as not all users may have HTTP enabled.

define service{
        use                             local-service         ; Name of service template to use
        host_name                       App1
        service_description             HTTP
        check_command                   check_http!-p 8080
        check_interval                  1
        notifications_enabled           0
        max_check_attempts               4
        event_handler                   check_stopjb

3.       Since PRODUCTION had multiple servers, defined multiple services.cfg file, specific to each host to be monitored. For example below, I have defined 2 services.cfg file for 2 Jboss hosts:

[ec2-user@prodnagios-server prod_services]$ ls -ltr
total 88
-rw-r--r-- 1 nagios nagios 4363 Jun 23  2015 services.cfg
-rw-r--r-- 1 nagios nagios 5130 Jul 21  2015 prodAP1_Services.cfg
-rw-r--r-- 1 nagios nagios 4832 Jul 21  2015 prodAP2_Services.cfg
-
[ec2-user@prodnagios-server prod_services]$ pwd
/usr/local/nagios/etc/objects/prod_services

And each file have entries of services to be monitored like example:

define service{
        use                             local-service         ; Name of service template to use
        host_name                       Prod***-APP1
        service_description             HTTP
        check_command                   check_http!-p 8080
        check_interval                  1
        notifications_enabled           1
         notification_options            w,u,c,r
        contact_groups                  jboss-admin
        max_check_attempts               4
#        event_handler                   check_stopjb
        }

4.       All the above changes, have defined in main Nagios config file:  Nagios.conf:

# prod services
cfg_file=/usr/local/nagios/etc/objects/prod_services/prodAP1_Services.cfg
cfg_file=/usr/local/nagios/etc/objects/prod_services/prodAP2_Services.cfg

# Definitions for monitoring the local (Linux) host
cfg_file=/usr/local/nagios/etc/objects/host.cfg

5.       Made sure 8080,22 and all required ports are reachable to Jboss hosts by putting Nagios to appropriate security group in AWS. And also Nagios nrpe plugin is installed on all Jboss servers.

And ran the verify to make sure the configuration worked fine :



[ec2-user@prodnagios-server etc]$ sudo /usr/local/nagios/bin/nagios -v /usr/local/nagios/etc/nagios.cfg

Nagios Core 4.0.8
Copyright (c) 2009-present Nagios Core Development Team and Community Contributors
Copyright (c) 1999-2009 Ethan Galstad
Last Modified: 08-12-2014
License: GPL

Website: http://www.nagios.org
Reading configuration data...
   Read main config file okay...
   Read object config files okay...

Running pre-flight check on configuration data...

Checking objects...
        Checked 77 services.
        Checked 12 hosts.
        Checked 1 host groups.
        Checked 0 service groups.
        Checked 6 contacts.
        Checked 1 contact groups.
        Checked 35 commands.
        Checked 5 time periods.
        Checked 0 host escalations.
        Checked 0 service escalations.
Checking for circular paths...
        Checked 12 hosts
        Checked 0 service dependencies
        Checked 0 host dependencies
        Checked 5 timeperiods
Checking global event handlers...
Checking obsessive compulsive processor commands...
Checking misc settings...

Total Warnings: 0
Total Errors:   0

Things look okay - No serious problems were detected during the pre-flight check

6.       Could monitor the Jboss through console:




7.       Also defined my email and Mobile to get alerts in contacts.cfg file:

The contacts.cfg file must be defined in Nagios.cfg file as well:

# host groups, contacts, contact groups, services, etc.
cfg_file=/usr/local/nagios/etc/objects/contacts.cfg


define contact{
        contact_name                    SandeshSMS          ; Short name of user
        use                             generic-contact         ; Inherit default values from generic-contact template (defined above)
        alias                           Nagios Admin            ; Full name of user

        email                          **********@txt.att.net       ; <<***** CHANGE THIS TO YOUR EMAIL ADDRESS ******
       service_notifications_enabled    1
        }


define contact{
        contact_name                    Sandesh         ; Short name of user
        use                             generic-contact         ; Inherit default values from generic-contact template (defined above)
        alias                           Nagios Admin            ; Full name of user

        email                           Sandesh.achar@********.com       ; <<***** CHANGE THIS TO YOUR EMAIL ADDRESS ******
       service_notifications_enabled    1
        }