Configure CISCO Catalyst 2960 ports to monitoring/mirroring mode

This will send all traffic that comes on the 0/1-0/9 ports to the Ga0/1 interface:

switch>enable
switch#configure terminal
Enter configuration commands, one per line.  End with CNTL/Z.

switch(config)#monitor session 1 source interface fastEthernet 0/1
switch(config)#monitor session 1 source interface fastEthernet 0/2
switch(config)#monitor session 1 source interface fastEthernet 0/3
switch(config)#monitor session 1 source interface fastEthernet 0/4
switch(config)#monitor session 1 source interface fastEthernet 0/5
switch(config)#monitor session 1 source interface fastEthernet 0/6
switch(config)#monitor session 1 source interface fastEthernet 0/7
switch(config)#monitor session 1 source interface fastEthernet 0/8

switch(config)#monitor session 1 destination interface gigabitEthernet 0/1

Show info:

switch#show monitor session 1
Session 1
---------
Type    : Local Session
Source Ports :
 Both: Fa0/1-8
Destination Ports : Gi0/1
 Encapsulation : Native
  Ingress : Disabled

And now we can capture this traffic:

# tcpdump -i eth0 -n

Monitor a processes memory and cpu usage over time

To do this I use the following command:

watch -d -n 1 'ps -o "user pid cmd pcpu pmem rss" $(pgrep apache)'

you can replace "apache" with the executable name of the process you want to monitor

How to Configure SNMP in Xenserver 5.x

Change firewall settings

You must change your firewall settings as follows to allow communication through the port that SNMP uses:

1. Open the file /etc/sysconfig/iptables in your preferred editor.

2. Add the following lines to the INPUT section after the line with -A RH-Firewall-1-INPUT –p udp –dport 5353... :

-A RH-Firewall-1-INPUT -p udp --dport 161 -j ACCEPT

3. Save and close the file.

4. Restart the firewall service:

# service iptables restart

Enable snmpd service
1. To enable snmpd service run the following command:

# chkconfig snmpd on

2. Start the snmpd service:

# service snmpd start

Change SNMP configuration
1. To change snmp configuration edit the /etc/snmp/snmpd.conf file.

2. Restart the snmpd service:

# service snmpd restart

SNMP configuration examples
Default settings
You can view only systemview subtree .1.3.6.1.2.1.1

View whole subtree

1. Change lines as follows:
After the lines lines starting with:

view systemview included (...)

Add this:

view all included .1

Change line:

access notConfigGroup “” any noauth exact systemview none none

To:

access notConfigGroup “” any noauth exact all none none

2. Save the file.

3. Restart the service:

# service snmpd restart

Change community string (default is “public”)
Change line:

com2sec notConfigUser default public

To:

com2sec notConfigUser default anything_you_need

Packet loss monitoring with zabbix

1. create a file named "packetloss" at this location "/etc/zabbix/externalscripts/"

vi /etc/zabbix/externalscripts/packetloss

note: you may need to create the external scripts directory:

mkdir -p /etc/zabbix/externalscripts

2. cut out and paste this in "packetloss" file

#!/bin/sh
if [ -z $1 ]
then
echo "missing ip / hostname address"
echo " example ./packetloss 192.168.201.1 10000"
echo "10000 = 10000 bytes to ping with. the more you use the harder the network will have to deliver it and you start see packetloss. ping with normal ping size is kinda pointless, on LAN networks I recommend to use 10000 - 20000 and on Internet around 1394 (1500 - 48(pppoe + IP + TCP) - 58(ipsec)"
echo "Remember some firewalls might block pings over 100"
echo " "
fi
if [ -z $2 ]
then
echo "missing ping size"
echo " example ./packetloss 192.168.201.1 10000"
echo "10000 = 10000 bytes to ping with. The more you use the harder the network will have to deliver
it and you start see packetloss. ping with normal ping size is kinda pointless, on LAN networks I recommend to use 10000 - 20000 and on Internet around 1394 (1500 - 48(pppoe + IP + TCP) - 58(ipsec)"
echo "Remember some firewalls might block pings over 100"
echo " "
exit
fi
PINGCOUNT = 10
tal=`ping -q -i0.30 -n -s $2 -c$PINGCOUNT $1 | grep "packet loss" | cut -d " " -f6 | cut -d "%" -f1`
if [ -z $tal ]
then
echo 100
else
echo $tal
fi

3. Make the file runnable by typing:

chmod u+x etc/zabbix/externalscripts/packetloss

4. in zabbix verify the host/template you want to monitor the packet loss on have a valid IP or host name and the correct "Connect to" selected.

Then under Item you create a new Item for that host/template

Type: External Check
Key: packetloss[10000]

SAVE

5. now check monitoring -> latest data for that host and you should start seeing packet loss values.

Done.

The number 10000 is Ping size, its very hard to spot packet loss when only sending a few bytes as a normal ping does.

Try increasing the size until you see packet loss then you know you pushing your equipment to the limit.

Performance Tuning MySQL for Zabbix

On my previous post I've shared some tips on how to tune ZABBIX configuration to get better results,however the most important tunning you have to do is to the data base server. Remember that this values depend on how much memory you have available on your server, here is how I've configured my MySQL server:

1. use a tmpfs tmpdir, create a folder like /mytmp and In /etc/my.cnf configure:

tmpdir=/mytmp

in /etc/fstab i put:

tmpfs /mytmp tmpfs size=1g,nr_inodes=10k,mode=700,uid=102,gid=105 0 0

You'll have to mkdir /mytmp and the numeric uid and gid values for your mysql user+group need to go on that line. Then you should be able to mount /mytmp and use tmpfs for mysql's temp directory. I don't know about the size and nr_inodes options there, I just saw those in linux tmpfs docs on the web and they seemed reasonable to me.

2. Buffer cache/pool settings.

In /etc/my.cnf jack up innodb_buffer_pool_size as much as possible. If you use /usr/bin/free the value in the "+/- buffer cache" row under the "free" column shows you how much buffer cache you have. I've also setup innodb to use O_DIRECT so that the data cached in the innodb buffer pool would not be duplicated in the filesystem buffer cache. So, in /etc/my.cnf:

innodb_buffer_pool_size=8000M
innodb_flush_method=O_DIRECT

3. Size the log files.

The correct way to resize this is documented here:

http://dev.mysql.com/doc/refman/5.0/en/adding-and-removing.html

In /etc/my.cnf the value I'm going to try is:

innodb_log_file_size=64M

A too small value means that MySQL is constantly flushing from the logfiles to the table spaces. It is better to increase this size on write-mostly databases to keep zabbix streaming to the logfiles and not flushing into the tablespaces constantly. However, the penalty is slower shutdown and startup times.

4. other parameters

innodb_file_per_table

Use file_per_table to keep tablespaces more compact and use "optimize table" periodically. And when you set this value in my.cnf you don't get an actual file_per_table until you run an optimize on all the tables. This'll take a long time on the large zabbix history* and trends* tables.
Turn on slow query logging:

log_slow_queries=/var/log/mysql.slow.log

This setting seems to affect the hit rate of Threads_created per Connection.

thread_cache_size=4

query_cache_limit=1M
query_cache_size=128M
tmp_table_size=256M
max_heap_table_size=256M
table_cache=256
max_connections = 400
join_buffer_size=256k
read_buffer_size=256k
read_rnd_buffer_size=256k 

This should help a lot for high volume writes.

innodb_flush_log_at_trx_commit=2

Simple Zabbix tunning tips

If you're getting gaps on ZABBIX's graphs and unknown status on some items, a little to often it might mean that you're monitoring server is low on performance, here are some general rules you can follow to boost ZABBIX performance:

• If the DB is located on the same host as zabbix, change zabbix_server.conf so it uses a Unix socket to connect to the DB

• Increase the number of pollers, trapers and pingers on the server config but don't overdo it.
  
  • General rule - keep value of this parameter as low as possible. Every additional instance of zabbix_server adds known overhead, in the same time, parallelism is increased. Optimal number of instances is achieved when queue, on average, contains minimum number of parameters (ideally, 0 at any given moment). This value can be monitored by using internal check zabbix[queue] or you can look at "Administration -> Queue" on the web interface.
• increase the number of processes on the agents configuration, again, don't overdo it.
• Change some of the items to use active checks (leave a few as regular checks so you can get availability information, leave stuff like host status as a regular check). Remember that the hostname set on the zabbix agent conf file must match the hostname given to the host on the web interface.
  
  • A regular check is initiated by ZABBIX server, it periodically sends requests to an agent to get latest info. The agent is passive, it just processes requests sent by the server.
  • An active check works the following way. ZABBIX agents connect to ZABBIX server to get a list of all checks for a host. Then, periodically, send required information to ZABBIX server. Note that ZABBIX server does not initiate anything. ZABBIX agent does all active work. This doesn't require polling on server side, thus it significantly (1.5x-2x) improve performance of ZABBIX server but if the host goes down the server won't get any information.
• monitor required parameters only
  
  However the most important tunning you have to make is to the DB server, in my next post I'll give you some advice on how to tune a MySQL server to boost ZABBIX performance.

Script to check if process is running

This is a skeleton of a watchdog script to check if a process is running:

#!/bin/bash
PROCESS="java"
log_found=`ps faux|grep -v grep|grep $PROCESS|grep -v $0|awk '{print $2}'`
if [ "$log_found" == "" ]; then
    echo "No process found"
else
    echo "Processes found:"
    for PID in $log_found; do
        echo $PID
    done
fi

You must change the PROCESS variable to your process name and add actions for when the process is or isn't found...

Zabbix - SMS with Gammu

Create an sms script (and make it executable) on the zabbix server in the AlertScriptsPath (=/etc/zabbix/alert.d/ on ubuntu)

#!/bin/sh
HOME=/etc
PATH=/bin:/sbin:/usr/bin:/usr/sbin
LOGFILE="/var/log/zabbix-server/zabbix-sms.log"
echo "Recipient='$1' Message='$3'" >> ${LOGFILE}
MOBILE_NUMBER=`echo "$1" | sed s#\s##`
# Log it
echo "echo $3 | /usr/bin/sudo /usr/bin/gammu --sendsms TEXT ${MOBILE_NUMBER}" >>${LOGFILE}
# Send it
echo "$3" | /usr/bin/sudo /usr/bin/gammu --sendsms TEXT "${MOBILE_NUMBER}" 1>>${LOGFILE} 2>&1
# EOF

Add:

zabbix ALL = NOPASSWD:/usr/bin/gammu

to the sudoers to make it available for the zabbix user

Configure a media type (menu administration) with the same name as your script (without path, without parameters)

Link this media type to a user (menu administration) and use the phone number as Send to parameter.

Dont forget to give zabbix a shell in /etc/passwd and permissions to write in the log file...

Note: if you have gammu configured as a daemon use gammu-smsd-inject instead of gammu --sendsms or

echo "$3" > /var/spool/gammu/outbox/OUT"${MOBILE_NUMBER}".txt

If you are using the files backend

I use a hawaii usb gsm modem and my /etc/gammurc looks like this:

[gammu]
port = /dev/ttyUSB0
model =
connection = at19200
synchronizetime = yes
logfile =
logformat = nothing
use_locking =
gammuloc =

Test Zabbix agent via telnet

From your server telnet to the agent... telnet host 10050

then type in a key to monitor such as the following

system.cpu.load[,avg1]

type the key on one line and press enter after. If your session gets closed without showing anything then it's likely a misconfiguration in your agent(d).conf file. Otherwise you should see a number.

Defunct Zabbix_Server Processes

Some times Zabbix stops collecting data and all zabbix_server processes are defunct.

Restarting the Zabbix server doesn't fix the defunct processes. According to the manual page (http://www.zabbix.com/documentation/1.8/manual/processes/zabbix_server) on the server config, we can set CacheSize to anywhere between 128K and 1G, the default being 8M. We set it to 32M, and restart the server. Now it's running again. Allegedly, in order to set it any higher, it is necessary to first adjust some kernel parameters with sysctl, like this:

sysctl -w kernel.shmmax=536870912

Or, according to thread 61016 on the Zabbix forum:

Spesific for FreeBSD machine:
kern.ipc.shmall=2097152
kern.ipc.shmmax=2147483648
kern.ipc.shmmni=4096
kern.ipc.semmsl=250
kern.ipc.semmns=32000
kern.ipc.semopm=100
kern.ipc.semmni=128

Install a monitoring Zabbix Server

Install the packages:

aptitude install zabbix-server-mysql zabbix-frontend-php

Set the php time zone:

vi /etc/php5/apache2/php.ini

• search for date.timezone and set it to "Europe/Lisbon"
• set "mbstring.func_overload" to 2
• set post_max_size to 16M
• set max_execution_time to 300

Restart apache

/etc/init.d/apache2 restart

SNMP Builder is a very useful add-on for Zabbix to help you to build custom SNMP based tamplates, to install-it follow the instructions on the following link:
http://www.zabbix.com/wiki/howto/monitor/snmp/snmp_builder

Look for CISCO MIBs here -> http://tools.cisco.com/Support/SNMP/do/BrowseMIB.do?local=en

goto http://<ip_address>/zabbix

Log in using:

admin/zabbix

Setup Auto Discovery:
http://www.zabbix.com/documentation/1.8/manual/auto-discovery

Template Linux:

there are several items in the Linux template that are not useful fore every server so I disable the following items and enable them only when needed:

• Email (SMTP) server is running
• FTP server is running
• IMAP server is running
• News (NNTP) server is running
• Number of running processes zabbix_server
• Number of running processes inetd
• Number of running processes mysqld
• POP3 server is running

Monitor Apache:

The Debian installation uses apache2 as the Apache process, so if you have a Debian based server farm, you should change your Number of running processes apache item to apache2. The SuSE apache process is named httpd2-prefork, so change the item, if needed. Maybe you need to clone the item, if this is the only server of this type.

Monitor SQUID from Zabbix:
http://www.zabbix.com/wiki/howto/monitor/web/squid

However I had to configure the zabbix_agentd.conf a little bit differently:

# Squid
UserParameter=squid.http_requests,squidclient -h 192.168.100.19 -p 80 mgr:info|grep 'Number of HTTP requests received:'|cut -d':' -f2| tr -d ' \t'
UserParameter=squid.clients,squidclient -h 192.168.100.19 -p 80 mgr:info|grep 'Number of clients accessing cache:'|cut -d':' -f2| tr -d ' \t'
UserParameter=squid.icp_received,squidclient -h 192.168.100.19 -p 80 mgr:info|grep 'Number of ICP messages received:'|cut -d':' -f2| tr -d ' \t'
UserParameter=squid.icp_sent,squidclient -h 192.168.100.19 -p 80 mgr:info|grep 'Number of ICP messages sent:'|cut -d':' -f2| tr -d ' \t'
UserParameter=squid.icp_queued,squidclient -h 192.168.100.19 -p 80 mgr:info|grep 'Number of queued ICP replies:'|cut -d':' -f2| tr -d ' \t'
UserParameter=squid.htcp_received,squidclient -h 192.168.100.19 -p 80 mgr:info|grep 'Number of HTCP messages received:'|cut -d':' -f2| tr -d ' \t'
UserParameter=squid.htcp_sent,squidclient -h 192.168.100.19 -p 80 mgr:info|grep 'Number of HTCP messages sent:'|cut -d':' -f2| tr -d ' \t'
UserParameter=squid.req_fail_ratio,squidclient -h 192.168.100.19 -p 80 mgr:info|grep 'Request failure ratio:'|cut -d':' -f2| tr -d ' \t'
UserParameter=squid.avg_http_req_per_min,squidclient -h 192.168.100.19 -p 80 mgr:info|grep 'Average HTTP requests per minute since start:'|cut -d':' -f2| tr -d ' \t'
UserParameter=squid.avg_icp_msg_per_min,squidclient -h 192.168.100.19 -p 80 mgr:info|grep 'Average ICP messages per minute since start:'|cut -d':' -f2| tr -d ' \t'
UserParameter=squid.request_hit_ratio,squidclient -h 192.168.100.19 -p 80 mgr:info|grep 'Hits as % of all requests:'|cut -d':' -f3|cut -d',' -f1|tr -d ' %'
UserParameter=squid.byte_hit_ratio,squidclient -h 192.168.100.19 -p 80 mgr:info|grep 'Hits as % of bytes sent:'|cut -d':' -f3|cut -d',' -f1|tr -d ' %'
UserParameter=squid.request_mem_hit_ratio,squidclient -h 192.168.100.19 -p 80 mgr:info|grep 'Memory hits as % of hit requests:'|cut -d':' -f3|cut -d',' -f1|tr -d ' %'
UserParameter=squid.request_disk_hit_ratio,squidclient -h 192.168.100.19 -p 80 mgr:info|grep 'Disk hits as % of hit requests:'|cut -d':' -f3|cut -d',' -f1|tr -d ' %'
UserParameter=squid.servicetime_httpreq,squidclient -h 192.168.100.19 -p 80 mgr:info|grep 'HTTP Requests (All):'|cut -d':' -f2|tr -s ' '|awk '{print $1}'
UserParameter=squid.process_mem,squidclient -h 192.168.100.19 -p 80 mgr:info|grep 'Process Data Segment Size via sbrk'|cut -d':' -f2|awk '{print $1}'
UserParameter=squid.cpu_usage,squidclient -h 192.168.100.19 -p 80 mgr:info|grep 'CPU Usage:'|cut -d':' -f2|tr -d '%'|tr -d ' \t'
UserParameter=squid.cache_size_disk,squidclient -h 192.168.100.19 -p 80 mgr:info|grep 'Storage Swap size:'|cut -d':' -f2|awk '{print $1}'
UserParameter=squid.cache_size_mem,squidclient -h 192.168.100.19 -p 80 mgr:info|grep 'Storage Mem size:'|cut -d':' -f2|awk '{print $1}'
UserParameter=squid.mean_obj_size,squidclient -h 192.168.100.19 -p 80 mgr:info|grep 'Mean Object Size:'|cut -d':' -f2|awk '{print $1}'
UserParameter=squid.filedescr_max,squidclient -h 192.168.100.19 -p 80 mgr:info|grep 'Maximum number of file descriptors:'|cut -d':' -f2|awk '{print $1}'
UserParameter=squid.filedescr_avail,squidclient -h 192.168.100.19 -p 80 mgr:info|grep 'Available number of file descriptors:'|cut -d':' -f2|awk '{print $1}'

Monitor Postfix from Zabbix:

http://www.zabbix.com/wiki/howto/monitor/mail/postfix/monitoringpostfix

but use the script from

http://www.zabbix.com/forum/showthread.php?p=74271

Configure Web senarios:
http://www.zabbix.com/documentation/1.8/manual/web_monitoring

Download extra templates:
http://www.zabbix.com/wiki/templates/start

Download Zabbix SNMP template generator:

This is a very useful script to automatically generate templates for your network equipment.
http://www.zabbix.com/wiki/scripts/gentemplate.pl

Create calculated Items:
http://www.zabbix.com/documentation/1.8/manual/config/items?s[]=calculated#calculated_items

Advanced parcing of SNMP Values:
http://www.zabbix.com/forum/showthread.php?t=19343

Install zabbix linux clients:

sudo aptitude install zabbix-agent
sudo dpkg-reconfigure zabbix-agent

Input the zabbix server IP address

Now edit the agent conf file:

vi /etc/zabbix/zabbix-agentd.conf

and change the hostname to the correct one

Restat the agent:

sudo /etc/init.d/zabbix-agent restart

For windows get the installer from:
http://www.suiviperf.com/zabbix/ or from http://www.zabbix.com/download.php

Please check my posts about Zabbix Performance:
Simple zabbix tunning tips
Performance tuning mysql for zabbix