Hierarchical/Distributed Nagios
Leland V. Lammert, PhD
Omnitec Corporation
St. Louis, MO
lvl@omnitec.net
@lvlammert
Introduction & Agenda
• The problem:
• Multiple Remote Sites
• System Status/Local Nagios
• Central Dispatch for probl...
Background
● Specialists in Open Source Solutions
● Core IT Infrastructure
● Multi-platform Nagios
● Leland V. Lammert...
SRM Hardware
SRM Hardware
● Raspberry Pi B
or
● Raspberry Pi B+
● R-Pi Camera
● Temperature Probe
● Cyntech case w/SD cover
SRM Hardware
SRM Hardware
● Raspberry Pi Camera
SRM Hardware
Temperature Probe Options
DHT11
Temperature and Humidity
DS18B20
OneWire
Temperature Only
SRM Hardware
SD Card
Camera Temperature
Sensor
Status
LEDs
Server Room Monitor [SRM]
Software Build
Software Build
● Debian for Pi - Raspian
http://www.raspbian.org/
● Download from Raspberry Pi Project
http://www.rasp...
Software Build
● Add packages for Nagios & admin:
– php5, php5-commin, php5-cli
– (apache2-mpm conflicts, will be repla...
Software Build
● Setup GPIO for the DS18B20
– Add to /etc/modules:
w1_gpio
w1_therm
– Use modprobe to install manuall...
Software Build
● Check the ID of the installed device:
● ll /sys/bus/w1/devices/
/sys/bus/w1/devices/28-00000489b929/w1...
Software Build
● Create startup script - /root/SRMStart
# Start the camera
raspistill -t 0 -tl 1500 -o /usr/local/nagi...
Software Build
● Add SRMStart to /etc/rc.local:
##
Start the SRM components
#
/root/SRMstart
Software Build
● Install Nagios – NagiosCore
● Download and build from Source
– ./configure -with-gd-lib=/usr/local/lib...
Software Build
● Install NSCA
– ./configure && make
– nsca deamon, send_nsca client
– Copy binaries to /usr/local/nagi...
Software Build
● Build a local copy of plugins
– ./configure, make, make install
– Installs to /usr/local/nagios/libexe...
Software Build
● Other system plugins
– Build in that environment
● Linux
● Windows
● Mac
● Verify operation
– ./ch...
Network Issues
Network Issues
Typical network configuration
?
Network Issues
● An ssh tunnel would normally be
required to traverse the firewall for
each remote host which is defini...
Network Issues
● How does NSCA allow one Nagios system
communicate with another?
● By forwarding notifications to a rem...
Network Issues
Remote Site(s) Central Dispatch
One open
port
required!
Nagios Configuration
Nagios Configuration
Nagios system communications
• Central system has NSCA Server which listens for events, normally in...
Nagios Configuration
● send_nsca Configuration
– password=fV5P0V8ggaFtNE
– encryption_method=1
● nsca daemon Configura...
Nagios Configuration
● Encryption Options
0 = None 1 = Simple XOR 2 = DES
3 = 3DES (Triple DES) 4 = CAST-128 5 = CAST-2...
Nagios Configuration
define command {
command_name notify-service-by-nsca
command_line /usr/bin/printf
"%st%st%st%sn" ...
Nagios Configuration
define command {
command_name notify-host-by-nsca
command_line /usr/bin/printf "%st%st%sn"
"$HOS...
Nagios Configuration
define contact {
contact_name Central_Dispatch
service_notification_period 24x7
host_notification...
Nagios Configuration
● Central Site
– Set the service definition or template to passive
passive_check_enabled=1, and ac...
Nagios Configuration
● Test service template on Central Server
define service {
name passive-service
use generic-servi...
Nagios Configuration
● Test service on Central Server
define service {
use passive-service
host_name localhost
servic...
Nagios Configuration
● Test service on Remote SRM
#!/usr/bin/perl
#####################################################...
Nagios Configuration
● Test service on Remote SRM (cont'd)
# DEFAULT RETURNS
$code=3;
$result="What's going on";
# CO...
Nagios Configuration
● System troubleshooting tips
– Check logs
● /usr/local/nagios/logs
● /var/log/apache2/error.log ...
Distributed/Hierarchical
Architecture
• Low cost
• Local Nagios Instance
• Central monitoring
and dispatch
Conclusio...
Ongoing work, ..
● Motion Video
– B+ hardware
– OpenGL accssible (non-proprietary)
● Configuration UI
– Adagios
– No...
Questions?
Thank you!
The End
Lee Lammert
lvl@omnitec.net
@lvlammert
of 42

Nagios Conference 2014 - Leland Lammert - Distributed Heirarchical Nagios

Leland Lammert's presentation on Distributed Heirarchical Nagios. The presentation was given during the Nagios World Conference North America held Oct 13th - Oct 16th, 2014 in Saint Paul, MN. For more information on the conference (including photos and videos), visit: http://go.nagios.com/conference
Published on: Mar 3, 2016
Published in: Technology      
Source: www.slideshare.net


Transcripts - Nagios Conference 2014 - Leland Lammert - Distributed Heirarchical Nagios

  • 1. Hierarchical/Distributed Nagios Leland V. Lammert, PhD Omnitec Corporation St. Louis, MO lvl@omnitec.net @lvlammert
  • 2. Introduction & Agenda • The problem: • Multiple Remote Sites • System Status/Local Nagios • Central Dispatch for problems • The solution: • SRM Hardware • Software Build • Network Issues • Nagios Configuration
  • 3. Background ● Specialists in Open Source Solutions ● Core IT Infrastructure ● Multi-platform Nagios ● Leland V. Lammert, PhD Chief Scientist
  • 4. SRM Hardware
  • 5. SRM Hardware ● Raspberry Pi B or ● Raspberry Pi B+ ● R-Pi Camera ● Temperature Probe ● Cyntech case w/SD cover
  • 6. SRM Hardware
  • 7. SRM Hardware ● Raspberry Pi Camera
  • 8. SRM Hardware Temperature Probe Options DHT11 Temperature and Humidity DS18B20 OneWire Temperature Only
  • 9. SRM Hardware SD Card Camera Temperature Sensor Status LEDs Server Room Monitor [SRM]
  • 10. Software Build
  • 11. Software Build ● Debian for Pi - Raspian http://www.raspbian.org/ ● Download from Raspberry Pi Project http://www.raspberrypi.org/downloads/ ● Or, purchase a pre-installed SD ● First task, update: aptitude update & upgrade
  • 12. Software Build ● Add packages for Nagios & admin: – php5, php5-commin, php5-cli – (apache2-mpm conflicts, will be replaced) – chkconfig – telnet – libgd2-xpm-dev – heirloom-mailx (MUA) – ssmtp (MTA)
  • 13. Software Build ● Setup GPIO for the DS18B20 – Add to /etc/modules: w1_gpio w1_therm – Use modprobe to install manually modprobe [w1_gpio, w1_therm]
  • 14. Software Build ● Check the ID of the installed device: ● ll /sys/bus/w1/devices/ /sys/bus/w1/devices/28-00000489b929/w1_slave ^^^^^^^^^^^^^^^^^^ ● ID on this device 28-00000489b929
  • 15. Software Build ● Create startup script - /root/SRMStart # Start the camera raspistill -t 0 -tl 1500 -o /usr/local/nagios/share/tmp/snapshot.jpg &>/dev/null & # Start the NSCA daemon /usr/local/nagios/libexec/nsca -c /usr/local/nagios/etc/nsca.cfg –daemon # Setup tunnel if required #/usr/bin/autossh -a -M 9005:1005 -N -R <remote>:127.0.0.1:<local> link@nagios.omnitec.net &
  • 16. Software Build ● Add SRMStart to /etc/rc.local: ## Start the SRM components # /root/SRMstart
  • 17. Software Build ● Install Nagios – NagiosCore ● Download and build from Source – ./configure -with-gd-lib=/usr/local/lib/, make, make install – make install-init - Install the init script in /etc/init.d – make install-commandmode - Install and sets permissions on the directory for external commands – make install-config - Install sample config files in /usr/local/nagios/etc
  • 18. Software Build ● Install NSCA – ./configure && make – nsca deamon, send_nsca client – Copy binaries to /usr/local/nagios/libexec – Configs to usr/local/nagios/etc ● Create password and add to both configs ● Initial encryption XOR, select as appropriate
  • 19. Software Build ● Build a local copy of plugins – ./configure, make, make install – Installs to /usr/local/nagios/libexec ● Verify operation – cd /usr/local/nagios/libexec – ./check_
  • 20. Software Build ● Other system plugins – Build in that environment ● Linux ● Windows ● Mac ● Verify operation – ./check_ [in that environment]
  • 21. Network Issues
  • 22. Network Issues Typical network configuration ?
  • 23. Network Issues ● An ssh tunnel would normally be required to traverse the firewall for each remote host which is definitely not ideal ● Solution – use NSCA! ● By forwarding data via NSCA, firewall changes are eliminated, .. greatly reducing load on central system!
  • 24. Network Issues ● How does NSCA allow one Nagios system communicate with another? ● By forwarding notifications to a remote system! – Notifications for Services – Notifications for Hosts – Using a specific Contact ● Only requires one open port, at the Central site!
  • 25. Network Issues Remote Site(s) Central Dispatch One open port required!
  • 26. Nagios Configuration
  • 27. Nagios Configuration Nagios system communications • Central system has NSCA Server which listens for events, normally in 5667 • Remote systems have NSCA Client which sends events to the Server on 5667 • Notifications are encrypted according to the NSCA configuration
  • 28. Nagios Configuration ● send_nsca Configuration – password=fV5P0V8ggaFtNE – encryption_method=1 ● nsca daemon Configuration – password=fV5P0V8ggaFtNE – encryption_method=1 1 = Simple XOR (Just obfuscation, but very fast)
  • 29. Nagios Configuration ● Encryption Options 0 = None 1 = Simple XOR 2 = DES 3 = 3DES (Triple DES) 4 = CAST-128 5 = CAST-256 6 = xTEA 7 = 3WAY 8 = BLOWFISH 9 = TWOFISH 10 = LOKI97 11 = RC2 12 = ARCFOUR 14 = RIJNDAEL-128 15 = RIJNDAEL-192 16 = RIJNDAEL-256 19 = WAKE 20 = SERPENT 22 = ENIGMA (Unix crypt) 23 = GOST 24 = SAFER64 25 = SAFER128 26 = SAFER+
  • 30. Nagios Configuration define command { command_name notify-service-by-nsca command_line /usr/bin/printf "%st%st%st%sn" "$HOSTNAME$" "$SERVICEDESC$" $SERVICESTATEID$" "$SERVICEOUTPUT$ | $SERVICEPERFDATA$" | tee -a /tmp/service_alert.log | /usr/local/nagios/libexec/send_nsca -H $CONTACTADDRESS1$ -c /usr/local/nagios/etc/send_nsca.cfg }
  • 31. Nagios Configuration define command { command_name notify-host-by-nsca command_line /usr/bin/printf "%st%st%sn" "$HOSTNAME$" "$HOSTSTATEID$" "$HOSTOUTPUT$" | /usr/sbin/send_nsca -H $CONTACTADDRESS1$ -c /usr/local/nagios/etc/send_nsca.cfg }
  • 32. Nagios Configuration define contact { contact_name Central_Dispatch service_notification_period 24x7 host_notification_period 24x7 service_notification_options w,u,c,r,f,s host_notification_options d,u,r,f,s service_notification_commands notify-service-by-nsca host_notification_commands notify-host-by-nsca address1 nagios.omnitec.net }
  • 33. Nagios Configuration ● Central Site – Set the service definition or template to passive passive_check_enabled=1, and active_checks_enabled=0. – Note: For any service to be monitored upstream, the hostname and service description must match exactly the data being sent via send_nsca!
  • 34. Nagios Configuration ● Test service template on Central Server define service { name passive-service use generic-service check_freshness 1 passive_checks_enabled 1 active_checks_enabled 0 is_volatile 0 flap_detection_enabled 0 notification_options w,u,c,s freshness_threshold 57600 ;12hr }
  • 35. Nagios Configuration ● Test service on Central Server define service { use passive-service host_name localhost service_description test check_command check_dummy!3!"No Data Received" }
  • 36. Nagios Configuration ● Test service on Remote SRM #!/usr/bin/perl ############################################################# # RETURN CODES: # 0-OK, 1-WARNING, 2-CRITICAL, 3-UNKNOWN ############################################################# #CONFIG FILES #$debug=1; $config="/usr/local/nagios/etc/send_nsca.cfg"; # LOCAL SYSTEM CONFIG OPTIONS $nsca_host="srm.omnitec.net"; $host="DevelSRM"; $service="test_service";
  • 37. Nagios Configuration ● Test service on Remote SRM (cont'd) # DEFAULT RETURNS $code=3; $result="What's going on"; # COMMAND LINE $send_nsca="/usr/local/nagios/bin/send_nsca -c $config -H $nsca_host"; # Start # INSERT YOUR FUN CODE HERE, Setting a $code and $result value # End if ($debug) {print "SENDING: $hostt$servicet$codet$resultn";} open(SEND,"|$send_nsca") || die "Could not run $send_nsca: $!n"; print SEND "$hostt$servicet$codet$resultn"; close SEND;
  • 38. Nagios Configuration ● System troubleshooting tips – Check logs ● /usr/local/nagios/logs ● /var/log/apache2/error.log – Verify UI user permissions ● /usr/local/nagios/etc/htpssword.users
  • 39. Distributed/Hierarchical Architecture • Low cost • Local Nagios Instance • Central monitoring and dispatch Conclusion
  • 40. Ongoing work, .. ● Motion Video – B+ hardware – OpenGL accssible (non-proprietary) ● Configuration UI – Adagios – Not fully NagiosCore 4 compatible yet
  • 41. Questions? Thank you!
  • 42. The End Lee Lammert lvl@omnitec.net @lvlammert

Related Documents