Nagios Conference 2011 - Daniel Wittenberg - Scaling Nagios At A Giant Insurance Company
Download as odp, pdf1 like4,847 views
Daniel Wittenburg' presentation on a reference story for a German Health Insurance Company. The presentation was given during the Nagios World Conference North America held Sept 27-29th, 2011 in Saint Paul, MN. For more information on the conference (including photos and videos), visit: http://go.nagios.com/nwcna
![Scaling Nagios At A Giant Insurance Company Daniel Wittenberg [email_address] https://github.com/dwittenberg2008/nagios](https://image.slidesharecdn.com/danielwittenberg-wittenbergpresentation-111003123211-phpapp02/85/Nagios-Conference-2011-Daniel-Wittenberg-Scaling-Nagios-At-A-Giant-Insurance-Company-1-320.jpg)
More Related Content
What's hot (20)
Viewers also liked (8)
Similar to Nagios Conference 2011 - Daniel Wittenberg - Scaling Nagios At A Giant Insurance Company (20)
More from Nagios (20)
Recently uploaded (20)
Nagios Conference 2011 - Daniel Wittenberg - Scaling Nagios At A Giant Insurance Company
- 1. Scaling Nagios At A Giant Insurance Company Daniel Wittenberg [email_address] https://github.com/dwittenberg2008/nagios
- 2. Personal Background Certified for HP-UX in mid 90's, then RHCE in '99, and AIX in early 2000's.
- 3. Worked on lots of different technologies and solutions including HA, SAN/iSCSI, Forensics/Security, Backups/Disaster Recovery, Performance Tuning, Capacity Planning, Monitoring/Trending, Networking/Protocol Analysis, Virtualization and Cloud Computing.
- 4. Consulted and worked in many industries include insurance, banking, accounting, construction, embedded hardware design, printing/publishing early education, higher education, and ISP/hosting providers.
- 7. Nagios Core
- 8. Plugins
- 10. Event Brokers
- 11. Other Software
- 13. General
- 14. Overview
- 15. Hardware Hardware vs VMware High forking rate not good fit for VMware CPU Requirements Quantity vs Quality Memory Typically memory efficient, but have enough for ramdisk(s)
- 16. Affected by your plugins if using active checks Disk I/O Faster the better!
- 17. VMware Performance Comparison Procs Memory Hosts Avg Svc Lat Avg CPU Util Avg CPU Load # Act Checks # Pass Checks 4 8 GB 1000 182 65 5 12084 6042 8 8 GB 1000 162 47 4 12084 6042 8 8 GB 600 87 60 8 7272 3636 Isolated VMWare ESX 4 Physical Dell PowerEdge R710 (new) Physical HP Proliant DL380 G4 (~ 8 years old) Procs Memory Hosts Avg Svc Lat Avg CPU Util Avg CPU Load # Act Checks # Pass Checks 4 16 GB 1000 0.19 10 1.25 12084 6042 8 8 GB 1000 0.38 25 1.15 12084 6042 Procs Memory Hosts Avg Svc Lat Avg CPU Util Avg CPU Load # Act Checks # Pass Checks 4 4 GB 800 0.29 32 1.95 9684 4842 8 4 GB 1000 0.47 37 4.43 12084 6042
- 18. Operating System CentOS / RHEL 5.6
- 19. Strip down the running services
- 20. Create ramdisks in Nagios RC script
- 21. - first one for status.dat, checkresults, temp_file
- 22. - second one for pnp processing directory, moved there by nagios perf command
- 23. - nagios.rc on github for full rc script ramdisk=`mount |grep "/var/nagios/ramcache"` if [ "$ramdisk"X == "X" ]; then mkdir -p -m 755 /var/nagios/ramcache mount -t tmpfs -o size=128m tmpfs /var/nagios/ramcache mkdir -p -m 755 /var/nagios/ramcache/checkresults chown -R nagios:nagios /var/nagios/ramcache fi
- 24. Operating System Make sure no ulimit restrictions
- 25. ulimit -a
- 26. Renice daemons and services
- 27. daemon -15 --user=$user $exec -d $config
- 28. perfdata_file_run_cmd =/bin/nice -n 20 /usr/libexec/pnp4nagios/process_perfdata.pl
- 29. puppet runs also re-niced (/etc/sysconfig/puppet – NICELEVEL=19)
- 30. Nagios Core Currently using Nagios 3.2.3 Empty hostgroups patch
- 31. Non-existent device patch (causes sigfault w/pipe) Large Scale Suggestions Doc Pre-caching objects
- 32. Re-write RC script to optimize restart time
- 33. Don't allow restart/stop if config broken
- 34. Limit use of macros (resources.cfg)
- 35. Nagios Core Remove use of CGI's Using Livestatus/Multisite/livestatus-slave Limit use of backups (crazy huh?)
- 36. Keep logging level low in all core and plugins/brokers
- 37. Keep comments limited, delete if X # or Y days old
- 39. (how often to update the status.dat in seconds)
- 41. (pass macros as ENV variables)
- 43. check_logfiles
- 44. check_hpasm / check_dell_sensors / check_dell_omreport
- 46. check_ps.sh (re-written for perf data, correct calculations)
- 48. Return perf data whenever possible
- 49. Many other custom and one-up plugins
- 50. Other Add-ons NRPE Patched to allow large buffer size NSCA – (NRDP future ?) max_packet_age=60, forward and back time patch
- 51. Run from xinetd to allow larger/faster connections
- 52. MUST use instances = UNLIMITED
- 53. Recommend per_source = UNLIMITED
- 54. Recommend cps = 5000 3 NSClient++ Many updates for buffering, data truncation, queueing PNP4Nagios 0.6 branch, since php53 RPM's available
- 57. MK Livestatus
- 58. Performance Data Splunker (custom)
- 59. Log separator (reduces grepping for messages) (custom)
- 60. Other Software Puppet Manage entire server Splunk Log files, performance data Graylog2 ? (not implemented yet)
- 61. Cacti Nagiostats template, updated to use livestatus instead of CGI Custom Control Panel Build host groups based on templates, auto-config based on host info ConSol Labs check_logfiles, check_hpasm, mod_gearman, check_mysql_health, check_oracle_health
- 62. Performance Monitoring How to watch your system to determine bottlenecks vmstat
- 63. iostat
- 64. top
- 65. iptraf
- 66. sar
- 67. strace
- 68. esxtop (if have to use VM)
- 69. General Configs Host config files are standalone configurations that tell everything about a host.
- 70. Hosts are tied to a hostgroup
- 71. Hostgroups are tied to a servicegroup
- 72. Services are tied to a servicegroup
- 73. host.cfg -> hostgroups -> servicegroups ← services
- 74. This allows for easy drop-in and removal of hosts, but also requires at least 1 host be assigned to a management server
- 75. Limitations – harder to make per-server per-service customizations
- 76. Hosts are built/assigned from control panel (round-robin distribution)
- 77. Parents built automatically from topology database, updated nightly, ESX hourly
- 78. Parent's only ping once a day unless there are problems, uses fping
- 79. Some alerts do trigger eventhandlers – auto-fix as much as possible
- 80. General Configs Types of things being monitored: cpu load, cpu stats (idle/wait/user/system), disk space, log files/Event Log, hardware, processes, swap, memory usage, service ports, NTP drift, cron job completion, UPS
- 81. Nagios configtest, livestatus connectivity
- 82. PNP4Nagios directory size (keeping up on processing)
- 83. Performance (cpu/memory) usage on certain processes
- 84. Puppet update time to make sure doesn't get behind
- 85. DB Response times (oracle/mysql/postgresql)
- 86. Apache Stats
- 87. Custom app status (user accounts, response times, loads, etc.)
- 88. Various SNMP/WMI values (most network related stats)
- 89. Links – where to find this stuff My Stuff – https://github.com/dwittenberg2008/nagios
- 90. MK Livestatus - http://mathias-kettner.de/checkmk_livestatus.html
- 91. LivestatusSlave - http://nagios.larsmichelsen.com/livestatusslave/
- 92. PNP4Nagios - http://docs.pnp4nagios.org/pnp-0.6/start
- 93. ConSol Labs - http://labs.consol.de/
- 94. Puppet - http://puppetlabs.com/
- 95. DNX - http://dnx.sourceforge.net/
- 96. Cacti Template (Base) - http://forums.cacti.net/about33806.html
- 97. Questions ? Suggestions ?