Linux default checks and policies
Agent Client Collector provides the following default checks and policies for Linux Metrics monitoring.
Linux monitoring metrics checks
| Check | Metric Name | Resource | Description | Units | Featured Metric | Anomaly Detection |
|---|---|---|---|---|---|---|
| os.linux.metrics-process-usage | proc.acc.running | process-name | Number of processes running with this name\(acc\) | count | ||
| proc.acc.cpuPercent | process-name | Percentage of cpu taken by the process. | percent | |||
| proc.acc.memPercent | process-name | Percentage of memory taken by the process. | percent | |||
| os.linux.metrics-reboot-count-today | reboot.count.today | empty | Number of reboot done on today | count | ||
| os.linux.metrics-system-cpu | cpu.total.user | total | Normal processes executing in user mode; cpu.total.user is the total of the cpuN.user metrics. | count | ||
| cpu.total.nice | total | Niced processes executing in user mode; cpu.total.nice is the total of the cpuN.nice metrics. | count | |||
| cpu.total.system | total | Time the CPU spent running the kernel; cpu.total.system is the total of the cpuN.system metrics. | sec | |||
| cpu.total.idle | total | Total time the CPU spent in an idle state.; cpu.total.idle is the total of the cpuN.idle metrics. | sec | |||
| cpu.total.iowait | total | Total time the CPU spent waiting for IO operations to complete.; cpu.total.iowait is the total of the cpuN.iowait metrics. | sec | |||
| cpu.total.irq | total | Total time that the processor is spending on handling Interrupts.; cpu.total.irq is the total of the cpuN.irq metrics. | sec | |||
| cpu.total.softirq | total | Time spent on servicing soft interrupt requests; cpu.total.softirq is the total of the cpuN.softirq metrics. | sec | |||
| cpu.total.steal | total | Total time the virtual CPU spent waiting for the hypervisor to service another virtual CPU. Only applies to virtual machines. | sec | |||
| cpu.total.guest | total | Total time the CPU spent running the virtual processor. Only applies to hypervisors. | sec | |||
| cpu.total.guest\_nice | total | Total time the CPU spent running as nice guest OS. cpu.total.guset\_nice si the total of the cpuN.guest\_nice metrics | sec | |||
| cpu.<cpu-core>.user | cpu-core | Time spent with normal processing in user mode. | sec | |||
| cpu.<cpu-core>.nice | cpu-core | Time spent with niced processes in user mode. | sec | |||
| cpu.<cpu-core>.system | cpu-core | Time spent running in kernel mode. | sec | |||
| cpu.<cpu-core>.idle | cpu-core | Time spent in vacations twiddling thumbs. | sec | |||
| cpu.<cpu-core>.iowait | cpu-core | Time spent waiting for I/O to completed. This is considered idle time too. | sec | |||
| cpu.<cpu-core>.irq | cpu-core | Time spent serving hardware interrupts. | sec | |||
| cpu.<cpu-core>.softirq | cpu-core | Time spent serving software interrupts. | sec | |||
| cpu.<cpu-core>.steal | cpu-core | Time stolen by other operating systems running in a virtual environment. | sec | |||
| cpu.<cpu-core>.guest | cpu-core | Time spent for running a virtual CPU or guest OS under the control of the kernel. | sec | |||
| cpu.<cpu-core>.guest\_nice | cpu-core | Total time the CPU spent running as nice guest OS. | sec | |||
| cpu.intr | empty | Interrupts serviced since boot time. | sec | |||
| cpu.ctxt | empty | Total number of context switches across all CPUs. | count | |||
| cpu.btime | empty | The time the system booted | sec | |||
| cpu.processes | empty | The number of processes and threads created, which includes \(but is not limited to\) those created by calls to the fork\(\) and clone\(\) system calls. | count | |||
| cpu.procs\_running | empty | The total number of processes running on all CPUs. | count | |||
| cpu.procs\_blocked | empty | The number of processes currently blocked, waiting for I/O to complete. | count | |||
| cpu.cpu\_count | empty | Number of cpu on the system | count | |||
| cpu.<cpu-core>.cores | cpu-core | The number of CPU cores | core count | |||
| os.linux.metrics-system-cpu-load | load\_avg.one | empty | The average system load over one minute. | thread count | yes | yes |
| load\_avg.five | empty | The average system load over five minutes. | thread count | yes | yes | |
| load\_avg.fifteen | empty | The average system load over fifteen minutes. | thread count | yes | yes | |
| load\_avg.norm.one | empty | The average system load over one minute normalized by the number of CPUs. | thread count | |||
| load\_avg.norm.five | empty | The average system load over five minutes normalized by the number of CPUs. | thread count | |||
| load\_avg.norm.fifteen | empty | The average system load over fifteen minutes normalized by the number of CPUs. | thread count | |||
| os.linux.metrics-system-cpu-percentage | cpu.avgutilization\_percentage | empty | Percent of cpu was used on average | percent | ||
| cpu.user\_percentage | empty | Percent of time total cpu was used by normal processes in user mode | percent | yes | yes | |
| cpu.nice\_percentage | empty | Percent of time all cpus used by niced processes in user mode | percent | yes | yes | |
| cpu.system\_percentage | empty | The percent of time the CPU spent running the kernel. | percent | yes | yes | |
| cpu.idle\_percentage | empty | Percent of time all cpus were idle | percent | yes | yes | |
| cpu.iowait\_percentage | empty | Percent of time all cpus waiting for I/O to complete | percent | yes | yes | |
| cpu.irq\_percentage | empty | Percent of time all cpus servicing interrupts | percent | yes | yes | |
| cpu.softirq\_percentage | empty | Percent of time all cpus servicing software interrupts | percent | yes | yes | |
| cpu.steal\_percentage | empty | Percent of time all cpus serviced virtual hosts operating systems | percent | yes | yes | |
| cpu.guest\_percentage | empty | Percent of time all cpus serviced guest operating system | percent | yes | yes | |
| os.linux.metrics-system-disk | disk.<disk-name>.reads | disk-name | Total number of reads completed successfully. | count | yes | yes |
| disk.<disk-name>.readsMerged | disk-name | Total number of reads merged | count | |||
| disk.<disk-name>.sectorsRead | disk-name | Total number of sectors read successfully. | count | |||
| disk.<disk-name>.readTime | disk-name | Total number of milliseconds spent by all reads. | millisec | |||
| disk.<disk-name>.writes | disk-name | Total number of writes completed successfully. | count | yes | yes | |
| disk.<disk-name>.writesMerged | disk-name | Total number of writes merged | count | |||
| disk.<disk-name>.sectorsWritten | disk-name | Total number of sectors written successfully. | count | |||
| disk.<disk-name>.writeTime | disk-name | Total number of milliseconds spent by all writes. | misllisec | |||
| disk.<disk-name>.ioInProgress | disk-name | Total number of I/Os currently in progress | count | |||
| disk.<disk-name>.ioTime | Total time spent doing I/Os | millisec | yes | yes | ||
| disk.<disk-name>.ioTimeWeighted | disk-name | Total time spent doing I/Os. This can provide an easy measure of both I/O completion time and the backlog that may be accumulating. | millisec | |||
| os.linux.metrics-system-disk-capacity | disk.<file-system-name>.total | file-system-name | The total size of the file system. | byte | ||
| disk.<file-system-name>.used | file-system-name | The total amount of space allocated to existing files in the file system. | byte | |||
| disk.<file-system-name>.avail | file-system-name | The total amount of space available within the file system. | byte | |||
| disk.<file-system-name>.used\_percentage | file-system-name | The percentage of the available space that currently allocated to all files on the file system. | percent | |||
| disk.<file-system-name>.itotal | file-system-name | The total number of inodes on the file system. | count | |||
| disk.<file-system-name>.iused | file-system-name | The number of used inodes. | count | |||
| disk.<file-system-name>.iavail | file-system-name | The number of free \(unused\) inodes. | count | |||
| disk.<file-system-name>.iused\_percentage | file-system-name | The percentage of used inodes. | percent | |||
| os.linux.metrics-system-disk-usage | disk\_usage.<disk>.total | disk-name | Total amount of space available on this disk | bytes | ||
| disk\_usage.<disk>.used | disk-name | Total amount of space used in this disk | bytes | |||
| disk\_usage.<disk>.avail | disk-name | Total amount of space available on this disk | bytes | |||
| disk\_usage.<disk>.used\_percentage | disk-name | The percentage of space used on this disk | percent | yes | yes | |
| os.linux.metrics-system-memoryos.linux.metrics-system-memory-percent | memory.total | empty | Total usable RAM. | KB | ||
| memory.free | empty | Total free RAM. | KB | |||
| memory.available | empty | An estimate of how much memory is available for starting new applications, without swapping. | KB | |||
| memory.buffers | empty | Temporary storage used for raw disk blocks. | KB | |||
| memory.cached | empty | In-memory cache for files read from disk \(the page cache\). Does not include mem\_swapcached. | KB | |||
| memory.swapTotal | empty | Total amount of swap space available. | KB | yes | yes | |
| memory.swapFree | empty | Amount of swap space that is currently unused. | yes | yes | ||
| memory.dirty | empty | Memory which is waiting to get written back to the disk. | KB | |||
| memory.swapUsed | empty | The amount of swap space in use. | KB | yes | yes | |
| memory.used | empty | The amount of RAM in use. | KB | |||
| memory.usedWOBuffersCaches | empty | The amount of memory in use. | KB | |||
| memory.freeWOBuffersCaches | empty | Value of MemAvailable from /proc/meminfo if present, but falls back to adding free + buffered + cached memory if not. | KB | |||
| memory.swapUsedPercentage | empty | Percent of swap space used. | percent | |||
| memory\_percent.free | empty | Percent of free RAM | percent | yes | yes | |
| memory\_percent.available | empty | Percent of Mem available | percent | yes | yes | |
| memory\_percent.buffers | empty | Precent of Memory used for raw disk blocks | percent | yes | yes | |
| memory\_percent.cached | empty | Percent of memory used for in-memory cache for files read from disk | percent | yes | yes | |
| memory\_percent.dirty | empty | Percent of memory waiting to get written back to the disk. | percent | yes | yes | |
| memory\_percent.swapUsed | empty | Percent of swap space used. | percent | yes | yes | |
| memory\_percent.usedWOBuffersCaches | empty | Percent of memory is being used | percent | yes | yes | |
| memory\_percent.freeWOBuffersCaches | empty | Percent of Memory available | percent | yes | yes | |
| os.linux.metrics-system-uptime | system.uptime\(sec\) | empty | The amount of time the system has been working and available. | sec | ||
| os.linux.metrics-memory-vmstat | vmstat.nr\_free\_pages | empty | Pages that are currently unused by the system. | pages | ||
| vmstat.nr\_alloc\_batch | empty | pages allocated to other domains due to insufficient memory in each domain of each NUMA | pages | |||
| vmstat.nr\_inactive\_anon | empty | memory pages in each domain of each NUMA node that have not been accessed for a long time | pages | |||
| vmstat.nr\_active\_anon | empty | Anonymous virtual memory pages that have been recently used | KB | |||
| vmstat.nr\_inactive\_file | empty | The memory page corresponding to the file that has not been accessed for a long time in each domain of each NUMA. | KB | |||
| vmstat.nr\_active\_file | empty | The memory page corresponding to the file that has been accesseed recently . | pages | |||
| vmstat.nr\_unevictable | empty | The number of pages is in the unevictable \(non-\)LRU list | count | |||
| vmstat.nr\_mlock | empty | Pages mapped into a VM\_LOCKED VMA - are a class of unevictable pages. | pages | |||
| vmstat.nr\_anon\_pages | empty | Memory mapped pages that is not part of a file. | pages | |||
| vmstat.nr\_mapped | empty | The number of memory mapped pages. | count | |||
| vmstat.nr\_file\_pages | empty | |||||
| vmstat.nr\_dirty | empty | Pages waiting to be written to disk | pages | |||
| vmstat.nr\_writeback | empty | Pages currently being written to disk | pages | |||
| vmstat.nr\_slab\_reclaimable | empty | Pages from the kernel slab memory usage that can be reclaimed | pages | |||
| vmstat.nr\_slab\_unreclaimable | empty | Pages from the kernel slab memory usage that cannot be reclaimed | pages | |||
| vmstat.nr\_page\_table\_pages | empty | Pages allocated to page tables | pages | |||
| vmstat.nr\_kernel\_stack | empty | Amount of memory allocated to kernel stacks. | KB | |||
| vmstat.nr\_unstable | empty | The number of unstable pages in each domain of each NUMA node | count | |||
| vmstat.nr\_bounce | empty | |||||
| vmstat.nr\_vmscan\_write | empty | The number of dirty pages written back during a scan of LRU\(s\) | count | |||
| vmstat.nr\_vmscan\_immediate\_reclaim | empty | |||||
| vmstat.nr\_writeback\_temp | empty | |||||
| vmstat.nr\_isolated\_anon | empty | The number of anonymous memory pages isolated in each domain of each NUMA node | count | |||
| vmstat.nr\_isolated\_file | empty | The number of pages of file storage pages isolated in each domain of each NUMA node | count | |||
| vmstat.nr\_shmem | empty | The number of shared memory pages | count | |||
| vmstat.nr\_dirtied | empty | The number of dirty pages in each domain of each NUMA node | count | |||
| vmstat.nr\_written | empty | |||||
| vmstat.numa\_hit | empty | The number of pages that were successfully allocated to this node. | count | |||
| vmstat.numa\_miss | empty | The number of pages that were allocated on this node because of low memory on the intended node. | count | |||
| vmstat.numa\_foreign | empty | The number of pages initially intended for this node that were allocated to another node instead. | count | |||
| vmstat.numa\_interleave | empty | The number of interleave policy pages successfully allocated to this node. | count | |||
| vmstat.numa\_local | empty | The number of pages successfully allocated on this node, by a process on this node | count | |||
| vmstat.numa\_other | empty | The number of pages allocated on this node, by a process on another node. | count | |||
| vmstat.workingset\_refault | empty | |||||
| vmstat.workingset\_activate | empty | |||||
| vmstat.workingset\_nodereclaim | empty | |||||
| vmstat.nr\_anon\_transparent\_hugepages | empty | |||||
| vmstat.nr\_free\_cma | empty | Free continuous memory allocator pages in each domain of each NUMA | ||||
| vmstat.nr\_dirty\_threshold | empty | |||||
| vmstat.nr\_dirty\_background\_threshold | empty | |||||
| vmstat.pgpgin | empty | The number of pages brought in from disk | count | |||
| vmstat.pgpgout | empty | The number of pages written out to disk | count | |||
| vmstat.pswpin | empty | The number of pages brought in from swap space | count | |||
| vmstat.pswpout | empty | The number of pages swapped out into swap space | count | |||
| vmstat.pgalloc\_dma | empty | |||||
| vmstat.pgalloc\_dma32 | empty | |||||
| vmstat.pgalloc\_normal | empty | |||||
| vmstat.pgalloc\_movable | empty | |||||
| vmstat.pgfree | empty | The number of pages are free since last boot | count | |||
| vmstat.pgactivat | empty | Number of page activations since last boot | count | |||
| vmstat.pgdeactivate | empty | Number of page deactivations since last boot | count | |||
| vmstat.pgfault | empty | Minor faults since last boot | pages | |||
| vmstat.pgmajfault | empty | Major faults since last boot | pages | |||
| vmstat.pglazyfreed | empty | |||||
| vmstat.pgrefill\_dma | empty | |||||
| vmstat.pgrefill\_dma32 | empty | |||||
| vmstat.pgrefill\_normal | empty | Number of page refills since last boot | count | |||
| vmstat.pgrefill\_movable | empty | |||||
| vmstat.pgsteal\_kswapd\_dma | empty | |||||
| vmstat.pgsteal\_kswapd\_dma32 | empty | |||||
| vmstat.pgsteal\_kswapd\_normal | empty | |||||
| vmstat.pgsteal\_kswapd\_movable | empty | |||||
| vmstat.pgsteal\_direct\_dma | empty | |||||
| vmstat.pgsteal\_direct\_dma32 | empty | |||||
| vmstat.pgsteal\_direct\_normal | empty | |||||
| vmstat.pgsteal\_direct\_movable | empty | |||||
| vmstat.pgscan\_kswapd\_dma | empty | |||||
| vmstat.pgscan\_kswapd\_dma32 | empty | |||||
| vmstat.pgscan\_kswapd\_normal | empty | Number of pages scanned by kswapd since boot | count | |||
| vmstat.pgscan\_kswapd\_movable | empty | |||||
| vmstat.pgscan\_direct\_dma | empty | |||||
| vmstat.pgscan\_direct\_dma32 | empty | |||||
| vmstat.pgscan\_direct\_normal | empty | Number of pages reclaimed since boot | count | |||
| vmstat.pgscan\_direct\_movable | empty | |||||
| vmstat.pgscan\_direct\_throttle | empty | |||||
| vmstat.zone\_reclaim\_failed | empty | |||||
| vmstat.pginodesteal | empty | |||||
| vmstat.slabs\_scanned | empty | |||||
| vmstat.kswapd\_inodesteal | empty | |||||
| vmstat.kswapd\_low\_wmark\_hit\_quickly | empty | |||||
| vmstat.kswapd\_high\_wmark\_hit\_quickly | empty | |||||
| vmstat.pageoutrun | empty | Number of times kswapd called page reclaim | count | |||
| vmstat.allocstall | empty | Number of times page reclaim was called directly \(low memory\) | count | |||
| vmstat.pgrotated | empty | |||||
| vmstat.drop\_pagecache | empty | |||||
| vmstat.drop\_slab | empty | |||||
| vmstat.numa\_pte\_updates | empty | |||||
| vmstat.numa\_huge\_pte\_updates | empty | |||||
| vmstat.numa\_hint\_faults | empty | |||||
| vmstat.numa\_hint\_faults\_local | empty | |||||
| vmstat.numa\_pages\_migrated | empty | |||||
| vmstat.pgmigrate\_success | empty | |||||
| vmstat.pgmigrate\_fail | empty | |||||
| vmstat.compact\_migrate\_scanned | empty | |||||
| vmstat.compact\_free\_scanned | empty | |||||
| vmstat.compact\_isolated | empty | |||||
| vmstat.compact\_stall | empty | The number of times a process stalls to run memory compaction so that a huge page is free for use. | count | |||
| vmstat.compact\_fail | empty | The number of times the system tries to compact memory but failed. | count | |||
| vmstat.compact\_success | empty | The number of times the system compacted memory and freed a huge page for use. | count | |||
| vmstat.htlb\_buddy\_alloc\_success | empty | |||||
| vmstat.htlb\_buddy\_alloc\_fail | empty | |||||
| vmstat.unevictable\_pgs\_culled | empty | |||||
| vmstat.unevictable\_pgs\_scanned | empty | |||||
| vmstat.unevictable\_pgs\_rescued | empty | |||||
| vmstat.unevictable\_pgs\_mlocked | empty | |||||
| vmstat.unevictable\_pgs\_munlocked | empty | |||||
| vmstat.unevictable\_pgs\_cleared | empty | |||||
| vmstat.unevictable\_pgs\_stranded | empty | |||||
| vmstat.thp\_fault\_alloc | empty | The number of huge pages is successfully allocated to handle a page fault. | count | |||
| vmstat.thp\_fault\_fallback | empty | The number of page fault fails to allocate a huge page and instead falls back to using small pages. | count | |||
| vmstat.thp\_collapse\_alloc | empty | The number of collapse of a range of pages into one huge page and then successfull allocation of a new huge page to store the data. | count | |||
| vmstat.thp\_collapse\_alloc\_failed | empty | The number of collapse of a range of pages into one huge page but failed the allocation. | count | |||
| vmstat.thp\_split | empty | The number of split of a huge page into base pages | count | |||
| vmstat.thp\_zero\_page\_alloc | empty | The number of successful allocation of huge zero page | count | |||
| vmstat.thp\_zero\_page\_alloc\_failed | empty | The number of times the kernel failed to allocate huge zero page and falls back to using small pages. | count | |||
| vmstat.balloon\_inflate | empty | |||||
| vmstat.balloon\_deflate | empty | |||||
| vmstat.balloon\_migrate | empty | |||||
| os.linux.metrics-process-status | proc.<process>.VmSize | process-name | The total amount of virtual memory used by the process | KB | ||
| proc.<process>.VmRSS | process-name | The non-swapped physical memory a process has used | KB | |||
| proc.<process>.VmSwap | process-name | The total amount of swap space used. | KB |
Linux network monitoring checks
Note: When upgrading from an earlier version, manually add the checks in this table to the Linux metrics policy.
| Type | Check | Description | Usage and usage example | Metrics collected | Featured Metric |
|---|---|---|---|---|---|
| Metric | os.linux.metrics-network-interface | Retrieves all network interface related metrics for Linux servers. | Usage: - -x, excludeinterface: List of interfaces to exclude \(comma separated\) - -i, includeinterface: List of interfaces to include \(comma separated\) - -I, includeinterfaceregex: Regex matching interfaces to include - -X, excludeinterfaceregex: Regex matching interfaces to exclude Usage example: `command: metrics-network-interface.rb` | - rxBytes \(featured metric\) - rxPackets \(featured metric\) - rxErrors - rxDrops \(featured metric\) - rxFifo - rxFrame - rxCompressed - rxMulticast - rxBytes \(featured metric\) - rxPackets \(featured metric\) - rxErrors - rxDrops \(featured metric\) - rxFifo - rxColls - rxCarrier - rxCompressed | yes |
| Metric | os.linux.metrics-netstat-tcp | Retrieves metrics on TCP socket states from netstat. Useful on high-traffic web or proxy servers with large numbers of short-lived TCP connections coming and going. | Usage: - -p, port: The port from which you want to receive metrics. Value range = 1-65535. - -t, type: The port type from which to receive metrics. Values=local or remote. Default = local - -d, disabletcp6: Disables the tcp6 check. Enter a value to set disabletcp6 = true. Usage example: `metrics-netstat-tcp.rb` | - tcp.UNKNOWN - tcp.ESTABLISHED - tcp.SYN\_SENT - tcp.SYN\_RECV - tcp.FIN\_WAIT1 - tcp.FIN\_WAIT2 - tcp.TIME\_WAIT - tcp.CLOSE - tcp.CLOSE\_WAIT - tcp.LAST\_ACK - tcp.LISTEN - tcp.CLOSING | no |
Parent Topic:Agent Client Collector Monitoring default checks and policies