Wiki source code of Logstash - Tips and Troubleshooting
Version 2.1 by Normann P. Nielsen on 2026/11/15 15:41
Show last authors
| author | version | line-number | content |
|---|---|---|---|
| 1 | {{toc/}} | ||
| 2 | |||
| 3 | = Tips = | ||
| 4 | |||
| 5 | === Key=Value === | ||
| 6 | to get Key=Value out from and input, I use this in a filter: | ||
| 7 | |||
| 8 | |||
| 9 | {{code}} | ||
| 10 | ..... | ||
| 11 | if [type] == "pinglog" { | ||
| 12 | kv {} | ||
| 13 | mutate { | ||
| 14 | convert => { "pingtime" => "float" } | ||
| 15 | } | ||
| 16 | } | ||
| 17 | else | ||
| 18 | { | ||
| 19 | if [type] == "syslog" | ||
| 20 | { | ||
| 21 | kv | ||
| 22 | { | ||
| 23 | include_keys => ["OUT","IN","SRC","SPT","DST","DPT","PROTO","ACTION"] | ||
| 24 | trim => "<>\[\]," | ||
| 25 | trimkey => "<>\[\]," | ||
| 26 | } | ||
| 27 | |||
| 28 | ..... | ||
| 29 | {{/code}} | ||
| 30 | |||
| 31 | According to [[http://logz.io/blog/5-logstash-pitfalls-and-how-to-avoid-them/]] there a danger in using "kv" without adding specific fields | ||
| 32 | |||
| 33 | As my "pingtime" is a value I want to visualize and use for an Y-Axis, it must be numeric - hence the mutate to a float: | ||
| 34 | |||
| 35 | [[image:Screen Shot 2016-04-13 at 18.59.59.jpg||width="900"||alt="Screen Shot 2016-04-13 at 18.59.59.jpg"]] | ||
| 36 | |||
| 37 | === Drop something === | ||
| 38 | THis is how I drop part of the syslog (google DNS lookup) after it has been "kv" (: | ||
| 39 | |||
| 40 | |||
| 41 | {{code}} | ||
| 42 | if [DST] == "8.8.8.8" | ||
| 43 | { | ||
| 44 | drop {} | ||
| 45 | } | ||
| 46 | {{/code}} | ||
| 47 | |||
| 48 | === Adding fields/metadata === | ||
| 49 | From the Router Syslog, I would like to add an owner and device of the Source (Identified by Source IP "SRC") - I have static IP's for the devices: | ||
| 50 | |||
| 51 | |||
| 52 | {{code}} | ||
| 53 | if [SRC] == "10.0.0.102" { | ||
| 54 | mutate | ||
| 55 | { | ||
| 56 | add_field => { "IPOwner" => "Toke" "Device" => "iPad" } | ||
| 57 | } | ||
| 58 | } | ||
| 59 | ... | ||
| 60 | ... | ||
| 61 | if [SRC] == "10.0.0.109" { | ||
| 62 | mutate | ||
| 63 | { | ||
| 64 | add_field => { "IPOwner" => "Christopher" "Device" => "Laptop LAN" } | ||
| 65 | } | ||
| 66 | } | ||
| 67 | {{/code}} | ||
| 68 | |||
| 69 | These metadata give me a possibility to Visualize traffic by IPOwner: | ||
| 70 | |||
| 71 | [[image:Screen Shot 2016-04-13 at 18.52.54.jpg||width="900"||alt="Screen Shot 2016-04-13 at 18.52.54.jpg"]] | ||
| 72 | |||
| 73 | |||
| 74 | |||
| 75 | |||
| 76 | |||
| 77 | = Trobleshooting = | ||
| 78 | |||
| 79 | === Cant send data to Elasticsearch - Elasticsearch wont recieve === | ||
| 80 | The /var/log/logstash/logstash.log shows: | ||
| 81 | |||
| 82 | |||
| 83 | {{code}} | ||
| 84 | {:timestamp=>"2016-04-08T08:32:32.217000+0200", :message=>"Beats input: the pipeline is blocked, temporary refusing new connection.", :reconnect_backoff_sleep=>0.5, :level=>:warn} | ||
| 85 | {:timestamp=>"2016-04-08T08:32:32.721000+0200", :message=>"Beats input: the pipeline is blocked, temporary refusing new connection.", :reconnect_backoff_sleep=>0.5, :level=>:warn} | ||
| 86 | {:timestamp=>"2016-04-08T08:32:33.232000+0200", :message=>"Beats input: the pipeline is blocked, temporary refusing new connection.", :reconnect_backoff_sleep=>0.5, :level=>:warn} | ||
| 87 | {:timestamp=>"2016-04-08T08:32:33.733000+0200", :message=>"Beats input: the pipeline is blocked, temporary refusing new connection.", :reconnect_backoff_sleep=>0.5, :level=>:warn} | ||
| 88 | {:timestamp=>"2016-04-08T08:32:34.282000+0200", :message=>"Beats input: the pipeline is blocked, temporary refusing new connection.", :reconnect_backoff_sleep=>0.5, :level=>:warn} | ||
| 89 | {:timestamp=>"2016-04-08T08:32:34.783000+0200", :message=>"Beats input: the pipeline is blocked, temporary refusing new connection.", :reconnect_backoff_sleep=>0.5, :level=>:warn} | ||
| 90 | {{/code}} | ||
| 91 | |||
| 92 | My Elasticseach was down - status "red": | ||
| 93 | |||
| 94 | |||
| 95 | {{code}} | ||
| 96 | curl localhost:9200/_cluster/health?pretty | ||
| 97 | { | ||
| 98 | "cluster_name" : "moselk", | ||
| 99 | "status" : "red", | ||
| 100 | "timed_out" : false, | ||
| 101 | "number_of_nodes" : 1, | ||
| 102 | "number_of_data_nodes" : 1, | ||
| 103 | "active_primary_shards" : 421, | ||
| 104 | "active_shards" : 421, | ||
| 105 | "relocating_shards" : 0, | ||
| 106 | "initializing_shards" : 0, | ||
| 107 | "unassigned_shards" : 25, | ||
| 108 | "delayed_unassigned_shards" : 0, | ||
| 109 | "number_of_pending_tasks" : 0, | ||
| 110 | "number_of_in_flight_fetch" : 0, | ||
| 111 | "task_max_waiting_in_queue_millis" : 0, | ||
| 112 | "active_shards_percent_as_number" : 94.39461883408072 | ||
| 113 | } | ||
| 114 | {{/code}} | ||
| 115 | |||
| 116 | See [[Elasticsearch - Tips and Troubleshooting>>Main.information-technology.elk-elasticsearch-logstash-kibana.elasticsearch-tips-and-troubleshooting]] | ||
| 117 | |||
| 118 | === Can send data to Elasticsearch - Congestion === | ||
| 119 | The /var/log/logstash/logstash.log shows: | ||
| 120 | |||
| 121 | |||
| 122 | {{code}} | ||
| 123 | {:timestamp=>"2016-04-08T08:32:32.217000+0200", :message=>"Beats input: the pipeline is blocked, temporary refusing new connection.", :reconnect_backoff_sleep=>0.5, :level=>:warn} | ||
| 124 | {:timestamp=>"2016-04-08T08:32:32.721000+0200", :message=>"Beats input: the pipeline is blocked, temporary refusing new connection.", :reconnect_backoff_sleep=>0.5, :level=>:warn} | ||
| 125 | {:timestamp=>"2016-04-08T08:32:33.232000+0200", :message=>"Beats input: the pipeline is blocked, temporary refusing new connection.", :reconnect_backoff_sleep=>0.5, :level=>:warn} | ||
| 126 | {:timestamp=>"2016-04-08T08:32:33.733000+0200", :message=>"Beats input: the pipeline is blocked, temporary refusing new connection.", :reconnect_backoff_sleep=>0.5, :level=>:warn} | ||
| 127 | {:timestamp=>"2016-04-08T08:32:34.282000+0200", :message=>"Beats input: the pipeline is blocked, temporary refusing new connection.", :reconnect_backoff_sleep=>0.5, :level=>:warn} | ||
| 128 | {:timestamp=>"2016-04-08T08:32:34.783000+0200", :message=>"Beats input: the pipeline is blocked, temporary refusing new connection.", :reconnect_backoff_sleep=>0.5, :level=>:warn} | ||
| 129 | {{/code}} | ||
| 130 | |||
| 131 | I have seen this twice, the first time setting ##[[congestion_threshold>>https://www.elastic.co/guide/en/logstash/current/plugins-inputs-beats.html#plugins-inputs-beats-congestion_threshold]] in the beats input to more than 5 (I use 25) - [[https://github.com/elastic/logstash/issues/4368]] - helped## | ||
| 132 | |||
| 133 | |||
| 134 | {{code}} | ||
| 135 | input { | ||
| 136 | beats { | ||
| 137 | port => 5044 | ||
| 138 | ssl => true | ||
| 139 | ssl_certificate => "/etc/pki/tls/certs/logstash-forwarder.crt" | ||
| 140 | ssl_key => "/etc/pki/tls/private/logstash-forwarder.key" | ||
| 141 | congestion_threshold => 25 | ||
| 142 | } | ||
| 143 | } | ||
| 144 | {{/code}} | ||
| 145 | |||
| 146 | And restart logstash |