Version 2.1 by Normann P. Nielsen on 2026/11/15 15:41

Show last authors
1 {{toc/}}
2
3 = Tips =
4
5 === Key=Value ===
6 to get Key=Value out from and input, I use this in a filter:
7
8
9 {{code}}
10 .....
11 if [type] == "pinglog" {
12 kv {}
13 mutate {
14 convert => { "pingtime" => "float" }
15 }
16 }
17 else
18 {
19 if [type] == "syslog"
20 {
21 kv
22 {
23 include_keys => ["OUT","IN","SRC","SPT","DST","DPT","PROTO","ACTION"]
24 trim => "<>\[\],"
25 trimkey => "<>\[\],"
26 }
27  
28 .....
29 {{/code}}
30
31 According to [[http://logz.io/blog/5-logstash-pitfalls-and-how-to-avoid-them/]] there a danger in using "kv" without adding specific fields
32
33 As my "pingtime" is a value I want to visualize and use for an Y-Axis, it must be numeric - hence the mutate to a float:
34
35 [[image:Screen Shot 2016-04-13 at 18.59.59.jpg||width="900"||alt="Screen Shot 2016-04-13 at 18.59.59.jpg"]]
36
37 === Drop something ===
38 THis is how I drop part of the syslog (google DNS lookup) after it has been "kv" (:
39
40
41 {{code}}
42 if [DST] == "8.8.8.8"
43 {
44 drop {}
45 }
46 {{/code}}
47
48 === Adding fields/metadata ===
49 From the Router Syslog, I would like to add an owner and device of the Source (Identified by Source IP "SRC")  - I have static IP's for the devices:
50
51
52 {{code}}
53 if [SRC] == "10.0.0.102" {
54 mutate
55 {
56 add_field => { "IPOwner" => "Toke" "Device" => "iPad" }
57 }
58 }
59 ...
60 ...
61 if [SRC] == "10.0.0.109" {
62 mutate
63 {
64 add_field => { "IPOwner" => "Christopher" "Device" => "Laptop LAN" }
65 }
66 }
67 {{/code}}
68
69 These metadata give me a possibility to Visualize traffic by IPOwner:
70
71 [[image:Screen Shot 2016-04-13 at 18.52.54.jpg||width="900"||alt="Screen Shot 2016-04-13 at 18.52.54.jpg"]]
72
73
74
75
76
77 = Trobleshooting =
78
79 === Cant send data to Elasticsearch - Elasticsearch wont recieve ===
80 The /var/log/logstash/logstash.log shows:
81
82
83 {{code}}
84 {:timestamp=>"2016-04-08T08:32:32.217000+0200", :message=>"Beats input: the pipeline is blocked, temporary refusing new connection.", :reconnect_backoff_sleep=>0.5, :level=>:warn}
85 {:timestamp=>"2016-04-08T08:32:32.721000+0200", :message=>"Beats input: the pipeline is blocked, temporary refusing new connection.", :reconnect_backoff_sleep=>0.5, :level=>:warn}
86 {:timestamp=>"2016-04-08T08:32:33.232000+0200", :message=>"Beats input: the pipeline is blocked, temporary refusing new connection.", :reconnect_backoff_sleep=>0.5, :level=>:warn}
87 {:timestamp=>"2016-04-08T08:32:33.733000+0200", :message=>"Beats input: the pipeline is blocked, temporary refusing new connection.", :reconnect_backoff_sleep=>0.5, :level=>:warn}
88 {:timestamp=>"2016-04-08T08:32:34.282000+0200", :message=>"Beats input: the pipeline is blocked, temporary refusing new connection.", :reconnect_backoff_sleep=>0.5, :level=>:warn}
89 {:timestamp=>"2016-04-08T08:32:34.783000+0200", :message=>"Beats input: the pipeline is blocked, temporary refusing new connection.", :reconnect_backoff_sleep=>0.5, :level=>:warn}
90 {{/code}}
91
92 My Elasticseach was down - status "red":
93
94
95 {{code}}
96 curl localhost:9200/_cluster/health?pretty
97 {
98 "cluster_name" : "moselk",
99 "status" : "red",
100 "timed_out" : false,
101 "number_of_nodes" : 1,
102 "number_of_data_nodes" : 1,
103 "active_primary_shards" : 421,
104 "active_shards" : 421,
105 "relocating_shards" : 0,
106 "initializing_shards" : 0,
107 "unassigned_shards" : 25,
108 "delayed_unassigned_shards" : 0,
109 "number_of_pending_tasks" : 0,
110 "number_of_in_flight_fetch" : 0,
111 "task_max_waiting_in_queue_millis" : 0,
112 "active_shards_percent_as_number" : 94.39461883408072
113 }
114 {{/code}}
115
116 See [[Elasticsearch - Tips and Troubleshooting>>Main.information-technology.elk-elasticsearch-logstash-kibana.elasticsearch-tips-and-troubleshooting]]
117
118 === Can send data to Elasticsearch - Congestion ===
119 The /var/log/logstash/logstash.log shows:
120
121
122 {{code}}
123 {:timestamp=>"2016-04-08T08:32:32.217000+0200", :message=>"Beats input: the pipeline is blocked, temporary refusing new connection.", :reconnect_backoff_sleep=>0.5, :level=>:warn}
124 {:timestamp=>"2016-04-08T08:32:32.721000+0200", :message=>"Beats input: the pipeline is blocked, temporary refusing new connection.", :reconnect_backoff_sleep=>0.5, :level=>:warn}
125 {:timestamp=>"2016-04-08T08:32:33.232000+0200", :message=>"Beats input: the pipeline is blocked, temporary refusing new connection.", :reconnect_backoff_sleep=>0.5, :level=>:warn}
126 {:timestamp=>"2016-04-08T08:32:33.733000+0200", :message=>"Beats input: the pipeline is blocked, temporary refusing new connection.", :reconnect_backoff_sleep=>0.5, :level=>:warn}
127 {:timestamp=>"2016-04-08T08:32:34.282000+0200", :message=>"Beats input: the pipeline is blocked, temporary refusing new connection.", :reconnect_backoff_sleep=>0.5, :level=>:warn}
128 {:timestamp=>"2016-04-08T08:32:34.783000+0200", :message=>"Beats input: the pipeline is blocked, temporary refusing new connection.", :reconnect_backoff_sleep=>0.5, :level=>:warn}
129 {{/code}}
130
131 I have seen this twice, the first time setting ##[[congestion_threshold>>https://www.elastic.co/guide/en/logstash/current/plugins-inputs-beats.html#plugins-inputs-beats-congestion_threshold]] in the beats input to more than 5 (I use 25) - [[https://github.com/elastic/logstash/issues/4368]] - helped##
132
133
134 {{code}}
135 input {
136 beats {
137 port => 5044
138 ssl => true
139 ssl_certificate => "/etc/pki/tls/certs/logstash-forwarder.crt"
140 ssl_key => "/etc/pki/tls/private/logstash-forwarder.key"
141 congestion_threshold => 25
142 }
143 }
144 {{/code}}
145
146 And restart logstash