本文檔記錄Telegraf解析nginx access.log的相關(guān)內(nèi)容

Tail Input Plugin

Tail Input Plugin 是 Telegraf項(xiàng)目 眾多Inputs插件之一。Tail Input Plugin 監(jiān)聽(tīng)一個(gè)日志文件并解析該文件的每個(gè)日志變更。默認(rèn)情況下，tail 插件的行為類似于以下 Unix tail 命令：

tail -F –lines=0 myfile.log

Telegraf收集nginx access.log 相當(dāng)于Unix tail命令：

tail -F –lines=0 access.log

并使用grok格式來(lái)解析日志。除此之外，Telegraf Tail Input Plugin支持多種 Data Format細(xì)節(jié)的東西可以跟下源碼，下面直接記錄運(yùn)行配置。

配置

telegraf.conf

[agent] interval = “10s” round_interval = true metric_batch_size = 1000 metric_buffer_limit = 10000 collection_jitter = “0s” flush_interval = “10s” flush_jitter = “0s” precision = “” hostname = “” omit_hostname = false[[inputs.tail]] files = [“/Users/enic/github/telegraf/testdata/access.log”]# from_beginning = false from_beginning = true grok_patterns = [“%{COMMON_LOG_FORMAT}”] name_override = “nginx_access_log” grok_custom_pattern_files = [] grok_custom_patterns = ”’ NGINX_ACCESS_LOG %{IP:remote_addr} – (-|%{WORD:remote_user}) [%{HTTPDATE:time_local}] %{QS:request} %{NUMBER:status:int} %{NUMBER:body_bytes_sent:int} %{QS:referrer} %{QS:agent} %{IPORHOST:xforwardedfor} %{IPORHOST:host} %{QS:upstream_addr} (-|%{NUMBER:upstream_status:int}) %{BASE10NUM:request_time:float} (-|%{BASE10NUM:upstream_response_time:float}) %{QS:cookie_uin} %{QS:cookie_luin} ”’ grok_timezone = “Local” data_format = “grok”[[outputs.file]] files = [“/Users/enic/github/telegraf/testdata/outputs.log”]

access.log

10.10.10.10 – – [19/Apr/2022:03:17:34 +0800] “POST /user/login HTTP/1.1” 400 63 “-” “Go-http-client/1.1” “-“10.10.10.10 – – [19/Apr/2022:03:17:34 +0800] “POST /user/login HTTP/1.1” 400 63 “-” “Go-http-client/1.1” “-“10.10.10.10 – – [19/Apr/2022:03:17:34 +0800] “POST /user/login HTTP/1.1” 400 63 “-” “Go-http-client/1.1” “-“10.10.10.10 – – [19/Apr/2022:03:17:34 +0800] “POST /user/login HTTP/1.1” 400 63 “-” “Go-http-client/1.1” “-“10.10.10.10 – – [19/Apr/2022:03:17:34 +0800] “POST /user/login HTTP/1.1” 400 63 “-” “Go-http-client/1.1” “-“10.10.10.10 – – [19/Apr/2022:03:17:34 +0800] “POST /user/login HTTP/1.1” 400 63 “-” “Go-http-client/1.1” “-“10.10.10.10 – – [19/Apr/2022:03:17:44 +0800] “POST /qps HTTP/1.1” 504 3693 “-” “Go-http-client/1.1” “-“10.10.10.10 – – [19/Apr/2022:03:17:54 +0800] “POST /latency HTTP/1.1” 504 3693 “-” “Go-http-client/1.1” “-“10.10.10.10 – – [19/Apr/2022:03:18:34 +0800] “POST /user/login HTTP/1.1” 400 63 “-” “Go-http-client/1.1” “-“10.10.10.10 – – [19/Apr/2022:03:18:34 +0800] “POST /user/login HTTP/1.1” 400 63 “-” “Go-http-client/1.1” “-“10.10.10.10 – – [19/Apr/2022:03:18:34 +0800] “POST /user/login HTTP/1.1” 400 63 “-” “Go-http-client/1.1” “-“10.10.10.10 – – [19/Apr/2022:03:18:34 +0800] “POST /user/login HTTP/1.1” 400 63 “-” “Go-http-client/1.1” “-“10.10.10.10 – – [19/Apr/2022:03:18:34 +0800] “POST /user/login HTTP/1.1” 400 63 “-” “Go-http-client/1.1” “-“10.10.10.10 – – [19/Apr/2022:03:18:34 +0800] “POST /user/login HTTP/1.1” 400 63 “-” “Go-http-client/1.1” “-“10.10.10.100 – – [19/Apr/2022:03:18:37 +0800] “GET / HTTP/1.1” 200 1839 “-” “Expanse, a Palo Alto Networks company, searches across the global IPv4 space multiple times per day to identify customers’ presences on the Internet. If you would like to be excluded from our scans, please send IP addresses/domains to: [email protected]” “-“10.10.10.10 – – [19/Apr/2022:03:18:44 +0800] “POST /qps HTTP/1.1” 504 3693 “-” “Go-http-client/1.1” “-“10.10.10.10 – – [19/Apr/2022:03:18:54 +0800] “POST /latency HTTP/1.1” 504 3693 “-” “Go-http-client/1.1” “-“10.10.10.10 – – [19/Apr/2022:03:19:34 +0800] “POST /user/login HTTP/1.1” 400 63 “-” “Go-http-client/1.1” “-“10.10.10.10 – – [19/Apr/2022:03:19:34 +0800] “POST /user/login HTTP/1.1” 400 63 “-” “Go-http-client/1.1” “-“10.10.10.10 – – [19/Apr/2022:03:19:34 +0800] “POST /user/login HTTP/1.1” 400 63 “-” “Go-http-client/1.1” “-“10.10.10.10 – – [19/Apr/2022:03:19:34 +0800] “POST /user/login HTTP/1.1” 400 63 “-” “Go-http-client/1.1” “-“10.10.10.10 – – [19/Apr/2022:03:18:34 +0800] “POST /user/login HTTP/1.1” 400 63 “-” “Go-http-client/1.1” “-“10.10.10.10 – – [19/Apr/2022:03:18:34 +0800] “POST /user/login HTTP/1.1” 400 63 “-” “Go-http-client/1.1” “-“10.10.10.10 – – [19/Apr/2022:03:18:34 +0800] “POST /user/login HTTP/1.1” 400 63 “-” “Go-http-client/1.1” “-“10.10.10.10 – – [19/Apr/2022:03:18:34 +0800] “POST /user/login HTTP/1.1” 400 63 “-” “Go-http-client/1.1” “-“10.10.10.10 – – [19/Apr/2022:03:19:34 +0800] “POST /user/login HTTP/1.1” 400 63 “-” “Go-http-client/1.1” “-“10.10.10.10 – – [19/Apr/2022:03:19:34 +0800] “POST /user/login HTTP/1.1” 400 63 “-” “Go-http-client/1.1” “-“10.10.10.10 – – [19/Apr/2022:03:19:34 +0800] “POST /user/login HTTP/1.1” 400 63 “-” “Go-http-client/1.1” “-“10.10.10.10 – – [19/Apr/2022:03:19:34 +0800] “POST /user/login HTTP/1.1” 400 63 “-” “Go-http-client/1.1” “-“10.10.10.10 – – [19/Apr/2022:03:18:34 +0800] “POST /user/login HTTP/1.1” 400 63 “-” “Go-http-client/1.1” “-“10.10.10.10 – – [19/Apr/2022:03:18:34 +0800] “POST /user/login HTTP/1.1” 400 63 “-” “Go-http-client/1.1” “-“10.10.10.10 – – [19/Apr/2022:03:18:34 +0800] “POST /user/login HTTP/1.1” 400 63 “-” “Go-http-client/1.1” “-“10.10.10.10 – – [19/Apr/2022:03:18:34 +0800] “POST /user/login HTTP/1.1” 400 63 “-” “Go-http-client/1.1” “-“10.10.10.10 – – [19/Apr/2022:03:18:34 +0800] “POST /user/login HTTP/1.1” 400 63 “-” “Go-http-client/1.1” “-“10.10.10.10 – – [19/Apr/2022:03:18:34 +0800] “POST /user/login HTTP/1.1” 400 63 “-” “Go-http-client/1.1” “-“10.10.10.10 – – [19/Apr/2022:03:18:34 +0800] “POST /user/login HTTP/1.1” 400 63 “-” “Go-http-client/1.1” “-“10.10.10.10 – – [19/Apr/2022:03:18:34 +0800] “POST /user/login HTTP/1.1” 400 63 “-” “Go-http-client/1.1” “-“10.10.10.10 – – [19/Apr/2022:03:18:34 +0800] “POST /user/login HTTP/1.1” 400 63 “-” “Go-http-client/1.1” “-“

操作

cd telegrafmake./telegraf –config=/Users/enic/github/telegraf/testdata/telegraf.conf

output展示

nginx_access_log,host=enicMBP,path=/Users/enic/github/telegraf/testdata/access.log,resp_code=400,verb=POST request=”/user/login”,http_version=1.1,ident=”-“,auth=”-“,resp_bytes=63i,client_ip=”10.10.10.10″ 1650309454000000000nginx_access_log,host=enicMBP,path=/Users/enic/github/telegraf/testdata/access.log,resp_code=400,verb=POST request=”/user/login”,resp_bytes=63i,client_ip=”10.10.10.10″,ident=”-“,auth=”-“,http_version=1.1 1650309454001000000nginx_access_log,host=enicMBP,path=/Users/enic/github/telegraf/testdata/access.log,resp_code=400,verb=POST ident=”-“,request=”/user/login”,auth=”-“,http_version=1.1,resp_bytes=63i,client_ip=”10.10.10.10″ 1650309454002000000nginx_access_log,host=enicMBP,path=/Users/enic/github/telegraf/testdata/access.log,resp_code=400,verb=POST http_version=1.1,client_ip=”10.10.10.10″,ident=”-“,resp_bytes=63i,auth=”-“,request=”/user/login” 1650309454003000000nginx_access_log,host=enicMBP,path=/Users/enic/github/telegraf/testdata/access.log,resp_code=400,verb=POST client_ip=”10.10.10.10″,auth=”-“,ident=”-“,request=”/user/login”,http_version=1.1,resp_bytes=63i 1650309454004000000nginx_access_log,host=enicMBP,path=/Users/enic/github/telegraf/testdata/access.log,resp_code=400,verb=POST ident=”-“,auth=”-“,request=”/user/login”,resp_bytes=63i,client_ip=”10.10.10.10″,http_version=1.1 1650309454005000000nginx_access_log,host=enicMBP,path=/Users/enic/github/telegraf/testdata/access.log,resp_code=504,verb=POST auth=”-“,request=”/qps”,http_version=1.1,resp_bytes=3693i,client_ip=”10.10.10.10″,ident=”-” 1650309464000000000nginx_access_log,host=enicMBP,path=/Users/enic/github/telegraf/testdata/access.log,resp_code=504,verb=POST ident=”-“,auth=”-“,request=”/latency”,http_version=1.1,resp_bytes=3693i,client_ip=”10.10.10.10″ 1650309474000000000nginx_access_log,host=enicMBP,path=/Users/enic/github/telegraf/testdata/access.log,resp_code=400,verb=POST auth=”-“,request=”/user/login”,resp_bytes=63i,client_ip=”10.10.10.10″,http_version=1.1,ident=”-” 1650309514000000000nginx_access_log,host=enicMBP,path=/Users/enic/github/telegraf/testdata/access.log,resp_code=400,verb=POST ident=”-“,auth=”-“,request=”/user/login”,client_ip=”10.10.10.10″,resp_bytes=63i,http_version=1.1 1650309514001000000nginx_access_log,host=enicMBP,path=/Users/enic/github/telegraf/testdata/access.log,resp_code=400,verb=POST resp_bytes=63i,client_ip=”10.10.10.10″,ident=”-“,auth=”-“,request=”/user/login”,http_version=1.1 1650309514002000000nginx_access_log,host=enicMBP,path=/Users/enic/github/telegraf/testdata/access.log,resp_code=400,verb=POST request=”/user/login”,resp_bytes=63i,ident=”-“,auth=”-“,http_version=1.1,client_ip=”10.10.10.10″ 1650309514003000000nginx_access_log,host=enicMBP,path=/Users/enic/github/telegraf/testdata/access.log,resp_code=400,verb=POST client_ip=”10.10.10.10″,auth=”-“,ident=”-“,request=”/user/login”,http_version=1.1,resp_bytes=63i 1650309514004000000nginx_access_log,host=enicMBP,path=/Users/enic/github/telegraf/testdata/access.log,resp_code=400,verb=POST resp_bytes=63i,ident=”-“,request=”/user/login”,client_ip=”10.10.10.10″,auth=”-“,http_version=1.1 1650309514005000000nginx_access_log,host=enicMBP,path=/Users/enic/github/telegraf/testdata/access.log,resp_code=200,verb=GET ident=”-“,request=”/”,resp_bytes=1839i,auth=”-“,http_version=1.1,client_ip=”10.10.10.100″ 1650309517000000000nginx_access_log,host=enicMBP,path=/Users/enic/github/telegraf/testdata/access.log,resp_code=504,verb=POST request=”/qps”,auth=”-“,ident=”-“,http_version=1.1,resp_bytes=3693i,client_ip=”10.10.10.10″ 1650309524000000000nginx_access_log,host=enicMBP,path=/Users/enic/github/telegraf/testdata/access.log,resp_code=504,verb=POST resp_bytes=3693i,auth=”-“,request=”/latency”,http_version=1.1,client_ip=”10.10.10.10″,ident=”-” 1650309534000000000nginx_access_log,host=enicMBP,path=/Users/enic/github/telegraf/testdata/access.log,resp_code=400,verb=POST http_version=1.1,resp_bytes=63i,auth=”-“,client_ip=”10.10.10.10″,ident=”-“,request=”/user/login” 1650309574000000000nginx_access_log,host=enicMBP,path=/Users/enic/github/telegraf/testdata/access.log,resp_code=400,verb=POST ident=”-“,auth=”-“,resp_bytes=63i,client_ip=”10.10.10.10″,request=”/user/login”,http_version=1.1 1650309574001000000nginx_access_log,host=enicMBP,path=/Users/enic/github/telegraf/testdata/access.log,resp_code=400,verb=POST ident=”-“,request=”/user/login”,auth=”-“,http_version=1.1,resp_bytes=63i,client_ip=”10.10.10.10″ 1650309574002000000nginx_access_log,host=enicMBP,path=/Users/enic/github/telegraf/testdata/access.log,resp_code=400,verb=POST auth=”-“,request=”/user/login”,client_ip=”10.10.10.10″,ident=”-“,http_version=1.1,resp_bytes=63i 1650309574003000000nginx_access_log,host=enicMBP,path=/Users/enic/github/telegraf/testdata/access.log,resp_code=400,verb=POST http_version=1.1,resp_bytes=63i,client_ip=”10.10.10.10″,auth=”-“,request=”/user/login”,ident=”-” 1650309514000000000nginx_access_log,host=enicMBP,path=/Users/enic/github/telegraf/testdata/access.log,resp_code=400,verb=POST ident=”-“,auth=”-“,client_ip=”10.10.10.10″,request=”/user/login”,http_version=1.1,resp_bytes=63i 1650309514001000000nginx_access_log,host=enicMBP,path=/Users/enic/github/telegraf/testdata/access.log,resp_code=400,verb=POST http_version=1.1,client_ip=”10.10.10.10″,request=”/user/login”,resp_bytes=63i,auth=”-“,ident=”-” 1650309514002000000nginx_access_log,host=enicMBP,path=/Users/enic/github/telegraf/testdata/access.log,resp_code=400,verb=POST resp_bytes=63i,ident=”-“,request=”/user/login”,http_version=1.1,client_ip=”10.10.10.10″,auth=”-” 1650309514003000000nginx_access_log,host=enicMBP,path=/Users/enic/github/telegraf/testdata/access.log,resp_code=400,verb=POST resp_bytes=63i,client_ip=”10.10.10.10″,ident=”-“,auth=”-“,request=”/user/login”,http_version=1.1 1650309574000000000nginx_access_log,host=enicMBP,path=/Users/enic/github/telegraf/testdata/access.log,resp_code=400,verb=POST client_ip=”10.10.10.10″,ident=”-“,auth=”-“,resp_bytes=63i,request=”/user/login”,http_version=1.1 1650309574001000000nginx_access_log,host=enicMBP,path=/Users/enic/github/telegraf/testdata/access.log,resp_code=400,verb=POST client_ip=”10.10.10.10″,resp_bytes=63i,ident=”-“,auth=”-“,request=”/user/login”,http_version=1.1 1650309574002000000nginx_access_log,host=enicMBP,path=/Users/enic/github/telegraf/testdata/access.log,resp_code=400,verb=POST resp_bytes=63i,client_ip=”10.10.10.10″,ident=”-“,auth=”-“,request=”/user/login”,http_version=1.1 1650309574003000000nginx_access_log,host=enicMBP,path=/Users/enic/github/telegraf/testdata/access.log,resp_code=400,verb=POST request=”/user/login”,resp_bytes=63i,client_ip=”10.10.10.10″,auth=”-“,http_version=1.1,ident=”-” 1650309514000000000nginx_access_log,host=enicMBP,path=/Users/enic/github/telegraf/testdata/access.log,resp_code=400,verb=POST auth=”-“,http_version=1.1,resp_bytes=63i,client_ip=”10.10.10.10″,request=”/user/login”,ident=”-” 1650309514001000000nginx_access_log,host=enicMBP,path=/Users/enic/github/telegraf/testdata/access.log,resp_code=400,verb=POST client_ip=”10.10.10.10″,auth=”-“,resp_bytes=63i,ident=”-“,request=”/user/login”,http_version=1.1 1650309514002000000nginx_access_log,host=enicMBP,path=/Users/enic/github/telegraf/testdata/access.log,resp_code=400,verb=POST auth=”-“,resp_bytes=63i,request=”/user/login”,http_version=1.1,client_ip=”10.10.10.10″,ident=”-” 1650309514003000000nginx_access_log,host=enicMBP,path=/Users/enic/github/telegraf/testdata/access.log,resp_code=400,verb=POST request=”/user/login”,resp_bytes=63i,client_ip=”10.10.10.10″,ident=”-“,http_version=1.1,auth=”-” 1650309514004000000nginx_access_log,host=enicMBP,path=/Users/enic/github/telegraf/testdata/access.log,resp_code=400,verb=POST request=”/user/login”,client_ip=”10.10.10.10″,http_version=1.1,resp_bytes=63i,ident=”-“,auth=”-” 1650309514005000000nginx_access_log,host=enicMBP,path=/Users/enic/github/telegraf/testdata/access.log,resp_code=400,verb=POST request=”/user/login”,http_version=1.1,resp_bytes=63i,client_ip=”10.10.10.10″,ident=”-“,auth=”-” 1650309514006000000nginx_access_log,host=enicMBP,path=/Users/enic/github/telegraf/testdata/access.log,resp_code=400,verb=POST ident=”-“,request=”/user/login”,resp_bytes=63i,client_ip=”10.10.10.10″,auth=”-“,http_version=1.1 1650309514007000000