ELK 日志分析平台搭建

2016-10-25 11:51:00

一、ELK介绍

本节内容来自开源实时日志分析ELK平台部署，感谢原作者详细的介绍。

日志主要包括系统日志、应用程序日志和安全日志。系统运维和开发人员可以通过日志了解服务器软硬件信息、检查配置过程中的错误及错误发生的原因。经常分析日志可以了解服务器的负荷，性能安全性，从而及时采取措施纠正错误。

通常，日志被分散的储存不同的设备上。如果你管理数十上百台服务器，你还在使用依次登录每台机器的传统方法查阅日志。这样是不是感觉很繁琐和效率低下。当务之急我们使用集中化的日志管理，例如：开源的syslog，将所有服务器上的日志收集汇总。

集中化管理日志后，日志的统计和检索又成为一件比较麻烦的事情，一般我们使用grep、awk和wc等Linux命令能实现检索和统计，但是对于要求更高的查询、排序和统计等要求和庞大的机器数量依然使用这样的方法难免有点力不从心。

开源实时日志分析ELK平台能够完美的解决我们上述的问题，ELK由ElasticSearch、Logstash和Kiabana三个开源工具组成。官方网站请戳这里

Elasticsearch是个开源分布式搜索引擎，它的特点有：分布式，零配置，自动发现，索引自动分片，索引副本机制，restful风格接口，多数据源，自动搜索负载等。

Logstash是一个完全开源的工具，他可以对你的日志进行收集、过滤，并将其存储供以后使用（如，搜索）。

Kibana 也是一个开源和免费的工具，它Kibana可以为 Logstash 和 ElasticSearch 提供的日志分析友好的 Web 界面，可以帮助您汇总、分析和搜索重要数据日志。

二、ELK架构

本图片引自https://my.oschina.net/davehe/blog/539886

三、ELK安装

系统环境 CentOS release 6.5 (Final)

Elasticsearch 2.4.1

Logstash 2.4.0

Kibana 4.6.1

java 1.7.0_79

本节主要介绍ELK的安装，至于java的安装和配置在这里不在进行阐述。

Elasticsearch

1、下载

wgethttps://download.elastic.co/elasticsearch/release/org/elasticsearch/distribution/tar/elasticsearch/2.4.1/elasticsearch-2.4.1.tar.gz

2、解压

[userwyh@userwyhelk]$tar -xzvf elasticsearch-2.4.1.tar.gz[userwyh@userwyhelk]$cd elasticsearch-2.4.1

3、安装Head插件

[userwyh@userwyh elasticsearch-2.4.1]$ ./bin/plugin install mobz/elasticsearch-head-> Installing mobz/elasticsearch-head...Trying https://github.com/mobz/elasticsearch-head/archive/master.zip...Downloading ...........................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................DONEVerifying https://github.com/mobz/elasticsearch-head/archive/master.zip checksumsifavailable...NOTE: Unable to verify checksumfordownloaded plugin (unable to find .sha1 or .md5 file to verify)Installed head into /mnt/elk/elasticsearch-2.4.1/plugins/head

4、配置

[userwyh@userwyh elasticsearch-2.4.1]$ vim config/elasticsearch.yml# 文件末尾添加以下四行，并修改你的IP地址path.data:/tmp/elasticsearch/datapath.logs:/tmp/elasticsearch/logs# ↓↓↓↓↓你的IP↓↓↓↓network.host:192.168.1.102network.port:9200

5、启动

[userwyh@userwyh elasticsearch-2.4.1]$ ./bin/elasticsearch[2016-10-22 18:21:45,994][WARN][bootstrap] unable to install syscall filter: seccomp unavailable: requires kernel 3.5+ with CONFIG_SECCOMP and CONFIG_SECCOMP_FILTER compiled in[2016-10-22 18:21:46,946][INFO][node] [Blue Bullet] version[2.4.1], pid[6821], build[c67dc32/2016-09-27T18:57:55Z][2016-10-22 18:21:46,946][INFO][node] [Blue Bullet] initializing ...[2016-10-22 18:21:48,437][INFO][plugins] [Blue Bullet] modules [lang-groovy, reindex, lang-expression], plugins [head], sites [head][2016-10-22 18:21:48,486][INFO][env] [Blue Bullet] using [1] data paths, mounts [[/ (/dev/mapper/vg_userwyh-lv_root)]], net usable_space [42gb], net total_space [49.2gb], spins? [possibly], types [ext4][2016-10-22 18:21:48,486][INFO][env] [Blue Bullet] heap size [1015.6mb], compressed ordinary object pointers [true][2016-10-22 18:21:48,486][WARN][env] [Blue Bullet] max file descriptors [4096] for elasticsearch process likely too low, consider increasing to at least [65536][2016-10-22 18:21:52,890][INFO][node] [Blue Bullet] initialized[2016-10-22 18:21:52,890][INFO][node] [Blue Bullet] starting ...[2016-10-22 18:21:53,007][INFO][transport] [Blue Bullet] publish_address {192.168.1.102:9300}, bound_addresses {192.168.1.102:9300}[2016-10-22 18:21:53,025][INFO][discovery] [Blue Bullet] elasticsearch/KuzVJMAlQ7WAsvZnunX0bw[2016-10-22 18:21:56,279][INFO][cluster.service] [Blue Bullet] new_master {Blue Bullet}{KuzVJMAlQ7WAsvZnunX0bw}{192.168.1.102}{192.168.1.102:9300}, reason: zen-disco-join(elected_as_master, [0] joins received)[2016-10-22 18:21:56,334][INFO][http] [Blue Bullet] publish_address {192.168.1.102:9200}, bound_addresses {192.168.1.102:9200}[2016-10-22 18:21:56,335][INFO][node] [Blue Bullet] started[2016-10-22 18:21:56,390][INFO][gateway] [Blue Bullet] recovered [0] indices into cluster_state

6、测试

使用curl:192.168.1.102:9200进行测试，回车后如果返回以下内容即为配置成功

[userwyh@userwyhelasticsearch-2.4.1]$curl192.168.1.102:9200{"name":"Blue Bullet","cluster_name":"elasticsearch","cluster_uuid":"fBQTeiZJRoS2S2FIpkmMFw","version":{"number":"2.4.1","build_hash":"c67dc32e24162035d18d6fe1e952c4cbcbe79d16","build_timestamp":"2016-09-27T18:57:55Z","build_snapshot":false,"lucene_version":"5.5.2"},"tagline":"You Know, for Search"}

通过浏览器访问http://192.168.1.102:9200/_plugin/head/我们可以看到如下的界面，这是我们刚刚安装的head插件，它是一个用浏览器跟ES集群交互的插件，可以查看集群状态、集群的doc内容、执行搜索和普通的Rest请求等。

Logstash

1、下载

wget https://download.elastic.co/logstash/logstash/logstash-2.4.0.tar.gz

2、解压

[userwyh@userwyhelk]$tar -xzvf logstash-2.4.0.tar.gz[userwyh@userwyhelk]$cd logstash-2.4.0

3、创建一个配置文件测试一下

[userwyh@userwyhlogstash-2.4.0]$ mkdir config[userwyh@userwyhlogstash-2.4.0]$ cd config/[userwyh@userwyhconfig]$ vim log.conf# 以下为配置文件内容,path为日志收集的地址input { file { type=>"tomcat-log"path=>["/mnt/apache-tomcat-7.0.61/logs/catalina.out"] start_position=>"beginning"}}output { elasticsearch { hosts=>"192.168.1.102:9200"index=>"logstash-%{type}"template_overwrite=>true} }

4、启动

[userwyh@userwyhlogstash-2.4.0]$./bin/logstash agent -f config/log.confSettings:Defaultpipelineworkers:1Pipelinemain started

5、测试

访问http://192.168.1.102:9200/_plugin/head/我们可以发现页面不在是空空如也了，大概是这样的画面，我们可以看到刚才在配置文件里面配置的logstash-tomcat-log索引，并看到上传了一些日志。

Kibana

1、下载

wget https://download.elastic.co/kibana/kibana/kibana-4.6.1-linux-x86_64.tar.gz

2、解压

[userwyh@userwyhelk]$tar -xzvf kibana-4.6.1-linux-x86_64.tar.gz [userwyh@userwyhelk]$cd kibana-4.6.1-linux-x86_64

3、关联Elasticsearch配置

[userwyh@userwyhkibana-4.6.1-linux-x86_64]$cd config/ [userwyh@userwyhconfig]$vim kibana.yml# 以下为配置文件内容server.port:5601server.host:192.168.1.102elasticsearch.url:http://192.168.1.102:9200

4、启动

[userwyh@userwyh kibana-4.6.1-linux-x86_64]$ ./bin/kibana log [19:20:46.485] [info][status][plugin:kibana@1.0.0] Status changed from uninitialized to green - Ready log [19:20:46.549] [info][status][plugin:elasticsearch@1.0.0] Status changed from uninitialized to yellow - Waiting for Elasticsearch log [19:20:46.644] [info][status][plugin:kbn_vislib_vis_types@1.0.0] Status changed from uninitialized to green - Ready log [19:20:46.673] [info][status][plugin:markdown_vis@1.0.0] Status changed from uninitialized to green - Ready log [19:20:46.683] [info][status][plugin:metric_vis@1.0.0] Status changed from uninitialized to green - Ready log [19:20:46.695] [info][status][plugin:spyModes@1.0.0] Status changed from uninitialized to green - Ready log [19:20:46.706] [info][status][plugin:statusPage@1.0.0] Status changed from uninitialized to green - Ready log [19:20:46.716] [info][status][plugin:table_vis@1.0.0] Status changed from uninitialized to green - Ready log [19:20:46.733] [info][listening] Server running at http://192.168.1.102:5601 log [19:20:51.682] [info][status][plugin:elasticsearch@1.0.0] Status changed from yellow to yellow - No existing Kibana index found log [19:20:54.729] [info][status][plugin:elasticsearch@1.0.0] Status changed from yellow to green - Kibana index ready

5、测试

通过浏览器访问http://192.168.1.102:5601/它会提示我们创建一个index pattern，输入上述配置的logstash-tomcat-log，点击Create按钮