这里分享的是ELK集群基础安装文档，这里是 ELK安装基础
环境：Centos 6.6
ElasticSearch 5.1.1
Logstash 5.1.1
Kibana 5.1.1
安装集群管理软件
#安装ElasticSearch [root@ELK ~]# yum -y install elasticsearch-5.1.1.rpm #ELK node1配置 [root@ELK ~]# mkdir /etc/elasticsearch/data -p [root@ELK ~]# id elasticsearch uid=498(elasticsearch) gid=499(elasticsearch) groups=499(elasticsearch) [root@ELK ~]# chgrp elasticsearch /etc/elasticsearch/data/ -R [root@ELK ~]# chmod 775 /etc/elasticsearch/data/ -R [root@ELK ~]# java -version java version "1.8.0_112" Java(TM) SE Runtime Environment (build 1.8.0_112-b15) Java HotSpot(TM) 64-Bit Server VM (build 25.112-b15, mixed mode) [root@ELK ~]# ln -s /usr/local/jdk1.8.0_112/bin/java /usr/bin/ [root@ELK ~]# grep -v ^# /etc/elasticsearch/elasticsearch.yml | grep -v ^$ cluster.name: "ES-cluster" node.name: "es-node01" node.master: true node.data: true http.enabled: true path.data: /etc/elasticsearch/data path.logs: /var/log/elasticsearch network.host: 0.0.0.0 http.port: 9200 http.cors.enabled: true http.cors.allow-origin: "*" discovery.zen.ping.unicast.hosts: ["192.168.31.100", "192.168.31.110"] discovery.zen.minimum_master_nodes: 1 #discovery.zen.minimum_master_nodes（默认是1）：这个参数控制的是，一个节点需要看到的具有master节点资格的最小数量，然后才能在集群中做操作。官方的推荐值是(N/2)+1，其中N是具有master资格的节点的数量（我们的情况是3，因此这个参数设置为2，但对于只有2个节点的情况，设置为2就有些问题了，一个节点DOWN掉后，你肯定连不上2台服务器了，这点需要注意） gateway.recover_after_nodes: 2 [root@ELK ~]# [root@ELK ~]# /etc/init.d/elasticsearch start [root@ELK ~]# netstat -tunlp | grep 9200 tcp 0 0 :::9200 :::* LISTEN 33156/java [root@ELK ~]# netstat -tunlp | grep 9300 tcp 0 0 :::9300 :::* LISTEN 33156/java [root@ELK ~]# #ELK node2配置 #node2配置和node1配置相同(略)
测试集群状态
[root@ELK ~]# curl localhost:9200/_cat/nodes?v #获取集群中节点列表 ip heap.percent ram.percent cpu load_1m load_5m load_15m node.role master name 192.168.31.100 4 81 99 4.32 3.28 3.46 mdi - es-node01 192.168.31.110 3 94 5 0.07 0.03 0.01 mdi * es-node02 [root@ELK ~]# [root@ELK ~]# curl localhost:9200/_cat/health?v #集群健康检查 epoch timestamp cluster status node.total node.data shards pri relo init unassign pending_tasks max_task_wait_time active_shards_percent 1482982586 11:36:26 ES-cluster green 2 2 22 11 0 0 0 0 - 100.0% [root@ELK ~]# [root@ELK ~]# curl 'localhost:9200/_cat/indices?v' #获取ElasticSearch索引 health status index uuid pri rep docs.count docs.deleted store.size pri.store.size green open logstash-message-2016.12.29 qkvr3jmWQei1oBhEy9VnCA 5 1 28 0 376.3kb 188.1kb green open logstash-nginx-2016.12.29 B9p9qwjsTlaE4fcZcIgJag 5 1 5 0 104kb 52kb green open .kibana bNLdONDMRdWK2-HdYkUuAA 1 1 3 0 34.1kb 17kb [root@ELK ~]# //注：当ElasticSearch配置完成后，node2会复制node1的索引 [root@ELK2 ~]# curl 'localhost:9200/_cat/indices?v' health status index uuid pri rep docs.count docs.deleted store.size pri.store.size green open logstash-message-2016.12.29 qkvr3jmWQei1oBhEy9VnCA 5 1 28 0 376.3kb 188.1kb green open .kibana bNLdONDMRdWK2-HdYkUuAA 1 1 3 0 34.1kb 17kb green open logstash-nginx-2016.12.29 B9p9qwjsTlaE4fcZcIgJag 5 1 5 0 104kb 52kb [root@ELK2 ~]# #ElasticSearch集群日志文件位置： [root@ELK ~]# ls /var/log/elasticsearch/ ES-cluster_deprecation.log ES-cluster_index_search_slowlog.log ES-cluster_index_indexing_slowlog.log ES-cluster.log [root@ELK ~]#
安装elasticsearch-head插件
由于Elasticsearch 5.0 head插件不能以插件形式安装，因此需要单独安装
参考： https://github.com/mobz/elasticsearch-head
Running as a plugin of Elasticsearch Install elasticsearch-head: – for Elasticsearch 5.x: site plugins are not supported. Run elasticsearch-head as a standalone server – for Elasticsearch 2.x – 4.x: sudo elasticsearch/bin/plugin install mobz/elasticsearch-head – for Elasticsearch 1.x: sudo elasticsearch/bin/plugin -install mobz/elasticsearch-head/1.x – for Elasticsearch 0.9: sudo elasticsearch/bin/plugin -install mobz/elasticsearch-head/0.9 open http://localhost:9200/_plugin/head/ Running with built in server： enable cors by adding http.cors.enabled: true in elasticsearch configuration. Don’t forget to also set http.cors.allow-origin because no origin allowed by default. http.cors.allow-origin: "*" is valid value, however it’s considered as a security risk as your cluster is open to cross origin from anywhere. Check Elasticsearch documentation on this parameter: https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-http.html#modules-http git clone git://github.com/mobz/elasticsearch-head.git cd elasticsearch-head npm install grunt server open http://localhost:9100/
#npm命令需要安装nodejs
[root@ELK ~]# wget https://nodejs.org/dist/v0.12.16/node-v0.12.16.tar.gz
[root@ELK ~]# tar -zxf node-v0.12.16.tar.gz
[root@ELK ~]# cd node-v0.12.16
[root@ELK node-v0.12.16]#
[root@ELK node-v0.12.16]# ./configure --prefix=/usr/local/node-v0.12.16
[root@ELK node-v0.12.16]# make && make install
[root@ELK ~]# ls -l /usr/local/node-v0.12.16/bin/npm
lrwxrwxrwx 1 root root 38 Dec 28 12:43 /usr/local/node-v0.12.16/bin/npm -> ../lib/node_modules/npm/bin/npm-cli.js
[root@ELK ~]# ln -s /usr/local/node-v0.12.16/bin/npm /usr/bin/
[root@ELK ~]# ll /usr/bin/npm
lrwxrwxrwx 1 root root 32 Dec 28 14:31 /usr/bin/npm -> /usr/local/node-v0.12.16/bin/npm
[root@ELK ~]#
[root@ELK ~]# ln -s /usr/local/node-v0.12.16/bin/node /usr/bin/
[root@ELK ~]# ll /usr/bin/node
lrwxrwxrwx 1 root root 33 Dec 28 15:06 /usr/bin/node -> /usr/local/node-v0.12.16/bin/node
[root@ELK ~]#
[root@ELK ~]# yum -y install git
[root@ELK ~]# cd /usr/local/
[root@ELK local]# git clone git://github.com/mobz/elasticsearch-head.git
[root@ELK local]# cd elasticsearch-head
[root@ELK elasticsearch-head]# npm install
#如果在elasticsearch-head目录下node_modules/grunt下如果没有grunt二进制程序，则需要执行
[root@ELK ~]# cd /usr/local/elasticsearch-head/
[root@ELK elasticsearch-head]# npm install grunt --save
[root@ELK elasticsearch-head]# ls
Dockerfile grunt_fileSets.js node_modules README.textile test
elasticsearch-head.sublime-project index.html package.json _site
Gruntfile.js LICENCE plugin-descriptor.properties src
[root@ELK elasticsearch-head]# ls -l node_modules/grunt/bin/grunt
-rwxr-xr-x 1 root root 53 Apr 6 2016 node_modules/grunt/bin/grunt
[root@ELK elasticsearch-head]#
[root@ELK elasticsearch-head]# head -98 Gruntfile.js | tail -9
connect: {
server: {
options: {
hostname: '0.0.0.0', #添加这行
port: 9100,
base: '.',
keepalive: true
}
}
[root@ELK elasticsearch-head]# /usr/local/elasticsearch-head/node_modules/grunt/bin/grunt server #如果执行报错看下面的解决办法
Running "connect:server" (connect) task
Waiting forever...
Started connect web server on http://localhost:9100
Q & A
如果服务器重启了，有时发现执行
server```报错```Fatal error: Unable to find local grunt.```，需要重新按照Question1的解决方法操作一下，也可以执行下面的脚本
``bash
[root@ELK ~]# cat check_es_head_grunt.sh
#!/bin/bash
# Auther: yfshare
# Date:2016-12-29
eshead_dir="/usr/local/elasticsearch-head"
grunt_dir="$eshead_dir/node_modules/grunt"
grunt_bin="$grunt_dir/bin/grunt"
kill -9 `ps -ef | grep -iw 'grunt' | grep -v grep | awk '{print $2}'` &>/dev/null
[ ! -x "$grunt_bin" ] && chmod 755 "$grunt_bin"
echo ''
echo 'Please wait a moment.'
cd "$eshead_dir"
npm install grunt --save-dev &> /dev/null
cd "$eshead_dir"
"$grunt_bin" server &
[ $? -eq 0 ] && echo 'start ok.'
[root@ELK ~]#
|
[root@ELK ~]# sh check_es_head_grunt.sh Please wait a moment. start ok. Running "connect:server" (connect) task Waiting forever... Started connect web server on http://localhost:9100 [root@ELK ~]#
Question1：如果报下面的错误：
[root@ELK ~]# /usr/local/elasticsearch-head/node_modules/grunt/bin/grunt server grunt-cli: The grunt command line interface (v1.2.0) Fatal error: Unable to find local grunt. If you're seeing this message, grunt hasn't been installed locally to your project. For more information about installing and configuring grunt, please see the Getting Started guide: http://gruntjs.com/getting-started [root@ELK ~]# Answer： [root@ELK ~]# cd /usr/local/elasticsearch-head/ [root@ELK elasticsearch-head]# npm install grunt --save-dev 再执行/usr/local/elasticsearch-head/node_modules/grunt/bin/grunt server就OK了
Question2：
之前修改ElasticSearch 5.1的network.host的IP时，不管修改成什么(注释network.host除外)，重启均报错，开始以为是不能绑定IP地址。之前有注意到日志里的报错的
checks failed. max number of threads [1024] for user [elasticsearch] is too low, increase to at least [2048]```因为注释了network.host后ElasticSearch能起来，所以没意识到是它的问题
``bash
#ElasticSearch日志报错
[2016-12-28T16:57:23,190][INFO ][o.e.n.Node ] [es-node01] starting ...
[2016-12-28T16:57:24,616][INFO ][o.e.t.TransportService ] [es-node01] publish_address {192.168.31.100:9300}, bound_addresses {192.168.31.100:9300}
[2016-12-28T16:57:24,686][INFO ][o.e.b.BootstrapCheck ] [es-node01] bound or publishing to a non-loopback or non-link-local address, enforcing bootstrap checks
[2016-12-28T16:57:24,708][ERROR][o.e.b.Bootstrap ] [es-node01] node validation exception
bootstrap checks failed
max number of threads [1024] for user [elasticsearch] is too low, increase to at least [2048]
[2016-12-28T16:57:24,739][INFO ][o.e.n.Node ] [es-node01] stopping ...
[2016-12-28T16:57:25,275][INFO ][o.e.n.Node ] [es-node01] stopped
[2016-12-28T16:57:25,277][INFO ][o.e.n.Node ] [es-node01] closing ...
[2016-12-28T16:57:25,500][INFO ][o.e.n.Node ] [es-node01] closed
|
Answer：修改
关于ElasticSearch5.1的network.host设置 参考：https://www.elastic.co/guide/en/elasticsearch/reference/5.1/modules-network.html#common-network-settings ```bash [root@ELK ~]# grep -v ^# /etc/security/limits.conf | grep -v ^$ elasticsearch soft nproc 2048 elasticsearch hard nproc 4096 [root@ELK ~]#
Question3：
如果执行 /usr/local/elasticsearch-head/node_modules/grunt/bin/grunt server
报下面的错误
[root@ELK-test ~]# /usr/local/elasticsearch-head/node_modules/grunt/bin/grunt server grunt-cli: The grunt command line interface (v1.2.0) Fatal error: Unable to find local grunt. If you're seeing this message, grunt hasn't been installed locally to your project. For more information about installing and configuring grunt, please see the Getting Started guide: http://gruntjs.com/getting-started [root@ELK-test ~]# Answer： #进入elasticsearch-head安装目录即可，因为在别的地方找不到Gruntfile.js文件 [root@ELK-test ~]# cd /usr/local/elasticsearch-head/
Question4：
如果执行 /usr/local/elasticsearch-head/node_modules/grunt/bin/grunt server
报下面的错误
[root@ELK-test elasticsearch-head]# /usr/local/elasticsearch-head/node_modules/grunt/bin/grunt server >> Local Npm module "grunt-contrib-clean" not found. Is it installed? >> Local Npm module "grunt-contrib-concat" not found. Is it installed? >> Local Npm module "grunt-contrib-watch" not found. Is it installed? >> Local Npm module "grunt-contrib-connect" not found. Is it installed? >> Local Npm module "grunt-contrib-copy" not found. Is it installed? >> Local Npm module "grunt-contrib-jasmine" not found. Is it installed? Warning: Task "connect:server" not found. Use --force to continue. Aborted due to warnings. [root@ELK-test elasticsearch-head]# Answer： 出现以下提示，为Gruntfile.js引用的，缺少以下包 [root@ELK-test elasticsearch-head]# npm install grunt-contrib-clean --registry=https://registry.npm.taobao.org [root@ELK-test elasticsearch-head]# npm install grunt-contrib-concat --registry=https://registry.npm.taobao.org [root@ELK-test elasticsearch-head]# npm install grunt-contrib-watch --registry=https://registry.npm.taobao.org [root@ELK-test elasticsearch-head]# npm install grunt-contrib-connect --registry=https://registry.npm.taobao.org [root@ELK-test elasticsearch-head]# npm install grunt-contrib-copy --registry=https://registry.npm.taobao.org [root@ELK-test elasticsearch-head]# npm install grunt-contrib-jasmine --registry=https://registry.npm.taobao.org [root@ELK-test elasticsearch-head]# /usr/local/elasticsearch-head/node_modules/grunt/bin/grunt server & [1] 22877 [root@ELK-test elasticsearch-head]#
注：集群配置完成后，需要把Logstash里的ElasticSearch的地址修改正确
即：Logstash配置文件的output关于ElasticSearch的配置
output {
elasticsearch {
hosts => ["192.168.31.100:9200"]
index => "logstash-nginx-%{+YYYY.MM.dd}"
}
注：本文内容来自互联网，旨在为开发者提供分享、交流的平台。如有涉及文章版权等事宜，请你联系站长进行处理。