Elasticsearch7.7部署

619次阅读
没有评论

共计 16391 个字符，预计需要花费 41 分钟才能阅读完成。

Elasticsearch7.7部署

最近遇到项目对平台用户的评论需要收集和搜索的需求，考虑用elasticsearch或mongodb，首选当然是elasticsearch，毕竟搜索一哥的地位。。。遂此处记录下elasticsearch的部署和配置

软件包链接

[root@elastic-01 src]# wget https://artifacts.elastic.co/downloads/kibana/kibana-7.7.0-x86_64.rpm
[root@elastic-01 src]# wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-7.7.0-x86_64.rpm

系统初始配置

配置内核参数

[root@elastic-01 ~]# cat /etc/sysctl.conf
vm.max_map_count = 655350
fs.file-max = 128000
net.ipv4.tcp_retries2 = 5

[root@elastic-01 ~]# sysctl -p

配置文件描述符

[root@elastic-01 ~]# cat /etc/security/limits.conf
root                soft    core            unlimited
root                hard    core            unlimited
root                soft    nproc           1000000
root                hard    nproc           1000000
root                soft    nofile          1000000
root                hard    nofile          1000000
root                soft    memlock         32000
root                hard    memlock         32000
root                soft    msgqueue        8192000
root                hard    msgqueue        8192000

*                soft    core            unlimited
*                hard    core            unlimited
*                soft    nproc           1000000
*                hard    nproc           1000000
*                soft    nofile          1000000
*                hard    nofile          1000000
*                soft    memlock         32000
*                hard    memlock         32000
*                soft    msgqueue        8192000
*                hard    msgqueue        8192000

关闭swap分区

# 注释fstab的swap分群配置，略

# 关闭当前swap分区
[root@elastic-01 ~]# swapoff -a

elasticsearch配置

配置elasticsearch配置文件

[root@elastic-01 elasticsearch]# egrep -v "^$|^#" /etc/elasticsearch/elasticsearch.yml
node.name: "node-1"
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
bootstrap.memory_lock: true
network.host: 0.0.0.0
http.port: 9200
discovery.seed_hosts: ["192.168.44.141"]
cluster.initial_master_nodes: "node-1"
http.cors.enabled: true
http.cors.allow-origin: "*"

调整jvm参数

# 根据自己机器去调整Xms和Xmx,官方建议Xmx最大不超过32G
[root@elastic-01 elasticsearch]#  egrep -v "^$|^#" jvm.options
-Xms1g
-Xmx1g
8-13:-XX:+UseConcMarkSweepGC
8-13:-XX:CMSInitiatingOccupancyFraction=75
8-13:-XX:+UseCMSInitiatingOccupancyOnly
14-:-XX:+UseG1GC
14-:-XX:G1ReservePercent=25
14-:-XX:InitiatingHeapOccupancyPercent=30
-Djava.io.tmpdir=${ES_TMPDIR}
-XX:+HeapDumpOnOutOfMemoryError
-XX:HeapDumpPath=/var/lib/elasticsearch
-XX:ErrorFile=/var/log/elasticsearch/hs_err_pid%p.log
8:-XX:+PrintGCDetails
8:-XX:+PrintGCDateStamps
8:-XX:+PrintTenuringDistribution
8:-XX:+PrintGCApplicationStoppedTime
8:-Xloggc:/var/log/elasticsearch/gc.log
8:-XX:+UseGCLogFileRotation
8:-XX:NumberOfGCLogFiles=32
8:-XX:GCLogFileSize=64m
9-:-Xlog:gc*,gc+age=trace,safepoint:file=/var/log/elasticsearch/gc.log:utctime,pid,tags:filecount=32,filesize=64m

配置启动elasticsearch systemed文件

[root@elastic-01 elasticsearch]# egrep -v "^$|^#" /usr/lib/systemd/system/elasticsearch.service
[Unit]
Description=Elasticsearch
Documentation=https://www.elastic.co
Wants=network-online.target
After=network-online.target
[Service]
Type=notify
RuntimeDirectory=elasticsearch
PrivateTmp=true
Environment=ES_HOME=/usr/share/elasticsearch
Environment=ES_PATH_CONF=/etc/elasticsearch
Environment=PID_DIR=/var/run/elasticsearch
Environment=ES_SD_NOTIFY=true
EnvironmentFile=-/etc/sysconfig/elasticsearch
# 增加这行，不然无法lock memory
LimitMEMLOCK=infinity
WorkingDirectory=/usr/share/elasticsearch
User=elasticsearch
Group=elasticsearch
ExecStart=/usr/share/elasticsearch/bin/systemd-entrypoint -p ${PID_DIR}/elasticsearch.pid --quiet
StandardOutput=journal
StandardError=inherit
LimitNOFILE=65535
LimitNPROC=4096
LimitAS=infinity
LimitFSIZE=infinity
TimeoutStopSec=0
KillSignal=SIGTERM
KillMode=process
SendSIGKILL=no
SuccessExitStatus=143
[Install]
WantedBy=multi-user.target

最后启动ealsticsearch

[root@elastic-01 elasticsearch]# systemctl daemon-reload
[root@elastic-01 elasticsearch]# systemctl enable elasticsearch
[root@elastic-01 elasticsearch]# systemctl start elasticsearch
[root@elastic-01 elasticsearch]# systemctl status elasticsearch
● elasticsearch.service - Elasticsearch
   Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; enabled; vendor preset: disabled)
   Active: active (running) since Sun 2020-06-12 23:00:04 CST; 20min ago
     Docs: https://www.elastic.co
 Main PID: 3814 (java)
   CGroup: /system.slice/elasticsearch.service
           ├─3814 /usr/share/elasticsearch/jdk/bin/java -Xshare:auto -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -XX...
           └─4048 /usr/share/elasticsearch/modules/x-pack-ml/platform/linux-x86_64/bin/controller

Jun 12 22:59:05 elastic-01 systemd[1]: Starting Elasticsearch...
Jun 12 23:00:04 elastic-01 systemd[1]: Started Elasticsearch.

[root@elastic-01 elasticsearch]# netstat -tunlp
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      911/sshd
tcp        0      0 127.0.0.1:25            0.0.0.0:*               LISTEN      1043/master
tcp6       0      0 :::9200                 :::*                    LISTEN      3814/java
tcp6       0      0 :::9300                 :::*                    LISTEN      3814/java
tcp6       0      0 :::22                   :::*                    LISTEN      911/sshd
tcp6       0      0 ::1:25                  :::*                    LISTEN      1043/master
udp        0      0 127.0.0.1:323           0.0.0.0:*                           656/chronyd
udp        0      0 0.0.0.0:68              0.0.0.0:*                           717/dhclient
udp6       0      0 ::1:323                 :::*                                656/chronyd

# 测试请求响应
[root@elastic-01 elasticsearch]# curl 127.0.0.1:9200
{
  "name" : "node-1",
  "cluster_name" : "elasticsearch",
  "cluster_uuid" : "ccmXpZK8S2in9DZfJM1HOQ",
  "version" : {
    "number" : "7.7.0",
    "build_flavor" : "default",
    "build_type" : "rpm",
    "build_hash" : "81a1e9eda8e6183f5237786246f6dced26a10eaf",
    "build_date" : "2020-05-12T02:01:37.602180Z",
    "build_snapshot" : false,
    "lucene_version" : "8.5.1",
    "minimum_wire_compatibility_version" : "6.8.0",
    "minimum_index_compatibility_version" : "6.0.0-beta1"
  },
  "tagline" : "You Know, for Search"
}

elasticsearch配置X-pack提高安全性

elasticsearch的一些工具及路径

[root@elastic-01 bin]# pwd
/usr/share/elasticsearch/bin
[root@elastic-01 bin]# ll
total 20172
-rwxr-xr-x 1 root root     2877 May 12  2020 elasticsearch
-rwxr-xr-x 1 root root      491 May 12  2020 elasticsearch-certgen
-rwxr-xr-x 1 root root      483 May 12  2020 elasticsearch-certutil
-rwxr-xr-x 1 root root      996 May 12  2020 elasticsearch-cli
-rwxr-xr-x 1 root root      433 May 12  2020 elasticsearch-croneval
-rwxr-xr-x 1 root root     4428 May 12  2020 elasticsearch-env
-rwxr-xr-x 1 root root     1828 May 12  2020 elasticsearch-env-from-file
-rwxr-xr-x 1 root root      121 May 12  2020 elasticsearch-keystore
-rwxr-xr-x 1 root root      440 May 12  2020 elasticsearch-migrate
-rwxr-xr-x 1 root root      126 May 12  2020 elasticsearch-node
-rwxr-xr-x 1 root root      172 May 12  2020 elasticsearch-plugin
-rwxr-xr-x 1 root root      431 May 12  2020 elasticsearch-saml-metadata
-rwxr-xr-x 1 root root      438 May 12  2020 elasticsearch-setup-passwords
-rwxr-xr-x 1 root root      118 May 12  2020 elasticsearch-shard
-rwxr-xr-x 1 root root      441 May 12  2020 elasticsearch-sql-cli
-rwxr-xr-x 1 root root 20565600 May 12  2020 elasticsearch-sql-cli-7.7.0.jar
-rwxr-xr-x 1 root root      426 May 12  2020 elasticsearch-syskeygen
-rwxr-xr-x 1 root root      426 May 12  2020 elasticsearch-users
-rwxr-xr-x 1 root root      332 May 12  2020 systemd-entrypoint
-rwxr-xr-x 1 root root      346 May 12  2020 x-pack-env
-rwxr-xr-x 1 root root      354 May 12  2020 x-pack-security-env
-rwxr-xr-x 1 root root      353 May 12  2020 x-pack-watcher-env

创建ca证书

# 创建ca证书
[root@elastic-01 bin]# elasticsearch-certutil ca
-bash: elasticsearch-certutil: command not found
[root@elastic-01 bin]# ./elasticsearch-certutil ca
This tool assists you in the generation of X.509 certificates and certificate
signing requests for use with SSL/TLS in the Elastic stack.

The 'ca' mode generates a new 'certificate authority'
This will create a new X.509 certificate and private key that can be used
to sign certificate when running in 'cert' mode.

Use the 'ca-dn' option if you wish to configure the 'distinguished name'
of the certificate authority

By default the 'ca' mode produces a single PKCS#12 output file which holds:
    * The CA certificate
    * The CA's private key

If you elect to generate PEM format certificates (the -pem option), then the output will
be a zip file containing individual files for the CA certificate and private key

Please enter the desired output file [elastic-stack-ca.p12]:
Enter password for elastic-stack-ca.p12 :

# 签证私钥
[root@elastic-01 bin]# ./elasticsearch-certutil cert --ca elastic-stack-ca.p12
This tool assists you in the generation of X.509 certificates and certificate
signing requests for use with SSL/TLS in the Elastic stack.

The 'cert' mode generates X.509 certificate and private keys.
    * By default, this generates a single certificate and key for use
       on a single instance.
    * The '-multiple' option will prompt you to enter details for multiple
       instances and will generate a certificate and key for each one
    * The '-in' option allows for the certificate generation to be automated by describing
       the details of each instance in a YAML file

    * An instance is any piece of the Elastic Stack that requires an SSL certificate.
      Depending on your configuration, Elasticsearch, Logstash, Kibana, and Beats
      may all require a certificate and private key.
    * The minimum required value for each instance is a name. This can simply be the
      hostname, which will be used as the Common Name of the certificate. A full
      distinguished name may also be used.
    * A filename value may be required for each instance. This is necessary when the
      name would result in an invalid file or directory name. The name provided here
      is used as the directory name (within the zip) and the prefix for the key and
      certificate files. The filename is required if you are prompted and the name
      is not displayed in the prompt.
    * IP addresses and DNS names are optional. Multiple values can be specified as a
      comma separated string. If no IP addresses or DNS names are provided, you may
      disable hostname verification in your SSL configuration.

    * All certificates generated by this tool will be signed by a certificate authority (CA).
    * The tool can automatically generate a new CA for you, or you can provide your own with the
         -ca or -ca-cert command line options.

By default the 'cert' mode produces a single PKCS#12 output file which holds:
    * The instance certificate
    * The private key for the instance certificate
    * The CA certificate

If you specify any of the following options:
    * -pem (PEM formatted output)
    * -keep-ca-key (retain generated CA key)
    * -multiple (generate multiple certificates)
    * -in (generate certificates from an input file)
then the output will be be a zip file containing individual certificate/key files

Enter password for CA (elastic-stack-ca.p12) :
Please enter the desired output file [elastic-certificates.p12]:
Enter password for elastic-certificates.p12 :

Certificates written to /usr/share/elasticsearch/elastic-certificates.p12

This file should be properly secured as it contains the private key for
your instance.

This file is a self contained file and can be copied and used 'as is'
For each Elastic product that you wish to configure, you should copy
this '.p12' file to the relevant configuration directory
and then follow the SSL configuration instructions in the product guide.

For client applications, you may only need to copy the CA certificate and
configure the client to trust this certificate.

# 生成的文件位置在
[root@elastic-01 bin]# ls /usr/share/elasticsearch/
bin  elastic-certificates.p12  elastic-stack-ca.p12  jdk  lib  LICENSE.txt  modules  NOTICE.txt  plugins  README.asciidoc

# 将文件放置/etc/elasticsearch/中
[root@elastic-01 elasticsearch]# mv elastic-* /etc/elasticsearch
[root@elastic-01 elasticsearch]# ll /etc/elasticsearch/
total 48
-rw------- 1 root root           3443 Dec 11 23:32 elastic-certificates.p12
-rw-rw---- 1 root elasticsearch   199 Dec 11 22:42 elasticsearch.keystore
-rw-rw---- 1 root elasticsearch  2891 Dec 11 22:58 elasticsearch.yml
-rw------- 1 root root           2527 Dec 11 23:29 elastic-stack-ca.p12
-rw-rw---- 1 root elasticsearch  2373 May 12  2020 jvm.options
drwxr-s--- 2 root elasticsearch     6 May 12  2020 jvm.options.d
-rw-rw---- 1 root elasticsearch 17419 May 12  2020 log4j2.properties
-rw-rw---- 1 root elasticsearch   473 May 12  2020 role_mapping.yml
-rw-rw---- 1 root elasticsearch   197 May 12  2020 roles.yml
-rw-rw---- 1 root elasticsearch     0 May 12  2020 users
-rw-rw---- 1 root elasticsearch     0 May 12  2020 users_roles

# 修改文件属主
[root@elastic-01 elasticsearch]# chown elasticsearch:elasticsearch -R  /etc/elasticsearch

# 设置密码，即为前面配置证书时输入的密码，漏此步骤，无法启动成功，会一直提示：failed to load SSL configuration [xpack.security.transport.ssl]

[root@elastic-01 elasticsearch]# /usr/share/elasticsearch/bin/elasticsearch-keystore add  xpack.security.transport.ssl.keystore.secure_password
Enter value for xpack.security.transport.ssl.keystore.secure_password:
[root@elastic-01 elasticsearch]# /usr/share/elasticsearch/bin/elasticsearch-keystore add  xpack.security.transport.ssl.truststore.secure_password
Enter value for xpack.security.transport.ssl.truststore.secure_password:

调整elasticsearch配置文件开启x-pack认证

[root@elastic-01 elasticsearch]# egrep -v "^$|^#" elasticsearch.yml
node.name: "node-1"
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
bootstrap.memory_lock: true
network.host: 0.0.0.0
http.port: 9200
discovery.seed_hosts: ["192.168.44.141"]
cluster.initial_master_nodes: "node-1"
http.cors.enabled: true
http.cors.allow-origin: "*"
xpack.security.enabled: true
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.keystore.path: /etc/elasticsearch/elastic-certificates.p12   # 步骤一中文件路径（默认从config文件夹中读取）
xpack.security.transport.ssl.truststore.path: /etc/elasticsearch/elastic-certificates.p12 # 步骤一中文件路径（默认从config文件夹中读取）
xpack.security.transport.filter.allow: "192.168.*"

# 重启服务
[root@elastic-01 elasticsearch]# systemctl restart elasticsearch

# 测试验证，此时未配置用户密码，会提示401
[root@elastic-01 elasticsearch]# curl 127.0.0.1:9200
{"error":{"root_cause":[{"type":"security_exception","reason":"missing authentication credentials for REST request [/]","header":{"WWW-Authenticate":"Basic realm=\"security\" charset=\"UTF-8\""}}],"type":"security_exception","reason":"missing authentication credentials for REST request [/]","header":{"WWW-Authenticate":"Basic realm=\"security\" charset=\"UTF-8\""}},"status":401}

配置elasticsearch用户密码

它会提示需要设置 elastic，apm_system，kibana，kibana_system，logstash_system，beats_system的密码，此处将密码都配置为elastic

[root@elastic-01 elasticsearch]# /usr/share/elasticsearch/bin/elasticsearch-setup-passwords interactive
Initiating the setup of passwords for reserved users elastic,apm_system,kibana,logstash_system,beats_system,remote_monitoring_user.
You will be prompted to enter passwords as the process progresses.
Please confirm that you would like to continue [y/N]y


Enter password for [elastic]:
Reenter password for [elastic]:
Enter password for [apm_system]:
Reenter password for [apm_system]:
Enter password for [kibana]:
Reenter password for [kibana]:
Enter password for [logstash_system]:
Reenter password for [logstash_system]:
Enter password for [beats_system]:
Reenter password for [beats_system]:
Enter password for [remote_monitoring_user]:
Reenter password for [remote_monitoring_user]:
Changed password for user [apm_system]
Changed password for user [kibana]
Changed password for user [logstash_system]
Changed password for user [beats_system]
Changed password for user [remote_monitoring_user]
Changed password for user [elastic]

测试验证用户密码

[root@elastic-01 elasticsearch]# curl -u elastic http://127.0.0.1:9200
Enter host password for user 'elastic':
{
  "name" : "node-1",
  "cluster_name" : "elasticsearch",
  "cluster_uuid" : "ccmXpZK8S2in9DZfJM1HOQ",
  "version" : {
    "number" : "7.7.0",
    "build_flavor" : "default",
    "build_type" : "rpm",
    "build_hash" : "81a1e9eda8e6183f5237786246f6dced26a10eaf",
    "build_date" : "2020-05-12T02:01:37.602180Z",
    "build_snapshot" : false,
    "lucene_version" : "8.5.1",
    "minimum_wire_compatibility_version" : "6.8.0",
    "minimum_index_compatibility_version" : "6.0.0-beta1"
  },
  "tagline" : "You Know, for Search"
}

Kibana配置连接

配置elasticsearch的pem证书

[root@elastic-01 elasticsearch]# pwd
/etc/elasticsearch
[root@elastic-01 elasticsearch]# openssl pkcs12 -in elastic-certificates.p12 -cacerts -nokeys -out elastic-ca.pem
Enter Import Password:
MAC verified OK
[root@elastic-01 elasticsearch]#
[root@elastic-01 elasticsearch]# ll
total 52
-rw-r--r-- 1 root          elasticsearch  1397 Dec 12 00:05 elastic-ca.pem
-rw-r--r-- 1 elasticsearch elasticsearch  3443 Dec 11 23:32 elastic-certificates.p12
-rw-rw---- 1 root          elasticsearch   335 Dec 11 23:49 elasticsearch.keystore
-rw-rw---- 1 elasticsearch elasticsearch  3387 Dec 11 23:42 elasticsearch.yml
-rw-r--r-- 1 elasticsearch elasticsearch  2527 Dec 11 23:29 elastic-stack-ca.p12
-rw-rw---- 1 elasticsearch elasticsearch  2373 May 12  2020 jvm.options
drwxr-s--- 2 elasticsearch elasticsearch     6 May 12  2020 jvm.options.d
-rw-rw---- 1 elasticsearch elasticsearch 17419 May 12  2020 log4j2.properties
-rw-rw---- 1 elasticsearch elasticsearch   473 May 12  2020 role_mapping.yml
-rw-rw---- 1 elasticsearch elasticsearch   197 May 12  2020 roles.yml
-rw-rw---- 1 elasticsearch elasticsearch     0 May 12  2020 users
-rw-rw---- 1 elasticsearch elasticsearch     0 May 12  2020 users_roles

# 将 elastic-ca.pem复制到kibana节点上/etc/kibana,略

配置kibana

安装nodejs，不然kibana有如下报错日志

Dec 21 01:07:50 localhost kibana: Browserslist: caniuse-lite is outdated. Please run the following command: `npm update`
Dec 21 01:07:50 localhost kibana: Browserslist: caniuse-lite is outdated. Please run the following command: `npm update`
Dec 21 01:07:50 localhost kibana: Browserslist: caniuse-lite is outdated. Please run the following command: `npm update`
Dec 21 01:07:51 localhost kibana: Browserslist: caniuse-lite is outdated. Please run the following command: `npm update`

安装kibana并配置

[root@elastic-01 ~]# yum install ./kibana-7.7.0-x86_64.rpm -y
[root@elastic-01 ~]# yum install nodejs -y
[root@elastic-01 ~]# cd /etc/kibana
[root@elastic-01 kibana]# egrep -v "^$|^#" kibana.yml
server.host: "0.0.0.0"
server.name: "kibana-server"
elasticsearch.hosts: ["http://192.168.44.141:9200"]
elasticsearch.username: "kibana"
elasticsearch.password: "elastic"
elasticsearch.ssl.certificateAuthorities: [ "/etc/kibana/elastic-ca.pem" ]
elasticsearch.ssl.verificationMode: certificate
i18n.locale: "zh-CN"

[root@elastic-01 kibana]# systemctl start kibana
[root@elastic-01 kibana]# netstat -tunlp
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 0.0.0.0:5601            0.0.0.0:*               LISTEN      16970/node
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      911/sshd
tcp        0      0 127.0.0.1:25            0.0.0.0:*               LISTEN      1043/master
tcp6       0      0 :::9200                 :::*                    LISTEN      15482/java
tcp6       0      0 :::9300                 :::*                    LISTEN      15482/java
tcp6       0      0 :::22                   :::*                    LISTEN      911/sshd
tcp6       0      0 ::1:25                  :::*                    LISTEN      1043/master
udp        0      0 127.0.0.1:323           0.0.0.0:*                           656/chronyd
udp        0      0 0.0.0.0:68              0.0.0.0:*                           717/dhclient
udp6       0      0 ::1:323                 :::*                                656/chronyd
Elasticsearch7.7部署
Elasticsearch7.7部署

正文完
elasticsearch
发表至： elasticsearch 运维
2020-06-14
 
xadocker
版权声明：本站原创文章，由 xadocker 2020-06-14发表，共计16391字。
转载说明：除特殊说明外本站文章皆由CC-4.0协议发布，转载请注明出处。
安装docker-ce
Mongodb 分片集群
Nginx正向代理
Mongodb 数据备份&恢复
评论（没有评论）