Elasticsearch常用运维API操作

892次阅读
没有评论

共计 7762 个字符，预计需要花费 20 分钟才能阅读完成。

Elasticsearch常用运维API操作

此篇记录下博主常用的操作命令，以下命令都是在kibana的dev tools控制台中运行

_cat API系列

此api提供了一系列查询es状态的接口

GET /_cat/

# 会得到以下api接口输出
=^.^=
/_cat/allocation
/_cat/shards
/_cat/shards/{index}
/_cat/master
/_cat/nodes
/_cat/tasks
/_cat/indices
/_cat/indices/{index}
/_cat/segments
/_cat/segments/{index}
/_cat/count
/_cat/count/{index}
/_cat/recovery
/_cat/recovery/{index}
/_cat/health
/_cat/pending_tasks
/_cat/aliases
/_cat/aliases/{alias}
/_cat/thread_pool
/_cat/thread_pool/{thread_pools}
/_cat/plugins
/_cat/fielddata
/_cat/fielddata/{fields}
/_cat/nodeattrs
/_cat/repositories
/_cat/snapshots/{repository}
/_cat/templates
/_cat/ml/anomaly_detectors
/_cat/ml/anomaly_detectors/{job_id}
/_cat/ml/trained_models
/_cat/ml/trained_models/{model_id}
/_cat/ml/datafeeds
/_cat/ml/datafeeds/{datafeed_id}
/_cat/ml/data_frame/analytics
/_cat/ml/data_frame/analytics/{id}
/_cat/transforms
/_cat/transforms/{transform_id}

查看节点状态

GET /_cat/nodes?v

# 输出
ip             heap.percent ram.percent cpu load_1m load_5m load_15m node.role master name
192.168.44.142           66          85   1    0.02    0.03     0.06 dilmrt    *      node-1

查看健康状态

GET /_cat/health?v

# 输出
epoch      timestamp cluster       status node.total node.data shards pri relo init unassign pending_tasks max_task_wait_time active_shards_percent
1670917717 07:48:37  elasticsearch yellow          1         1     19  19    0    0        1             0                  -                 95.0%

获取索引状态

GET /_cat/indices?v

# 输出
health status index                           uuid                   pri rep docs.count docs.deleted store.size pri.store.size
green  open   .monitoring-es-7-2021.02.11     aT1GiQiZThCdBDuVy1w1AQ   1   0        903          570    996.9kb        996.9kb
green  open   .monitoring-es-7-2021.02.12     xlmlmtaJQJuln0NziYnmSQ   1   0      71730        40983     37.6mb         37.6mb
green  open   .monitoring-es-7-2021.02.13     Ccg1vqIVQxCFYEMMJYolqQ   1   0      11957         9679      8.3mb          8.3mb
green  open   .apm-agent-configuration        2hYg_V5xRySvGyuCUPTibw   1   0          0            0       208b           208b
green  open   project_test_v2                 7jGRoDdORaiLv1jgCQ8mig   1   0          5            0      4.7kb          4.7kb
green  open   .kibana_1                       fJuJq3ZZSJiPeeJYbdWrlg   1   0        285            2      1.3mb          1.3mb
green  open   kibana_sample_data_flights      B01_e446RzmbOWbQ2ep4XQ   1   0      13059            0      6.3mb          6.3mb
green  open   .tasks                          6eaM0vs1TNiDc-ItpnRzUA   1   0         12            0     20.1kb         20.1kb
green  open   .security-7                     7pTnk2QWTkWoDkTps2xfWQ   1   0         42            0    107.3kb        107.3kb
yellow open   testindex                       JMJoQcmjToKYovs8Y1rzvA   1   1          5            0        7kb            7kb
green  open   .apm-custom-link                NXd5wQmCS3eJWLDtzkaoiQ   1   0          0            0       208b           208b
green  open   .monitoring-kibana-7-2021.02.12 C-iZhn3ZT7-uwciTEe9ktQ   1   0       3918            0    948.1kb        948.1kb
green  open   .kibana_task_manager_1          FK44fN-tSjWG0yudyGzlCQ   1   0          5            1     58.8kb         58.8kb
green  open   .monitoring-kibana-7-2021.02.13 PGVq4B-dRS2EdpcvYKq5KQ   1   0        533            0    164.1kb        164.1kb
green  open   kibana_sample_data_ecommerce    3MEYodTnTEixMcQCZ9fGpw   1   0       4675            0      4.3mb          4.3mb
green  open   .monitoring-kibana-7-2021.02.11 OZNHeIm0QJinzjrkqJjXcA   1   0         58            0    135.5kb        135.5kb
green  open   project_test                    P5xA7xPlQ6-WyxfRz-F6wA   1   0          5            0     16.3kb         16.3kb
green  open   kibana_sample_data_logs         ozxMqYgNSFCg88DzKaaHaQ   1   0      14074            0     11.2mb         11.2mb
green  open   test_logs                       onVkDs9aSD-EIAFXmVNZDw   1   0      14074            0        9mb            9mb

获取插件信息

GET /_cat/plugins?v

# 输出
name   component   version
node-1 analysis-ik 7.7.0

获取别名

GET /_cat/aliases?v

# 输出
alias                index                  filter routing.index routing.search is_write_index
.kibana_task_manager .kibana_task_manager_1 -      -             -              -
project_test_alias   project_test_v2        -      -             -              -
.security            .security-7            -      -             -              -
.kibana              .kibana_1              -      -             -              -

获取索引分片

GET /_cat/shards?v

# 输出
index                           shard prirep state       docs   store ip             node
testindex                       0     p      STARTED        5     7kb 192.168.44.142 node-1
testindex                       0     r      UNASSIGNED                              
test_logs                       0     p      STARTED    14074     9mb 192.168.44.142 node-1
project_test_v2                 0     p      STARTED        5   4.7kb 192.168.44.142 node-1
.monitoring-kibana-7-2021.02.12 0     p      STARTED     3918 948.1kb 192.168.44.142 node-1
.monitoring-kibana-7-2021.02.11 0     p      STARTED       58 135.5kb 192.168.44.142 node-1
.kibana_task_manager_1          0     p      STARTED        5  58.8kb 192.168.44.142 node-1
.monitoring-es-7-2021.02.11     0     p      STARTED      903 996.9kb 192.168.44.142 node-1
.monitoring-kibana-7-2021.02.13 0     p      STARTED      533 164.1kb 192.168.44.142 node-1
.security-7                     0     p      STARTED       42 107.3kb 192.168.44.142 node-1
kibana_sample_data_flights      0     p      STARTED    13059   6.3mb 192.168.44.142 node-1
kibana_sample_data_ecommerce    0     p      STARTED     4675   4.3mb 192.168.44.142 node-1
.tasks                          0     p      STARTED       12  20.1kb 192.168.44.142 node-1
.monitoring-es-7-2021.02.13     0     p      STARTED    11957   7.8mb 192.168.44.142 node-1
.monitoring-es-7-2021.02.12     0     p      STARTED    71730  37.6mb 192.168.44.142 node-1
kibana_sample_data_logs         0     p      STARTED    14074  11.2mb 192.168.44.142 node-1
.apm-custom-link                0     p      STARTED        0    208b 192.168.44.142 node-1
.kibana_1                       0     p      STARTED      276   1.1mb 192.168.44.142 node-1
.apm-agent-configuration        0     p      STARTED        0    208b 192.168.44.142 node-1
project_test                    0     p      STARTED        5  16.3kb 192.168.44.142 node-1

查看task

GET /_cat/tasks?v

# 输出
action                         task_id                      parent_task_id               type      start_time    timestamp running_time ip             node
cluster:monitor/tasks/lists    Vgd49fa4Qrir7oHXUGW_zw:68900 -                            transport 1670917454160 07:44:14  212.8micros  192.168.44.142 node-1
cluster:monitor/tasks/lists[n] Vgd49fa4Qrir7oHXUGW_zw:68901 Vgd49fa4Qrir7oHXUGW_zw:68900 direct    1670917454160 07:44:14  43micros     192.168.44.142 node-1

获取线程池

GET /_cat/thread_pool?v

# 输出
node_name name                active queue rejected
node-1    analyze                  0     0        0
node-1    ccr                      0     0        0
node-1    fetch_shard_started      0     0        0
node-1    fetch_shard_store        0     0        0
node-1    flush                    0     0        0
node-1    force_merge              0     0        0
node-1    generic                  0     0        0
node-1    get                      0     0        0
node-1    listener                 0     0        0
node-1    management               1     0        0
node-1    ml_datafeed              0     0        0
node-1    ml_job_comms             0     0        0
node-1    ml_utility               0     0        0
node-1    refresh                  0     0        0
node-1    rollup_indexing          0     0        0
node-1    search                   0     0        0
node-1    search_throttled         0     0        0
node-1    security-token-key       0     0        0
node-1    snapshot                 0     0        0
node-1    transform_indexing       0     0        0
node-1    warmer                   0     0        0
node-1    watcher                  0     0        0
node-1    write                    0     0        0

_cluster集群API

cluster集群健康

GET /_cluster/health

# 输出
{
  "cluster_name" : "my-application",
  "status" : "green",
  "timed_out" : false,
  "number_of_nodes" : 3,
  "number_of_data_nodes" : 3,
  "active_primary_shards" : 19,
  "active_shards" : 36,
  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 0,
  "delayed_unassigned_shards" : 0,
  "number_of_pending_tasks" : 0,
  "number_of_in_flight_fetch" : 0,
  "task_max_waiting_in_queue_millis" : 0,
  "active_shards_percent_as_number" : 100.0
}

其他_cluster常用命令

# 获取集群索引级别信息
GET /_cluster/health?level=indices 

# 获取集群分片级别信息
GET /_cluster/health?level=shards  

# 获取集群状态信息
GET /_cluster/stats?pretty=true
GET /_cluster/state?pretty=true

# 获取堆积任务信息
GET /_cluster/pending_tasks?pretty=true

task有关的API

# 查看当前任务
GET /_tasks?detailed=true

# 查看所有任务
GET /_tasks/

# 查看正在执行的任务
GET /_nodes/hot_threads

# 查看待处理的任务
GET /_cat/pending_tasks

其他API命令

开启慢日志

PUT _settings
{
  "index.search.slowlog.threshold.query.warn": "10s",
  "index.search.slowlog.threshold.query.info": "5s",
  "index.search.slowlog.threshold.query.debug": "2s",
  "index.search.slowlog.threshold.query.trace": "500ms",
  "index.search.slowlog.threshold.fetch.warn": "1s",
  "index.search.slowlog.threshold.fetch.info": "800ms",
  "index.search.slowlog.threshold.fetch.debug": "500ms",
  "index.search.slowlog.threshold.fetch.trace": "200ms",
  "index.indexing.slowlog.threshold.index.warn": "10s",
  "index.indexing.slowlog.threshold.index.info": "5s",
  "index.indexing.slowlog.threshold.index.debug": "2s",
  "index.indexing.slowlog.threshold.index.trace": "500ms",
  "index.indexing.slowlog.level": "info",
  "index.indexing.slowlog.source": "1000"
}

加速节点故障恢复

若存在节点故障，会存在很多unassigned的分片，此时将故障节点重启恢复后会进行分片恢复，如果想要加快分片恢复速度，可以调高恢复进程的并发数，结合实际情况调整，不要调太高导致集群拒绝写入

  • cluster.routing.allocation.node_concurrent_incoming_recoveries ：节点上最大接受的分片恢复并发数。一般指分片从其它节点恢复至本节点
  • cluster.routing.allocation.node_concurrent_outgoing_recoveries ：节点上最大发送的分片恢复并发数。一般指分片从本节点恢复至其它节点
  • cluster.routing.allocation.node_concurrent_recoveries ：该参数同时设置上述接受发送分片恢复并发数为相同的值
  • cluster.routing.allocation.node_initial_primaries_recoveries：初始化数据恢复时，并发恢复线程的个数
# 使用transient，则是零时设置，重启节点后配置失效
PUT _cluster/settings
{
  "transient": {
    "cluster.routing.allocation.node_concurrent_incoming_recoveries": 10,
    "cluster.routing.allocation.node_initial_primaries_recoveries": 10,
    "cluster.routing.allocation.node_concurrent_recoveries": 10
  }
}

大数据量索引重建

对大数据量的索引重建时，则在创建索引时，关闭刷新时间间隔，将refresh_interval设置为-1即可

PUT project_test_v2
{
  "settings": {
     "refresh_interval": "-1",
     "number_of_shards": 5, 
     "number_of_replicas": 2
  },
  "mappings": {
    "dynamic": false, 
    "properties": {
    "content": {
        "type": "text",
        "analyzer": "ik_max_word",
        "search_analyzer": "ik_smart"
      },
    "title": {
        "type": "text",
        "analyzer": "ik_max_word",
        "search_analyzer": "ik_smart"
      }
    }
  }
}

对于打数据量的索引重建，若需要控制重建索引的速度，则可以设置size来控制，同时设置wait_for_completion为false，这样执行该api立即返回一个任务id，而不是终端一直夯着等待

POST _reindex?wait_for_completion=false
{
  "source": {
    "index": "old_index",
    "size": 5000
  },
  "dest": {
    "index": "new_index"
  }
}

此时可以通过以下API获取任务细节

GET /_tasks?detailed=true&actions=*reindex

若要取消任务则可以执行

GET /_tasks/node_id:task_id/_cancel

正文完
elasticsearch
发表至： elasticsearch 运维
2021-03-07
 1
xadocker
版权声明：本站原创文章，由 xadocker 2021-03-07发表，共计7762字。
转载说明：除特殊说明外本站文章皆由CC-4.0协议发布，转载请注明出处。
用dnsmasq做私有dns解析
Vue中的nginx配置
Openresty中使用ngx.log()来调试
Centos根分区扩容
评论（没有评论）