ELK 高可用

[moooofly]

• Small, Medium, or Large - Scaling Elasticsearch and Evolving the Elastic Stack to Fit
• How to Deploy the ELK Stack in Production
• Adventures In High Availability Logging — Elasticsearch, Logstash, and Kibana (ELK) on Kubernetes

[moooofly]

Small, Medium, or Large - Scaling Elasticsearch and Evolving the Elastic Stack to Fit

We will demonstrate how to expand from a minimalist pattern to multi-data centers logging platform.

本文将展示从 最小模式部署 到 多数据中心部署；

The Minimalist

主要需求：exploring and visualizing logs, events, and metrics.

• Filebeat is simple tools for a simple task.
• we can make this better with Elasticsearch ingest node processors. We can parse log events and enrich the documents with custom fields.
• Include Elastic’s X-Pack security in the stack, we have encryption and secured transmission between Beats and the Elasticsearch cluster.

Doing More

• Why stop at logs and metrics? What about other inputs, like Kafka, AWS Kinesis, Google Pub/Sub, TCP, UDP, JDBC, S3, SQS? Add Logstash to the stack.
• Besides being able to collect data from a variety of inputs, if we direct the Beats output through Logstash before reaching Elasticsearch, we can now do advanced transformations like aggregate or enrich data using lookup filters such as translate, dns, JDBC streaming, or Elasticsearch, which provide the ability to gather data from multiple external sources into a single document.
• Furthermore, Logstash can send data to S3 or HDFS for archiving in addition to indexing into Elasticsearch.

Safety First

There are a number of built-in resiliency mechanisms that provide some safety net.

• Filebeat and Winlogbeat use a back-pressure sensitive protocol when sending data to Elasticsearch or Logstash so they won’t overwhelm the pipeline. Also, both guarantees at least once delivery. -- 反压机制
• Logstash has a configurable persistent queue that provides data durability in events of abnormal shutdown of Logstash or its host, ensuring at least once delivery. Also, it can buffer bursts of events, eliminating the need for an external message broker, thereby avoiding adding an extra component to the stack. -- 持久化 queue
• Enable the Logstash dead letter queue to persist events or messages that failed to process due to mapping errors or alike to disk along with meta information like time, source and reasons for the failures. A Logstash pipeline can use the dead_letter_queue input plugin to read from the queue and reprocess the events at a later time. -- 死信 queue

That said, this pattern does have limitations:

• Logstash input plugins that do not use a request-response protocol cannot be protected from data loss. For example: TCP, UDP, and many other Logstash input plugins that do not acknowledge receipt to the sender. -- logstash 的 input 插件可能导致数据丢失
• The Logstash persistent queue does not provide queue level replication. Therefore, it cannot achieve high availability or protect data from permanent failure like disk corruption or machine loss without additional hardware level protections. -- logstash 的持久化 queue 不提供 queue 级别的复制

Adding a Message Queue to the Stack

To address the Logstash persistent queue limitations, we will need to introduce a queuing mechanism like Kafka to the stack. The added benefit of having a message broker is to decouple the indexing module from the collection module. The collection tier or the collectors, are the Beats and Logstash instances that sit between the inputs and the message queue. The indexing tier or the indexers are the Logstash instances between the message queue and Elasticsearch. We can add or subtract from each tier independently. This will be very valuable as more inputs and use cases feed into the stack.

解决 Logstash 持久化 queue 局限的办法是引入 Kafka 作为 broker ；

Centralized Logstash

One pattern is to let each user group control their own Elasticsearch document schema and the configuration of their Beats and Logstash instances to transform log events before arriving at the message queue. This approach achieves two things:

• Configuration changes for each group are independent from others.
• A single uniform indexer configuration greatly simplifies its management.

Multiple Logstash Instances

Multi Data Centers with a Queue at Each DC

[moooofly]

How to Deploy the ELK Stack in Production

Overall, ELK implementation needs to:

• Save and index all of the log files that it receives (sounds obvious, right?)
• Operate when the production system is overloaded or even failing (because that’s when most issues occur, isn’t it?)
• Keep the log data protected from unauthorized access
• Have maintainable approaches to data retention policies, upgrades, and more

---

Use Redis (Use Kafka)

• Elasticsearch is the engine at the heart of ELK. It is very susceptible to load, which means you need to be extremely careful when indexing and increasing your amount of documents. When Elasticsearch is busy, Logstash works slower than normal — which is where Redis comes in, accumulating more documents that can then be pushed to Elasticsearch. This is critical not to lose log events. -- 这里将 Redis 换成 Kafka 也成立

Keep Track of Logstash/Elasticsearch Exceptions

• Logstash may fail when it tries to index logs to Elasticsearch that can’t fit into the automatically-generated mapping.

一个例子

For example, let’s say you have a log entry that looks like this:

Timestamp=time, type=my_app, error=3,….

And later, the system generates a similar log:

Timestamp=time, type=my_app, error=”Error”,….

In the first case, a number is used. In the second case, a string is used.

As a result, Elasticsearch will NOT index the document — it will just return a failure message and the log will be dropped.

解决办法：

To make sure that such logs are still indexed, you need to:

• Use the “Type” field for each type of log. “Type” is somewhat similar to an SQL table. For a given “type,” a field can have one schema (integer OR string). -- 要求使用一个 Type 域来明确指明
• Work with developers to make sure they’re keeping log formats consistent. If a log schema change is required, just change the “Type” field according to the type of log. -- 要求 developers 确保 log 格式不要发生变化
• Ensure that Logstash is consistently fed with information and monitors Elasticsearch exceptions to ensure that logs are not shipped in the wrong formats. Using mapping that is fixed and less dynamic is probably the only solid solution here (that doesn’t require you to start coding). -- 通过某种手段确保 logstash 接收和输出一致的格式，并对 es 进行出错监控

At Logz.io, we solve this problem by building a pipeline to handle mapping exceptions that eventually index these documents in manners that don’t collide with existing mapping. -- 自己构建一个 pipeline 用于确保格式没有问题

Keep Up with Growth and Bursts

• In general, log management solutions consume large amounts of CPU, memory, and storage.
• Log systems are bursty by nature, and sporadic bursts are typical.
• If a file is purged from your database, the frequency of logs that you receive may range from 100 to 200 to 100,000 logs per second. As a result, you need to allocate up to 10 times more capacity than normal.
• When there is a real production issue, many systems generally report failures or disconnections, which cause them to generate many more logs. This is actually when log management systems are needed more than ever.

ELK Elasticity

• we recommend having a cluster of Elasticsearch nodes that run in different availability zones, or in different segments of a datacenter, to ensure high availability.

Logstash

• You should also separate Logstash and Elasticsearch by using different machines for them. This is critical because they both run as JVMs and consume large amounts of memory, which makes them unable to run on the same machine effectively. I also recommend allocating half of the memory on each machine to Elasticsearch or Logstash. - Logstash 和 Elasticsearch 不可部署在同一台机器；建议只分配物理机器一半的内存给 Logstash 或 Elasticsearch 使用；

Cluster Elasticsearch

• Elasticsearch is composed of a number of different components, two of which are the most important: the master nodes and the data nodes.
• The master nodes are responsible for cluster management while the data nodes, as the name suggests, are in charge of the data.
• We recommend clustering Elasticsearch with at least three master nodes because of the common occurrence of split brain, which is essentially a dispute between two nodes regarding which one is actually the master. As a result, using three master nodes prevents split brain from happening. - master 节点至少 3 个
• As far as the data nodes go, we recommend having at least two data nodes so that your data is replicated at least once. - node 节点至少 2 个
• This results in a minimum of five nodes: the three master nodes can be small machines, and the two data nodes need to be scaled on solid machines with very fast storage and a large capacity for memory. - data 节点的机器需要 fast storage 以及较大的 memory ；
• A quick note about AWS: Since multicast doesn’t work with AWS, Elasticsearch’s cluster discovery protocol will not work. You need to install the Elasticsearch AWS plugin or configure unicast cluster discovery.

Run in Different AZs (But Not in Different Regions)

• We recommend having your Elasticsearch nodes run in different availability zones or in different segments of a datacenter to ensure high availability. This can be done through the Elasticsearch setting that allows you to configure every log to be replicated between different AZs. -- 建议跨 AZ 或同一个数据中心不同 segments 部署
• While it may seem odd, we don’t recommend running Elasticsearch clusters across AWS regions. Due to the increased latency that is experienced when synchronizing data between regions, it doesn’t work well… — at least according to the tests we have done. -- 不建议跨 AWS regions 部署 es

Access Control

• Due to the fact that logs may contain sensitive data, it is crucial to protect who can see what. How can you limit access to specific dashboards, visualizations, or data inside your log analytics platform? As far as we know, there is no simple way to do this in the ELK Stack. One option is to use the Nginx reverse proxy to access your Kibana dashboard, which entails a simple nginx configuration that requires those who want to access the dashboard to have a username and password. This quickly blocks access to your Kibana console.
• Last but not least, be careful when exposing Elasticsearch because it is very susceptible to attacks. We recommend limiting all access to Elasticsearch inbound port 9200 from the internet. -- Elasticsearch 是易受攻击的，建议只开发 9200 端口（最好前面再挡点别的什么）；

Maintainability

Log Data Consistency

• Logstash has a component that parses logs and tokenizes them in accordance with a set of rules. Therefore, if you have an access log from nginx, you want the ability to view each field and have visualizations and dashboards built based on specific fields. You need to apply the relevant parsing abilities to Logstash — which has proven to be quite a challenge, particularly when it comes to building groks, debugging them, and actually parsing logs to have the relevant fields for Elasticsearch and Kibana. -- 在使用 Logstash 的各种组件进行日志解析和 token 分割时很复杂
• At the end of the day, it is very easy to make mistakes using Logstash, which is why you should carefully test and maintain all of your log configurations by means of version control. -- 基于版本控制管理 logstash 配置

Data Retention

• Another aspect of maintainability comes into play with excess indices. Depending on how long you want to retain data, you need to have a process set up that will automatically delete old indices — otherwise, you will be left with too much data and your Elasticsearch will crash, resulting in data loss. To prevent this from happening, you can use Elasticsearch Curator to delete indices. We recommend having a cron job that automatically spawns Curator with the relevant parameters to delete any old indices, ensuring you don’t end up holding too much data. It is commonly required to save logs to S3 in a bucket for compliance, so you want to be sure to have a copy of the logs in their original format. Copying should be done before logs are parsed by Logstash.
• 需要一个进程自动删除老旧 indices ，可以使用 Curator ；
• 建议使用 cron job 自动的、定时启动 Curator 进行索引清理；
• 一般还会使用 S3 保存一份日志，用于持久化和离线分析；

Upgrades

• Performing Elasticsearch upgrades can be an endeavor on its own. First and foremost, you don’t want to lose any data in the lengthy process. New versions are released every few weeks, so pay careful attention to the upgrade process if you choose to go that route. Make sure you have at least one replication on Elasticsearch and perform warm upgrades (sync data and upgrade one node at a time). -- 在升级的过程中，你需要确保至少保留一个 Elasticsearch 副本，然后再执行 warm upgrades ；同步数据，一次升级一个 node ；
• Logstash upgrades are generally easier, but pay close attention to the compatibility between Logstash and Elasticsearch. -- 关注兼容性问题
• Kibana upgrades are hard, but possible. Kibana 3 does not have an upgrade path to Kibana 4. We’ve just met a user who has 600 different Kibana 3 dashboards and now has to rewrite them to use Kibana 4. I guess that’s one of the drawbacks of free open-source solutions.

[moooofly]

Adventures In High Availability Logging — Elasticsearch, Logstash, and Kibana (ELK) on Kubernetes