![]() A detailed analysis of different hash functions is beyond the scope of this blog, but the hash function used in the fingerprint filter should be considered carefully as it will have an impact on ingest performance and on the number of hash collisions.Ī simple Logstash configuration to dedupe an existing index using the fingerprint filter is given below. For most practical cases, the probability of a hash collision is likely very low. an _id that is not generated by Elasticsearch) will have some impact on the write performance of your index operations.Īlso, it is worth noting that depending on the hash algorithm used, this approach may theoretically result in a non-zero number of hash collisions for the _id value, which could theoretically result in two non-identical documents being mapped to the same _id, thus causing one of these documents to be lost. This could be accomplished by changing the input section in the example below to accept documents from your real-time input source rather than pulling documents from an existing index.īe aware that using custom _id values (i.e. In the example below I have written a simple Logstash configuration that reads documents from an index on an Elasticsearch cluster, then uses the fingerprint filter to compute a unique _id value for each document based on a hash of the fields, and finally writes each document back to a new index on that same Elasticsearch cluster such that duplicate documents will be written to the same _id and therefore eliminated.Īdditionally, with minor modifications, the same Logstash filter could also be applied to future documents written into the newly created index in order to ensure that duplicates are removed in near real-time. This technique is described in this blog about handling duplicates with Logstash, and this section demonstrates a concrete example which applies this approach. Logstash may be used for detecting and removing duplicate documents from an Elasticsearch index. Using Logstash for deduplication of Elasticsearch documents Given this example document structure, for the purposes of this blog we arbitrarily assume that if multiple documents have the same values for the fields that they are duplicates of each other. This corresponds to a dataset that contains documents representing stock market trades. Therefore, in this blog post we cover how to detect and remove duplicate documents from Elasticsearch by (1) using Logstash, or (2) using custom code written in Python.įor the purposes of this blog post, we assume that the documents in the Elasticsearch cluster have the following structure. If this occurs then it may be necessary to find and remove such duplicates. ![]() However, if the data source accidentally sends the same document to Elasticsearch multiple times, and if such auto-generated _id values are used for each document that Elasticsearch inserts, then this same document will be stored multiple times in Elasticsearch with different _id values. Many systems that drive data into Elasticsearch will take advantage of Elasticsearch’s auto-generated id values for newly inserted documents.
0 Comments
Leave a Reply. |
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |