Solution Document

 Duplicate documents in Elasticsearch

  Overview

General/Customer specific	General
Author	Rukmini
Reviewer	Aman,Siva
Approver
Release date	03/08/2022
Product Version	9.1r3

Audience: CSG/TechWarriors/PAC/Platform/Product teams

What’s the Issue?  In BMCSL setup we saw duplicate documents are indexing at the same time in network-health and network-element-health index.

Describe the Issue in Detail We observed wrong input/output bandwidth utilization, while analyzing issue, we saw duplicates documents are indexing at the same time in network-health and network-element-health index it will impact on disk space and calculation of bandwidth utilization (because we are using arithmetic plugin in bandwidth calculation, Arithmetic plugin gives period as the output between two documents, if two documents indexed at same time obviously, we will get wrong bandwidth). After analyzing input(snmp-data) and output topics (network-element-health, network-health) of snmp-app we saw one input message and 2 corresponding output messages with only difference of one field (@correlate_metadata).  We discussed with Rahul (bigdata team) and he suggested to use the snmp-app with old image. When we checked with old image, we didn’t see any issue. We need to test the correlate plugin in latest image (product team) to avoid issue temporarily we ran snmp-app using old vunetstream image. docker image ls We should make copy file create-stream-v1.sh as create-stream-v1-test.sh And change image with old image in the file create-stream-v1-test.sh at line number 131 we can see image path .( ghcr.io/vunetsystems/vustreams:latest) replace latest image path with old image(ghcr.io/vunetsystems/vunetstreams) and save file. Stop snmp-app using docker service rm snmp-app Run snmp-app with new image (./create-stream-v1-test.sh /deployment/config_snmp_app.conf) Once we ran snmp-app using old vunetstream image duplicates document issue resolved but we should fix this on latest image as well because some other latest required configuration may not available on old one.

Solution Summary After fixing the issue in latest image we need to load the latest image into environment, issue was a missing filter in the program which led to the data duplication.

Solution Steps in Detail To load the latest image into environment should follow the below steps. Copy latest vustream .tar image file into server using winscp.(/data/vuDocker) cd /data/vuDocker/ ll docker image ls docker load -i vusteams-latest.tar.gz (to load image in main server) To load image into all the servers mv vusteams-latest.tar.gz vustream02Aug.tar mv vustream02Aug.tar /data/vuDocker/docker-script/docker-images/ cd /data/vuDocker/ ls -ltr vim test.sh scp /data/vuDocker/docker-script/docker-images/vustream02Aug.tar vunet@$ip:/data/vuDocker/docker-script/docker-images ssh vunet@$ip "docker load -i /data/vuDocker/docker-script/docker-images/vustream02Aug.tar" Save the file ls -ltrh /data/vuDocker/docker-script/docker-images/vustream02Aug.tar bash test.sh cd ../vusmartmaps/bin/kafka-streams/ docker service ls docker service rm snmp-app (we are stopped running snmp-app ,because due to duplicate documents issue snmp-app started using older image, in other situation we need restart all applications ). ./create-stream-v1.sh deployment-configs/config_snmp_app.sh docker service ls docker service logs -f snmp-app by refering below image we can understand issue got resolved there is no duplicate documents at time (image 2) and also there is no interval in input topic(image 1) and same as es indexing(image2) , latest image working properly

Corrective actions if any to avoid in future No

More Help: 

References: