unable to read data from ES having nested fields 505

unable to read data from ES having nested fields 505






 Solution Document


ICICI || unable to read data from ES having nested fields

 

Overview

General/Customer specific

General

Author

Tejaswi Botla

Reviewer

NA

Approver

 NA 

Release date

18/08/2022

Product Version

8.5r5


Audience: CSG/TechWarriors/PAC/Platform/Product teams


What’s the Issue? 

unable to read data from ES having nested feilds

Describe the Issue in Detail

Requirement :

We have a requirement such that client is storing all the application related logs using ElasticSearch and hence we have to read from client ES.

Problem Statement :

1. As we are having multiple nested feilds we are unable to fetch the data using query filter in input plugin.

2. Post fetching some data we are seeing that each document is getting read multiple times and all feilds are coming in array format.



Solution Summary

  1. We have used below query to fetch the data using nested feilds.

  

curl -XGET -H 'Content-Type:application/json' 'http://{eshost}:9200/indexname/_search?pretty'

-d '{"query":{"range":{"event.api_message_metadata.request_datetime":{"gte":"now-1h","lte":"now"}}}}'


  1. As kafka input and output are placed in same logstash instance, data is getting looped. Hence we maintained seperate logstash for input and output of kafka post which we are able to see proper document count.

Solution Steps in Detail

Note - 


  1. We have used below query to fetch the data using nested feilds.

  

curl -XGET -H 'Content-Type:application/json' 'http://{eshost}:9200/indexname/_search?pretty'

-d '{"query":{"range":{"event.api_message_metadata.request_datetime":{"gte":"now-1h","lte":"now"}}}}'


  1. As kafka input and output are placed in same logstash instance, data is getting looped. Hence we maintained seperate logstash for input and output of kafka post which we are able to see proper document count.


Corrective actions if any to avoid in future


More Help: .


References: 






    • Related Articles

    • Unable to read data from logbeat Agent 389

      VuNet Systems Private Limited Unable to read data from logbeat Agent 389 Overview General/Customer specific RBL Author Rachitha H V Reviewer Seema Approver Tejaswi B Release date Product Version 8.5r5 Audience: CSG/TechWarriors/PAC/Platform/Product ...

    • ES Has Become Readonly

      Ques - ES has become readonly and unable to index any data. How to resolve this? Ans - https://kb.objectrocket.com/elasticsearch/how-to-fix-the-forbidden-12-read-only-api-error-in-elasticsearch-282   ...

    • Unable to process data in shipper1 510

      VuNet Systems Private Limited Unable to process data in shipper1 510 Overview General/Customer specific RBL Author Rachitha H V Reviewer Seema Approver Deepak G Release date Product Version 8.5r5 Audience: CSG/TechWarriors/PAC/Platform/Product teams ...

    • How To Dynamically Move Nested Key Value To Root Level?

      My logstash data output is { “host” => “127.0.0.1”, “@timestamp” => 2021-06-25T21:00:00.000Z, “@version” => “1”, “fields” => { “data1” => “mydata1”, “data2” => “mydata2”, “data3” => “mydata3” } } I wanna move all fields within “fields” to root level. ...

    • Duplicate documents are reflecting for snmp data in ES - 536

                   Solution Document  Duplicate documents in Elasticsearch   Overview General/Customer specific General Author Rukmini Reviewer Aman,Siva Approver   Release date 03/08/2022 Product Version 9.1r3 ...