How to delete multiple open indices in Elasticsearch?

How to delete multiple open indices in Elasticsearch?

Question:
Sometimes in the development environment, we create a code change to store certain data in Elasticsearch. It creates a new index in Elasticsearch on a regular interval of time. For a large period of time, the number of open indices may increase. This causes a slow execution of applications in our system. To solve this issue we need to remove those unused indices. In general, we use a curl command to delete an index in Elasticsearch which can be viewed as :

curl -X DELETE ‘http://localhost:9200/<index_name>’

To delete few indices, this method can be manageable. But for a large number of open indices, it might take huge time to follow the above method. Do we have an option to achieve this task in an easy way?

Answer1:
Please use the following command to list all the open indices: curl -XGET localhost:9200/_cat/indices | grep open | awk ‘{print $3}’

Copy all these indexes and paste it in indices.txt and then use this command: cat indices.txt | xargs -I {} curl -XDELETE localhost:9200/{}

You can also use this command to list closed indexes: curl -XGET localhost:9200/_cat/indices | grep close | awk ‘{print $2}’

Answer2:
generally use a script to perform this task.

from elasticsearch import Elasticsearch

index_string = 'vunet-1-1-data-source-configuration'

def es_connection(es_server, es_port):
client = Elasticsearch(es_server + ':' + str(es_port))
client.ping()
return client

if __name__ == '__main__':
es_server = 'localhost'
es_port = 9200
client = es_connection(es_server, es_port)
index_list = []
for index in client.indices.get(index_string):
print("removing index " + index)
client.indices.delete(index=index, ignore=[400, 404])

- We have to replace the index_string with the index name or index pattern (vunet-1-1-data*).

- Activate the virtual environment and run the script.

Answer3:
The other method could be:

1. copy all the list of indices in a file indices.txt using following command: (you can change your grep condition in 2nd pipe below based on your requirements)
1.1 curl -XGET localhost:9200/_cat/indices | grep open | awk ‘{print $3}’ > indices.txt
1.2 curl -XGET localhost:9200/_cat/indices | grep close | awk ‘{print $2}’ >> indices.txt
2. cat indices.txt | xargs -I {} curl -XDELETE localhost:9200/{}
3. You can change localhost to Analyser IP if you are doing this in Shipper or some other server.

This will delete indexes one by one.

Can someone create a script out of this which take the ‘grep condition’ as argument?

  • The other method could be:

    1. copy all the list of indices in a file indices.txt using following command: (you can change your grep condition in 2nd pipe below based on your requirements)
    1.1 curl -XGET localhost:9200/_cat/indices | grep open | awk ‘{print $3}’ > indices.txt
    1.2 curl -XGET localhost:9200/_cat/indices | grep close | awk ‘{print $2}’ >> indices.txt
    2. cat indices.txt | xargs -I {} curl -XDELETE localhost:9200/{}
    3. You can change localhost to Analyser IP if you are doing this in Shipper or some other server.

    This will delete indexes one by one.

  • Answer4:

  • curl es_ipaddress:9200_/cat/indices?v | grep <index-name> | grep open | awk ‘{print $3}’ | xargs -I{} curl -XDELETE localhost:9200/{}

  • Answer5: If we want to delete an index (say for example RIB) for all occurrence for the month of September (for all dates in September). we can use the below curl command.

    curl -XGET localhost:9200/_cat/indices | grep RIB | grep 2020.09 | awk {‘print $3’} | xargs -i curl -XDELETE “http://localhost:9200/{}”

    GREP can be modified accordingly to suit ur needs.