Unable to delete archive index when we are facing space issue in archive partition 399

Unable to delete archive index when we are facing space issue in archive partition 399




   Solution Document


  Unable to delete archive index when we are facing space issue in archive partition

       Overview

General/Customer specific

General

Author

Rukmini

Reviewer

     Nishanth

Approver

 Ravi 

Release date

13/09/2021

Product Version

8.5r11


           Audience:CSG/TechWarriors/PAC/Platform/Product teams


   What’s the Issue? 

     Unable to delete archive index when we are facing space issue in archive partition

   Describe the Issue in Detail

   * At TSL percentage usage of /data and /vunetindex or /archive partition more            than the 80%.
   * /vunetindex usage percentage  reached 89%, the issue here is because of large amount of syslog data, 20 days of archive index was deleted, as we know from customer that they don't need syslog data more than 15 days.
    * Retention setting for syslog data was, active data for 30 days and archive data for 31 days=(20 days last week deleted + 11 days we kept in archival) , we don't need these many days data for syslog so data retention setting for syslog data is changed to active data for 15 days and archive data for 30 days.


           *Once we applied this new data retention setting 15 days of active data now /data partition usage              percentage reduced but /archive partition usage percentage is 100% because 11 days of data we                kept previously and from last week to today 6 days of archive data added and as per new data                    retention archive data for 30 days so in archive we have total 11days + 6 days + 15 days=32 days                of data.

  We tried to delete archive index from UI but it is giving error  already one data retention running because of that it is not able to perform any other operation.



     Solution Summary

 


                     * While deleting archive index from UI we captured below logs, from that we can understand the                             issue is with already one retention running.
                     * vulog


*Then another way to resolve this is we must stop running snapshot.                                                  If Storage is full we can't delete archive   will get below error.

vunet@TSL-DC-VU-ANL-01:~$ curl -X DELETE "localhost:9200/_snapshot/vunet-repo/vunet-1-1-syslog-2022.01.01?pretty"

{

 "error" : {

   "root_cause" : [

     {

       "type" : "repository_exception",

       "reason" : "[vunet-repo] failed to delete snapshot [vunet-1-1-syslog-2022.01.01/g2sRrnq6SI2LGS_B-qQOFA]"

     }

   ],

   "type" : "repository_exception",

   "reason" : "[vunet-repo] failed to delete snapshot [vunet-1-1-syslog-2022.01.01/g2sRrnq6SI2LGS_B-qQOFA]",

   "caused_by" : {

     "type" : "i_o_exception",

     "reason" : "No space left on device"

   }

 },

 "status" : 500

}


So check if any file can be deleted there, there were two jpg file there which i have deleted.


      Corrective actions if any to avoid in future

  1. No


           More Help: 

              Please review /var/log/VuSmartmaps/vusoft_logging.<issue reported time >  for     more details.


           References: 




    • Related Articles

    • Can we restore an archived index and close it and then again archive it?

      Question: I want to understand that can we restore an archived index and then close it and archive it again after using the information required? For example: We are keeping the data open for last 2 days for an index and archiving the older data than ...

    • NFS Mount Point Disk Space Usage Breached The Threshold 357

          Solution Document           NFS Mount Point Disk Space Usage Breached The Threshold        Overview   General/Customer specific Sify TSL Author Rukmini Reviewer Seema Approver  Rukmini Release date 01/07/2022 Product Version                  ...

    • Unable to Mount Partition

      Getting error while mounting the partition Error – Already mounted or busy Answer: check if there is any process using that device (/dev/sda1). It is often a fsck process which runs automatically on system startup. You can check it quickly e.g. by ...

    • Issue with analyser logstash

      Question: In intellect analyser logstash is in excited state. Even after restarting the service it’s again going into excited state. There is no space issue also . Data has stopped coming from last night 11:00pm.  Following errors seen in logstash ...

    • [RBL] Issue In Telnet (Unable To Telnet Through Port 5050 From Source Server)

      In RBL there is an issue in telnet (unable to telnet through port 5050 from source server) Source :: ebanking servers Destination :: Vunet shipper Port :: 5050 RBL network team said everything thing is fine from source end but we are not getting ...