Timestamp field in Elasticsearch

Consider the following scenario, that you want to know when a specific document is added to Elasticsearch. In initial versions of Elasticsearch, each document contained a meta field named @timestamp that contained information about the date the document was indexed.

However, it is currently impossible to tell when a document was added to Elasticsearch. There are two methods for achieving this.

• From the Application level, you can add time to each document.

• To add time to each document, use the ingest pipeline. To do that, follow the instructions below:

  Steps to add timestamp field in each document using Ingest pipeline

• Ingest pipeline is created as depicted below. There are 3 processors in the ingest pipeline :

1. The timestamp field's value for "ingest.timestamp" is set using the set processor command.
   NOTE: ingest.timestamp stores the time at which the document is added in Elasticsearch. The time is in this format 2022-11-12T07:48:45.760165787Z

2. The timestamp can be changed to Epoch time by a script processor.

3. The timestamp field can be removed by the delete processor. Since, according to the script, now @timestamp stores the epoch time.

PUT /_ingest/pipeline/ts-pipeline
{
   "description": "Add timestamp field to all documents",
   "processors": [
       {
           "set": {
               "field": "timestamp",
               "value": "{{_ingest.timestamp}}"
           }
       },
       {
           "script": {
               "id":"calculate-timestamp"
           }
       },
       {
           "remove": {
               "field": "timestamp"
           }
       }
   ]
}

• The following script is used by the script processor to change the timestamp from date format to epoch format.

•   DateTimeFormatter dtf = DateTimeFormatter.ofPattern(\"yyyy-MM-dd'T'HH:mm:ss.SSSSSSSSS'Z'\"); LocalDateTime ldt = LocalDateTime.parse(ctx['timestamp'], dtf); ZonedDateTime zdt = ldt.atZone(ZoneId.of('Z')); ctx['@timestamp']=zdt.toInstant().toEpochMilli();
    When documents are indexed, @timestamp the field is added to each document (in epoch format)
  

  Working Example

• Using the previously created ingest pipeline, add a document to the index.

•   POST timestamp-test/_doc/1?pipeline=ts-pipeline
    {
      "city":"indore"
    }
  

  Retrieve the added document using the Get API

•   GET /timestamp-test/_doc/1
    {
        "_index": "timestamp-test",
        "_id": "1",
        "_version": 1,
        "_seq_no": 0,
        "_primary_term": 1,
        "found": true,
        "_source": {
            "city": "indore",
            "timestamp": "2022-11-12T07:48:45.760165787Z"
        }
    }
  

  You'll notice that Elasticsearch has added a new field called @timestamp that contains the time the document was added.