Using eventTime

You can expose a timestamp when an event is persisted to S3.

When you define a Dataset with Incremental Data Access, you can expose the timestamp indicating when the events are persisted to S3 (not just the import timestamp). For this purpose, you can use the eventTime field in your Dataset.

Exposing Event Timestamp

The following example illustrates the use of Incremental Data Access with the eventTime field:

val writeDF = entData.select("Id", "eventTime", "createdTime", "updatedTime")

Example of the timestamps returned from this Dataset:

{
  "Id": "wS1qfhX",
  "eventTime": 1515338020000,
  "createdTime": "2018-01-07T15:12:37.887Z",
  "updatedTime": "2018-01-07T15:12:37.887Z"
}

In this example, the timestamp is exposed as the eventTime field where the eventTime value is greater than updateTime and less than currentTime.

Note: The eventTime timestamp is returned in milliseconds.