Add inserted_at column for samples_v4 and time_series_v4 tables

[srikanthccv]

I initially pushed against the idea to save some space (~50gb for 100 billion samples). But a lot of time spent time in analysing the false alerts could simply be explained if we know exactly when the sample is stored on the database vs when it is produced on client side.

At that, scale additional 50gb should be fine for the time it will save us.

[ankitnayan]

The intent is to help easy analysis or an attempt to solve them too?
I see 2 types of false alerts due to ingestion lag:

1. Incorrect No Data alerts - cannot be solved without ingestion guarantees
2. Incorrect alerts due to some timeseries needed by the alert are already ingested and some take time to get ingested. When evaluating an alert, we can check if all the timeseries that are needed by the alert available at the evaluation interval. IMO we discussed this once sometime back. Do you see instances of this type of false alerts?

[ankitnayan]

IMO we should be fine with the storage costs

[srikanthccv]

The intent is to help easy analysis or an attempt to solve them too?

Both, but the solution part scope is independent and requires more work.

Incorrect alerts due to some timeseries needed by the alert are already ingested and some take time to get ingested. When evaluating an alert, we can check if all the timeseries that are needed by the alert available at the evaluation interval. IMO we discussed this once sometime back. Do you see instances of this type of false alerts?

This kind of false alerts have become the primary source of false alerts.

[ankitnayan]

got it