Update ingestion workflow to use new ingestion history table#623
Update ingestion workflow to use new ingestion history table#623vish-cs wants to merge 1 commit into
Conversation
Up to standards ✅🟢 Issues
|
| Metric | Results |
|---|---|
| Complexity | 0 |
NEW Get contextual insights on your PRs based on Codacy's metrics, along with PR and Jira context, without leaving GitHub. Enable AI reviewer
TIP This summary will be updated as you push new changes.
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request introduces tracking for a new metric, TimeSeriesCount (or ts_count), across the ingestion helper application, Spanner schema, and client tests. It also updates the Spanner ingestion workflow to record ingestion history status updates (PENDING, RUNNING, SUCCESS, RETRY) at various stages. Feedback on these changes highlights a potential database migration issue where the new column may not be applied to existing databases, and a workflow bug where an undefined import_info variable in the error-handling block could cause a crash if the initial fetch fails.
| auth: | ||
| type: OIDC | ||
| body: | ||
| workflowId: '${sys.get_env("GOOGLE_CLOUD_WORKFLOW_EXECUTION_ID")}' |
There was a problem hiding this comment.
[optional] Prefer saving as workflow variable and using everywhee '${sys.get_env("GOOGLE_CLOUD_WORKFLOW_EXECUTION_ID")}'
| workflowId: '${sys.get_env("GOOGLE_CLOUD_WORKFLOW_EXECUTION_ID")}' | ||
| status: 'PENDING' | ||
| stage: 'dataflow' | ||
| importList: '${import_info.body}' |
There was a problem hiding this comment.
should we use ${import_list} instead of '${import_info.body}' ? will import_info.body contain details of all import_list - even if its a new import?
| workflowId: '${sys.get_env("GOOGLE_CLOUD_WORKFLOW_EXECUTION_ID")}' | ||
| jobId: ${default(dataflow_job_id, default(map.get(e, "job_id"), "N/A"))} | ||
| importList: '${default(import_info, {"body": []}).body}' | ||
| status: 'RETRY' |
There was a problem hiding this comment.
there is no failed status?
2 participants
Uh oh!
There was an error while loading. Please reload this page.