Introduce additional instrumentation around Git operations

[jamietanna]

Tell us more.

As part of #38346, we're introducing a new option to perform shallow cloning on repositories, as an alternative to blobless cloning.

As there are performance trade-offs of both options, it would be great to provide self-hosters the ability to understand which option is better for them, by providing more meaningful information in Renovate's debug logs (as a ...Stats type under lib/util/stats.ts).

This would provide a clearer understanding about which operations do actually take the most time, and help clarify for users testing out the differences in shallow clones and blobless clones which one makes sense for their workload.

This would look something like:

// NOTE that I've taken a look through call sites to SimpleGit, and determined that these are the most important subcommands to measure, as well as lower priority, ungrouped subcommands
export type GitOperationType =
  /**
   * The `git clone` sub-command.
   */
  'clone' |
  /**
   * The `git reset` sub-command.
   */
  'reset' |
  /**
   * The `git checkout` sub-command.
   */
  'checkout' |
  /**
   * The `git fetch` sub-command.
   */
  'fetch' |
  /**
   * The `git push` sub-command.
   */
  'push' |
  /**
   * The `git clean` sub-command.
   */
  'clean' |
  /**
   * The `git merge` sub-command.
   */
  'merge' |
  /**
   * The `git submodule` sub-command.
   */
  'submodule' |
  /**
   * The `git commit` sub-command.
   */
  'commit' |
  /**
   * Any "plumbing" **??** // TODO
   *
   * - `git update-index`
   *
   * See also: https://git-scm.com/book/en/v2/Git-Internals-Plumbing-and-Porcelain
   */
  'plumbing' |
  /**
   * Any other operations  i.e.
   *
   * - `git log`
   * - `git remote`
   * - `git rev-parse`
   * - `git status`
   * - `git config`
   * - `git branch`
   *
   * See also: https://git-scm.com/book/en/v2/Git-Internals-Plumbing-and-Porcelain
   */
  'other';


// the report would then output something in this shape, in the DEBUG logs:
{
  durations: { // or `durationsMs`?
    'clone': 0,
    'reset': 0,
    'checkout': 0,
    'fetch': 0,
    'push': 0,
    'clean': 0,
    'merge': 0,
    'submodule': 0,
    'commit': 0,
    'plumbing': 0,
    'other': 0,
  },
  cloned: false,
  branchesProcessed: 0,
};

As an ideal case, we'd provide a wrapper around SimpleGit, which would auto-instrument each of the operations, which would provide a more ergonomic experience for contributions, as well as more centrally tweaking the configuration.

[viceice]

sounds good to me

[secustor]

We should also add the OTEL spans for these operations too.

[secustor]

Also probably worth to unify the tracing setup and stats setup in a single unified abstraction

[jamietanna]

How would something like:

    res = await instrument('other', async () => {
      await git.raw(['rev-parse', '--abbrev-ref', 'origin/HEAD']);
    })

look/feel? The other is the GitOperationType.

Not quite sure what we'd want to do for the span name - require the caller set it? Use the GitOperationType?

[jamietanna]

Under the hood, I've got this calling instrument that creates an OTel span - #38582

[secustor]

Are you aware of this?

renovate/lib/instrumentation/decorator.ts

Line 10 in 95abcda

export function instrument<T>({

Also something to consider is to use simply a custom Spanprocessor to generate the stats we have

renovate/lib/instrumentation/decorator.ts

Line 10 in 95abcda

export function instrument<T>({

[jamietanna]

This is now done - we get a lot more insight i.e.