Use db_session retry and more exception handling in upgrade

[qstokkink]

Fixes #8320

This PR:

• Updates the upgrade script to use db_session's retry* instead of manually committing and rolling back.
• Updates the try .. catch constructions to be even more defensive.
• Removes the ignored ignore ANN101 (ruff complained about this).

*To my dismay, retry cannot be used in a context manager. So, I had to refactor all the function bodies, the bits inside db_session, into separate functions with a db_session decorator.