How to change annotationsDiffBaseStorage prefix and clean old one

[DrummyFloyd]

Hi, i'm trying to change domain of my prefix ..
eg:

# from 
NS_API_DEPRECATED="crui.foo.github.com"
# to
NS_API="crui.bar.github.com"

i used to start the operator like this

@kopf.on.startup(id="crui-startup")
async def configure(settings: kopf.OperatorSettings, **_):
    settings.persistence.diffbase_storage = kopf.AnnotationsDiffBaseStorage(
        prefix=K8S_API_NS,
        key="last-handled-configuration",
    )

i added a deprecated function

@kopf.on.resume(
    "", "v1", "secret", labels=LABELS_MATCH_DEPRECATED, id="crui-deprecated"
)
async def on_deprecated_labels(body: kopf.Body, **_):  # pragma: no cover
    """Handle deprecated label events"""
    raise kopf.PermanentError(
        f"deprecated labels detected on secret {body.metadata.name}, please update to use {K8S_API_NS} instead of {K8S_API_NS_DEPRECATED}"
    )

but i'm struggling to remove old annotationsDiffBaseStorage

any idea how to make this properly?

thank you

[DrummyFloyd]

i found the solution, digging in issue/PR and discussion with that

#319 (comment)

@kopf.on.resume(
    "", "v1", "secret", labels=LABELS_MATCH_DEPRECATED, id="crui-deprecated"
)
async def on_deprecated_labels(body: kopf.Body, patch, **_):  # pragma: no cover
    """Handle deprecated label events"""
    patch.metadata.annotations[
        f"{K8S_API_NS_DEPRECATED}/last-handled-configuration"
    ] = None
    patch.metadata.annotations[f"{K8S_API_NS_DEPRECATED}/kopf-managed"] = None
    raise kopf.PermanentError(
        f"deprecated labels detected on secret {body.metadata.name}, please update to use {K8S_API_NS} instead of {K8S_API_NS_DEPRECATED}"
    )

work as expected

[nolar]

For the record and later readers, I would like to extend the answer. The transition of this kind is documented here:

• https://kopf.readthedocs.io/en/stable/configuration/#storage-transition

The solution is slightly more complex, and consists of 3 stages of arbitrary duration:

• Add a new label (prefix), while keeping the old supported.
• Let it run for an arbitrary period, until everything is upgraded AND the possibility of the rollback is excluded forever-ever-ever.
• Remove the old labels (mind the k8s api overload in case of too many resources).

As long as there is a possibility of a rollback, you want the old labels/annotations to be present. The multi-source storages solve this exact problem: write to all configured places, read from the first successful one. Apply to both "progress storage" and "diffbase storage".

For this particular reason (and this reason alone), Kopf itself still uses the kopf.zalando.org prefix instead of its own newer kopf.dev for the out-of-the-box configuration: I cannot reliably upgrade the annotations/labels without causing a k8s api havoc for the Kopf users (with no benefit for them).

[DrummyFloyd]

thank for the explanations, help a lot ( for the actual and future project )

in my case ( do no hesitate if 'im wrong ofc), it's way more simple

i don't use kopf as a real operator it's more like a easy way to control action upon labels
https://github.com/DrummyFloyd/crunchy-userinit-controller

based on following labels

K8S_API_NS_DEPRECATED = "crunchy-userinit.ramblurr.github.com"
LABELS_MATCH_DEPRECATED = {
    "postgres-operator.crunchydata.com/role": "pguser",
    LABEL_ENABLED_DEPREACTED: "true",
}

i change owner a of Database in postgres operated by crunchyData-Operator, because old project has been archived.

the change is non destructive, because everything is already made by previous labels, and the controller, do not make any action on cleanup(once the DB owership is made, nothing will be done.

i want to change the crunchy-userinit.ramblurr.github.com to crunchy-userinit.drummyfloyd.github.com to sync the labels with new orgs

that's why i made a pr-BREAKING-CHANGE

with following logic

@kopf.on.startup(id="crui-startup")
async def configure(settings: kopf.OperatorSettings, **_):
    settings.peering.standalone = True
    settings.persistence.diffbase_storage = kopf.AnnotationsDiffBaseStorage(
        prefix=K8S_API_NS, key="last-handled-configuration", v1=False
    )
    if DEV_MODE:
        logger.warning("running in dev mode")
        _ = await config.load_kube_config()
    else:
        config.load_incluster_config()


@kopf.on.update(
    "", "v1", "secret", labels=LABELS_MATCH_DEPRECATED, id="update-crui-deprecated"
)
@kopf.on.create(
    "", "v1", "secret", labels=LABELS_MATCH_DEPRECATED, id="create-crui-deprecated"
)
@kopf.on.resume(
    "", "v1", "secret", labels=LABELS_MATCH_DEPRECATED, id="resume-crui-deprecated"
)
async def on_deprecated_labels(body: kopf.Body, patch: kopf.Patch, **_):
    """Handle deprecated label events"""
    for annotation in body.metadata.annotations:
        if annotation.startswith(K8S_API_NS_DEPRECATED):
            logger.warning(
                f"removing deprecated annotation {annotation} from secret {body.metadata.name}"
            )
            patch.metadata.annotations[annotation] = None
    patch.metadata.annotations[ANNOTATION_MIGRATED] = "False"
    raise kopf.PermanentError(
        f"deprecated labels detected on secret {body.metadata.name}, please update to use {K8S_API_NS} instead of {K8S_API_NS_DEPRECATED}"
    )


@kopf.on.create("", "v1", "secret", labels=LABELS_MATCH, id="create-crui")
@kopf.on.update("", "v1", "secret", labels=LABELS_MATCH, id="update-crui")
@kopf.on.resume("", "v1", "secret", labels=LABELS_MATCH, id="resume-crui")
async def on_pguser_secret_created(body: kopf.Body, patch: kopf.Patch, **_):
    """Handle PostgreSQL user secret creation/update events"""

    if any(K8S_API_NS_DEPRECATED in key for key in body.metadata.labels):
        logger.warning(
            f"Secret '{body.metadata.name}' contains unused deprecated label '{K8S_API_NS_DEPRECATED}'.Consider removing it for clarity since the current labels are already in use."
        )
        patch.metadata.annotations[ANNOTATION_MIGRATED] = "True"
    else:
        patch.metadata.annotations[ANNOTATION_MIGRATED] = None

    try:
      # main logic here
      ...

thank again for you clarification.