x509: certificate has expired or is not yet valid

[asalan316]

We often see that the credhub x509: certificate are expired and the input resources in the pipelines do not work properly.
Credhub credentials are expired if they are older than 30 days. As a result, the following error messages are occurs

• Credhub pod: Get "https://credhub.concourse.svc.cluster.local:9000/info": x509: certificate has expired or is not yet valid: current time 2023-02-27T10:14:45Z is after 2023-02-25T15:05:44Z
• Concourse input resources x509: certificate has expired or is not yet valid

Credub Container Logs

....
....
2023-02-25T15:06:09.108178082Z 2023-02-25T15:06:09.108Z [https-jsse-nio-9000-exec-29861] INFO  CEFAudit - CEF:0|cloud_foundry|credhub|2.12.18|GET /api/v1/data|GET /api/v1/data|0|rt=1677337569103 suser=credhub_admin_client suid=uaa-client:credhub_admin_client cs1Label=userAuthenticationMechanism cs1=uaa request=/api/v1/data?name-like=%2Fconcourse%2Fcapi-team%2Fcf-performance-tests-go%2Fcf-perf-github-user-token requestMethod=GET cs3Label=versionUuid cs3=null cs4Label=httpStatusCode cs4=200 src=10.104.4.10 dst=credhub.concourse.svc.cluster.local cs2Label=resourceName cs2=null cs5Label=resourceUuid cs5=null deviceAction=FIND cs6Label=requestDetails cs6={"nameLike":"/concourse/capi-team/cf-performance-tests-go/cf-perf-github-user-token","path":null,"paths":null,"expiresWithinDays":""} 
2023-02-25T15:06:09.108905197Z 2023-02-25T15:06:09.108Z [https-jsse-nio-9000-exec-29847] INFO  CEFAudit - CEF:0|cloud_foundry|credhub|2.12.18|GET /api/v1/data|GET /api/v1/data|0|rt=1677337569085 suser=credhub_admin_client suid=uaa-client:credhub_admin_client cs1Label=userAuthenticationMechanism cs1=uaa request=/api/v1/data?current=true&name=%2Fconcourse%2Fapp-autoscaler%2Fautoscaler_git_key requestMethod=GET cs3Label=versionUuid cs3=484b25cf-266e-4ddc-b85d-99804176d2b7 cs4Label=httpStatusCode cs4=200 src=10.104.4.10 dst=credhub.concourse.svc.cluster.local cs2Label=resourceName cs2=/concourse/app-autoscaler/autoscaler_git_key cs5Label=resourceUuid cs5=ccefdf42-859f-44bc-8157-ac9bd30fe9ba deviceAction=GET cs6Label=requestDetails cs6={"name":"/concourse/app-autoscaler/autoscaler_git_key","versions":null,"current":true} 
2023-02-25T15:06:09.109982195Z 2023-02-25T15:06:09.109Z [https-jsse-nio-9000-exec-29825] INFO  CEFAudit - CEF:0|cloud_foundry|credhub|2.12.18|GET /api/v1/data|GET /api/v1/data|0|rt=1677337569106 suser=credhub_admin_client suid=uaa-client:credhub_admin_client cs1Label=userAuthenticationMechanism cs1=uaa request=/api/v1/data?name-like=%2Fconcourse%2Fcapi-team%2Fcf-performance-tests-rails%2Fcf-perf-github-username requestMethod=GET cs3Label=versionUuid cs3=null cs4Label=httpStatusCode cs4=200 src=10.104.4.10 dst=credhub.concourse.svc.cluster.local cs2Label=resourceName cs2=null cs5Label=resourceUuid cs5=null deviceAction=FIND cs6Label=requestDetails cs6={"nameLike":"/concourse/capi-team/cf-performance-tests-rails/cf-perf-github-username","path":null,"paths":null,"expiresWithinDays":""} 
2023-02-25T15:06:09.113414576Z 2023-02-25T15:06:09.113Z [https-jsse-nio-9000-exec-29857] INFO  CEFAudit - CEF:0|cloud_foundry|credhub|2.12.18|GET /api/v1/data|GET /api/v1/data|0|rt=1677337569110 suser=credhub_admin_client suid=uaa-client:credhub_admin_client cs1Label=userAuthenticationMechanism cs1=uaa request=/api/v1/data?name-like=%2Fconcourse%2Fcapi-team%2Fcf-perf-github-user-token requestMethod=GET cs3Label=versionUuid cs3=null cs4Label=httpStatusCode cs4=200 src=10.104.4.10 dst=credhub.concourse.svc.cluster.local cs2Label=resourceName cs2=null cs5Label=resourceUuid cs5=null deviceAction=FIND cs6Label=requestDetails cs6={"nameLike":"/concourse/capi-team/cf-perf-github-user-token","path":null,"paths":null,"expiresWithinDays":""} 
2023-02-25T15:06:09.114327995Z 2023-02-25T15:06:09.114Z [https-jsse-nio-9000-exec-29819] INFO  CEFAudit - CEF:0|cloud_foundry|credhub|2.12.18|GET /api/v1/data|GET /api/v1/data|0|rt=1677337569106 suser=credhub_admin_client suid=uaa-client:credhub_admin_client cs1Label=userAuthenticationMechanism cs1=uaa request=/api/v1/data?name-like=%2Fconcourse%2Fcapi-team%2Fcf-load-test%2Faws-access-key-id requestMethod=GET cs3Label=versionUuid cs3=null cs4Label=httpStatusCode cs4=200 src=10.104.4.10 dst=credhub.concourse.svc.cluster.local cs2Label=resourceName cs2=null cs5Label=resourceUuid cs5=null deviceAction=FIND cs6Label=requestDetails cs6={"nameLike":"/concourse/capi-team/cf-load-test/aws-access-key-id","path":null,"paths":null,"expiresWithinDays":""} 
2023-02-25T15:06:09.115389264Z 2023-02-25T15:06:09.115Z [https-jsse-nio-9000-exec-29825] INFO  CEFAudit - CEF:0|cloud_foundry|credhub|2.12.18|GET /api/v1/data|GET /api/v1/data|0|rt=1677337569111 suser=credhub_admin_client suid=uaa-client:credhub_admin_client cs1Label=userAuthenticationMechanism cs1=uaa request=/api/v1/data?name-like=%2Fconcourse%2Fcapi-team%2Fcf-perf-github-username requestMethod=GET cs3Label=versionUuid cs3=null cs4Label=httpStatusCode cs4=200 src=10.104.4.10 dst=credhub.concourse.svc.cluster.local cs2Label=resourceName cs2=null cs5Label=resourceUuid cs5=null deviceAction=FIND cs6Label=requestDetails cs6={"nameLike":"/concourse/capi-team/cf-perf-github-username","path":null,"paths":null,"expiresWithinDays":""} 
2023-02-25T15:06:09.117152707Z 2023-02-25T15:06:09.117Z [https-jsse-nio-9000-exec-29841] INFO  CEFAudit - CEF:0|cloud_foundry|credhub|2.12.18|GET /api/v1/data|GET /api/v1/data|0|rt=1677337569113 suser=credhub_admin_client suid=uaa-client:credhub_admin_client cs1Label=userAuthenticationMechanism cs1=uaa request=/api/v1/data?name-like=%2Fconcourse%2Fapp-autoscaler%2Fapp-autoscaler-release-github-private-workers%2Fautoscaler_git_key requestMethod=GET cs3Label=versionUuid cs3=null cs4Label=httpStatusCode cs4=200 src=10.104.4.10 dst=credhub.concourse.svc.cluster.local cs2Label=resourceName cs2=null cs5Label=resourceUuid cs5=null deviceAction=FIND cs6Label=requestDetails cs6={"nameLike":"/concourse/app-autoscaler/app-autoscaler-release-github-private-workers/autoscaler_git_key","path":null,"paths":null,"expiresWithinDays":""} 
2023-02-25T15:06:09.123557574Z 2023-02-25T15:06:09.123Z [https-jsse-nio-9000-exec-29841] INFO  CEFAudit - CEF:0|cloud_foundry|credhub|2.12.18|GET /api/v1/data|GET /api/v1/data|0|rt=1677337569119 suser=credhub_admin_client suid=uaa-client:credhub_admin_client cs1Label=userAuthenticationMechanism cs1=uaa request=/api/v1/data?name-like=%2Fconcourse%2Fapp-autoscaler%2Fautoscaler_git_key requestMethod=GET cs3Label=versionUuid cs3=null cs4Label=httpStatusCode cs4=200 src=10.104.4.10 dst=credhub.concourse.svc.cluster.local cs2Label=resourceName cs2=null cs5Label=resourceUuid cs5=null deviceAction=FIND cs6Label=requestDetails cs6={"nameLike":"/concourse/app-autoscaler/autoscaler_git_key","path":null,"paths":null,"expiresWithinDays":""}
2023-02-25T15:06:09.127263127Z 2023-02-25T15:06:09.127Z [https-jsse-nio-9000-exec-29784] INFO  org.cloudfoundry.credhub.services.RetryingEncryptionService - Attempting decrypt
....
....

Workaround

Restart the credhub Kubernetes deployment in the concourse namespace. It will destroy the old pod and create a new one.