[FIXED] Server-side Validation of Kubernetes yaml

Issue

I would like to do server-side validation of Kubernetes yaml files before applying them.

I know that in my Jenkins agent, I could use the following kubectl command for validating yaml files at the server-side but I am a bit concerned about access-control:

• Kubernetes < v1.18: kubectl apply --server-dry-run -f ...
• Kubernetes >= v1.18: kubectl apply --dry-run=server -f ...

The Kubernetes documentation says the following:

Authorization for dry-run and non-dry-run requests is identical. Thus, to make a dry-run request, the user must be authorized to make the non-dry-run request.

I don't want any Jenkins agents to have super powers over my EKS cluster. A bad actor could use my Jenkins agent maliciously and apply any manifests they wanted. Right now for security/stability/management reasons, creating Kubernetes objects is done by a different system not Jenkins.

I checked a few other options but I can see drawbacks:

• Kubeval is not aware of any CRDs installed in the actual cluster.
• Client Validation is not really end-to-end validation
• I could develop a rest api that exposes a validation rest endpoint and hits Kubernetes APIs or runs a kubectl --run-dry under the hood. However, this requires more dev work than we have capacity for.

Do you have any ideas or are you aware of any validation tools that I could use in our CI system securely for the purpose of validating end-2-end Kubernetes yaml files?

Solution

I've been looking for this myself and did not find a sufficient tooling. However, there are few workarounds:

• Deploy all objects to a temporary ci-job-id namespace in dev/stage clusters. They should be the same as a prod, but will not impose the security risks you mentioned. This gives an additional benefit - you can check if everything got created, all pods are running. It helps to catch issues like insufficient resource requests, missing images, misconfigured Service selectors, etc. Also it let's you add a smoke test on top.
• Spin a small minikube with all the CRDs specifically for CI validations. This approach gives you less coverage, but it is much cheaper to maintain.

Answered By - Max Lobur