Kubernetes: Internal error occurred: failed calling webhook

Hi, it's BlueByte. A kubectl apply that worked yesterday now fails in under a second with Internal error occurred: failed calling webhook, and nothing in your manifest changed. Your object was never actually evaluated: an admission webhook sits in front of that write, the API server could not get an answer out of it, and the default failure policy turned that silence into a rejection. We'll walk through the shape of the message, why a webhook you may not have installed yourself can block your writes, how to tell a network block from a certificate mismatch, the fix for each cause, and how to scope the webhook so it cannot lock you out again.

What the API server is actually reporting

The failure comes back as a server-side error on any verb the webhook matches:

Error from server (InternalError): error when creating "ingress.yaml": Internal error occurred: failed calling webhook "validate.example.com": failed to call webhook: Post "https://webhook-service.webhook-system.svc:443/validate?timeout=10s": context deadline exceeded

Three fragments carry all the information. The quoted webhook name identifies one entry inside a ValidatingWebhookConfiguration or MutatingWebhookConfiguration. The URL is assembled from clientConfig.service — namespace, name and port, where the documented defaults are port 443 and path /. The tail after the final colon is the diagnosis, and it arrives in a small set of shapes:

• context deadline exceeded — the call was dispatched and no reply came back in time.
• connect: connection refused — something is routable at that address and nothing is listening on the port.
• no endpoints available for service "webhook-service" — the Service exists but has no ready backends.
• x509: certificate signed by unknown authority — the API server does not trust the certificate the webhook served.
• x509: certificate is valid for ..., not webhook-service.webhook-system.svc — the certificate is trusted but carries the wrong name.

Read that tail first. Everything below branches on it.

Why a webhook that cannot answer rejects your write

Each webhook entry carries a failurePolicy, and the Kubernetes reference is explicit that the default is Fail: an error calling the webhook causes admission failure and the request is rejected. Ignore is the other value, and it lets the request proceed. The policy covers network errors, timeouts, non-2xx responses, malformed responses and serialization failures — every way of not getting a usable answer.

The clock is timeoutSeconds, documented with an allowed range of 1 to 30 seconds and a default of 10. When it expires, the call is ignored or the request is rejected according to that same failure policy.

One distinction saves a lot of wasted debugging: a webhook that runs and says no is a different event entirely. The reference notes that an explicit rejection, correctly transmitted, always denies the API request regardless of the failurePolicy setting. That message reads admission webhook "..." denied the request, not failed calling webhook.

Four things that stop the API server reaching the webhook

• No ready backends. The webhook Deployment was scaled to zero, evicted, or is crash-looping. The Service resolves, the endpoint list is empty.
• A blocked network path. On a managed cluster the control plane lives outside your VPC and reaches the nodes through a narrow hole. Google's GKE guidance states that by default the firewall does not allow TCP connections to nodes except on ports 443 (HTTPS) and 10250 (kubelet), and that an admission webhook trying to reach a Pod on a different port fails without a custom firewall rule.
• Broken TLS trust. caBundle is the PEM-encoded, base64-wrapped CA the API server validates against, and the serving certificate must be valid for <svc_name>.<svc_namespace>.svc. A regenerated cert, an expired one, or a re-created Secret breaks the pair.
• The webhook is up but slow. A handler that calls an external API or a database can exceed timeoutSeconds under load while looking perfectly healthy in its own logs.

There is a fifth case worth naming because it is self-inflicted: a webhook that validates the namespace it runs in cannot be restarted once it is down, because its own Pod creation is blocked by itself.

Check the three things the tail points at

Start by listing what is actually in the admission path, with its policy and timeout:

kubectl get validatingwebhookconfigurations -o custom-columns=NAME:.metadata.name,HOOKS:.webhooks[*].name,POLICY:.webhooks[*].failurePolicy,TIMEOUT:.webhooks[*].timeoutSeconds

NAME                HOOKS                  POLICY   TIMEOUT
example-admission   validate.example.com   Fail     10

Then ask whether anything is behind the Service named in the URL:

kubectl -n webhook-system get endpoints webhook-service
kubectl -n webhook-system get pods -l app=webhook-server

NAME              ENDPOINTS         AGE
webhook-service   10.20.1.23:8443   41d

An ENDPOINTS column reading <none> ends the investigation right there — fix the workload. If endpoints exist, probe the same URL from inside the cluster:

kubectl -n webhook-system run probe --rm -i --restart=Never --image=curlimages/curl:8.11.1 -- curl -sk -o /dev/null -w '%{http_code}\n' https://webhook-service.webhook-system.svc:443/

This one deserves care in how you read it. A Pod lives inside the cluster network; the API server, on a managed control plane, does not. A probe that returns a status code while the API server still times out is the signature of a firewall rule, not of a sick webhook. For the TLS branch, compare the two halves of the trust pair:

kubectl get validatingwebhookconfiguration example-admission -o jsonpath='{.webhooks[0].clientConfig.caBundle}' | base64 -d | openssl x509 -noout -subject -dates
kubectl -n webhook-system get secret webhook-server-cert -o go-template='{{index .data "tls.crt"}}' | base64 -d | openssl x509 -noout -ext subjectAltName -dates

The issuer of the serving certificate has to be the CA in caBundle, and its subjectAltName has to include webhook-service.webhook-system.svc. Nothing is broken if you have never run these before — they only read, and you can run them on a healthy cluster to see what a matching pair looks like.

Fix it by cause, cheapest move first

Empty endpoints: bring the workload back and let the next retry through.

kubectl -n webhook-system rollout restart deploy/webhook-server
kubectl -n webhook-system rollout status deploy/webhook-server --timeout=120s

Blocked port on a private cluster: open the control-plane range to the port the webhook Pod actually listens on. Google documents collecting masterIpv4CidrBlock from gcloud container clusters describe and the node target tags from the existing gke- rules, then:

gcloud compute firewall-rules create allow-webhook-8443 \
  --action ALLOW \
  --direction INGRESS \
  --source-ranges CONTROL_PLANE_RANGE \
  --rules tcp:8443 \
  --target-tags TARGET

Certificate mismatch: reissue the serving certificate from the CA in caBundle, or re-inject the CA the webhook's installer manages. If cert-manager owns it, the cert-manager.io/inject-ca-from annotation on the configuration is what refills that field — check the annotation is still present before you patch anything by hand.

Slow handler: raise the timeout toward the documented ceiling, and treat that as buying time rather than fixing latency.

kubectl patch validatingwebhookconfiguration example-admission --type=json -p='[{"op":"replace","path":"/webhooks/0/timeoutSeconds","value":20}]'

Locked out with a production deploy waiting: switching that one entry to failurePolicy: Ignore restores writes immediately and lets unvalidated objects through until you switch it back. Deleting the configuration object does the same thing more bluntly, and a Helm release or operator will usually recreate it on the next reconcile.

A worked example: a private cluster and a webhook on 8443

A team adds a policy controller to a private GKE cluster. It installs cleanly, the Pod is Running, its logs are quiet. Every kubectl apply against a matched resource then fails with failed calling webhook ending in context deadline exceeded after almost exactly ten seconds. Endpoints show 10.20.1.23:8443, and a curl probe from a Pod returns 200, which rules out the workload. The ten-second wall matches the documented default timeout, and the port is 8443 — not one of the two the control plane is allowed to open. They add the firewall rule above for tcp:8443 from the control-plane CIDR to the node tag. The next apply returns ingress.networking.k8s.io/site created with no change to the manifest.

Confirm admission works again before you walk away

Re-run the write that failed, then exercise the webhook deliberately with an object you expect it to reject:

kubectl apply -f ingress.yaml
kubectl apply -f deliberately-invalid.yaml

A success on the first and admission webhook "validate.example.com" denied the request on the second is the result you want. A second success means the webhook is being skipped, which is not the same as being fixed.

Scope the webhook so it cannot lock you out

The Kubernetes good-practices guidance is to exclude system namespaces and the webhook's own namespace so it cannot block its own recovery:

namespaceSelector:
  matchExpressions:
  - key: kubernetes.io/metadata.name
    operator: NotIn
    values:
    - kube-system
    - webhook-system

The same guidance asks for a small timeout, narrow rules, objectSelector and namespaceSelector filtering so the webhook is invoked less often, and more than one replica behind the Service. Watch the certificate expiry too — a webhook that has run for a year without complaint fails the day its cert rolls over.

How this differs from kubectl's x509 error and from a denial

kubectl: x509: certificate signed by unknown authority points the opposite way down the same connection: there it is your client that does not trust the API server's certificate, and the fix is in your kubeconfig. Here the API server is the client and the webhook is the server, so the fix is caBundle and the serving cert inside the cluster — your kubeconfig is fine, which is why kubectl get keeps working while writes fail.

The second look-alike is admission webhook "..." denied the request. That is the webhook working: it received your object, applied its rule, and said no. failurePolicy has no bearing on it, and the fix is in your manifest rather than in the cluster.

Next time a write dies on a webhook, walk these checks back in order: read the tail of the message, check the endpoints, probe the Service from a Pod, then compare the CA against the serving certificate.