Troubleshooting Kubernetes Connectivity: Using exec and port-forward Effectively

When a Kubernetes service times out, you usually need to answer one plain question before you can fix anything: where does the connection stop working? kubectl exec lets you test from inside the cluster, close to the application. kubectl port-forward lets you bring one pod or service back to your laptop without changing an Ingress, LoadBalancer, firewall rule, or DNS record.

Used together, these two commands help you avoid guessing. You can test whether the app is listening, whether cluster DNS works, whether a Service points at the right pods, and whether the problem lives in Kubernetes networking or in the application itself.

Understanding kubectl exec

The kubectl exec command allows you to execute commands inside a running container within a pod. This is incredibly useful for inspecting logs, checking configurations, and running diagnostic tools directly where your application lives.

Use it carefully in production. You are running a command inside a real workload container. A harmless env, cat, curl, or ss is normal. Installing packages, changing files, or running expensive diagnostics inside a live pod can hide the original problem or create a new one.

Basic Syntax

The fundamental syntax for kubectl exec is:

kubectl exec <pod-name> -- <command> [args...]

• <pod-name>: The name of the pod you want to execute a command in.
• --: This separator is crucial. It distinguishes between kubectl flags and the command you want to run inside the container.
• <command>: The command to execute within the container (e.g., ls, cat, ping).
• [args...]: Any arguments for the command.

Interactive Shell Access

One of the most common uses of kubectl exec is to get an interactive shell (like bash or sh) inside a container. This allows you to explore the container's filesystem and run multiple commands.

To get an interactive shell:

kubectl exec -it <pod-name> -- /bin/bash

• -i (or --stdin): Keeps stdin open even if not attached.
• -t (or --tty): Allocates a pseudo-TTY, which is necessary for interactive shell sessions.

Example: Accessing a bash shell in a pod named my-app-pod:

kubectl exec -it my-app-pod -- /bin/bash

Once inside, you can use standard Linux commands. To exit the shell, type exit or press Ctrl+D. If /bin/bash is not present, try /bin/sh; many small images do not ship with Bash.

Executing a Single Command

You can also execute a single command without an interactive shell. This is useful for quick checks or scripting.

Example: Checking files in the /app directory of my-app-pod:

kubectl exec my-app-pod -- ls /app

Example: Viewing the contents of a configuration file config.yaml:

kubectl exec my-app-pod -- cat /etc/my-app/config.yaml

Specifying a Container within a Pod

If your pod has multiple containers, you need to specify which container to execute the command in using the -c flag.

kubectl exec <pod-name> -c <container-name> -- <command>

Example: Executing env in the sidecar-container of multi-container-pod:

kubectl exec multi-container-pod -c sidecar-container -- env

Understanding kubectl port-forward

The kubectl port-forward command allows you to establish a secure tunnel from your local machine to a specific pod or service in your Kubernetes cluster. This is invaluable for debugging applications that are not exposed externally, accessing databases, or testing internal APIs.

It is not a production traffic path. It is a debugging tunnel over your Kubernetes API connection. If the API server connection drops, your local tunnel drops too.

Basic Syntax

The general syntax is:

kubectl port-forward <pod-name> <local-port>:<remote-port>

• <pod-name>: The name of the pod you want to connect to.
• <local-port>: The port on your local machine that will listen for connections.
• <remote-port>: The port on the pod that will receive the forwarded traffic.

Example: Forwarding local port 8080 to port 80 of my-app-pod:

kubectl port-forward my-app-pod 8080:80

Once this command is running, you can access your application by navigating to http://localhost:8080 in your web browser or using tools like curl on your local machine.

Forwarding to a Service

You can also forward traffic to a Kubernetes Service instead of a specific pod. kubectl will automatically select a pod backing that service.

kubectl port-forward service/<service-name> <local-port>:<service-port>

Example: Forwarding local port 3000 to port 80 of the my-service service:

kubectl port-forward service/my-service 3000:80

When forwarding to a Service, remember that kubectl chooses one backing pod. If only one replica is broken, service-level forwarding can miss it. For instance, a Deployment with three pods may have two healthy pods and one pod with a bad config. Forwarding to service/my-service might pick a healthy pod and make the service look fine. When you suspect a replica-specific issue, forward to the exact pod name.

Forwarding to Deployments or StatefulSets

Similarly, you can forward to Deployments or StatefulSets. kubectl will select one of the pods managed by the specified resource.

kubectl port-forward deployment/<deployment-name> <local-port>:<container-port>

kubectl port-forward statefulset/<statefulset-name> <local-port>:<container-port>

Binding to a Specific Address

By default, port-forward binds to localhost. You can specify a different local address using the --address flag.

kubectl port-forward --address 127.0.0.1 <pod-name> <local-port>:<remote-port>

Multiple Port Forwarding

kubectl port-forward can forward multiple ports simultaneously.

kubectl port-forward my-app-pod 8080:80 9090:90

This command forwards local port 8080 to pod port 80 and local port 9090 to pod port 90.

Common Troubleshooting Scenarios and Solutions

Scenario 1: Application is unresponsive, but pod appears healthy.

• Problem: The pod is running, but requests to its service are failing or timing out. The application might have internal configuration issues or be stuck.
• Solution with kubectl exec:
  1. Get an interactive shell into the pod: kubectl exec -it <pod-name> -- /bin/bash
  2. Inside the shell, check application logs (e.g., tail -f /var/log/myapp.log).
  3. Verify the application's internal configuration files.
  4. Check network connectivity from within the pod to other services using ping or curl (if installed).
• Solution with kubectl port-forward:
  1. Forward a port to the application's listening port: kubectl port-forward <pod-name> 8080:<app-port>
  2. Attempt to access the application locally via http://localhost:8080. This helps determine if the issue is with the Kubernetes service discovery or ingress, or if the application itself is not responding.

If the port-forward works but the normal service URL fails, inspect the Service selector and endpoints:

kubectl get service <service-name> -n <namespace> -o yaml
kubectl get endpoints <service-name> -n <namespace>
kubectl get pods -n <namespace> --show-labels

A very common failure is a label mismatch. The pods are healthy, the service exists, but the selector does not match the pod labels, so the Service has no endpoints.

Scenario 2: Need to debug a database running in a pod.

• Problem: You need to connect your local database client to a database running inside a Kubernetes pod to inspect data or run queries.
• Solution with kubectl port-forward:
  1. Identify the pod running the database and its port (e.g., mysql-pod, port 3306).
  2. Forward a local port to the database port: kubectl port-forward mysql-pod 3306:3306
  3. Configure your local database client to connect to localhost:3306 using the appropriate database credentials.

Scenario 3: Diagnosing DNS resolution issues within a pod.

• Problem: An application inside a pod cannot reach other services by their service names, suggesting a DNS problem.
• Solution with kubectl exec:
  1. Get an interactive shell into the pod: kubectl exec -it <pod-name> -- /bin/bash
  2. Inside the shell, try to resolve a known service name: nslookup <service-name>.<namespace>.svc.cluster.local or dig <service-name>.<namespace>.svc.cluster.local.
  3. Check the contents of /etc/resolv.conf to ensure the cluster's DNS configuration is correct within the pod.

If the image does not include nslookup, dig, or curl, use a temporary debug pod in the same namespace:

kubectl run net-debug -n <namespace> --rm -it --image=curlimages/curl -- sh

From there, test the same service name your application uses. This helps separate "my application image has no tools" from "cluster DNS is broken."

Scenario 4: Port-forward connects, then immediately closes.

• Problem: kubectl port-forward prints a forwarding message, but the connection closes when you open the browser or run curl.
• Likely causes: The target process is not listening on the remote port, the application only binds to 127.0.0.1 inside the container, or the pod restarts while the tunnel is open.
• Checks:

  kubectl exec <pod-name> -n <namespace> -- ss -lntp
  kubectl get pod <pod-name> -n <namespace> -w
  kubectl logs <pod-name> -n <namespace> --tail=100
  

If the process listens on port 8080 but you forward to 80, the tunnel itself is fine; the target is wrong. If the pod restarts while you test, fix the restart cause before chasing networking.

Scenario 5: Service works from one pod but not another.

• Problem: A backend can reach Redis, but a worker pod in another namespace cannot.
• Checks with exec:

  kubectl exec <source-pod> -n <source-namespace> -- curl -v http://<service>.<target-namespace>.svc.cluster.local:<port>
  kubectl exec <source-pod> -n <source-namespace> -- cat /etc/resolv.conf
  

If DNS resolves but TCP fails, check NetworkPolicies, service endpoints, and whether the target application accepts traffic from that namespace. If DNS does not resolve, test the fully qualified service name before blaming the app.

A Simple Connectivity Ladder

When a request fails, test from nearest to farthest:

1. Inside the pod, test the local app process:

   kubectl exec <pod> -n <namespace> -- curl -v http://127.0.0.1:<port>/health
   

2. From a different pod in the same namespace, test the Service:

   kubectl run curl-test -n <namespace> --rm -it --image=curlimages/curl -- curl -v http://<service>:<port>/health
   

3. From your laptop, test through port-forward:

   kubectl port-forward service/<service> -n <namespace> 8080:<service-port>
   curl -v http://localhost:8080/health
   

4. Only after those pass, move outward to Ingress, load balancers, DNS, and firewall rules.

This order saves time because each step proves one layer. If localhost inside the pod fails, Ingress is irrelevant. If pod-local succeeds but Service access fails, the app is probably fine and Kubernetes service discovery needs attention.

One more habit helps: keep the namespace explicit while debugging. Many confusing sessions come from running kubectl exec in the default namespace while the broken workload lives somewhere else. Either set the namespace in your context for the session or add -n <namespace> to every command. The extra typing is cheaper than testing the wrong pod.

Also save the exact command that proved the failure. The next person on call can rerun it without reconstructing your context from memory.

Best Practices and Tips

• Keep port-forward running: kubectl port-forward runs in the foreground. You'll need to keep the terminal window open. To run it in the background, you can use tools like nohup or screen/tmux.
• Use specific pods when debugging: While forwarding to services is convenient, for pinpointing issues with a specific instance, forwarding to a particular pod using its name is often more effective.
• Security: Be mindful of what ports you expose. Avoid forwarding sensitive ports unless absolutely necessary and ensure your local machine is secured.
• Resource Usage: kubectl exec can consume resources. Use it judiciously, especially on production clusters.
• Permissions: Ensure your kubectl context has the necessary permissions to execute commands in pods or forward ports.

What to Write Down After You Fix It

Good debugging leaves a trail. Capture the failing URL, the source pod, the target service, the namespace, the exact command that reproduced the issue, and the layer where it failed. "Connectivity issue" is too vague to help next time. "Service selector did not match pods after label rename" is a fixable pattern.