Published 08 Sep, 2022

Java - Can't connect to kubernetes API from inside the cluster

Category Java
Modified : Sep 29, 2022
20

I'm trying to use the fabric8io/kubernetes-client in a simple example on my local minikube cluster where I get an IP of a pod

import io.fabric8.kubernetes.client.Config;
import io.fabric8.kubernetes.client.ConfigBuilder;
import io.fabric8.kubernetes.client.DefaultKubernetesClient;
import io.fabric8.kubernetes.client.KubernetesClient;

public class PodLogExample {

  public static void main(String[] args) throws InterruptedException {

    String podName = "some-pod-name-jnfen3";
    String namespace = "default";
    String master = "https://localhost:32780/";

    Config config = new ConfigBuilder().withMasterUrl(master).build();
    KubernetesClient client = new DefaultKubernetesClient(config);
    String podIP = client.pods().inNamespace(namespace).withName(podName).get().getStatus().getPodIP();
    System.out.println("Pod IP is: " + podIP);
    Thread.sleep(5 * 1000);
    client.close();
  }
}

some-pod-name-jnfen3 is a name of a pod which finished execution and visible if I execute kubectl get pods.

https://localhost:32780/ is a kubernetes API host which I get from kubectl config view -o jsonpath="{.clusters[?(@.name==\"minikube\")].cluster.server}".

  • minikube has only 1 node.
  • fabric8io/kubernetes-client version is 4.9.1
  • Java 11

I deploy this code as a Job on the same cluster as the some-pod-name-jnfen3. Here's the Exception:

Exception in thread "main" io.fabric8.kubernetes.client.KubernetesClientException: Operation: [get]  for kind: [Pod]  with name: [some-pod-name-jnfen3]  in namespace: [default]  failed.
    at io.fabric8.kubernetes.client.KubernetesClientException.launderThrowable(KubernetesClientException.java:64)
    at io.fabric8.kubernetes.client.KubernetesClientException.launderThrowable(KubernetesClientException.java:72)
    at io.fabric8.kubernetes.client.dsl.base.BaseOperation.getMandatory(BaseOperation.java:225)
    at io.fabric8.kubernetes.client.dsl.base.BaseOperation.get(BaseOperation.java:168)
    at PodLogExample.main(PodLogExample.java:16)
Caused by: java.net.ConnectException: Failed to connect to localhost/127.0.0.1:32780
    at okhttp3.internal.connection.RealConnection.connectSocket(RealConnection.java:249)
    at okhttp3.internal.connection.RealConnection.connect(RealConnection.java:167)
    at okhttp3.internal.connection.StreamAllocation.findConnection(StreamAllocation.java:258)
    at okhttp3.internal.connection.StreamAllocation.findHealthyConnection(StreamAllocation.java:135)
    at okhttp3.internal.connection.StreamAllocation.newStream(StreamAllocation.java:114)
    at okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.java:42)
    at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
    at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121)
    at okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.java:93)
    at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
    at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121)
    at okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.java:93)
    at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
    at okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.java:127)
    at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
    at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121)
    at io.fabric8.kubernetes.client.utils.BackwardsCompatibilityInterceptor.intercept(BackwardsCompatibilityInterceptor.java:134)
    at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
    at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121)
    at io.fabric8.kubernetes.client.utils.ImpersonatorInterceptor.intercept(ImpersonatorInterceptor.java:68)
    at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
    at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121)
    at io.fabric8.kubernetes.client.utils.HttpClientUtils.lambda$createHttpClient$3(HttpClientUtils.java:112)
    at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
    at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121)
    at okhttp3.RealCall.getResponseWithInterceptorChain(RealCall.java:257)
    at okhttp3.RealCall.execute(RealCall.java:93)
    at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:469)
    at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:430)
    at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleGet(OperationSupport.java:395)
    at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleGet(OperationSupport.java:376)
    at io.fabric8.kubernetes.client.dsl.base.BaseOperation.handleGet(BaseOperation.java:845)
    at io.fabric8.kubernetes.client.dsl.base.BaseOperation.getMandatory(BaseOperation.java:214)
    ... 2 more
Caused by: java.net.ConnectException: Connection refused (Connection refused)
    at java.base/java.net.PlainSocketImpl.socketConnect(Native Method)
    at java.base/java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:399)
    at java.base/java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:242)
    at java.base/java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:224)
    at java.base/java.net.SocksSocketImpl.connect(SocksSocketImpl.java:403)
    at java.base/java.net.Socket.connect(Socket.java:609)
    at okhttp3.internal.platform.Platform.connectSocket(Platform.java:129)
    at okhttp3.internal.connection.RealConnection.connectSocket(RealConnection.java:247)
    ... 34 more

For some reason it can't connect to the kubernetes API host.

How can I fix it?

UPDATE

If I don't pass the config to the client, it gets the correct master IP and then fails with the following exception

Exception in thread "main" io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: GET at: https://10.96.0.1/api/v1/namespaces/default/pods/some-pod-name-jnfen3. Message: Forbidden!Configured service account doesn't have access. Service account may have been revoked. pods "some-pod-name-jnfen3" is forbidden: User "system:serviceaccount:default:default" cannot get resource "pods" in API group "" in the namespace "default".
    at io.fabric8.kubernetes.client.dsl.base.OperationSupport.requestFailure(OperationSupport.java:568)
    at io.fabric8.kubernetes.client.dsl.base.OperationSupport.assertResponseCode(OperationSupport.java:505)
    at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:471)
    at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:430)
    at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleGet(OperationSupport.java:395)
    at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleGet(OperationSupport.java:376)
    at io.fabric8.kubernetes.client.dsl.base.BaseOperation.handleGet(BaseOperation.java:845)
    at io.fabric8.kubernetes.client.dsl.base.BaseOperation.getMandatory(BaseOperation.java:214)
    at io.fabric8.kubernetes.client.dsl.base.BaseOperation.get(BaseOperation.java:168)
    at PodLogExample.main(PodLogExample.java:17)

Answers

There are 2 suggested solutions here and each one has been listed below with a detailed description. The following topics have been covered briefly such as Java, Kubernetes, Minikube, Java 11. These have been categorized in sections for a clear and precise explanation.

17

kubectl config view -o jsonpath="{.clusters[?(@.name==\"minikube\")].cluster.server}" returns the relative IP. It is different for the Pod.

Not passing config will make the library find the correct IP.

Also a new service account should be created and set in the job.yaml.

The default service account doesn't have permissions to fetch k8s entities.

Related GitHub issue


14

It’s localhost from the point of view of your laptop, from inside the cluster it is not. Use the InClusterConfig helpers instead.