I hate that you hate to write this, but good work doing it. I never understood why people perceive k3s as hard and then write pages of docker compose yaml instead. Admittedly my day job got me a CKA, but running k3s at home is barely a step up from docker compose.
Curious what you would use instead? I can only think of one wrong answer and that’s Jsonnet.
Are you running coredns in your cluster? This sounds like a Corefile change but maybe it’s just missing a service?
Pods won’t have any entries in DNS aside from the pod IP addresses you’ve found. A Service will be in DNS and should have several entries depending on the namespace you’re calling from, eg:
service-a
service-a.namespace
service-a.namespace.svc.cluster.local
I’m on mobile so haven’t looked above to see if you’ve created a service for the pod. A service will have endpoints which are the pod ips and ports (and you can ignore endpoint slices for now)
Edit: I see coredns in there now. I’d check Corefile or the kubelet configs, seems like it can contact DNS because you’re getting an NXDOMAIN response at least.
Is the firewall config blocking any ports flannel might use? As far as Google can tell vxlan uses port 8472.
No yaml, no helm, no operators? Using Pulumi as a layer of abstraction but not using ingress because its a layer of abstraction?
I don’t know the equivalent in GCP, but in AWS this would be treating EKS like ECS. Missing out on 90% of the benefits of kubernetes by ignoring that 90%.
But also, small company. Kubernetes is a better base to start from and expand from there.
Yeah for reference I’d probably never run the full open source Kubernetes distribution unless I had to, and that would mean having access to millions of dollars of hardware in a datacenter.
K3s is a lightweight Kuberbetes distribution that implements the full Kuberbetes API (full-ish? Maybe?). It’s super easy to run on Linux, I run a 3 node cluster with GPUs at home. Its only real downside is the backend is a single point of failure, but that’s ok for me cause it’s run from my storage node with all the disks, so if that disappears I have bigger problems.
There are others like microk8s which can handle control plane failures, but it’s for that reason that I also dislike it - they wrote their own distributed sqlite instance and it failed on me, a story for another time.
Minikube can run on your desktop, it’s also an option.
But if you have docker desktop, you also have a built in Kuberbetes API server too, just have to enable it with one checkbox (not a full API server, but good enough for installing helm charts).
Kind is a docker based Kubernetes server but I think that’s in the realm of testing not running. I believe K0s is in this camp too but could be wrong.
At work the daily driver will be one of EKS, GKE, AKS, or whichever cloud providers implementation. They’re effectively free and a loss leader because you’ll pay for instances anyway (at least on EKS, I’m most familiar with that one).
But if you’re interested in learning, start with docker desktops k8s API, or minikube, or k3s if you have a Linux host or raspberry Pi lying around.
🌈The more you know!🌈