Minikube L&D Lab with In-Cluster Caddy and Ceph

This guide sets up a full Kubernetes L&D environment on a single high-performance VM using Minikube, CRI-O, in-cluster Caddy ingress with wildcard DNS, Ceph storage via Rook, and monitoring stack with Prometheus + Grafana.

📦 System Requirements

🧰 Tech Stack

• Minikube (Kubernetes Lab)
• CRI-O (Container Runtime)
• Caddy (Ingress Controller inside Kubernetes)
• Let’s Encrypt with Wildcard DNS via Route 53
• Rook-Ceph for dynamic PVs
• Prometheus + Grafana for monitoring

☁️ DNS Setup in Route 53

Add a wildcard A record:

*.sudo-samurai.com → <Minikube VM Public IP>

Create an IAM user in AWS with Route 53 DNS permissions and note the AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY.

🚀 Installation Steps

Run the setup script included in this package: setup.sh

It installs:

• CRI-O
• kubectl, Helm, Minikube
• Starts Minikube with CRI-O
• Deploys Ceph (Rook)
• Sets up Grafana + Prometheus
• Installs Caddy as a Kubernetes Ingress Controller (wildcard SSL via Route 53)

🔑 AWS Credentials Secret

The script will prompt you to create a secret like:

kubectl create secret generic route53-credentials \
  --from-literal=AWS_ACCESS_KEY_ID=YOUR_KEY \
  --from-literal=AWS_SECRET_ACCESS_KEY=YOUR_SECRET \
  -n caddy-system

🌐 Sample Ingress for Grafana

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: grafana-ingress
  namespace: monitoring
  annotations:
    caddy.ingress.k8s.io/host: grafana.sudo-samurai.com
spec:
  rules:
    - host: grafana.sudo-samurai.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: kube-prometheus-grafana
                port:
                  number: 80

You can similarly create Ingress rules for Jenkins, GitLab, or any custom app.

🔐 Optional: Basic Auth with Caddy

Caddy supports HTTP Basic Auth via basicauth directive. This is not included by default in the Helm chart but can be added with Caddyfile extensions.

✅ Final Checks

Happy hacking with your fully automated, production-like K8s lab 🎉