Managing Multiple Kubernetes Contexts

Overview

When working with multiple Kubernetes clusters, managing kubeconfig files for each cluster can be challenging. The KUBECONFIG environment variable allows you to specify the location of multiple kubeconfig files, and Kubernetes will merge them at runtime. However, tools like kubectx, which are commonly used to switch between Kubernetes contexts, do not natively support multiple kubeconfig files. This document explains how to handle this scenario effectively using the Fish shell on your Garuda Linux setup.

1. Setting Up KUBECONFIG in Fish Shell

In Fish shell, environment variables are set using the set command. Here’s how to specify the KUBECONFIG environment variable when you have multiple kubeconfig files:

Command to Set KUBECONFIG:

set -x KUBECONFIG /home/sudo-samurai/.kube/home-lab:/home/sudo-samurai/.kube/prod-2024

• x: This flag exports the variable, making it available globally to all child processes (similar to export in Bash).
• /home/sudo-samurai/.kube/home-lab:/home/sudo-samurai/.kube/prod-2024: This path contains two kubeconfig files separated by a colon (:). Kubernetes can use multiple kubeconfig files at once, combining the configurations.

Making it Persistent:

To persist this environment variable across all Fish sessions, add the following line to your config.fish file:

echo 'set -x KUBECONFIG /home/sudo-samurai/.kube/home-lab:/home/sudo-samurai/.kube/prod-2024' >> ~/.config/fish/config.fish

After doing this, every time you open a new Fish shell session, the KUBECONFIG variable will be automatically set.

2. Verifying the KUBECONFIG Setup

To ensure that KUBECONFIG is set correctly, you can check its value:

echo $KUBECONFIG

This should output:

/home/sudo-samurai/.kube/home-lab:/home/sudo-samurai/.kube/prod-2024

You can also verify that Kubernetes is loading and merging both kubeconfig files:

kubectl config view --merge

This command will show the combined kubeconfig data from both files, ensuring that both configurations are available.

3. Addressing the kubectx Limitation

While Kubernetes supports multiple kubeconfig files, some tools, like kubectx, do not handle multiple kubeconfig files defined in the KUBECONFIG variable. If you try running kubectx with multiple kubeconfig files, you might encounter this error:

error: kubeconfig error: failed to load: cannot determine kubeconfig path: multiple files in KUBECONFIG are currently not supported

Solution: Merging Kubeconfig Files

To work around this limitation, you can merge your kubeconfig files into a single file. Here’s how to do it:

Step 1: Merge the Kubeconfig Files

You can merge the kubeconfig files using the kubectl command:

KUBECONFIG=/home/sudo-samurai/.kube/home-lab:/home/sudo-samurai/.kube/prod-2024 kubectl config view --merge --flatten > /home/sudo-samurai/.kube/merged-config

• -merge: Combines the contexts, clusters, and users from both kubeconfig files.
• -flatten: Removes duplicate or redundant entries.
• > /home/sudo-samurai/.kube/merged-config: Writes the output to a new file (merged-config).

Step 2: Update the KUBECONFIG Environment Variable

After merging the files, update the KUBECONFIG variable to point to the new merged file:

set -x KUBECONFIG /home/sudo-samurai/.kube/merged-config

For persistence, add this to your config.fish:

echo 'set -x KUBECONFIG /home/sudo-samurai/.kube/merged-config' >> ~/.config/fish/config.fish

Step 3: Verify the Merged Configuration

Run the following command to verify that the merged kubeconfig file contains all the required contexts:

kubectl config get-contexts

This command should now list the contexts from both the original home-lab and prod-2024 kubeconfig files.

4. Working with kubectx

Once the kubeconfig files are merged, you can use kubectx to switch between contexts without issues:

kubectx

This command will now display all available Kubernetes contexts in the merged kubeconfig file, allowing you to switch between them easily.

5. Fish Variables and Universal Variables

While we’re setting KUBECONFIG using set -x, it’s essential to understand the fish_variables file. The fish_variables file is where Fish shell stores universal variables, which are accessible across all Fish sessions.

• Universal variables can be set using the set -U command. They are stored in the fish_variables file, which is located in ~/.config/fish/.
• This file is managed by Fish internally and is not meant to be edited manually. To persist the KUBECONFIG variable globally across sessions and instances without editing the config.fish file, you could use a universal variable:

set -U KUBECONFIG /home/sudo-samurai/.kube/merged-config

This command will save the KUBECONFIG variable to the fish_variables file, making it available in all Fish instances without needing to modify config.fish.

Conclusion

Managing multiple kubeconfig files in Fish shell requires a workaround due to the limitations of tools like kubectx. By merging kubeconfig files and updating the KUBECONFIG environment variable, you can seamlessly manage multiple Kubernetes contexts. The steps provided ensure that your configuration is persistent and manageable across Fish shell sessions.

Here’s a summary of the commands covered:

1. Set the KUBECONFIG variable in Fish:

   set -x KUBECONFIG /home/sudo-samurai/.kube/home-lab:/home/sudo-samurai/.kube/prod-2024

2. Merge the kubeconfig files:

   KUBECONFIG=/home/sudo-samurai/.kube/home-lab:/home/sudo-samurai/.kube/prod-2024 kubectl config view --merge --flatten > /home/sudo-samurai/.kube/merged-config

3. Set the new KUBECONFIG variable:

   set -x KUBECONFIG /home/sudo-samurai/.kube/merged-config

4. Use kubectx to switch contexts:

   kubectx

This guide ensures you can effectively manage multiple kubeconfig files while maintaining compatibility with tools like kubectx in Fish shell.