### Overview When working with multiple Kubernetes clusters, managing kubeconfig files for each cluster can be challenging. The `KUBECONFIG` environment variable allows you to specify the location of multiple kubeconfig files, and Kubernetes will merge them at runtime. However, tools like `kubectx`, which are commonly used to switch between Kubernetes contexts, do not natively support multiple kubeconfig files. This document explains how to handle this scenario effectively using the Fish shell on your Garuda Linux setup. --- ### 1. Setting Up `KUBECONFIG` in Fish Shell In Fish shell, environment variables are set using the `set` command. Here's how to specify the `KUBECONFIG` environment variable when you have multiple kubeconfig files: ### Command to Set `KUBECONFIG`: ```bash set -x KUBECONFIG /home/sudo-samurai/.kube/home-lab:/home/sudo-samurai/.kube/prod-2024 ``` - **`x`**: This flag exports the variable, making it available globally to all child processes (similar to `export` in Bash). - **`/home/sudo-samurai/.kube/home-lab:/home/sudo-samurai/.kube/prod-2024`**: This path contains two kubeconfig files separated by a colon (`:`). Kubernetes can use multiple kubeconfig files at once, combining the configurations. ### Making it Persistent: To persist this environment variable across all Fish sessions, add the following line to your `config.fish` file: ```bash echo 'set -x KUBECONFIG /home/sudo-samurai/.kube/home-lab:/home/sudo-samurai/.kube/prod-2024' >> ~/.config/fish/config.fish ``` After doing this, every time you open a new Fish shell session, the `KUBECONFIG` variable will be automatically set. --- ### 2. Verifying the `KUBECONFIG` Setup To ensure that `KUBECONFIG` is set correctly, you can check its value: ```bash echo $KUBECONFIG ``` This should output: ```bash /home/sudo-samurai/.kube/home-lab:/home/sudo-samurai/.kube/prod-2024 ``` You can also verify that Kubernetes is loading and merging both kubeconfig files: ```bash kubectl config view --merge ``` This command will show the combined kubeconfig data from both files, ensuring that both configurations are available. --- ### 3. Addressing the `kubectx` Limitation While Kubernetes supports multiple kubeconfig files, some tools, like `kubectx`, do not handle multiple kubeconfig files defined in the `KUBECONFIG` variable. If you try running `kubectx` with multiple kubeconfig files, you might encounter this error: ```bash error: kubeconfig error: failed to load: cannot determine kubeconfig path: multiple files in KUBECONFIG are currently not supported ``` ### Solution: Merging Kubeconfig Files To work around this limitation, you can merge your kubeconfig files into a single file. Here’s how to do it: ### Step 1: Merge the Kubeconfig Files You can merge the kubeconfig files using the `kubectl` command: ```bash KUBECONFIG=/home/sudo-samurai/.kube/home-lab:/home/sudo-samurai/.kube/prod-2024 kubectl config view --merge --flatten > /home/sudo-samurai/.kube/merged-config ``` - **`-merge`**: Combines the contexts, clusters, and users from both kubeconfig files. - **`-flatten`**: Removes duplicate or redundant entries. - **`> /home/sudo-samurai/.kube/merged-config`**: Writes the output to a new file (`merged-config`). ### Step 2: Update the `KUBECONFIG` Environment Variable After merging the files, update the `KUBECONFIG` variable to point to the new merged file: ```bash set -x KUBECONFIG /home/sudo-samurai/.kube/merged-config ``` For persistence, add this to your `config.fish`: ```bash echo 'set -x KUBECONFIG /home/sudo-samurai/.kube/merged-config' >> ~/.config/fish/config.fish ``` ### Step 3: Verify the Merged Configuration Run the following command to verify that the merged kubeconfig file contains all the required contexts: ```bash kubectl config get-contexts ``` This command should now list the contexts from both the original `home-lab` and `prod-2024` kubeconfig files. --- ### 4. Working with `kubectx` Once the kubeconfig files are merged, you can use `kubectx` to switch between contexts without issues: ```bash kubectx ``` This command will now display all available Kubernetes contexts in the merged kubeconfig file, allowing you to switch between them easily. --- ### 5. Fish Variables and Universal Variables While we’re setting `KUBECONFIG` using `set -x`, it’s essential to understand the `fish_variables` file. The `fish_variables` file is where Fish shell stores universal variables, which are accessible across all Fish sessions. - Universal variables can be set using the `set -U` command. They are stored in the `fish_variables` file, which is located in `~/.config/fish/`. - This file is managed by Fish internally and is not meant to be edited manually. To persist the `KUBECONFIG` variable globally across sessions and instances without editing the `config.fish` file, you could use a universal variable: ```bash set -U KUBECONFIG /home/sudo-samurai/.kube/merged-config ``` This command will save the `KUBECONFIG` variable to the `fish_variables` file, making it available in all Fish instances without needing to modify `config.fish`. --- ### Conclusion Managing multiple kubeconfig files in Fish shell requires a workaround due to the limitations of tools like `kubectx`. By merging kubeconfig files and updating the `KUBECONFIG` environment variable, you can seamlessly manage multiple Kubernetes contexts. The steps provided ensure that your configuration is persistent and manageable across Fish shell sessions. Here’s a summary of the commands covered: 1. Set the `KUBECONFIG` variable in Fish: ```bash set -x KUBECONFIG /home/sudo-samurai/.kube/home-lab:/home/sudo-samurai/.kube/prod-2024 ``` 2. Merge the kubeconfig files: ```bash KUBECONFIG=/home/sudo-samurai/.kube/home-lab:/home/sudo-samurai/.kube/prod-2024 kubectl config view --merge --flatten > /home/sudo-samurai/.kube/merged-config ``` 3. Set the new `KUBECONFIG` variable: ```bash set -x KUBECONFIG /home/sudo-samurai/.kube/merged-config ``` 4. Use `kubectx` to switch contexts: ```bash kubectx ``` This guide ensures you can effectively manage multiple kubeconfig files while maintaining compatibility with tools like `kubectx` in Fish shell.