On this blog I talked in the past about Keystone authentication for your Kubernetes cluster. The solution described works great if you have Openstack users stored in the keystone mysql database. However, in real production systems, it is common to access Openstack with a Federated login. The web login works with a redirect to an Identity provider that will confirm the user identity and will redirect again to the Openstack dashboard.
It is a well known problem that the federated login process needs to go through web pages redirects to enter the necessary information, and this does not work for users that need to authenticate with CLI tools. In our case the CLI tool is kubectl.
The good news is that the Keystone developers already implemented a solution for this problem, the Keystone Application Credentials. This is a feature available since the Queens release of Openstack. The key idea is that a user can login on the web interface with the federated login process, and then from the dashboard identity panel he can create new credentials to be consumed directly from CLI tools.
The following three screenshoots show the user journey to create an Application Credential in the Openstack Horizon dashboard:
So what is missing to authenticate with kubectl using a keystone application credential ?
Starting at kubectl v1.11 the specific cloud providers API implementations were moved out of the kubernetes source tree. You have to modify our client configuration as follows:
contexts: - context: cluster: kubernetes namespace: keystone-daadb4bcc9704054b108de8ed263dfc2 user: openstackgarr name: garr users: - name: openstackgarr user: exec: apiVersion: client.authentication.k8s.io/v1alpha1 command: /bin/client-keystone-auth args: - --domain-name=myDomain - --keystone-url=https://keystone.doman.tld:5000/v3 env: - name: OS_USERNAME value: username - name: OS_PASSWORD value: secret - name: OS_PROJECT_NAME value: test
Note that in this configuration snippet we are still using username and password. If you want to test this setup make sure the client-keystone-auth version is newer than v0.2.0 or that is patched including commit 66961abd. The version v0.2.0 is not able to request keystone project scoped token, so your setup will not work.
The software client-keystone-auth uses the golang library gophercloud to talk to the Kubernetes API.
To reach our goal, the first step was to patch gophercloud to implement the application_credential authentication method described in the Queens spec.
This patch enables any golang application to easily access the application credentials authentication method, so it could be useful to other golang software tools, like for example Terraform.
Please note that the patch implements just the possibility to issue a token authenticating with the application credential. I did not propose a gophercloud patch that implements the full create/update/delete workflow for application credentials, because this went beyond the scope of this work.
At the time of this writing the latest PR is still not merged, so you will need to compile the code yourself to test it.
Now that all the code is there, we can change the client configuration to use the application credential instead of username and password as we show in the example below:
contexts: - context: cluster: kubernetes namespace: keystone-daadb4bcc9704054b108de8ed263dfc2 user: openstackgarr name: garr users: - name: openstackgarr user: exec: apiVersion: client.authentication.k8s.io/v1alpha1 command: /bin/client-keystone-auth args: - --domain-name=myDomain - --keystone-url=https://keystone.doman.tld:5000/v3 - --user-name=username - --application-credential-name=kuberneteslogin - --application-credential-secret=thisismysecret
You can also use environment variables instead of command line arguments. The client-keystone-auth supports the same variable names as the official openstack client.
Conclusion: you can today use kubectl with your keystone application credentials, this is really useful if you have a federated login to the Openstack cloud.
I would like to thank the people that participated to this development work with me, especially Giuseppe Attardi , Roberto Di Lallo and Joe Topjian that helped in implementation, discussion and code review.