Keycloak for AuthN and AuthZ

Hey, welcome back 👋😊

🧵 This is the part 04 of the Home Lab 2.0 series.

When I started this project to rebuild my home lab, I wanted to have a way of managing Authentication and Authorization and provide SSO to my applications. I really wanted to try something like Microsoft Entra ID but obviously it demands serious $$$ 🔥💵 to have access to all the features. Hence I looked for alternatives and found Keycloak. I’ve never used Keycloak until now and it seems like a fairly popular tool, even amongst larger organisations. Also, nothing beats the fact that its open-source and free😊.

Initially, I was planing to use a VM to run Keycloak in docker but after considering my hardware limitations (poor mans lab setup🫠), I decided to use two Unprivileged LXC containers to deploy Keycloak and ProgreSQL. It is surprisingly efficient with resources and secure enough for my purpose. I would only use a VM in my setup when it is absolutely necessary. Otherwise, LXC would be my first choice.

Providing the permissions to ProgreSQL schema was a bit of thing with Ansible. I kept getting Argument "schema" is not allowed for type "schema" error with community.postgresql.postgresql_privs module. I followed the documentation to the letter but had no luck. Therefore I switched to the community.postgresql.postgresql_query module to move forward.

It took me a while to figure out some of the nuance of the Keycloak application. The documentation was a lot to digest. journalctl -u keycloak.service -f was my best friend 😅.

Below is one of those very helpful error messages 🤪. Since I’m using Traefik as the reverse proxy, I had to add proxy-headers=xforwarded to explicitly tell Keycloak to read X-Forwarded-* headers from the proxy 😅.

Apart from the above proxy flag, I added a new router to the Traefik and updated my DNS with Keycloak info. Basically all my applications should point to the reverse proxy, Traefik. Below is how my pfsense DNS looks like at the moment. It will grow when I add the rest of the apps.

With that, all the ducks were aligned 𓅬🦢🦆🪿𓅼 and I could successfully log into Keycloak as admin 🤩.

Once the Keycloak is up and running, I decided to integrate Proxmox as my first Keycloak Client.

Integrating Proxmox 🚀

This involved two main steps.

1. Configure an OIDC Auth Realm in Proxmox - Hand rolled ( backlog item to automate 😊)

2. Create an OIDC Client in Keycloak - using Ansible

The Proxmox Realm configuration looks like below.

I spent many hours trying to troubleshoot below error 💀💀💀 when configuring Keycloak for Proxmox. It’s worth documenting as anyone could inadvertently walk into the same trap.

The lay of the land 🏜️

1. NO firewall rules enabled on Traefik LXC or Proxmox nodes. Basically it is all open.

2. Traefik is listening on port 443

root@traefik:~# ss -tulpn | grep 443

tcp   LISTEN 0      4096               *:443             *:*    users:(("traefik",pid=28531,fd=6))

3. Unable to reach Keycloak from Proxmox LXC

root@pmox01:~# curl -vvv https://keycloak.jayforweb.com/realms/homelab/.well-known/openid-configuration

*   Trying 192.168.193.50:443...

4. TCP dump on the Proxmox LXC shows that the TCP handshake is stuck in a loop of SYN and SYN-ACK without the final ACK. Maybe this is a telltale sign a network engineer would spot in a heartbeat🙃. But to me, it just confirmed that I could rule out any firewall issues as traffic is not blocked!.

So.. I knew something does not add up but I could not see a logical reason until I stumbled upon something about routing confusions. Then.. Aha..💡🔔

It was all my fault (d’oh)🙊

I defined my Traefik LXC IP as 192.168.193.50/32 in a 192.168.193.0/24 VLAN 💀🔫

Let me explain the implications of my genius act (sarcasm emoji here 🤪), using networking 101.

The IP 192.168.193.50/32 has no distinct network address (192.168.193.0) for routing or broadcast address (192.168.193.255) to send traffic to other devices in the VLAN. Basically it effectively defines a network with one IP address - the device itself. This creates problems with ARP and routing in a network segment.

So I was trying to communicate with my Traefic LXC while it was marooned in an isolated network island 🏝, all alone, waiting to hear from someone 😅

Once I could get through this point, I ended up making the rest of the configurations using the Keycloak UI as well as Ansible. I rebuild the Keycloak LXC and PostgreSQL a few times as I messed up the things with all the experimentations (I mean poking around). When I was sure that everything works as expected I took a full dump of the Realm configuration using the kc.sh, redacted anything sensitive and tried to use that as a baseline to generate some Ansible. Google’s Gemini🤖 did a half decent job with most of the boilerplate code but some of its suggestions were blatantly wrong 🙄.

I think the current AI tools have a long way to go. They certainly can generate some code which can be useful, only if you know what you are doing. Also they may reduce the troubleshooting time by working as a Search engine on steroids. However I don’t see that this is enough to qualify something as Artificial Intelligence. Code generation is nothing new 🤷. Despite all the hype, I feel like these tools still lack the very thing they claim to have - Intelligence! But, I digress.

Below is how you authenticate with keycloak API and export the full realm configuration. The export function from UI is limited and not useful in this context.

root@keycloak:~# cd /opt/keycloak/bin/
root@keycloak:/opt/keycloak/bin# ./kcadm.sh config credentials --server "http://localhost:8080/" --realm master --user admin --password "my-super-secret-pw" --client admin-cli
root@keycloak:/opt/keycloak/bin# ./kc.sh export --realm homelab --dir /tmp --users realm_file

Once I completed the code, I deleted the Keycloak, PostgreSQL and redeployed them using the new code. Everything worked like a charm, this time!

I think thats about it. If you are interested, a fully working copy of the source code is on my Github. Check it out and let me know what you think.

Next, I will be working on Hashicorp Vault to facilitate secret management.

On a side note, did you notice the banner of this post? In the background, its Anorak, the avatar of James Halliday in Ready Player One. I thought it is fitting when we are discussing Key + Cloak 😎

Until next time 🫡

Source Code 🧩

This homelab code and other things on my Github 😊

References 📖

1. Keycloak official documentation

2. Keycloak repo by ansible-middleware

3. Keycloak CLI - Adding users to group without sed

4. Proxmox-services by briandipalma