LangFuse Upgrade Experience: Navigating from v3.29 to v3.54

LangFuse is awesome. If you're working with Large Language Models (LLMs), you probably know how useful it is for seeing what your AI is actually doing. It helps us track everything, keep an eye on costs, get feedback, and manage the prompts we feed our models. We love it.

We were using an older version (v3.29.1) and started seeing a weird bug where sometimes the tracking data (called "traces") just wouldn't load. Annoying! So, we decided to upgrade to the latest version (v3.54.0) hoping it would fix things.

We thought it would be easy – just tell our system (which uses Kubernetes and Helm, think of Helm as a package manager for Kubernetes apps) to use the new Langfuse version. Spoiler: it wasn't that easy. It turned into a bit of a detective story figuring out configuration changes.

Here's what happened and how we fixed it, in case you run into similar bumps:

Problem 1: The Upgrade Just Wouldn't Start

We tried the basic command to upgrade, and BAM! Errors everywhere.

• Password Demands: The system complained about needing passwords for things like Redis (a speedy database Langfuse uses) and S3 (for storage). It turns out, when you upgrade these kinds of tools using popular setup charts (from Bitnami), they make you re-enter the current password as a safety check.

  • Fix: We had to dig into our Kubernetes setup to find the existing passwords (using a specific command) and then explicitly give those passwords back to the upgrade command. We even had to provide the Redis password in a few different places in the command for it to be happy!

• Config Format Confusion: We also got warnings about how some settings were written down in our configuration file (values.yaml). Things like secret keys (salt, nextauth.secret) used to be just plain text. The new version wanted them wrapped in a slightly different format (like salt: { value: "our-secret-here" }).

  • Fix: We had to go into our configuration file and change the format for those specific settings to match what the new version expected.

Problem 2: It Tried to Install Storage We Didn't Need

We use AWS S3 for storage, which is separate from our Kubernetes cluster. But the Langfuse upgrade setup assumed we wanted to install internal storage (MinIO/S3) by default. This triggered more password errors for storage we weren't even using!

• Fix: Easy one! We just added a couple of lines to our configuration file explicitly telling Helm: "Nope, don't install your own S3/MinIO." (s3: { deploy: false }, minio: { deploy: false }).

Problem 3: Where Do the Settings Go Now?!

Okay, we cleared the first hurdles, but then hit more errors. This time, it was about how we configured connections to things like our S3 storage and our main database (ClickHouse).

• The Big Change: In the older version, we just dumped a lot of settings (like S3 keys, database usernames) into a general list of "environment variables." The new version is much more organized. It has dedicated sections in the configuration file specifically for S3 settings, ClickHouse settings, Redis settings, etc.

• The Problem: The upgrade process got confused because it either couldn't find settings in their new dedicated spots, or it found them in both the old general list and the new sections.

• Fix: We had to carefully move all those connection settings out of the old general list and put them into their proper new sections in the configuration file (e.g., all S3 details went under the s3: section).

Problem 4: Okay, It Upgraded... But Now It's Broken!

Finally, the Helm upgrade command finished without errors! But when we tried to use Langfuse...

• App Can't Find the Database: The main Langfuse web part couldn't connect to its helper database (Redis/Valkey). The error logs showed it was looking for a server name that didn't exist.

  • Fix: We realized the upgrade had slightly changed the internal network name for the Redis service. We just had to find the correct new name and update our connection setting to point to it.

• Database Out of Disk Space: When Langfuse tried to update its main database structure (ClickHouse), it failed because the virtual disk was full.

  • Fix: We needed to give the ClickHouse database more disk space. This was tricky because you often can't just change a setting in Helm to resize an existing disk. We had to:

    1. Make sure our underlying cloud storage system allowed resizing.

    2. Use a direct Kubernetes command (kubectl patch pvc...) to tell the existing virtual disk to grow bigger.

    3. Then update our Helm configuration file with the new size so it matches reality for the future.

• Can't Log In! After all that, we couldn't even log in. Just kept getting "Invalid credentials."

  • Checks: We checked everything – was the Langfuse web address set correctly? Did the secret key change? Was the database connection really okay? Everything looked fine.

  • The (Embarrassing) Fix: ...I was typing my email address wrong. Yep. Sometimes the simplest explanation is the right one. Always check the basics!

Success and What We Learned

With the login fixed, Langfuse v3.54.0 was finally up and running! And the good news? The original problem with traces not loading seems to be gone.

This whole process taught us a few things:

1. Read the Upgrade Notes: The people who make these tools often write guides for major changes. Read them!

2. Dependencies Have Needs: Langfuse relies on other tools (like Redis). Be ready to handle their specific quirks during upgrades (like needing passwords).

3. Configuration Isn't Static: How you set things up can change between versions. Pay attention to new formats or sections in the config files.

4. Some Fixes Need Direct Intervention: Helm is great, but for some things (like resizing existing disks), you might need to use direct Kubernetes commands.

5. Check for Typos! Don't spend hours debugging complex configurations before checking if you just misspelled your own login.

Even though it was a bit more work than expected, the new way LangFuse organizes its configuration is actually cleaner. Hopefully, sharing our little adventure helps someone else have a smoother upgrade!