Two Recent Engineering Challenges

I wasted a lot of time recently on two engineering challenges: I couldn't create signed GCS URLs in Google Cloud Run and my email attachments weren't being added to the outgoing emails.

These two issues stonewalled my progress to getting the #MVP out.

Creating Signed GCS URLs in Google Cloud Run

The first was super painful to debug. I followed all the docs, best practices, and had it running locally. To test in the cloud required a deployment + product regeneration which full cycle took about 20 minutes. That means, I could only repeat the debug cycle 3 times an hour! Developer hell! Had I STOPPED and thought for twenty minutes, I could have implemented a skeleton project that had a much faster turnaround time. I could have debugged probably twice as fast. But, I kept telling myself — you’re almost there — just one more thing to fix. No need to stop now; keep pushing. Not sure if that’s “momentum” or the “sunk cost fallacy” or just fatigue from being up since 4 am plowing through features. Probably a mixture of all three.

In the end, no LLM-based interactions nor model swapping helped. None. The reason was because the solution was buried in a stackoverflow post where a responder to the query says, “I made the commit in the client libraries 3.5 years ago and I’m still surprise[d] the signature or requirements didn’t change[]”:

Reading that, then following the deep links to three other posts, I finally found what I needed.

But, I ask myself: did it really need to be this hard? Why wasn’t Google clearer about the nuance required for Google Cloud Run?

Google even has a blog about it.Attachments

The next issue was all me. My MVP — to be announced shortly — does deeply personalized research for people. Because this runs out of a container with limited file system storage I needed a guaranteed way to clean up after myself when the process was done. I initially tried a try, except, finally to clean up. That was silly so I moved to a TemporaryDirectory context manager, which is an elegant solution designed to solve this very problem:

    # Create temporary directory that will be used throughout the process
    # This will raise an exception if it fails - no silent errors
    with tempfile.TemporaryDirectory(dir=REPORTS_DIR) as temp_dir:
        try:
            logger.info(f"Starting report generation for {user_email}")

When Python exits the with block, it cleans up after itself every time. I don’t need to do anything.

BUT…. my overall application logic was this:

    # Create temporary directory that will be used throughout the process
    # This will raise an exception if it fails - no silent errors
    with tempfile.TemporaryDirectory(dir=REPORTS_DIR) as temp_dir:
        try:
            logger.info(f"Starting report generation for {user_email}")
            report = ...

    # By this point, Python cleaned up the TemporaryDirectory and my attachement
    # was lost.
    send_email(report)

This confused me even more b/c I had the report in GCS, the file path to where it should have been on disk, but it would attach to the emails. So, I thought it was an email problem and focused on solving the wrong issue.

I then inspected Mailcatcher locally, which is amazing, and found that a simple skeleton/mock test worked fine. So, now I’m back to the logic which I figured out pretty quickly.

No LLM interactions helped here, either. I think either the context was still too big, the model too weak, or the complexity of the code still too high for them to understand.

So:

• As experienced as I am, whenever I spend 30-45 minutes on an issue and can’t get through it, I need to stop and step away from the keyboard. If LLMs aren’t helping — I definitely need to step away.

• LLMs are great tools, but human thought still required. They’re like having mid-level engineers that get stuck. (They also add code like crazy but don’t like removing it as much.)

• When I start spinning my wheels, I gotta’ remember that quote — and step away from the keyboard — “Spend 5 minutes thinking, and you’ll save 5 hours coding” — or something like that.

• Know what’s really in your way or not. I didn’t actually need the signed URLs for MVP. Hah. I do need them, yes, but not right now. :smh.

Happy thinking,

Jason