AI Assisted Software Development

I have been using ChatGPT, Github Copilot, and recently Claude for code generation, increasingly since ~beginning of 2023. These are my raw personal experiences and takeaways in this space:

• Currently, I find Claude 3.5 Sonnet the best of the ones I have tried for most AI tasks. Github Copilot is also useful for small in-place edits in VS Code, but not for more complex tasks. ChatGPT-4o is also good, but I find Claude better at alignment.

• Whether or not a coding task works depends on the task complexity, language(s) used, how we approach it, etc.

  • More and more complex coding tasks are working well the first time I ask, with the latest models compared to earlier ones.

  • You can ask the AI for everything - not just code - related to a programming project. For example, picking a tech task, how to set up the project, how to run it, what is an ideal project structure, to add tests, to ensure code coverage, to add profiling, to check for memory leaks (depending on language), to fix path resolutions, improve algorithm efficiency - anything and everything really, at various levels of complexity and abstraction and expertise.

  • If a coding task doesn't work out of the box, just paste the error you get while running back into the AI chat, and do what it says, more or less mechanically, and that works many (but still not all) times.

  • If it still doesn't work, then actually look at the code and the result, and either fix it, or ask more specific questions where you suspect the issue might be.

  • Longer conversations around code (as with other things) tend to mess up the AI's memory of the conversation, so later responses around same code fail more and in sillier ways than earlier ones. Claude's artifact approach seems to work well to make this better - since edits are made to an existing file. ChatGPT's saved file also works, but I found it a bit clunkier - I have to give more explicit instructions to generate a file, more often than with Claude.

  • Multi (programming) language code seems to struggle more than single language code

• There is one downside to AI generating code - when most of the code is generated by AI, and we can (and being lazy, do) paste it into our editor and use it, changing one thing in the code gets harder. There is one important value that we lose when we use AI - when we type out the code or at least copying and pasting code one function at a time, that we tend to commit the overall structure and details to memory. This is useful while making changes, because we have a mental model of the structure of the code. If AI delivers us a complex piece of code, and we paste it into our project, now we do not know where and what the effects of a change we want could be. One way to improve on this is to use Test Driven Development with Code Coverage for AI generated code projects - have AI always give the tests, read and review and understand the tests, and if it looks good, get the code for it. Then add tests one (or as you get adept at this, a few) at a time, and have them pass before moving on. We can also extend this to higher levels of complexity using Behavior Driven Development, by having unit tests, functional tests and integration tests. As the complexity increases (from TDD towards BDD), just remember to break down the asks into steps, do not ask for all of it at once. This is not pedantic fandom of test driven development - this approach makes it possible to manage complexity one bit at a time particularly around human+AI coding, which would otherwise be easy to get lost in.

• I have become lazy - increasingly dependent on AI to give me the code I need

  • The things that matter to get the best out of human+ai, though very different from what we could do pre-ai coding, are improving:

    • how to phrase the question

    • settling on tech stack and its parameters upfront before diving into code

    • what parts of the design to specify vs what parts to leave to the AI to decide

    • when to ask AI and when to go elsewhere (or even actually code myself :-) )

    • when to break down the overall task into steps and manage externally when to ask for the whole thing in one shot

  • The things that don't matter as much, things that AI can do for me well, are being forgotten, which is a good thing.

• The most interesting things are on the edge between what AI can and cannot do well - this is where most effort is spent. Trying to get AI to do something that it struggles with, but seems to be within its reach. However, this is messy - I have spent a lot of time in this zone, trying to get AI to do something - the joy when it works is immense, as is the frustration when it doesn't. This is overall net positive, because even for cases when I fail, I have learnt to not ask AI for similar things in the future. Also, as new updates arrive, these edge cases are the ones to first try out.

• Education, testing, job interviews, etc. need to adapt to this new paradigm of human+AI for coding (assuming each of them embraces AI for coding in the first place) fast - unfortunately, I fear this change won't happen soon, or optimally.

In summary, I think availability of AI is rewiring our brains in coding (and several other tasks) - this seems scary, but then I take comfort in the knowledge that rewiring the brain is not new - happens all the time for all our lives - some positive, but a lot negative - when people, media and other systems mostly with personal agendas, manipulate our minds with carefully cherry picked, crafted, packaged, timed and placed messages to make us believe in what they want us to (love for their brand, fear from their warnings, importance of their entity, etc.). In this case, the rewiring is leading to a net positive for me (in terms of getting better at coding with AI to help, which then leads to benefit to others who use the fruits of this coding) and I believe it's all good.