↓Skip to main content
  1. Blogs/

On Programming with LLMs

··5 mins
Table of Contents
Always understand your code.

Initial Thoughts #

My favorite way of thinking about LLMs is as translators. In programming tasks, the goal is to translate natural language into raw code. The more similar examples an LLM saw in training, the more robust it will be at solving that task. The impressive part is that there is a large mounting of evidence that LLMs represent world states such as location or game board states in its parameters1, so as we continue to increase the number of training examples, I believe their ability to solve problems in dimensions between well known problems will increase (Effectively, interpolating between known problems).

For this reason, LLMs are superb in tiny standalone projects. When the context is small enough and requirements specific enough, you can one-shot entire classes or web pages. The next decision then is when to start a new chat to prevent the context window from growing out of control. This involves creating a fresh chat and pasting in the current error.

My favorite thing to do is to utilize a difference checker when updating files - it highlights exactly what the LLM changed within the file, allowing me to very clearly read and understand what is changing. For this, I usually use this website2, but any similar tool will work. It would be cool to see this done automatically or some sort of tool to speed up the process of pasting in the code into such a tool!

Keep in mind: Almost always, asking an LLM should I do X? results in a positive response, asking an LLM should I *not* do X results in a negative response.

Large Projects #

When working on React projects or, for example, a large Unity project, you are left with tens of files that could be possibily relevant to your query. Here are some techniques for handling these projects with a large context window:

  • For help with a specific error - paste in relevant files using your expert judgement. After spending too much time hitting ctrl+A ctrl+C ctrl+V, I created this tool3, which allows you to upload an arbitrary number of files, and it will format the contents into an easily copy and pastable chunk to prompt with.
  • If your project has grown over the course of the chat, at the end of a chat, say Summarize our project and the latest of what we've done into one output that I can use to reprompt to keep the context window short. Take this output to a new chat, and either paste in relevant files - or use the next tip:
  • In a new chat, say Do not output code yet, just tell me which files you need to debug this issue.

You can get as creative as your mind allows with this process!

Have a singular focus in each chat - it helps keep the LLM on track.

Caveats #

One problem I foresee with the use of LLMs is specificity. As an example, I was working on adding Firebase Authentication flow to my Unity game, and I encountered a strange error: Unable to load DLL 'native-googlesignin'. The specific error is not the point, rather how niche of a problem it is. As APIs and technologies continue to rapidly change, it will be incredibly difficult to keep LLMs populated with up to date information. As for this specific error, I was lucky to find a small stackoverflow thread4 mention it, with specific instructions from someone who had faced the same issue as me.

My point is, for common issues on popular tech stacks, LLMs are superb. As you move into the long tail of more niche scenarios and tech stacks, the amount of training data and subsequently quality of answers can only go down.

Another thing to keep in mind is the bias towards popularity. For instance, if you ask an LLM to create a website for you, I can almost guarantee it will attempt to guide you to use the React framework, even if the task does not necessitate a complicated framework, purely because of the sheer amount of React training examples it ingested. As long as this is true, LLMs cannot be your oracle for larger trends in programming - keep in mind their limitations.

Beyond that, if you were a company hosting your own LLM, for example Microsoft running Copilot, it would certainly be in your own interest to for it to reccomend Azure services when the user asks the LLM how to host their project. You could completely vertically integrate a user into your whole product suite without them even realizing it, and the LLM would guide them every step of the way! Beyond your own products, this opens up the possibility for an even more insidious form of advertising. You could offer other companies a service to boost salience of their products in your LLM, such that when a user asks for shoe reccomendations, for instance, it suggests Nike instead of Adidas.

Parting Thoughts #

  • As is commonly said, LLMs are a tool which require a careful hand to apply.
  • LLMs are very sensitive to your input, and can be easily swayed by your choice of words
  • Generated code can look very convincing. It’s up to you to test it to ensure problems don’t accumulate down the line.

  1. Language Models Represent Space and Time ↩︎

  2. Diff Checker, © 2024 Checker Software Inc. https://www.diffchecker.com/ ↩︎

  3. Justin Dachille, “LLM File Formatter,” 2024. https://justindachille.github.io/LLM-File-Formatter/ ↩︎

  4. https://stackoverflow.com/a/79465697 ↩︎