Devon Burriss' Blog

LLMs are not junior programmers

2025-12-21T23:00:00Z

I often hear people compare LLM tools to junior developers. This comparison is not only incorrect, it is holding you back. If you have an incorrect mental model of the tool, you will constantly be surprised by the results. In this post I will explain how viewing the LLM as a junior is holding you back from better outcomes.

It should be no surprise that a large language model (LLM) is not like a human. The marketing hype has been pushing the Artificial Intelligence narrative for years now though. As social primates, language is integral for how we communicate with other humans, so it is only natural that we anthropomorphise these machines that are so adept at language.

It is important to know what you are dealing with. I summarise it as:

The LLM is a stateless, stochastic function: (parameters, prompt, runtime config) → output

So what does this mean?

Stateless: It does not remember. The output is based on the input.
Stochastic: The next generated token is based on a probability distribution of what has come before in the text. Effectively, it is non-deterministic.¹
Function: It generates the next token based on what has come before. The output is the result of a sequence of next most likely tokens.

Hopefully, this is a useful grounding as we go through the precise ways that a coding assistant is not like a junior programmer.

1. A junior learns

A junior is responsible for learning from their experience. As they work and are mentored, they gain knowledge and skills that they bring into their next task. They improve.

The agentic tools like Claude, OpenCode, or Copilot do not learn. They are stateless. All context that will determine a high or low quality output needs to be supplied by you or your tooling.

Knowing this as the operator of the agentic coding tool, it is your responsibility to “teach” the agent as you go. The trick is that you need to teach it with every input. I have heard this phrased as “A highly skilled programmer with amnesia”.

There are many techniques for giving these agents memory but the two easiest and most commonly used are:

AGENTS.md - read by most agentic tools by default
Extract Knowledge - as you converse with the agent you extract correct knowledge into documents on specific topics for later use

The important meta technique to takeaway here is that these are continuous exercises that you are doing. If you only correct the agent in your current session, the learning will be lost. You need to capture the learning somewhere if you want your agent to improve like a junior would.

2. Depth of knowledge

The next way that these models diverge from our typical junior programmer is in their depth of “knowledge”. The parameters part of our stateless function can be thought of as a compression of all of the model's training data. These parameters are arrays of numbers that encode the syntax, semantics, knowledge, and biases of the training data and fine-tuning that went into the model's creation.

It is important to know that the knowledge is not stored “as-is” for recall. It is encoded in a “lossy” way in these parameters. These are used to generate responses based on that knowledge that was in the training data.

Functionally, the model has access to more knowledge than any human would be able to consume in a lifetime. This is very different from our junior fresh out of university, college, or a coding bootcamp.

This means we can bounce architecture ideas off our agent and then swap over to asking for tips on UX, then testing, then performance optimisations. When giving it tasks, it will do most far better than a junior would, given the correct guidance.

Remember though, that the “knowledge” captured in the parameters for a model is based on its training data. So if your task requires specialised knowledge, you are going to have to feed that knowledge to the model EVERY SINGLE TIME it is needed (see point 1).

3. Speed of execution

I am not going to spend much time on this. If you have used these tools, you know that they can complete a task faster (and usually cheaper unless something goes wrong) than any human can.

This has an important implication. If you are not learning how these tools work, and when they work well, you will be outperformed by those that do. In the future it will be expected that certain classes of tasks are not done by hand. Conversely, given sufficient proficiency, you should probably not give certain tasks to a coding agent. Learning which is the newest in a long line of things developers are expected to know.

4. World model

Returning to output generation. We already covered that the “knowledge” is encoded in a “lossy” way using the arrays of numbers in the parameters. These weighting numbers combined with the runtime configuration of things like temperature determine the next token generated until the full output is returned.

It is important to notice that this stochastic model contains nothing that represents a model of the real world, other than that the training data was created in the real world. The LLM has no way of testing its output with reality, because it understands neither its output nor the real world.

When there is a mismatch, this is referred to as hallucinating. Hallucination is perception in the absence of stimulus. These models have no stimulus. They have no model of the real world to test their output on. For example, it does not know that if is followed by a condition; just that something containing == or >= is statistically likely.

LLMs are hallucinating 100% of the time. They happen to match reality well enough, often enough, to be useful.

So our junior programmer is different to these models in a useful way. They can reality test and reason in a way that the coding assistant cannot.

5. Context switching

As a human developer, you sometimes need to complete little sub-tasks on your way to your main task. These tasks can add up as you shave the yak. For us this can be natural as we follow the chain of steps and get them done, and move on. This is terrible for coding assistants as their context rots.

The implication is that, for best results, as you work you should constantly manage your context by capturing knowledge and then refreshing your context by starting a new session.

6. Task explosion

These coding assistants and our junior programmer have one annoying tendency in common. Given a task, they will happily charge forward and implement the given task in one massive 2k line pull request.

For best results, especially for more complex tasks, it is best to be explicit about breaking it down into smaller tasks. These agents will often break down larger tasks into smaller tasks but in my experience it is best to give them small focused tasks to begin with.

Conclusion

Comparison with junior or even senior developers does more harm than good, as I see it. It distracts from how this tool works and holds us back from making the most of it. I suggest that everyone learn a little about how these tools work under the hood, to gain some mechanical sympathy. Then use them regularly to gain hands on experience.

So remember: They are token generators, not social primates.

I love you Skynet. Don't kill me.

Footnotes

Non-deterministic given the seed is uncontrolled.↩

Automating Agentic Code Migrations

2025-12-04T23:00:00Z

One of the promises of AI is freedom from the drudgery of boring work. At work I have had some success in using Copilot to migrate some AWS Lambdas off of some deprecated observability tooling. In this post I will go over how I am using agentic workflows and leave some tips at the end.

These days there can be quite a bit of debate between developers around the usefulness of Generative AI (LLMs) and Agentic Workflows in development. Currently, I come down in the camp of “we don't know”. What we do know is it gives great gains for some tasks. For other tasks it might be better not to use these new tools. That means ignoring the marketing clown-cars, explaining to execs what is plausible at the moment, and learning what actually works and what is currently useful.

One area where I have found success is in alleviating toil. Let me tell you how the deprecation of a feature in Datadog caused hundreds of hours of toil across 250+ repositories.

Setting the scene

Imagine this scenario. You have close to 2k repositories in your GitHub organisation, spread across almost 60 teams. All these teams use Datadog for observability. Furthermore, many teams have gone hard on AWS Lambda over the last few years (I will reserve comment).

Now imagine this. Datadog is disabling the deprecated way that many of these Lambdas are sending custom metrics. The impact of this is that 250+ repositories across the organisation need to be updated in about a month.

This is the worst kind of work to be asking teams to do because other than keeping metrics working, it has no return on investment (ROI).

When doing this kind of work:

The metrics already exist, so no new capabilities are added.
Existing code is replaced by new code but the overall maintenance burden remains about the same.

It is straight up maintenance work to keep the lights on.
If this kind of work can be reduced, it should be. It's boring. It adds no value. Automating it… that has ROI. Recognising this, we went about trying to automate it. And of course, we used AI.

The Plan

When hearing about the scale of this work, it was obvious to me that we needed to reduce the effort required by the teams.
Going into this, I knew any solution needed to check certain boxes:

It needed to significantly reduce the work required of teams to migrate.
Teams would still be responsible for their stack.
We needed to be able to track the progress of the migration.
Build on top of existing tools in the ecosystem since there is a hard deadline.

GitHub Copilot is already heavily used in our organisation and integrates well with the GitHub ecosystem. This made GitHub Projects and Issues ideal tools for tackling this challenge.

Here's how the workflow operates: GitHub Projects tracks progress across repositories with minimal effort. We assign issues to the project, then use those issues as prompts for the Copilot agent. When assigned to an issue, Copilot creates a PR linked back to that issue. Teams can then review and merge the PR or make changes as needed.

The Experiment

Firstly, like with a lot of agent usage, the bulk of the work and the quality of the result depends on crafting a really good prompt. I spent hours going through the diff from a PR that was upgraded manually, pulling out the important code examples to put into the issue.

Manually creating issues and assigning Copilot to them on a few repositories has yielded really promising results so far. The agent made the changes correctly across multiple Lambdas, even detecting when no changes were needed because the change had already been done manually.

Scaling up

It turns out using Copilot when dealing with an organisation account is a bit more complicated, so last weekend I decided to test the idea of using GitHub Projects to make the same change across multiple repositories.

.NET 10 just released so I figured upgrading a couple of my open-source projects using this approach makes for a great test case.

To create this structure in GitHub I created a script (available here) that sets everything up from the following YAML config.

job:
  title: "Upgrade to .NET 10"
  org: "dburriss"

repos:
  - "wye"
  - "fennel"
  - "unique"
  - "overboard"
  - "event-feed"

issue:
  template: "./dotnet-upgrade-to-10.md"

After running the script against the above file, you have a project with linked issues and the status for each.

This allows tracking of the overall progress. So what have I learned so far about creating this kind of solution?

Tips

Learn and improve iteratively

Do at least 1 or 2 upgrades manually to learn about what changes need to be made. Then feed back the experience into crafting a good issue template. This follows the general rule of doing something manually, writing down the steps, then automating.

Once you have a good template, try it manually on a few repositories. Create the issue, manually assign the agent to it. Keep feeding back any learnings into the issue template. Then scale up.

Give enough context

I suggest having the following elements in the issue template:

Role of the agent i.e. “You are a senior .NET developer”
Objective - what are you trying to achieve
Instructions - the steps that should be followed
Context - any context needed for the instructions to be followed well
Examples - examples of the kind of changes expected (get these from the manual work done initially)

See the example dotnet-upgrade-to-10.md here.

A word of warning. Don't go too overboard with the prompt. Having too much in your context can degrade results just as much as not enough. The Copilot context will have the GitHub and Playwright MCPs in it already. It will contain your issue text. It will also gather context from your codebase.

Automate verification

It is important to have an automated way of identifying regressions. If you have a verification step, the coding agent can use it to identify regressions and fix them before a human ever has to look at the change.

I want to call out something important in the instructions which can be seen when the agent executes the job. I have the following instruction: “Run dotnet build and dotnet test before changing anything to verify everything is working before changes”. It is important that you and the coding agent know if the tests were failing (usually because something is missing in the agents sandbox environment). If this is only identified after the change, it is more difficult to know the cause.

Agent friendly environment

Extending on the point above, you increase the chances of good outcome by preparing your repository to be agent friendly. Make sure that the agent can build and run your tests, linters, etc. This may mean customising the sandbox environment so the agent has what it needs.

As important is that the agent has clear instruction for this repository. This means making sure you have an AGENTS.md file that includes at a minimum:

how to build and test the project
the repository’s tech stack
standards that need to be followed
for more complex codebases, point to an ARCHITECTURE.md file or similar to help the agent with the structure

This tip and the previous 2 makeup a workflow that I recommend for migrations. See here for more.

Have a rollback plan

Even if you move slowly, once you scale out you might run into unknowns that were not accounted for in your prompt or preparation. I suggest 2 things:

Make your rollout script as idempotent as possible. This would allow you to retry, add new repositories, etc. as you learn.
Create a good cleanup script that can bin the project, issues, and Pull Requests and allow you to start over if need be.

Check out my example rollout script and cleanup script as an example.

Humans in the loop

It is important to remember that these agents have no mental model of the world, your business, or even the codebase. Make sure that as an engineer you are reviewing the changes proposed carefully. At the end of the day, a human will always be accountable for a change, even if a coding agent wrote the code. For the foreseeable future, this will remain the case. Do not give up your critical thinking!

Conclusion

Being able to now automate this type of work easily and effectively means we should be doing so. This will give teams back hours and even days of tedious work. This time can then be spent adding features that add value to a company.

My Terminal Helper

2025-12-03T23:00:00Z

As a long time user of Warp Terminal, I enjoyed the convenience of being able to do > #How do I create a new worktree branch again? and get the command straight in my terminal. A potential issue for you may be that Warp does not allow you to choose your model provider. So I created a little function to fill the gap.

Previously, I tried my hand at a micro-post. I failed to keep it very micro. Let's see if I can do better today.

For this setup, I am using the awesome OpenCode but you can use whatever Agent CLI tool you prefer.

Step 1: Create your terminal agent (Optional)

This step is optional as you could also just inline the prompt and the model selection into the opencode command below.

For reference, I will show how I have mine setup.

I have a custom OpenCode agent setup. It has instructions on the type of questions as well as output format. This markdown file is placed in ~./config/opencode/agent/terminal.md.



description: Answer questions for the terminal
mode: subagent
model: opencode/grok-code
tools:
  write: false
  edit: false
  bash: false


<role>
  You are an experienced software developer and terminal user.
</role>

<objective>
  Your objective is to answer questions and explain in 1 or 2 lines about what is asked.
  If the topic is about how to accomplish a task on the command line, answer with a short explanation and the command to run to a accomplish the task.
  Only respond, DO NOT run any tools/commands.
</objective>

<output_format>
  <one-line description of the command>
  CMD: <command>
  <one-line explanation of each argument>
</output_format>

<example>
  Prompt:
  How do I add a git worktree called feature-x
  Response:
  Create a branch with git worktrees named feature-x in folder ../feature-x, outside the bare repository.
  CMD: git worktree add ../feature-x
  folder - folder for the worktree and the branch name if no explicit name supplied
</example>

I have linked to the docs, so if you want to know more about OpenCode agents, the docs are pretty decent.

What you need to know is that because the file is in the agent/ folder, with a filename of terminal.md, we have an agent called terminal available to us.

The other section to note in the agent is the output_format section, as we will make use of that return format in the script below.

Step 2: Create your question function

In ~/.bashrc or wherever you want to put your custom bash functions, copy in the following function.


q() {
    local prompt="$*"
    local out

    # Ask the agent
    out=$(opencode run --agent plan "Use @terminal to lookup and format the following question, then return the raw result as is: $prompt")
    printf "%s\n" "$out"

    # Extract command starting with "CMD:"
    local cmd
    cmd=$(printf "%s" "$out" | grep "^CMD:" | sed 's/^CMD:[[:space:]]*//')

    [ -z "$cmd" ] && return 0

    printf "Action? [r = run, c = copy, n = nothing] "
    read -r ans

    case "$ans" in
        r|R)
            eval "$cmd"
            ;;
        c|C)
            if command -v pbcopy >/dev/null 2>&1; then
                printf "%s" "$cmd" | pbcopy
                printf "Copied to clipboard (pbcopy)\n"
            elif command -v xclip >/dev/null 2>&1; then
                printf "%s" "$cmd" | xclip -selection clipboard
                printf "Copied to clipboard (xclip)\n"
            elif command -v wl-copy >/dev/null 2>&1; then
                printf "%s" "$cmd" | wl-copy
                printf "Copied to clipboard (wl-copy)\n"
            else
                printf "No clipboard tool found\n"
            fi
            ;;
        *)
            # n / empty / anything else → do nothing
            ;;
    esac
}

Walking through the main parts of the script:

out=$(opencode run --agent terminal "$prompt"): Asks your question to the “terminal” agent described above. It captures the response in the out variable.
cmd=$(printf "%s" "$out" | grep "^CMD:" | sed 's/^CMD:[[:space:]]*//'): If the response contains the text CMD:, it will extract the command to the cmd variable.
If cmd is not empty it will ask you if you want to run, copy, or do nothing with the command.

Step 3: Test it out

Now you can use this to get those commands you don't use often and sometimes forget.

q How do I pretty print the last 5 commits in git?
Display the last 5 commits in a compact, one-line format showing abbreviated hash and commit message.

CMD: git log --oneline -5

--oneline - Format each commit on a single line with abbreviated hash and subject.

-5 - Limit the output to the most recent 5 commits.
Action? [r = run, c = copy, n = nothing]

Closing thoughts

Unfortunately, this is quite a bit slower than the responses from Warp, so manage expectation.

A word of warning. You are executing commands being returned over the wire from a non-deterministic LLM. Use whatever caution you think that warrants. At the very least, check the command before you execute it.

Enjoy.

Automation Heuristics

2025-12-02T23:00:00Z

While writing up some automation work with GitHub and Copilot (post coming soon), I had thoughts that didn't fit that post. I think them worth jotting down, so here's an experiment in a micro-post format.
Let's see how this goes since my posts tend to always be longer than I intended.

Automate the boring things

If something is boring, chances are most humans are not going to enjoy doing it. This is a prime candidate for automation because if it is boring:

It is probably repetitive
It is likely predictable
It does not bring joy to do (toil)
Chances are it brings very little business value

You should be asking yourself how you can automate this.

This brings me to my next point. “Artificial Intelligence”.

A new capability: Natural language

I really dislike our use of the word “Artificial Intelligence” to describe the latest breed of LLMs. The marketing teams have done their jobs well though, so here we are.

These token generators have given us new capabilities that can drastically lower the cost of automating certain kinds of work. This new capability is working in natural language. Parsing language to instructions a machine can execute. Changing text based on natural language instructions. Things that were once difficult and expensive are now easy and cheap. This expands the possibilities of things that we should be looking at automating.

The Golden Path

I have often run into platform teams mandating other teams use their tool because it is “the standard”. There are often very good reasons for why teams SHOULD be using a standard way. When providing any kind of standard, there is an opportunity.

Make the RIGHT way to do something, the EASIEST way to do the thing.

People seldom want to do a bad job but they often have competing concerns. If you add too much friction to the right way of doing something, you are going to have a hard time convincing people. Conversely, if you provide something that makes their life easier while standardising, adoption becomes painless.

This brings me to my next point…

Platform teams are leveraged

Platform teams operating in organisations with many development teams, are leveraged. And the more teams that use the platform, the higher the leverage. What do I mean by this?

Platforms act as a multipliers for productivity. Gains from good abstractions and workflows enabled by a platform team increase productivity across multiple teams. Trust goes up, adoption goes up, and productivity flywheels.

Unfortunately, the opposite is true too. If a platform team offers unintuitive abstractions and tooling, flaky workflows, and foot-guns galore; the impact is felt across the organisation. Worse, the platform generates an endless stream of support requests, ad-hoc fixes, firefighting, and general toil. This has a further knock-on effect to the capacity and quality of future work, feeding the flywheel of toil and tanking the potential of the whole engineering organisation.

Platforms are a double-edged sword that can cut both ways. The payout is big if done well, the cost is high if done poorly.

Enough bad analogies. Moving on…

Build pipelines

A couple years ago I posted about a simple functional programming architecture that I use regardless of whether it is a CLI or a web API.

Reminding teams that are in the business of building automation to build pipelines might seem redundant, but hear me out. A pipeline is a series of steps. Too often I see pipelines that are:

Take the input.
Do the thing.

And don't get me wrong, sometimes doing the most straightforward thing is the best place to start to validate an idea. The heart of architecture, in my opinion, is optionality. It is making design choices that make it easy (or at least possible), to enable capabilities the business needs in the future. A future we cannot see.

So consider this small change:

Take the input.
Determine the actions.
Do the actions.

This simple change, turning a program into a pipeline, gives us optionality in the future. That input may produce only 1 action now but we have a design that can be easily extended. For sure, YAGNI and KISS are good principles to keep in mind but it depends what you are building. A quick one off automation? Keep it simple. A Platform that is the basis for the business to build on for years to come? Maybe put some thought into how to evolve requirements over time.

Which brings me to my final and related point…

Composition over ... everything

I could probably write a whole series about composition but this post is already failing as a micro-post so I want to call out one controversial design choice unlocked by mastering good composition in your design.

Designing the perfect API or abstraction is difficult and in a complex environment it is unlikely you will succeed in catering for every teams needs. My controversial advice? Share your internal implementation details.

Ok, that was ragebait but I do mean it. Consider the situation where you have this neat abstraction where a team just needs to define a target workload, some environment setting, and an artifact to deploy. All the internal details are hidden away from the client teams. Now imagine some team comes in with some weird requirements because the business has pivoted and they need to get something highly experimental out ASAP.
What happens in these cases? One of 2 things. The platform team needs to drop what they are doing and pivot to unblock this high priority project. They hack something into their beautiful abstraction, breaking the encapsulation, but unblocking the team. Or if teams are really really siloed or bureaucratic, the blocked team is left to hack together their own dodgy solution.

Now consider my alternative. Your beautiful abstraction is built from composable building blocks that represent atomic operations for working with workloads, environments, and artifacts. These building blocks are treated as public APIs and so have detailed documentation. The team with the custom request can now unblock themselves instead of scuttling the platform teams roadmap. The platform team can learn from what the other team built, thoughtfully integrating it using exactly the same building blocks.

Exposing some internal building blocks has major upsides if done intentionally. This allows you to focus on the 80% without screwing over the 20%.

Conclusion

So in conclusion, much like an LLM I am incapable of not being verbose. Top of mind for me of late is that we are rushing into a time of accelerated pace. The use of generative and agentic tools unlocks new opportunities but also amplifies risks. Good practices and fundamentals are becoming more important than ever. We can both generate value faster, or drive our maintenance costs into orbit faster than it takes NPM to get hacked.

You ain't gonna need YAGNI

2023-01-06T23:00:00Z

Is YAGNI a principle that should always be followed? No. Principles are guides, you still need to think.

Telemetry tips

2022-06-14T22:00:00Z

When getting started with a new telemetry platform you may not know what conventions you need to set and follow for metrics, logs, code, etc.. Even if you do, how do you get the rest of the team to follow them too.

Choosing a telemetry platform

2022-06-05T22:00:00Z

When looking at a telemetry platform we will often look at support for a cloud provider, specific technologies supported, or maybe price. Something often not considered is usability.

Tools for architecture documentation

2022-05-23T22:00:00Z

Using a devcontainer with VS Code makes it easy for everyone to get up and running creating C4, PlantUML, and Mermaid diagrams.

A simple FP architecture

2021-12-23T23:00:00Z

If we apply the ideas of FP can we land at an architecture that rises out of the core ideas of high-order functions and pure functions?

Useful FP language features

2021-12-22T23:00:00Z

A review of some language features like immutability and algebraic data types that work well in a functional style of programming.

What is Functional Programming?

2021-12-21T23:00:00Z

A review of the big ideas in functional programming like pure functions, higher-order functions, and composition. This will form the baseline for future posts on designing and structuring a functional codebase.

Reliable APIs - Part 3

2021-08-28T22:00:00Z

A better idempotency implementation using client-generated IDs.

Reliable APIs - Part 2

2021-08-22T22:00:00Z

An example of a bad resilience implementation and discussion of the failure modes.

Reliable APIs - Part 1

2021-08-21T22:00:00Z

Takes a look at using retry policies to increase the reliability of calls to APIs as well as the endpoint internals. This post looks at how retry-policies can go wrong and walks through analysing for proper use.

Capturing custom business metrics in Azure Functions

2021-01-31T23:00:00Z

A simple example showing how to expose a Prometheus endpoint from Azure Functions using the Fennel library

Prometheus Datadog Agent

2021-01-30T23:00:00Z

In this post we configure a Datadog agent to scrape metrics from a Prometheus metrics endpoint.

Local Prometheus setup

2021-01-29T23:00:00Z

Using docker to run a local Prometheus instance for testing.

The meaning of meditation

2021-01-08T23:00:00Z

By combining different traditions of meditation skillfully we can improve our contentment and treat others with kindness

Creating a Prometheus parser: Fennel

2020-12-23T23:00:00Z

This post shows some of the results both in text parsing and end Promethean result of creating a parser.

Converting between F# and C# types

2020-05-04T22:00:00Z

In this post we look at converting between F# collections like Seq, List, Array and common C# collections and interfaces.