When to merge duplicate code

Copies, shared code, or a checked master#

Imagine a product manual published in English and in French. The two editions must say the same thing, but they can’t be one text, because each reader needs their own language. There are three ways to keep them in step.

You can translate every edit by hand into both. Each edition is its own document, so every change has to be made twice, and one day it isn’t: someone corrects a step in the English manual, and the French one keeps the old step. That’s the missed update, the way duplication fails, and it’s what happened to httpx’s logging line.

You can publish one bilingual document, with both languages on every page. Every edit then reaches both readers, including the ones meant for one. Eventually the document fills up with notes like “French edition only” and “English readers skip this paragraph”.

Or you can write the English as the master, generate the French from it, and check before each release that the two match. The editions stay separate, nobody edits the French by hand, and the check catches the day they drift. The cost is a generator and a step in the release.

httpx’s two clients are in the same position as the two editions. They must behave the same, but a Python function is either sync or async, so they can’t literally be one text. The same three choices appear in code. Duplication risks missed updates. A shared abstraction risks collecting exceptions for callers that never quite agreed. Generation keeps separate copies but makes the rule that they must match something a machine checks.

Sandi Metz described the second failure in The Wrong Abstraction. A new requirement comes along that the shared code almost fits, and the next programmer, feeling bound to keep the abstraction, will “alter the code to take a parameter, and then add logic to conditionally do the right thing based on the value of that parameter”. After enough of those, nobody can read it. The line people quote, “duplication is far cheaper than the wrong abstraction”, comes from the talk that prompted her post. Her conclusion is about what to do once you’re there: “the fastest way forward is back”, which means inlining the shared code into its callers again.

So duplication is the cheaper mistake, but it is still a mistake when the copies are meant to match. The missed logging update shipped in a release.

”Premature” means the reason to share is still a guess#

The rule of three is usually traced to Martin Fowler’s Refactoring, which credits it to Don Roberts: do something once and just do it, do something similar a second time and wince but duplicate it, and refactor the third time. Kent C. Dodds’s version is AHA, “Avoid Hasty Abstractions”, with a rule he adds beside it: “Optimize for change first”.

I agree with both. Waiting for a third copy gives you time to learn, but the count itself doesn’t tell you what you learned. Two copies can express one rule that must change everywhere. Five copies can serve five callers whose needs are starting to diverge. What you want to know is whether the copies are one idea, or two ideas that happen to look alike today. One idea changes as one: when the rule changes, every copy changes. Two ideas drift: each changes when its own callers need something. An abstraction is premature when you build it before you can tell which you have.

Fowler’s Yagni narrows it further, and I think this is the sentence people skip: “Yagni only applies to capabilities built into the software to support a presumptive feature, it does not apply to effort to make the software easier to modify.” An abstraction that exists so the code is easier to change isn’t a premature feature. On a real-time media platform I architected, I put signaling behind a plugin contract before a second transport existed. When server-mediated telephony came later, it was a factory addition rather than a rewrite. That boundary came from a known source of variation, even though only one implementation existed at first. The premature abstractions are different: they merge two pieces of code because they look alike, and bet that they’ll keep changing alike.

You don’t have to bet. Whether two copies have changed alike is a fact about the past, and it’s sitting in version control. History can’t promise that callers will agree forever, but it can show whether past edits served one requirement or several. That is a better starting point than “these look the same”.

Ask which commits changed both#

The question is simple to state. Take two copies. For every commit since both existed, did it change both, or only one? Copies that keep showing up in the same commits are coupled whether or not they share code, and that pattern is called co-change.

httpx’s two request methods show it plainly. Each of these commits changed both copies, the same way, in one go:

Each row is one reason to change, and it reached both copies at once. The logging change from the opening is the same kind of commit with one half missing.

I wrote cochange.py to answer the question for a whole repository. It finds similar functions, traces their histories, and counts the commits each pair shares.

The first pass finds the copies. Every function with a body of six or more lines is turned into a stream of tokens, with every name, string and number replaced by a placeholder, so get(url) and head(link) come out the same:

1
def shape(src: str) -> list[str]:
2
    """The token stream with names, strings and numbers replaced, so `get(url)` and `head(link)` look the same."""
3
    out = []
4
    for tok in tokenize.generate_tokens(io.StringIO(src).readline):
5
        if tok.type == tokenize.NAME:
6
            out.append(tok.string if keyword.iskeyword(tok.string) else "ID")
7
        elif tok.type == tokenize.STRING:
8
            out.append("STR")
9
        elif tok.type == tokenize.NUMBER:
10
            out.append("NUM")
11
        elif tok.type == tokenize.OP:
12
            out.append(tok.string)
13
    return out

Keywords such as return and await stay as they are, because they’re part of what the code does. The token streams are compared with Python’s difflib ratio, which is twice the matching tokens divided by the total. Two functions at least 85% alike count as near-duplicates.

Replacing names and literals deliberately throws information away, so this finds code that looks alike, which is all it is meant to do. The next pass asks git where each candidate changed, with git log -L, which follows a range of lines back through every commit that changed it. Then it counts, and that is where it’s easy to get wrong.

Start after both copies existed#

Run git log -L on httpx’s sync _send_single_request and the trace reaches back to April 2019, long before the method had its current form. -L follows lines, not functions, so it keeps tracing those lines back into whatever code they used to belong to. The count therefore has to start when both copies existed.

Count from where the trace starts and you get nonsense: commits that look like changes to “only the async copy”, made at a time when there was no sync copy to change. A change made while there was one copy isn’t a one-sided change; it’s just a change. So the script applies two cuts:

1. Each function’s history stops at the first commit, going back, where the traced lines weren’t mostly inside a function of the same name.
2. A pair’s commits only count strictly after the younger copy’s first commit. Creating that copy is not a later change to it.

Here is the second cut as the script applies it to a pair:

1
def count_pair(git: Git, x: Function, y: Function) -> PairCount:
2
    hx, hy = git.history(x), git.history(y)
3
    ...
4
    birth_x = min(hx.values(), key=lambda c: c.ts)
5
    birth_y = min(hy.values(), key=lambda c: c.ts)
6
    since = max(birth_x, birth_y, key=lambda c: c.ts)
7
    after = lambda h: {s for s, c in h.items() if c.ts > since.ts}  # noqa: E731 - strictly after: the birth is not a change
8
    ax, ay = after(hx), after(hy)
9
    ...
10
    return PairCount(
11
        since=since,
12
        together=by_time(ax & ay),
13
        only_first=by_time(ax - ay),
14
        only_second=by_time(ay - ax),
15
        ...

Each copy’s history starts at its oldest commit after the first cut. The pair starts at the later of the two, and the > means the commit that made the younger copy doesn’t count as a change to it. The lines left out set sweeping commits aside, which comes up below.

For this pair, the first cut moves the start to July 2020, at e107e0f, the commit that gave both methods their current name.

httpx at b5addb6
httpx/_client.py:Client._send_single_request  ~  httpx/_client.py:AsyncClient._send_single_request
  99.6% alike, 34 and 33 lines
  history stops where each copy got its name: 22 earlier commits traced other code
  both exist from e107e0f 2020-07-17 Tighten public client methods (#997)
  after that: together 15, only first 1, only second 6
  left out: 1 commit from before both existed, 1 sweeping (over 20 files)
    ...
    only first  f1157db 2023-03-20 Use standard logging style (#2547)
    ...
    only second 4b5a92e 2023-04-12 set logging request lines to INFO level for async method also (#2656)

The two copies are 99.6% alike, not 100%, because the async copy’s await is one token the sync copy doesn’t have. Since e107e0f, 15 commits changed both, 1 changed only the sync copy, and 6 changed only the async one.

The cuts matter far more than that small difference in similarity. The first cut removed 22 commits where the traced lines were other code. The second left out 1 more, e107e0f itself, where the names were set. Put those back and count from where git’s tracing starts, April 26, 2019, and the same pair reads 29 together and 15 one-sided (4 sync, 11 async). The counted history doubles, from 22 commits to 44, and the extra half is code that wasn’t either method.

The script also sets aside sweeping commits, any commit that touched more than 20 files. Here that removes one 29-file change to how httpx configures TLS. It touched both copies, but it would have touched them whether or not they belonged together.

Read the seven one-sided changes#

The count only tells us where to look. A commit that changed one copy is either a missed update (the other copy needed it too) or a sign the two do different jobs (it only ever belonged in one). Only the diff says which, so I read all seven:

Four of the seven were the copies falling out of step and being pulled back, or not. Three were cosmetic. None of them gave one method a job the other didn’t have. The similarity score said these two look alike; the history says they are one idea written twice.

The same check on another pair tells a similar story. A transport in httpx is the object underneath a client that actually opens connections and sends bytes, again one sync and one async. The two transports’ __init__ methods read 12 together and 4 one-sided, and all four one-sided changes are the same kind of miss. Passing the proxy’s TLS settings through, part of HTTPS proxy support, reached the sync transport in September 2023 and the async one eight months later.

One commit counted as “together” shows the limit of counting. fd60b18, a formatting change, split a long error message in both transports and kept the f that makes it an f-string in only one. The async transport’s error then printed {proxy.url.scheme!r} literally until 336204f, 18 months later. Both copies changed in the same commit; they didn’t change the same way.

The maintainers know all this, and they manage it. In January 2024, #3042, “Keep clients in sync”, says it plainly: “Except for the async/await syntax, AsyncClient and Client are supposed to be the same.” Its author found where they had drifted the way you’d expect: “I isolated them into separate files and used git diff --no-index --word-diff asyncclient.py syncclient.py to find and fix differences.” That is a person doing by hand the check that the third answer automates.

Similar bodies, different reasons to change#

Now the opposite case. httpx has module-level functions, httpx.get(url), for one-off requests, and Client.get for code that keeps a client around. Both build a request and pass it on. Their bodies are 87.8% alike, above the 85% threshold, so by the rule of three they’re overdue for merging. Their history says otherwise:

httpx/_api.py:get  ~  httpx/_client.py:Client.get
  87.8% alike, 34 and 28 lines
  history stops where each copy got its name: 11 earlier commits traced other code
  both exist from ee37a76 2020-01-08 Reintroduce sync API. (#735)
  after that: together 3, only first 5, only second 7
  left out: 12 commits from before both existed, 2 sweeping (over 20 files)
    ...

Three commits touched both, and twelve touched only one. The twelve make sense once you think about the callers. A one-off request has nowhere else to put its configuration, so the module-level function gained proxies, then proxy, then a reworked verify. A client already holds defaults, so its method gained a “use the client’s default” marker, an extensions parameter and auth=None.

Merge them and you get the bilingual manual: one function with a parameter for whichever kind of caller it’s serving today. Keeping them apart gives each job its own home. The twelve one-sided changes are the reasons for that separation, and an 88% match doesn’t outweigh them.

What happens across the repository?#

Run the script over all of httpx:

httpx at b5addb6: 201 functions of 6+ lines under httpx/
  128 near-duplicate pairs (>= 85% alike), in 28 clone groups
  ...
clone groups: changes to every existing member / to only some of them
  G1  12 members  consistent 11, partial 20  (4 sweeping left out)
  ...
  G12  2 members  consistent 15, partial 7  (1 sweeping left out)
        httpx/_client.py:AsyncClient._send_single_request
        httpx/_client.py:Client._send_single_request
  ...
  all groups: 232 consistent, 73 partial (24% partial), 38 sweeping left out

It finds 128 near-duplicate pairs among 201 functions. Pairs overcount: a family of five copies makes ten pairs, and one commit that changes all five would be counted ten times. So the script also joins connected pairs into clone groups, 28 of them, and counts each commit once per group. One commit against one group is a group change. It is consistent if it changed every copy in the group that existed by then, and partial if it changed only some. “Consistent” only says every copy was touched.

Across httpx, 232 group changes were consistent and 73 partial, about a quarter. G12 is our request pair again, with the same 15 and 7.

That 24% is a share of group changes, and some of them are on purpose. To see what was inside it, I drew 20 of the 73 partial changes at random and read each diff:

All nine “on purpose” changes were in the two big groups where the near-duplicate threshold had joined the module-level functions to the client methods. That is the httpx.get and Client.get story, nine times over. The six out-of-step changes include a missing http1 argument in the async client’s proxy setup, and an empty-chunk fix that reached one decoder 22 months after its twin had it. Twenty is enough to show several different causes, but too few to read 6 in 20 as a rate.

The research agrees, and it’s worth knowing which study says what. Jens Krinke studied five open-source systems in 2007 and found that “usually half of the changes to code clone groups are inconsistent changes” (what the script calls partial), and that once a group goes inconsistent, “it is rarely the case” that later changes bring it back. Juergens and colleagues (ICSE 2009) went through over 700 inconsistent clone groups with the systems’ own developers and concluded that “nearly every second unintentionally inconsistent change to a clone leads to a fault”. Kim and colleagues (ESEC/FSE 2005) found many clones short-lived: of those that disappeared, 48% to 72% did so within an average of eight check-ins, so “aggressive, immediate refactoring may not be necessary”.

These findings aren’t in conflict. Some copies are one idea, and diverging is a bug. Others are two ideas, and diverging is the point. Co-change is how you tell which you have.

One project family uses all three answers#

httpx and httpcore, the library underneath it that handles connections, come from the same organisation. Between them they use all three answers from the manual.

Share what doesn’t vary. httpx’s BaseClient holds what both clients do the same way and never waits on the network: merging URLs, headers and cookies, building requests, and working out where a redirect goes. Both clients inherit it. That’s the bilingual document, kept to the part that reads the same for every caller, the way a manual’s diagrams and part numbers need no translation.

Copy what can’t be shared. The methods that do I/O are written twice, sync and async, because in Python a function is either one or the other. That is a deliberate trade, and #3042 and the history above show its cost being paid.

Generate one and check it. httpcore has the same problem and answers it differently. Its async code is the master, and scripts/unasync.py writes the sync copy from it by rewriting async def to def, deleting await, turning aclose into close, and so on. Run with --check, it writes nothing and exits with an error if any line of the sync files differs from what it would have written. The project’s scripts/check ends by running it. The copies stay separate, nobody edits the sync one by hand, and a missed update fails the check instead of shipping.

That third answer is the one I’d reach for more often than people do. On a media platform where I designed the developer toolchain, the agent surfaces, the way each AI agent runtime sees the toolchain, had to exist for three different runtimes, and we generated all three from one source definition. A generator like that is usually a small script, and a check that its outputs are current is one more step in CI. Unlike an abstraction, it doesn’t ask you to predict how the copies will change.

Decide from the reason for the differences#

Put together, the decision isn’t about how many copies there are. “Enough history” is a judgement. The tool below calls fewer than five changes since both existed too young to tell, which is a practical cut-off rather than a statistical one. More history only helps if it really belongs to these functions and you have read what changed.

Read “changed together” as a question about intent as well as commits: a formatting change that touched both says nothing about a shared rule.

I’ve had to make this kind of call at both ends. On a large communications platform, calling was implemented separately for each device platform and was heading towards eight parallel implementations. I set the strategy that converged them onto one write-once architecture. In a media contract across three operating systems, I drew the line the other way on purpose: a narrow portable interface where portability was achievable, and native code per OS where compositor and graphics differences made a shared abstraction unsound.

Counting copies didn’t decide either one. Both came down to where the platforms really agreed and where each needed room to change on its own, which is the question co-change asks of your history.

Try it on a pair you already suspect#

For a whole Python repository, the script lists every near-duplicate pair and clone group with its counts. For one pair, it lists every commit:

python3 cochange.py path/to/repo src
python3 cochange.py path/to/repo src --pair src/a.py:Client.get src/b.py:AsyncClient.get

For one pair you already suspect, in Python, JavaScript or TypeScript, or the C family of languages, there’s a browser tool. Get each copy’s history with its current first and last line numbers:

git log --no-patch --format='%h %cI %s' -L <first>,<last>:<file>

Paste the two functions and the two histories, and press Run. The tool leads with how many commits changed only one copy, draws the three counts as bars, and lists every one-sided commit, newest first, so you can read them. Below that are where the two differ today, line by line, which differ only in names and which in what they do, and how alike they are with names set aside. It reads the language from the code and compares bodies only, at the script’s 85%. The “Left out” line in its detail is the script’s two cuts added up: 22 commits of other code plus e107e0f makes 23.

Its first sample is the httpx request pair from this post, counted from e107e0f, and Run gives the 15 together, 1 sync-only and 6 async-only from above. Then clear Count from and run it again to watch the counts jump to 29 together and 15 one-sided (4 and 11). Ignore comes filled with 8e36f2b, the 29-file TLS change the script sets aside on its own. Clear that too and it counts as one more change to both, 30 together. Put e107e0f back in Count from, and you’re at 16 together, with only the TLS change added:

The tool reads tokens rather than fully parsing each language, and its one rule, that fewer than five changes is too early to tell, is stated rather than tuned. It can’t tell whether a one-sided change was a bug, whether a shared change did the same thing to both copies, or whether a commit swept the whole repository unless you name it in Ignore.

The script, its tests, the hand-checked sample and the CI that reruns all of it against httpx at a pinned commit are in premature-abstraction/.

httpx’s logging line took 23 days and a second contributor to reach its twin. So the next time someone in a review says “these three look the same, let’s pull them together”, don’t count them. Ask git how they’ve been changing, and read the changes that went to only one copy, because that is often where the design decision is already written down. If they move as one, merge them, or generate one from the other. If they don’t, they were never the same thing.