What I Learned from Reimplementing 40 Multi-Agent LLM Papers

7 min read3 days ago--I recently re-implemented the core workflow of 40 published multi-agent LLM papers as runnable reference scripts on h5i-python.Here is what re-implementing all of them taught me.1. Most of these papers are ~100 lines of control flowStrip away the evaluation sections and the framework advertising, and the median paper’s contribution is: a loop shape, a prompt discipline, and an aggregation rule. Multiagent debate is “answer independently, then read the others and update, then vote”. Skeleton-of-Thought is one planning call and one parallel gather . Even MetaGPT’s standard-operating-procedure, once you have a work primitive and typed documents, is a hundred lines.This is not a criticism. Small algorithms with real effects are the best kind. But it changed how I read new agent papers: I now look for the loop shape and the aggregation rule first, and I’m suspicious when a paper can’t be summarized that way.# Example of "More Agents Is All You Need (Li et al. 2024, arXiv:2402.05120)"def parse_answer(value: Any) -> str: if not isinstance(value, Mapping) or "answer" not in value: raise ValueError('reply must be {"reasoning": "...", "answer": "..."}') return str(value["answer"]).strip()def canon(answer: str) -> str: return answer.strip().lower()def majority(samples: list[str]) -> tuple[str, int]: winner, n = Counter(canon(s) for s in samples).most_common(1)[0] return winner, nasync def main(question: str) -> None: prompt = ( f"{question}\n\nReason step by step, then reply as JSON: " '{"reasoning": "<your reasoning>", ' '"answer": "<final answer, as short as possible>"}' ) async with Conductor(".", "agent-forest-demo", launcher="resident", isolation="supervised") as c: forest = [ await c.hire(f"tree{i}", runtime="claude", model="claude-haiku-4-5") for i in range(N_AGENTS) ] # The whole method: N independent samples, then vote.

samples = list( await asyncio.gather(*(seat.ask(prompt, parse=parse_answer) for seat in forest)) ) # The scaling curve, from prefixes of the same sample set. for n in range(1, N_AGENTS + 1, 2): winner, votes = majority(samples[:n]) print(f"ensemble size {n}: '{winner}' ({votes}/{n} votes)") winner, votes = majority(samples) await c.note(f"agent-forest: '{winner}' won {votes}/{N_AGENTS} votes") print(f"\nfinal answer: {winner}")2. Independence is the load-bearing invariantA surprising number of papers silently depend on samples being independent: self-consistency’s vote is meaningless if the samples saw each other; CodeT’s “dual execution agreement” requires tests written blind to the implementations; Chain-of-Verification’s best variant (“factored”) requires the verification questions to be answered by a context that never saw the draft.In single-context frameworks, this invariant is easy to break by accident: one shared history and your five “independent samples” are five paraphrases. If you build agent infrastructure, make independence a first-class, checkable property. It’s the invariant the most robust methods lean on. For example, h5i-python offers sandboxed workspaces for agents, which makes it easier for users to implement sealed environments.3. The refine-loop family is one loop with different feedback sourcesSelf-Refine, Reflexion, CRITIC, Self-Debug, and Constitutional AI are the same two-step loop, attempt, then revise against feedback, and differ only in where the feedback comes from:Self-Refine: the same model, critiquingReflexionthe: model’s own reflection on a failure signalCRITIC: external tool outputSelf-Debug: the model explaining its own code line by lineConstitutional AI: critiques against written principlesImplementation-wise, the useful move was making feedback a first-class object: whatever its source, it becomes a structured review the agent revises against, and every retry is a recorded turn. Once feedback is an object, the five papers are one function with a parameter.

The rubber-duck detail in Self-Debug is worth stealing: before fixing, the model must explain what its code actually does, not what it was meant to do. The paper reports this helps even with no error message at all, which is a strong claim about where the signal actually lives.4. Prefer execution over opinion, and never verify in the author’s environmentThe methods that hold up best replace an LLM’s opinion with a real signal wherever one exists: CRITIC runs tools, AgentCoder runs tests, LATS uses execution results as its search reward, CodeT ranks candidates by whether they pass a blind test suite. The LLM-judge papers themselves are mostly about compensating for the absence of such a signal — with persona diversity (ChatEval), verifier count (Multi-Agent Verification), or peer weighting (PRD).One engineering detail matters more than it looks: verification has to be a neutral re-execution — apply the candidate to a fresh sandbox and run the command there — never “the agent says its tests pass.” Every test-driven paper (AgentCoder, CodeT, AlphaCodium, MapCoder) assumes this property, usually implicitly.5. Aggregation rules are ten lines and decide everythingMajority vote, confidence-weighted vote (ReConcile), mean-of-judges (ChatEval), approval counting across aspect verifiers (BoN-MAV), win counts over pairwise comparisons (LLM-Blender), dual execution agreement (CodeT): each is about ten lines of host-language code, and each is the actual difference between papers that otherwise share a skeleton.Two practical notes. First, pairwise comparison beats absolute scoring, but only if you present each pair in both orders and count a win only when it survives the swap — position bias is real and this is the cheap fix (LLM-Blender and PRD both need it). Second, keep the aggregation in ordinary code, not in a prompt. It’s the part you’ll want to unit test.6. The debate family is a visibility functionMultiagent debate, MAD, ReConcile, and Exchange-of-Thought differ mainly in who sees whose messages between rounds.

EoT makes this explicit with four topologies, and the entire difference fits in one function:def visible_peers(topology: str, me: int, n: int) -> list[int]: if topology == "memory": # bus: everyone return [j for j in range(n) if j != me] if topology == "report": # star: spokes <-> hub return [j for j in range(n) if j != me] if me == 0 else [0] if topology == "relay": # ring: predecessor only return [(me - 1) % n] if topology == "debate": # tree: siblings pair, root hears all return [j for j in range(n) if j != me] if me == 0 else [me % 2 + 1]Everything else — the rounds, the update prompt, the final vote — is shared. Add a stopping rule (unanimity, judge ruling, or confidence streaks) and an aggregation rule, and you can generate most of the debate literature by picking one item from each column.7. Dynamic team structure needs mutable state, not a workflow graphTwo papers were the strongest argument for define-by-run orchestration over static workflow graphs. DyLAN prunes the team as it goes — a ranker scores each round’s contributions and the weakest agent is deactivated, which in code is literally active.remove(weakest). AgentVerse goes further: an evaluator can reject the result, dissolve the team, and recruit a differently-shaped one, which requires hiring new agents mid-run based on LLM output.Neither fits a DAG you compile up front, because the structure is computed by the workflow itself. In plain Python both were trivial.8. Search over thoughts is cheap; search over attempts is notTree of Thoughts, Graph of Thoughts, and LATS form a cost ladder. ToT and GoT search over text — proposing and scoring thoughts are cheap data turns, and only the winning plan pays for real implementation work. LATS searches over actual attempts, where every tree node is a real submission plus a real test run. That’s an order of magnitude more expensive per node, and it buys you a reward signal that isn’t an opinion.The practical recipe that emerged: search broad over text, commit narrow over execution.

ToT-style planning to pick a direction, then a LATS-style loop only on the final candidate.9. Validate at the turn boundaryEvery structured reply in all 40 scripts goes through the same mechanism: the agent must return JSON, a parser validates it (and checks semantic constraints — “you scored a candidate that doesn’t exist”, “your citation ids aren’t in the evidence”), and a failed parse triggers a bounded re-ask with the error attached. This single pattern eliminated the majority of flakiness. LLM-judge outputs especially need the semantic checks: judges will cite evidence that doesn’t exist, and a validation loop that rejects hallucinated citations is cheap insurance.10. Forty papers is really about eight papersThe honest meta-lesson. When I selected the second batch of 20, the hardest part was finding papers that weren’t already implied by the first batch. Several well-cited works reduce to compositions of others: a review-cycle plus a judge panel covers whole “author/reviewer/meta- reviewer” frameworks; the EoT topology function subsumes the communication-topology papers; multi-persona single-model methods are the role-playing papers with fewer sessions.Again, not a dig — consolidation is what a healthy field looks like from inside. But if you’re building products rather than publishing: implement the eight families, parameterize them, and you have coverage of most of the literature. The families, roughly: refine loops, sample-and-vote, debate-with-a-visibility-function, judge panels, generative fusion, text-search-then-commit, staged pipelines (software and long-form writing), and dynamic team management.What didn’t mapFor fairness, the categories I deliberately excluded: anything requiring training or weight access (RLHF-style methods, learned rankers — I implemented LLM-Blender’s PairRanker as a prompted judge instead), methods needing token-level logits, environment-heavy simulations (Generative Agents), and the workflow-search meta-papers (AFlow, ADAS, GPTSwarm) — those treat everything above as their search space and are projects in themselves, not afternoon scripts.Also, one honest engineering caveat: two paper mechanics needed workarounds. Self-Refine wants a model to review its own artifact, which the engine forbids for provenance reasons (solved: a second seat pinned to the same model).