Thoroughly Understanding C++ ABI

This article was translated by AI using Gemini 2.5 Pro from the original Chinese version. Minor inaccuracies may remain. Application Binary Interface, or ABI as we commonly call it, is a concept that feels both familiar and unfamiliar. Familiar in what sense? It’s often discussed when troubleshooting, frequently mentioned in articles, and sometimes we even have to deal with compatibility issues it causes. Unfamiliar in what sense? If someone asks you what an ABI is, you’ll find that you know what it’s about, but describing it in precise language is quite difficult. In the end, you might just resort to saying, as WIKI does: an ABI is an interface between two binary program modules. Is there a problem with that? No, as a general description, it’s sufficient. But it can feel a bit hollow. This situation is not uncommon in the field of Computer Science. The author encountered the exact same situation in a previous article discussing reflection. Fundamentally, CS is not a discipline that strives for absolute rigor; many concepts lack strict definitions and are more often conventional understandings. So, instead of getting bogged down in definitions, let’s look at what these so-called binary interfaces actually are and what factors affect their stability. CPU & OS The final executable file ultimately runs on a specific operating system on a specific CPU. If the CPU instruction sets are different, it will certainly lead to binary incompatibility. For example, programs on ARM cannot run directly on x64 processors (unless some virtualization technology is used). What if the instruction sets are compatible? For instance, x64 processors are compatible with the x86 instruction set. Does that mean an x86 program can definitely run on an x64 operating system? This is where the operating system comes into play. Specifically, factors such as Object File Format, Data Representation, Function Calling Convention, and Runtime Library must be considered. These points can be regarded as ABI regulations at the operating system level. We will discuss the fourth point in a dedicated section later. Below, taking the x64 platform as an example, we will discuss the first three points. x64, x86-64, x86_64, AMD64, and Intel 64 all refer to the 64-bit version of the x86 instruction set. There are two main common ABIs on the x64 platform: Windows x64 ABI for 64-bit Windows operating systems x86-64 System V ABI for 64-bit Linux and various UNIX-like operating systems Calling a function from a dynamic library can be simply viewed as the following three steps: Parse the dynamic library according to a certain format. Look up the function address from the parsed result based on the symbol name. Pass function parameters and call the function. Object File Format How to parse a dynamic library? This is where the ABI’s regulations on Object File Format come into play. If you want to write your own linker, the final executable file must meet the format requirements of the corresponding platform. Windows x64 uses the PE32+ executable file format, which is the 64-bit version of PE32 (Portable Executable 32-bit). The System V ABI uses the ELF (Executable Linkable Format) executable file format. By using parsing libraries (or writing your own if interested), such as pe-parse and elfio, to parse actual executable files and obtain their symbol tables, we can get the mapping between function names and function addresses. Data Representation After obtaining the function address, the next step is how to call it. Before calling, parameters must be passed, right? When passing parameters, special attention must be paid to the consistency of Data Representation. What does this mean? Suppose I compile the following file into a dynamic library: struct X{ int a; int b; }; int foo(X x){ return x.a + x.b; } Then, a subsequent version upgrade changes the structure content, and the structure definition seen in the user’s code becomes: struct X{ int a; int b; int c; }; And then it still tries to link to the dynamic library compiled from the old version code and call its function: int main(){ int n = foo({1, 2, 3}); printf("%d\n", n); } Will it succeed? Of course, it will fail. This type of error can be considered a so-called ODR (One Definition Rule) violation. More examples will be discussed in later sections. The above situation is an ODR violation caused by the user actively changing the code. But what if I don’t actively change the code, can I ensure the stability of the structure layout? This is guaranteed by the Data Representation in the ABI. For example, it specifies the size and alignment of basic types. Windows x64 specifies long as 32 bits, while System V specifies long as 64 bits. It also specifies the size and alignment of struct and union, and so on. Note that the C language standard still does not specify an ABI. For the System V ABI, it is primarily written using C language terminology and concepts, so it can be considered to provide an ABI for the C language. The Windows x64 ABI does not have a very clear boundary between C and C++. Function Calling Convention Next, we come to the step of passing function parameters. We know that a function is just a piece of binary data. Executing a function simply means jumping to the function’s entry address, executing that piece of code, and then jumping back when finished. Parameter passing is nothing more than finding a place to store data, so that this location can be accessed to retrieve data both before and after the call. What locations can be chosen? There are mainly four options: global (global variables) heap (heap) register (registers) stack (stack) Using global variables for parameter passing sounds magical, but in practice, when writing code, parameters that need to be passed repeatedly, such as config, are often changed to global variables. However, it’s clear that not all parameters are suitable for global variable passing, and thread safety needs to be considered even more carefully. Using the heap for parameter passing also seems incredible, but in fact, C++20’s stackless coroutines store coroutine states (function parameters, local variables) on the heap. However, for ordinary function calls, if dynamic memory allocation is required every time parameters are passed, it is indeed a bit extravagant. So we mainly consider using registers and the stack for parameter passing. Having more options is always good, but not here. If the caller thinks parameters should be passed via registers, it stores the parameters in registers. But the callee thinks parameters should be passed via the stack, so it retrieves data from the stack. Inconsistency arises, and it’s very likely that garbage values are read from the stack, leading to logical errors in the code and program crashes. How to ensure that the caller and callee pass parameters to the same location? I believe you’ve already guessed: this is where the Function Calling Convention comes into play. Specifically, the calling convention specifies the following: Order of function parameter passing: left-to-right or right-to-left? Method of function parameter and return value passing: via stack or registers? Which registers remain unchanged before and after the caller’s call? Who is responsible for cleaning up the stack frame: the caller or the callee? How to handle C language variadic functions? ... In 32-bit programs, there were many calling conventions, such as __cdecl, __stdcall, __fastcall, __thiscall, etc., and programs at that time suffered greatly from compatibility issues. In 64-bit programs, unification has largely been achieved. There are mainly two calling conventions, those specified by the Windows x64 ABI and the x86-64 System V ABI respectively (though they don’t have formal names). It needs to be emphasized that the function parameter passing method is only related to the calling convention, not to the code optimization level. You wouldn’t want code compiled with different optimization levels to fail when linked together, would you? Introducing specific regulations can be tedious. Interested readers can refer to the relevant sections of the corresponding documentation. Below, we mainly discuss some more interesting topics. Note: The following discussions only apply when function calls actually occur. If a function is fully inlined, the act of passing function parameters does not happen. Currently, C++ code inlining optimization mainly occurs within the same compilation unit (single file). For code across compilation units, LTO (Link Time Optimization) must be enabled. Code across dynamic libraries cannot be inlined yet. Passing struct values smaller than 16 bytes is more efficient than passing by reference. This statement has been around for a long time, but I’ve never found the basis for it. Finally, while researching calling conventions recently, I found the reason. First, if the struct size is less than or equal to 8 bytes, it can be directly placed into a 64-bit register for parameter passing. Passing parameters via registers involves fewer memory accesses than passing by reference, making it more efficient, which is fine. What about 16 bytes? The System V ABI allows a 16-byte struct to be split into two 8-byte parts and then passed using registers separately. In this case, passing by value is indeed more efficient than passing by reference. Observe the following code: #include <cstdio> struct X { size_t x; size_t y; }; extern void f(X); extern void g(const X&); int main() { f({1, 2}); // pass by value g({1, 2}); // pass by reference } The generated code is as follows: main: sub rsp, 24 mov edi, 1 mov esi, 2 call f(X) movdqa xmm0, XMMWORD PTR .LC0[rip] mov rdi, rsp movaps XMMWORD PTR [rsp], xmm0 call g(X const&) xor eax, eax add rsp, 24 ret .LC0: .quad 1 .quad 2 The System V ABI specifies that the first six integer parameters can be passed using rdi, rsi, rdx, rcx, r8, r9 registers, respectively. The Windows x64 ABI specifies that the first