Calling conventions

[fastman92 2,503]

I decided to make a documentation about calling conventions for beginners.

Later I'll add the details about different architectures.

What is calling convention?
Calling convention is the standarized way of passing the arguments to function and getting a result from called function.

x86

Like on every architecture there are registers that must be left unmodified when called function returns and the registers of which the values can be messed up.

Temporary CPU registers: EAX, ECX, EDX
Preserved registers: all other registers (EBX, ESI, EDI, EBP)

When a function is called, any of the registers considered temporary may change the value upon the return of called function.


There are calling conventions that will fix the stack upon the return and there is a calling convention that will not fix the stack pointer.

In x86 the function arguments are stored on stack.

Sample function definition:

FILE *__cdecl CFileMgr::OpenFile(const char *filename, const char *mode);

push    offset mode     ; "rb"push    offset aTimecyc_dat ; "TIMECYC.DAT"call    _ZN8CFileMgr8OpenFileEPKcS1_ ; CFileMgr::OpenFile(char const*,char const*)add  esp, 8

There are two arguments, so two values are pushed.

What push really does? It lowers the value of ESP (stack pointer) register by 4.
Then it puts the value into memory address pointed by ESP register.

In example there are two pushes, call and then comes "add esp, 8".
__cdecl calling convention is the only one that doesn't fix the stack pointer upon the return and thus the calling function needs to do.
Because there were 2 pushes, the ESP register is badly lowered by 8, so it needs to have 8 added again.

Since the __cdecl doesn't fix the ESP register it's the only good calling convention for functions with variable number of arguments like printf.

Similar calling convention to __cdecl is __stdcall, with the only difference that ESP register would be fixed upon the return of called __stdcall function and thus, the "add esp, 8" would be unneccessary in result.

Finally, there's a __thiscall calling convention to be mentioned.
Let's consider this function:

class CMemoryAddressCalculator{public:    // Gets actual virtual address from virtual prefered address given.    uintptr_t GetCurrentVAbyPreferedVA(uintptr_t virtualAddress);}

The function will use a __thiscall calling convention.
__thiscall is very similar to __stdcall with one addition: it needs an object pointer (this) to be passed.
ECX register with the pointer to object (here pointer to object of type CMemoryAddressCalculator) needs to be set up before calling such a function.

Example:

push 0x6D1A59   // virtualAddressmov ecx, offset g_mScannercall CMemoryAddressCalculator::GetCurrentVAbyPreferedVA // now a result will be in EAX after calling a function.// Since it's __thiscall, which is very similar to __stdcall, there's no need to do  "add esp, 4".

Result
How the result will be returned depends on the type of returned value.
Any integral values or pointers/references will be returned in EAX register.

The floating point values (when the type of returned values resolves into float/double) will be returned on the FPU stack in the ST0 register.
The value is pushed by the called function and thus needs to be popped off by the caller.
In most cases it's done by using "fstp" instruction.

When the result is the structure of data, the C++ compiler will append yet another argument to be pushed. That will be a first stack argument and should contain the address to which the structure data will be written.

 

MIPS

 

Like on every architecture there are registers that must be left unmodified when called function returns and the registers of which the values can be messed up.

Temporary CPU registers: $t, $a
Preserved registers: all other registers

 

In MIPS the arguments are primarily passed with the use of registers.

 

$a0, $a1, $a2, $a3, $t0, $t1, $t2

These registers will be used for all the integral values or pointers/references.

If there are more values like this to be passed than the number of registers above, the stack will be used for the remaining values.

 

Floating point values are passes in these registers:

$f12, $f13, $f14, $f15, $f16, $f17

 

The remaining arguments are passed on the stack.

 

Result
How the result will be returned depends on the type of returned value.
Any integral values or pointers/references will be returned in $v0 register.

If result is 64-bit, it will be returned in $v0 and $v1.

The floating point values (when the type of returned values resolves into float/double) will be returned in the $f0/$f1 registers.

When the result is the structure of data, the C++ compiler will append yet another argument to be pushed. That will be a first integral argument and should contain the address to which the structure data will be written.

 

x64

Like on every architecture there are registers that must be left unmodified when called function returns and the registers of which the values can be messed up.

List of volatile and non-volatile registers can found here: https://msdn.microsoft.com/en-us/library/9z1stfyw.aspx

When a function is called, any of the registers considered volatile may change the value upon the return of called function.

 

In x64, if caller has some function to be called, there must be 32 bytes reserved for the purpose of callee to back up the 4 of non-volatile registers:

 

Author: fastman92

Edited November 2, 2015 by fastman92