The calling convention used within Lisp code on SBCL/x86 was, for the longest time, really bad. If it weren't for the fact that it predates modern x86 CPUs, one might almost believe it to have been designed explicitly to defeat the branch-prediction hardware therein. This chapter is somewhat of a brain-dump of information that might be useful when attempting to improve the situation further, mostly written immediately after having made a dent in the problem.
Assumptions about the calling convention are embedded throughout the system. The runtime knows how to call in to Lisp and receive a value from Lisp, the assembly-routines have intimate knowledge of what registers are involved in a call situation, src/compiler/target/call.lisp contains the VOPs involved in implementing function call/return, and src/compiler/ir2tran.lisp has assumptions about frame allocation and argument/return-value passing locations.
The current round of changes has been limited to VOPs, assembly-routines, related support functions, and the required support in the runtime.
Note that most of this documentation also applies to other CPUs, modulo the actual registers involved, the displacement used in the single-value return convention, and the fact that they use the “old” convention anywhere it is mentioned.