x64 assembly sample program

To get a better understanding of how the new Microsoft x64 calling convention works, I thought it would be a good idea to reacquaint myself with assembly language, this time using ml64.exe. While I do not write assembly language in my day-to-day job, I still find it surprisingly useful to be able to read and decipher assembly language. Understanding calling conventions and stack usage are a critical part of being able to debug assembly language.

A key takeaway for me from this process is that the x64 convention passes the first four arguments in volatile registers, therefore making it difficult to review the parameters as passed to a function because they will not be stored in memory. Note that space is allocated on the stack for the parameters that are passed via register, but the memory is generally left uninitialised in release-mode builds. The Advanced Windows Debugging and Troubleshooting blog has a good run-down of the difficulties this poses.

Below is a sample program I wrote, along with the command line to compile it:

extrn ExitProcess: PROC 
extrn MessageBoxA: PROC
extrn WriteConsoleA: PROC
extrn AllocConsole: PROC
extrn GetStdHandle: PROC

.data
caption db 'x64 Assembly Program', 0
message db 'Hello World!', 0

.code
main proc
  sub rsp, (5 + 1 + 1) * 8 ;5 parameters + return + stdhandle + [written length]; 

  call AllocConsole
  test rax, rax
  jz error

  mov rcx, 0FFFFFFF5h
  call GetStdHandle ;eax = handle

  mov rcx, rax
  lea rdx, message
  mov r8, 12 ; length of string
  lea r9, [rsp + 6 * 8]
  mov qword ptr [rsp + 4 * 8], 0

  call WriteConsoleA

  call DisplayMessage

  add rsp, 56
  ret

error:
  mov rcx, -1
  call ExitProcess
main endp

DisplayMessage proc
  sub rsp, 28h
  mov   rcx, 0 
  lea   rdx, message
  lea   r8, caption
  mov   r9d, 0
  call  MessageBoxA
  add   rsp, 28h
  ret
DisplayMessage endp
end

This is the command line that can be used to compile the code above:

ml64.exe /Zi Prog.asm /link /subsystem:windows /defaultlib:kernel32.lib /defaultlib:user32.lib /entry:main

Note that, in my installation of Visual Studio 2010, ml64.exe is in the C:\Program Files (x86)\Microsoft Visual Studio 10.0\VC\bin\x86_amd64 folder. Make sure that you use the x64 version of the Visual Studio command prompt, otherwise the link process will silently fail.