Mac OS X x86 debugging

It’s been ages since I’ve had to do anything dealing with assembly language. And while I may not have to do much programming in it, it’s still a useful thing to know when it comes to debugging.

Today I’ve been trapped in debugging hell. We’ve got some code that works fine on Mac OS X versions prior to Snow Leopard (10.6) but seems to hang under 10.6. The code flat out executes differently. I watch how our code is invoked and things just don’t happen in the same order across the OS versions. Apple must have done some major under-the-hood changes to NSBrowser and NSTreeController. And since we’re using Cocoa Bindings, it’s a bitch to debug. Everything that’s happening is happening because of Key-Value Observing (KVO). I get somewhere into the bowels of NSBrowserBinder and it appears to be looping… looping… and never exiting the loop for some reason.

Now that we’re in the new x86-based world order of Macs, I went looking for some tips to help me work with debugging the x86 assembly so I could see what the OS was doing. I found a few useful resources.

Greg Parker at the Hamster Emporium has a great article about crashing in objc_msgSend and how to decipher the crash. I wasn’t crashing, but still there are some nice gdb commands and interpretation of the registers.

Then over at the Google Mac Blog, a 2 part article on spelunking. Part 1 and Part 2. These aren’t debugging articles (more like trying to hack around and discover how undocumented functions work), but they still explain a bit about how the x86 ABI works, AT&T syntax, and how Mac all mesh together. Note that Avi Drissman previously published the same article in his own personal blog, but edited it a bit for republishing in the Google blog.

Clark Cox provides a handy table on how to inspect Objective-C parameters in gdb.

And you thought the only thing I knew how to blog about was guns. 😉