Engineering

Snow Leopard’s file system changes

/ hsoi / 2 Comments

Mac OS X 10.6 “Snow Leopard” brought a slew of under the hood changes. There are two distinct but related changes that Apple made that most users will probably never notice and many developers probably never will either. But if you happen to be someone who works with the file system, these changes may affect you. These changes are:

1. KB vs. KiB. That is, is a kilobyte 1000 bytes or 1024 bytes.

2. file-system-level compression of files

Since I recently went around with these, I felt it’d be worthwhile to share what I discovered, since there isn’t much information out there on the topic. The only thing I found that remotely discussed the matter was this page of Ars Technica’s Snow Leopard review.
Continue reading →

But what if you need more when installing?

/ hsoi / 6 Comments

As a developer of Mac OS X software, how to deliver that software is an issue of constant battle.

Long long ago, everything was shipped as either an installer or just in a StuffIt archive. Typically applications came in installers and the only thing in StuffIt archives would be non-applications (documents, pictures, other data).

Mac OS X tries to change this by simplifying things using drag-installs. It’s actually a very simple process of download, drag the application icon to the Applications folder, and off you go. But, even something as simple as that can prove too confusing for the lay person, or at least it allows for a host of problems. Installers still exist as well, and there are many other modes of delivery and installation out there.

Various bloggers have been discussing the issue. It starts with FireFox distribution issues, where they ultimately conclude that you should just ship an installer and piggy back off other mechanisms (e.g. automounting of .dmg’s). John Gruber counters that we should just use .zip files, and makes rather compelling arguments for that approach. Chris Clark also agrees with using zip archives. Andy Kim takes it a bit further and actually starts to touch on what I’m going to hit at.

All of these solutions are nice, but they all seem to target one thing:

Very simple applications.

If all your application is is itself, then I guess that’s fine. But what if your application has more than just the application itself? What if it needs to install drivers or other kernel extensions? What if it had launch agents or daemons? How about other plugins that need to be installed? Maybe something that requires a restart of the machine? What if you need to run scripts to deal with housekeeping (e.g. to help upgrading past users to the new version of the software)? All of these simplistic approaches just do not cut it.

Some would argue this should be a part of the application itself. That is, self-healing applications. I see merit in that approach and it could be a solution that solves this and the initial download-and-install problem. But it is not a panacea.

The ultimate problem is there is no one-size-fits-all solution, which seems to be what people are proposing… or at least, they’re not considering the wider scope that some developers may need in the install process. Apple seems to have the tools in place, but isn’t quite there. What ought to come would be:

  • A new installer format. It could be an archive holding all the parts to be deployed, but it should allow for logic to be applied during the install (e.g. use of shell scripts, etc.).
  • This format is understood by the OS, so when applications such as Safari download them, they can start the install process. Of course, we have to be mindful that some people could use this for nefarious purposes, so being fully automatic is not ideal… asking the user would be helpful. Using code signing and other forms of trusted computing may help here.
  • The notion of installing then isn’t just some file format, but is also considered an end-to-end process, from either downloading the software or inserting the removable media that has the software on it, to getting the software installed, configured, registered, and cleaned up after (optionally removing older versions of the software and its support files, removing any archives, trying to leave the machine in a cleaner state than before it started). Get the users going the whole way through. It should “just work”; Gruber’s mention of how the iPhone works is apt.
  • To publish a set of tools AND guidelines on installer creation. Apple goes through a lot of trouble to establish human interface guidelines on how applications should work. Apple can and should do the same with installers so that the experience of getting software to users can be as simple and pain free as possible, but also as consistent as possible.

The big problem we have today is too many approaches to installing software. Everyone has different needs and different thoughts on how best to approach it. I’ve had debates at my office about how we should deliver our software, and it’s difficult to achieve consensus because all approaches have strengths and weaknesses. This only serves to hurt the user because there isn’t one way that “just works” for them to get to know and be comfortable with. Every install process is different and unique, and for the non-geeky types, just frustrating and confusing.

I hate to say it but the movement to improve this will ultimately have to come from Apple. It’s their platform, it’s their user experience that they wish to control and dictate. Ultimately Apple must be the one to improve this.

Can we 3rd party developers do something about it? Sure. We could work to tackle the problem, publish our own guidelines, create tools to help out. But I’m not sure how far it would get. Efforts like Growl, Sparkle, solve some great problems and didn’t come from Apple and have gained a tremendous amount of traction in the developer world. Still, some venture to do it themselves. Code solutions are good, but guidelines are better so if you have to do it yourself, at least some guiding principles can be followed with the hope of still making things good for the user.

Where to start?

Mac OS X x86 debugging

/ hsoi

It’s been ages since I’ve had to do anything dealing with assembly language. And while I may not have to do much programming in it, it’s still a useful thing to know when it comes to debugging.

Today I’ve been trapped in debugging hell. We’ve got some code that works fine on Mac OS X versions prior to Snow Leopard (10.6) but seems to hang under 10.6. The code flat out executes differently. I watch how our code is invoked and things just don’t happen in the same order across the OS versions. Apple must have done some major under-the-hood changes to NSBrowser and NSTreeController. And since we’re using Cocoa Bindings, it’s a bitch to debug. Everything that’s happening is happening because of Key-Value Observing (KVO). I get somewhere into the bowels of NSBrowserBinder and it appears to be looping… looping… and never exiting the loop for some reason.

Now that we’re in the new x86-based world order of Macs, I went looking for some tips to help me work with debugging the x86 assembly so I could see what the OS was doing. I found a few useful resources.

Greg Parker at the Hamster Emporium has a great article about crashing in objc_msgSend and how to decipher the crash. I wasn’t crashing, but still there are some nice gdb commands and interpretation of the registers.

Then over at the Google Mac Blog, a 2 part article on spelunking. Part 1 and Part 2. These aren’t debugging articles (more like trying to hack around and discover how undocumented functions work), but they still explain a bit about how the x86 ABI works, AT&T syntax, and how Mac all mesh together. Note that Avi Drissman previously published the same article in his own personal blog, but edited it a bit for republishing in the Google blog.

Clark Cox provides a handy table on how to inspect Objective-C parameters in gdb.

And you thought the only thing I knew how to blog about was guns. 😉

Why did Apple do this?

/ hsoi / 3 Comments

No sorry… nothing about today’s Apple fan-boy event. More as to why I didn’t post much today.

Apple’s new OS version is 10.6, named “Snow Leopard.” Snow Leopard brought about a lot of under the hood changes to the OS. One of them is fairly well covered on this page of the Ars Technica review.

Basically, Apple did some stuff very very low level to help with reclaiming some disk space but also taking advantage of the volume format layout of HFS+ and how they can use that to their advantage to speed things up… RAM and CPU’s are wicked fast these days, and disk drives are still abysmally slow by comparison (physics can only go so far). So Apple did some neat things to improve speed and access times, and for the most part it works out great. Most people will never notice.

But in the line of work I do… I’m not most people.

The software I develop in my day job does a lot of working with the file system. So all of these changes that Apple made are actually wreaking havoc and hell on me right now. Long held maxims like a file’s logical size will never be larger than its physical size…. out the door. That calculating sizes is now base-10 instead of base-2 (i.e. 1 kilobyte is now being calculated as 1000 bytes instead of 1024 bytes)… changes a lot of things.

I’ve been reevaluating our entire codebase (which is huge) and this just doesn’t play well with us. All I can do at this point is formulate a lengthy email to Apple’s Developer Technical Support and ask for some help and guidance.

It’s been a trying couple of days.

I’m glad I have Kali class tonight. That should relieve a little stress. 🙂

Snow Leopard cometh

/ hsoi

Mac OS X 10.6 “Snow Leopard” is due tomorrow, August 28, 2009.

Macworld has a series of articles discussing the new OS version.

As a Mac software developer I have to keep up with such things. Frankly, the Snow Leopard update isn’t all that sexy for end users. You’ll see there’s no massive nor major feature list upgrade, which is a change. Every upgrade of Mac OS X so far has been a big step forward, so this is the first small step. Frankly what that points to is a maturing of the OS, and I think that’s welcome. We don’t have to be chasing the latest thing all the time and plunking down a lot of money to do so. Well, at least end-users don’t have to be… what Snow Leopard really brings about are a lot of under-the-hood changes that affect us developers far more.

Still, I have people asking me if they should upgrade. That’s up to you. Some people like to have the latest stuff, so if you like being on the cutting edge, then by all means be there at midnight when it goes on sale. One big feature for end-users is built-in support for Microsoft Exchange. If you have to work in an Exchange-based environment, you may find this quite welcome.

I must admit that I’m still a bit of a hold-out from converting fully to Apple’s own software. I still use Microsoft Entourage for my email because frankly it’s been the best Mac email client for a number of years. I have moved my calendaring to iCal (tho ToDo’s remain in Entourage because iCal’s doesn’t cut it), and with the purchase of the iPhone I have to use the Mac’s Address Book a lot more (thankfully Entourage can sync between its address book and Address Book.app). But with the massive amount of email, folder organization, rules, automation, and other such things it’s just tough for me to make the transition… probably take me a day or two just to try to get it all set up, and when I’ve attempted it in the past I discover Mail.app just isn’t quite there. But now with the Exchange support built-in, there may be more feature parity and it may work out. So I’ll probably reevaluate it at some point. The reality is, you do get further on the Mac working with Apple’s technologies because things like Address Book, iCal, Mail, etc. are all well-integrated into the system. They mesh with your iPhone better. Plus since developers can have access to such things (e.g. the Address Book API’s) the greater user experience can be enhanced if you’re taking advantage of what Apple provides. While some people don’t like this “monopoly-style” approach to things, it’s actually what makes the Mac such a great end-to-end experience. Instead of lots of disparate parts that don’t really hook together well causing the end-user a lot of frustration in accomplishing simple tasks, you’ve got parts that integrate well together from the hardware level through the OS lower layers to the higher layers to the end applications, and with everything hooking together, things can “just work.” User experience is so important and that’s what really differentiates Mac from other platforms.

So we’ll see. For me the key is being able to get my work done in a day with minimal-to-no hassle. So far so good, so it’s tough to want to change unless I can see improvement in that area.

But being a developer, I have been using Snow Leopard for a while and find that it’s just another upgrade. Being a “minor” upgrade for users, the impacts to me as a developer mean that I’ll still be supporting Mac OS X 10.4 “Tiger” for some time. Believe it or not, users are still using Tiger a fair amount. I can’t say why, but one possible reason is Tiger is the last version of the OS to include “Classic” support. If people are still relying upon Mac software from years gone by that was never rebuilt to be Mac OS X native, they would want to stick with Tiger. Trouble is, it’s going to be harder and harder to maintain that. Snow Leopard itself is Intel-only: PowerPC-based Macs are officially heading off into the sunset. Still, there’s not a whole lot at least in the work I do that would force one of my products to be Snow Leopard-only, and while I’m certainly working on a few Leopard-only things, so long as Tiger users exist and continue to buy software while holding out on that 4+ year old OS well… we’ll keep supporting Tiger users. Of course one potentially limiting factor will be Apple’s developer tools. Snow Leopard’s version of Xcode 3.2 does not install things for targeting 10.4… you have to do a custom install to get them (seems to be SDK 10.4 and gcc 4.0.x); so the writing is on the wall even from Apple: move forward.

Frankly, that makes sense for Apple. You see, at the heart of it all Apple is a hardware company. To make things like Mac OS X, iWork, and other bits of software, that’s really only done to make the hardware functional and useful. Macs, iPods, iPhones — it’s all hardware. The family iMac is a PowerPC G5-based iMac. It’s long in the tooth, no doubt. I’ve been wanting to replace it with a new Intel-based iMac for some time, but I have a rule with hardware purchases. You see, everyone always wants the best and wants to minimize obsolescence. There will always be something bigger, better, faster, cheaper somewhere on down the line, so there’s really nothing you can do about it. So you use what you’ve got as long as you can, push it until you truly cannot wait any more… not just desire wait, but some larger need. Then buy the best you can afford. So now that PowerPC is officially dead to Apple, eventually the family iMac just won’t be able to cut it any more. It should last a bit longer, but push is coming to shove and a new purchase is in the wings. Works well for Apple’s bottom-line, doesn’t it? 🙂

There’s cool stuff in Austin

/ hsoi

Waterloo Labs has more details. National Instruments is here in town. So is Stunt Ranch.

Who says Austin isn’t home to fun gun stuff? 🙂

And why not try this at home with real guns? Of course, as long as you can do it legally and safely. Add this to the long list of reasons why I want a lot of land.

4.5 out of 5 stars

/ hsoi

I write Mac utility software for a living. A software bundle the company creates, CheckIt 2 Performance Suite, was just reviewed and received 4.5 out of 5 stars. The particular product I work on, Spring Cleaning, is the flagship software in that bundle.

What stood out to me in this review versus other reviews of the product is the reviewer seemed to look at the entire product and didn’t overlook key features. Many reviewers seem to only focus on the “pretty icons” and the basic tools, as if they just gave the product a 5 minute look over and then wrote a review. This reviewer appeared to fully examine the product and review based upon that:

Spring Cleaning comes with other features that are used to modify and limit how the other features work, allowing you to choose areas or files that you don’t want it to include. Finally, and very importantly, is the ability to schedule any or all of these functions to be performed automatically. For instance, it is probably a good idea to repair disk permissions once a month, and it is a good idea to allow the Unix maintenance routines to run. Using Spring Cleaning, you can easily set these up to automatically run when it works for your schedule.

Talked about things like Exclusions, and most importantly, the Scheduler. That’s one of the biggest strengths of Spring Cleaning, that you can easily set up automated schedules to keep your Mac running in top shape. As Ron Popeil was fond of saying, “Just set it and forget it!”

We understand a lot of functionality here is something an über-geek could do on their own, but not everyone is an über-geek. We strive to bring a lot of power and functionality “for the rest of us” so that anyone can keep their Mac running well.

I was just happy at the excellent review. Now go buy a copy and keep me in a job. 😉

Treemaps for space-constrained visualization of hierarchies

/ hsoi

If the title made your eyes glaze over, no need to click this link to read Ben Shneiderman’s paper on treemaps.

I’ve been buried in working with them all day today at work. Just conceptual and design at this point, haven’t started to write algorithms just yet. It’s pretty cool stuff, at least for geeks like me.

Even if you’re not, the graphical representation of treemaps can look pretty cool. I really like these circular treemaps.

Who knew examining the composition and state of your computer’s filesystem could be so trippy. 😉

WWDC 2009 Keynote Thoughts

/ hsoi / 9 Comments

Apple’s World Wide Developer Conference starts today, and of course the keynote is the big deal the world watches.

I’ve attended WWDC many times in the past, but I intentionally skipped it this year. Saving the company a little money, WWDC has turned into an iPhone luv-fest, the quality of the conference has gone down over the years (yes, I’m an old-timer), it’s packed, the fan-boys are in way too much effect here… it’s just not worth it. I mean, people were lining up outside the convention center at 4:30 AM to be “first in line” for the keynote. Sheesh! I’d rather spend the week being productive and working. Any of the few sessions that’d be worth attending I can catch videos of later.

Oh well, so what came out of the keynote.

  • Updated the entire portable/notebook line. That’s good stuff and looks like they’re moving the whole line towards their new form factor. Good deal. I’d love to buy a new one to get that sort of new battery life.
  • Simple Snow Leopard improvements, like Exposé in the Dock, faster install and recovery of disk space is good. Enhanced “previewing” support directly in the Finder. They are small things, but the devil is always in the details.
  • Chinese input via drawing on the trackpad? That’s interesting. I need to check on how that works and if it works for other “symbolic” languages such as Japanese, Korean, etc..
  • I like Safari 4’s “crash resistance” setup, because without question the highest crashing thing on my Mac is the Flash plugin.
  • That QuickTime has finally become “X” and gotten the much overdue overhaul is great.
  • Grand Central, OpenCL, 64-bit hardware and the OS fully committed to 64-bit, multiple core CPU’s and wicked powerful GPUs, just awesome stuff for a geek like me.
  • While I think MS Exchange is a huge steaming turd, I do know how many organizations are based upon it for all of their electronic communication and organizational needs. So that Apple is embracing this and putting Exchange support directly into the OS is good, both for Apple and for Microsoft.
  • Overall I’m liking that Snow Leopard is going to be what they said it was going to be: refinement. Leopard really brings about a maturing Mac OS X in terms of features and user experience, so now you need to stop adding on features and making the features that you have really solid and stable and refined. Snow Leopard is doing that in a big way. Thing is, as a geek I know it’s good and will move to it. But from the early days of Snow Leopard (as a developer, I’ve had access) I was always curious how Apple was going to make this appeal and be marketed towards consumers because the non-geeks well… I don’t know how it will appeal to them to plunk down the money for the OS upgrade that does a lot of awesome stuff under the hood which then enables us developers to do awesome stuff…. but that doesn’t equate to simple whiz-bang features that you can list on a box or in an advertisement. What’s there to really entice the consumer? 
    • Now I know. Pricing it at $29, or $49 for a family pack. Smart move on Apple’s part. They know there aren’t any whiz-bang flashy features to really sell to consumers, but make it a cheap upgrade that gets you lots of things towards the future. It mainly helps developers out, so now we can write software that use the new technology, the $29 OS upgrade price is a negligent barrier then for people to upgrade the OS to use our softwares. This is all good stuff. Glad Apple did this.
  • Snow Leopard will be available in September.
  • iPhone. Cut/Copy/Paste/Undo. Gosh, such essential parts… about time they’re here. Landscape mode all around, good. Spotlight, neat. Of course more iTunes/Store integration, which is great for them and AT&T.
  • “Tethering”. Nice. But AT&T won’t support it and who is the iPhone carrier? Sheesh. 
  • “Find My iPhone”, only through MobileMe. Way to drum up business! 🙂  I actually think this is pretty cool, but on the surface appears ripe for abuse or other evil things, so hopefully they thought about that (likely so) and it’s a fairly controlled and unabusable experience. But yeah, this is cool.
  • P2P support is good, gamers will like that. They’ll also like the in-app purchasing ability. Hardware accessory support is very good.
  • Push notification. ’bout time.
  • Other little iPhone OS 3.0 things that are good, especially the expanded language support. It’s nice to see the OS coming along, maturing, catching-up.
    • Available June 17.
  • OO… TomTom for iPhone.
  • But 8 iPhone demos. Ugh. I know Apple wants to trot out a lot of people and cool things, but when you’re at the show, sitting in cramped quarters (the chairs are all clamped together, they’re barely wide enough for a child to sit in, then consider most geeks aren’t exactly svelte and it makes for a very uncomfortable 2 hours), man… this is painful. And I’m not there and it’s painful, but from dealing with it in the past I know the pain.
  • iPhone 3GS. Faster, better camera, video, digital compass, voice control, encryption and data wiping. Awesome.
    • $299 for the biggest (32GB) version. Nice.
    • Also, the existing 3G iPhone (8GB) for $99.
    • Available when? Depends: 3G today, 3GS June 19. 
  • There was no “one more thing”, and for that I’m glad. While it was a cute thing in its day, I’m glad it went away. People got way too hyped up and it got ridiculous. Fanboys get all excited prior to the show and work up expectations that there’s no way Apple can live up to, then people moan and grow because their unrealistic expectations weren’t exceeded. So I’m glad they’re doing away with it. In the end, I think the keynote offered some great stuff and really showed where Apple is going. They are focusing on making the OS very solid towards the future. They are making their hardware platforms grow and improve, and working to make both the hardware and the OS work very well together (one of Apple’s strengths since they sell them both… it’s part of what makes a Mac a Mac, vs. using Windows atop any old hardware or using any old hardware with whatever OS). Plus it’s evident iPhone is #1 at Apple, because you know the revenue streams are ridiculous here. Many of the new things demoed during the keynote rang of $$$ for them, carriers, and developers.

While I was watching various live webcast coverages, when Bertrand Serlet was up I found myself reciting all of his quotes with his accent.  Then Scott Forstall came up for iPhone stuff and gosh, I remember when Scott was just a low-level guy at Apple; it’s really neat to see how every year or two he’s climbed further up the ladder there at Apple. That’s quite awesome for him. Then while watching the keynote, near the end, Wife IM’d me and said “one more thing”… Wife doesn’t understand what “one more thing” means in the context of an Apple keynote, it was just a funny coincidence. All this stuff? It was just like I was there, but with a better seat. 🙂

I will say, this is one thing tough about my current job. I have to ship product that works for the existing users, so I often end up being behind the curve. All these new things are great, but I still have to support Tiger users (Mac OS X 10.4). I don’t get to really take advantage of new things for quite some time. Frustrating, but I can see what I can look forward to.

That said, one thing people find hard to believe about me is that while I’ve been an Apple user since I was a kid and have spent a good portion of my life developing software for the Mac, I don’t own an iPod or an iPhone and never have. Main reason? I don’t have a need. But the iPhone has been full of want, and that iPhone 3GS seems like maybe a good time and place to start. In fact, maybe a 3G for Wife and a 3GS for me (Wife isn’t as techno geek).

of SQL and other programmer joys

/ hsoi

I know it’s been a light posting day. I’ve been buried in code.

I’m an application programmer. I tend to write most of my stuff in Objective-C and C++ (and Objective-C++ and sometimes straight C). Sometimes I do some work in Python. A few other bits of dabbling now and again.

I haven’t looked at serious database work in almost a decade.

But today I needed to consider using SQLite as a solution, so I had to pull out my SQL book and remind myself just how it all worked.

Of course by the end of the day I came to the conclusion that going straight to a relational database wasn’t going to be the best avenue to solve my problem. Instead, I think I’m going to use Core Data. I’ve used it numerous times in the past but didn’t feel it was the right fit for the work at hand (given Apple’s design intent/constraints on Core Data). But the more I thought through things, the more I think it’s going to be the way to go. At least, that’s my thinking right now. Could change tomorrow when I start prototyping.

Yeah, this probably doesn’t make a lot of sense to most of you. But I’ve been nose-down in that all day, thus minimal blogging.