Computers

Setting up Jenkins… things I’ve learned so far

/ hsoi / 3 Comments

At the day job I’ve been tasked with setting up an automated build system. This is a Good Thing™ and I actually volunteered to do this task. I’m interested in doing what can help us make our code better, deliver our products faster, and make our lives easier in terms of automating things we don’t necessarily need to be involved in. So a Mac Mini was obtained to be a build machine, and I’ve been working to set things up.

I should note that I’m not a dedicated buildmeister. I know at larger companies they can have a person (or persons) whose full-time job is dealing with builds, but I’ve never been one of those people. I’ve done some build hacking in the past, but it was always homebrewed scripts and systems. This time around, let’s use an established system instead of homebrew.

It seems the modern hotness is Jenkins. Yes there are other options, but all signs via Google searching point to it. As well, it seemed (note the verb tense) like the more Mac-friendly solution. If nothing else, it had a nice Mac OS X installer. 🙂

I’m still far from having our complete build system, and I reckon as I learn more things on this list will change. But I’ve already learned a few useful things and I felt like sharing and adding to the greater knowledge-base out there that, through Google-Fu, helped me get this far. Maybe with my additions, someone else can be helped and maybe with a little less frustration and time sink.

The Project

A little context about the project being built.

Working on Mac OS X, developing iOS apps. Thus we’re using Mac OS X 10.7 Lion as our dev runtime environment. We’re using Xcode 4.3.3. Jenkins is version 1.474. We use git and a mix of private and public repositories on github.com.

First Time Problems

Started with a brand new, fresh from the box Mac Mini. Of course, before attempting any of this CI-specific work, the box was brought up to date with OS updates, Mac App Store updates, and so on. Note! Jenkins is a Java app and Java is NOT installed by default. So after you run the Jenkins installer and it tries to start Jenkins, things will probably fail. The OS will prompt you to install Java, so you’ll have to do that, but then Jenkins should end up running. Not a big hurdle, but it’s there.

Make sure to launch Xcode.app (the GUI app) and get it properly happy. This is mostly ensuring the various downloadable components get downloaded and installed, like command line tools and such.

You will be using command line tools, thus you will have to run xcode-select. But being as this is Xcode 4, things are different.

$ sudo /usr/bin/xcode-select -switch /Applications/Xcode.app/Contents/Developer

This will ensure xcodebuild can be found. And note, the first time you run xcodebuild, it won’t run! While you agreed to the license agreement when you first launched the GUI Xcode.app, that’s not good enough for xcodebuild; you have to agree to it all over again via the command line. Just be aware. It’s not a big deal, it’s a first-time-only occurrence, but it’s there to deal with.

The User (login) Matters

This was the source of much fun and frustration for me.

The first time I installed Jenkins I had no idea what to expect. That it had a proper Mac OS X Installer .pkg was cool, but it was also hiding a secret. Any time there’s a “Customize” button I like to click it and see what options there are. I noticed it provided two options:

  • Start at boot as “daemon”
  • Start at boot as “jenkins”

I had no idea what the relevance of these two options were. The default was “daemon” and “jenkins” was unchecked. I just figured to trust the default installer settings. Ha ha ha…  This actually caused me the most trouble and pain. I won’t recount the many hours spent dealing with this, but I will explain the issues.

I opted to go with the installer’s default of “daemon” but that creates a problem because “daemon” is a special user. When Jenkins needs to do things, it’s going to look in non-typical places, like /var/root for ssh keys or the like. Basically, it’s going to cause you a lot of headache.

When you search around for information about this, everyone starts to talk about using the “dscl” command line tool to create a hidden secret “jenkins” user and run Jenkins that way. This makes sense because it creates a user of restricted ability so it helps to keep the system secure and minimize chance of damage should someone gain access to the system via the Jenkins system or login. But in practice, this turned out to be a big pain in the ass because of what we’re doing. I’m sure there are some projects where this won’t be that problematic. But writing iOS apps brings issues. Apple makes great products, but you find they are great as long as you color within the lines; the lines might be really wide and vast, but still you must color within them. Trying to deal with this secret “jenkins” user created various issues.

For Xcode (or xcodebuild) to do code signing, it needs access to private keys and certificates and other such things. Thus the Keychain is involved, but this hidden “jenkins” user doesn’t have one. Again, more searching will turn up possible solutions, but they are not ideal solutions. In fact, one solution of putting these into the “system” keychain really defeats the purpose doesn’t it? Then there are the .mobileprovision files needed during the signing process, and those must exist somewhere in the “jenkins” user structure. In my case, the provisioning files may be updated fairly often due to the addition of device UDID’s, and setting up the initial .mobileprovision files by downloading to the logged in user then copying it all into the hidden jenkins user locations… it was just turning into a massive pain in the ass. And you’ll have to do this at least once a year, when you renew your iOS account with Apple.

So I fumbled with various permutations of these user setups and it was all just frustrating to me.

In the end, I backed everything out and started over (for the umpteenth time). I created a “jenkins” user via the System Preferences. That is a full-on login-able user named “jenkins”, admin user. I then logged in as the “jenkins” user. I ran the Jenkins installer, customize, and selected the “start at boot as ‘jenkins'” option (and deselected the “start at boot as ‘daemon'” option). I ran the installer. That did not immediately succeed, but the resolution was simple. Stop the daemon (using launchctl). Edit the .plist for the Jenkins launchd job and changed the location of JENKINS_HOME to /Users/Jenkins. Then restarted the machine. Life was good.

I think today I’m going to try doing it “yet again” tho… deleting the Jenkins user and creating it again, then installing again, and this time making a subfolder, like /Users/Jenkins/Jenkins_CI or something like that. That’ll contain all the stuff in one folder and not litter the actual home directory.

Is this the right thing to do? Well, from a pure security standpoint, no. It’s now an admin user that can log in and have all sorts of fun. But my feeling is this machine will be behind our firewall. If this box gets compromised, we have bigger problems anyways.  Could it mean bigger problems for me down the line with Jenkins-CI itself? Maybe. We’ll see. I’m not necessarily advocating taking this approach, but it’s just the one I’ve presently taken. It appears that it will create less long-term hell for me, especially in dealing with Xcode and the OS.

Authentication

Another source of pain was dealing with authentication issues.

Since we use private repositories on github, we need to authenticate. This proved to be more difficult than it should have been.

When I first tried a basic build of our project, I installed the various git plugins within Jenkins, created the new job, set it to obtain the source via the http protocol, and off we go. Well, that failed. The job just hung forever, I reckon waiting for authentication. So how can we authenticate? I saw no configuration options for it. So I started playing with ssh.

ssh would actually be a good thing in many ways, but with all the aforementioned user/login issues, it was becoming a massive pain. The way Mac OS X seems to handle ssh is by launching the ssh-agent on demand. Well… it can’t do that for the secret jenkins user. I tried all sorts of things, but I just could NOT get it working. *sigh*   Another vote for making a fully-realized jenkins users instead of this secret hacked one.

The one thing I had to then do was edit my ssh keys to not have a passphrase, and that would work (no need for ssh-agent, no need for interactive issues). But lacking a passphrase ain’t so hot. But it seemed to work… well, at first.

The next problem? Our git project has submodules. In the .gitmodules list, the “url” for each submodule used http as the protocol. And so, when the Jenkins-CI git plugin tried to obtain the submodules, we were back to the original problem. FML.

I tried fiddling with the repository URL as specified in the “git” section of the Jenkins-CI job to be something like:

https://:@github.com/path/to/repo.git

and while that worked for the top-level repository, the submodules still tripped me up. Plus, I did not like the idea of storing a password in plain text and sight like that.

I was able to solve it tho… through the magic of the .netrc file.

I created the following file in /Users/jenkins/.netrc

machine github.com
login <github username>
password <github password>

Saved it. Did a “chmod 600” on it. And lo…. everything worked. Huzzah!

I think the git plugin should have a way of doing authentication, but I’m still too much of a Jenkins n00b to know really where the fault lies.

More Ahead

At this point, I’m able to clone our git repository and build the app. It’s very basic at this point, with much configuration and fiddling yet to come. But I spent a good deal of time on these matters this past week and tried to sift through a bunch of search engine results to wind up here. So much of the information out there is Linux or Windows based, which is fine for the general but some of the little details aren’t there. And I can’t imagine we’re the first people to need to build in such a way, but I just couldn’t find anything specific to the particulars of our setup. So here’s hoping this helps contribute back.

As well, if you the reader have any information, insight, comment, etc. please share. I’m still learning, and constructive contribution is appreciated.

Trying Trello

/ hsoi / 3 Comments

The new hotness in software development is “agile“. At my prior day job, it was waterfall. Sure they tried to adopt agile processes, but it really wasn’t going to happen. Due to the nature of the products and process, it just can’t be agile, tho they could try to adopt a few things and make some improvements. But at my new day job? Agile can make a LOT of sense. Take home: you can’t impart the process merely because it’s the new hotness or you think if you just adopt X process it’ll solve problems. Like any problem to be solved, you have to understand your problem fully and then apply the right tools to solve it, and that includes what processes you use.

However, it’s tough getting folks on board, so I’ve desire to try to sneak in agile stuff as we can. It’s nothing more than a commentary on human condition — we tend to resist change. If all this change is dropped on folks all at once, we’ll balk. I would too. Massive sudden change, especially when you’ve still got daily chaos and stress to manage, well… the change will be rejected. But if we can make little changes here, little changes there, over time we get there.

One thing I can tell is we all need a way to see the whole picture. We’ve got so many things going on at once, and it changes on a daily and sometimes sub-daily basis. I find myself often making lists and (re)telling my todo lists to my teammates merely to help ensure 1. I know what I’m doing and am on track, 2. that we’re all on the same page. By sharing with them I hope that if there’s a mismatch, they’ll speak up and correct. I’d rather be perceived as over-communicating than under-communicating.

But all this talk doesn’t solve the problem for everyone. The dev team is one thing, testing another, production another, sales another, marketing another… there’s so many things. Sure, we could use our issue tracking system, and there’s a lot of sense in that approach. But the issue tracker doesn’t work as well for non-dev folk PLUS it’s harder to get a 1000′ view. Yes, pictures/diagrams can make a big difference.

One thing from the agile/scrum world is having the daily stand-up meetings at “the wall”. Let’s set aside the meeting for now (again, baby steps), and just focus on the wall. I’ve suggested the wall, because I think that would be useful. Pick a wall in the office, divide it up in whatever way makes sense for us, then start populating it with sticky notes to represent all the tasks to be done. I think that’ll be useful at keeping a somewhat permanent record of the state of things (unlike a whiteboard, which will be erased eventually). Plus it allows anyone to just look at the wall at any time of the day to see where things are. CEO wants to know what’s up with Customer X? Just look at the wall. Did we ship Product Y yet? Just look at the wall.

Alas, one shortcoming is 1. we don’t have a lot of free wall space left in the office, 2. the wall is restricted to the physical. I’ve been searching for a digital wall solution and haven’t found much that thrills me. Something that we could access from any computer via a web browser (or even a platform-based app), and it would look good. But then we could also access from iOS or Android devices. In the office. Out of the office. At 3 AM, during office hours. From my desk, in a meeting on the shared screen. Whatever. Something with power to do what we need, but flexibility to work for anyone, not just the geeks.

So… I recently read this article from Joel Spolsky about “Software Inventory”. While I read it, it sounded like it was talking directly to me and our situation. I went looking at their solution:

Trello

I’ve only just started to play with it, but it seems like it could be the answer to my problems. I could see using this in my personal life, for Hsoi Enterprises, and for the day job stuff. Even if the rest of the office doesn’t buy into it, it could be useful for my own management of my tasks and issues to deal with. Being able to SEE everything instead of sorting through a bunch of text notes and to-do lists is sometimes much more useful, even if it ends up being redundant and a little more maintenance to keep 2 data stores in sync.

It’s still preliminary, but it’s promising.

DR Performance Practice Deck for iOS version 1.1 Now Available

/ hsoi

The DR Performance Practice Deck for iOS version 1.1 is now available.

Lots of good things in this update, the biggest news being score tracking is now available.

Please spread the good word!

Thank you for your support.

Apple TV or not?

/ hsoi / 13 Comments

Any of you have experience with the Apple TV?

I’ve been thinking about ditching cable, and the Apple TV seems rather compelling (Apple lifer that I am). For $99 I reckon there’s no harm in just buying one and trying it (first world problem, I know). Still, it’d be nice to get some input from folks that may have one.

Your input requested. 🙂

 

Fun with SleepWatcher

/ hsoi

I do the majority of my computing work on a MacBook Pro — a laptop computer. The primary reason for a laptop is portability. Sure most of my work gets done at my desk, but sometimes I have to move or want to move. But moving isn’t always so quick and easy because I have lots of things hooked up to the MacBook Pro, like an external FireWire hard drive so I can run Time Machine backups. Trouble is, you can’t just yank out the cable for the disk, you must eject it, then you can remove the cable, else you could risk corruption of the data on the drive. Of course when you’re in a hurry to get out, or just consider we all get absent-minded from time to time gee… this is a pain in the neck and wouldn’t it be nice if we could automate it so we didn’t have to worry about it! Isn’t that what computers are useful for?

As I searched about for a solution, I found a little app called autoEJECT. It’s not bad, but after using it for a few days it just didn’t work like I needed it to. It did unmount things fine, but it failed to mount my FireWire drive. It has an option to “Force Mount”, which I then enabled but when I did so it mounted everything including the secret hidden partitions Apple has for recovery and booting. This is not desired as that 1. clutters things up, 2. exposes volumes that you do NOT want to risk to change. Even then it was a little quirky in how it would remember volumes to mount and unmount and just got in the way more than it was being useful. I figured there had to be a better way.

I found something called SleepWatcher.

SleepWatcher is a system daemon that watches for various events on the system, like sleep and wake, and allows you to execute scripts in response to those events. The app isn’t for novices, and the author actually goes out of his way to make it a bit complicated to use because it’s not a tool for the masses to use. But if you know what you’re doing, it can be useful. So I figured to give it a shot.

Of course, SleepWatcher itself isn’t what does the voodoo. You need scripts.

This website listed a couple scripts.

His sleep script was:

#!/bin/sh
osascript -e 'tell application "Finder" to eject disks'

and his wake script was:

#!/bin/sh
/usr/sbin/diskutil list | \
 awk '/Apple_HFS/ {print $NF}' | \
 xargs -I{} /usr/sbin/diskutil mount {}

Those are reasonable starting scripts. I thought about changing the wake script to just look for my one volume, because as it is now is ends up mounting all my partitions. Since they’re already mounted it really makes no difference, but it’d just be more efficient to scan for my lone volume. Still tho, this is useful for being complete.

The sleep script I modified slightly based upon what I found here. The change I made was simple:

#!/bin/sh
osascript -e 'tell application "Finder" to eject (every disk whose ejectable is true)'

which works a bit better in my situation. Granted, this will kick out optical discs, but as I rarely use them it’s not a big deal to me. The author also notes how this will not work if the Finder isn’t running, but I use the Finder and it should always be running (else there’s a problem).

Along the same lines as these is this hint from Mac OS X Hints, but he takes some vastly different approaches in what is selected for unmounting and what is selected for mounting. Not exactly what I want, but it shows what else you can do.

Anyways, SleepWatcher is pretty nifty and reading the code seems to be pretty well written too. Cheers, Bernhard Baeher!

Announcing kwikkEmail – the kwikkest way to send email

/ hsoi / 7 Comments

I’m proud to announce kwikkEmail – the kwikkest way to send email!

Read all about it over at Hsoi Enterprises.

(Hsoi Enterprises is my company, as you might guess from the name. Thus…).

Finally made the leap

/ hsoi / 2 Comments

I finally did it. I upgraded to Lion.

Seems a little late in coming eh?

I’ve been writing software for a long time. I know how the process goes, and well… I have enough of my own bugs in a day to deal with, I don’t need to be hampered by someone else’s. Snow Leopard (Mac OS X 10.6.8) was working fine for me, and out of the gate Lion had lots problems. Things are better in Lion, with Mac OS X 10.7.3, so maybe it’s time to upgrade.

Still, I didn’t. The main reason was stability. When you’re going through a development cycle, you don’t change your tools. You do this so you don’t introduce change and risk that’s outside of your control. Wait until you ship, then you can upgrade everything. Yes it might mean you get held back for a while, but that’s how it goes. But sometimes you can violate this, and I have because daily use of Snow Leopard was getting to be a problem.

See, at my day job I have to work on a MacBook Pro that runs Lion. The requirements of that project required Lion, Xcode 4.3, iOS 5 target, and so on. So I had to be all on the latest. Very cool for what it is, but well… let’s put it this way. When you use that computer all day and are forced to scroll “the other way”, when you switch to your other computer you get all messed up. I got to a point where I stopped knowing how to scroll and which way went which way on which machine, so any scrolling was always a “ok, just flick one way and if it doesn’t work then reverse direction”. Very inefficient and annoying since you become VERY aware of how much you scroll in a day. 🙂

Plus, since I had to work in Lion all day, I got to see that, for the most part, it’s working fine and not getting in my way. So I finally opted to upgrade.

Oye… what a slow, laborious process. Started it last night after supper, still working on it this morning. But now I’m turning on the whole-disk encryption and it’s slowly chugging away at encrypting everything.

The sad part? I think my current MacBook Pro isn’t going to make the upgrade to Mountain Lion. Hopefully that “Apple Event” next week will shed some light on new hardware options. 🙂

 

 

Technology failures

/ hsoi / 4 Comments

Too much computer fail right now.

Time-Warner Cable’s RoadRunner Internet has been horrible for me lately. It’ll be up for a while, then go down for a few minutes, then back up, then back down, and just repeat this for too long. That’s been going on for some weeks but I tried to do my best to ignore it. But then yesterday she gave up the ghost. I’m guessing the modem has failed, or something bigger. It’s not an outage, it’s local. Tech won’t come until Wednesday.

This doesn’t bode well for someone who works from home and relies upon Internet connectivity.

It’s good that my day job has me writing connection manager software, so I have lots of 3G/4G USB dongles around. I set up an old iMac with one of the 4G dongles and turned on Internet Sharing to share the 4G connection through the Ethernet port. Then hooked my router up to that iMac. And viola… the household is back online, every device. It’s a lot slower, but at least we can limp along for now.

And then on my server… one of the hard drives is making a funny noise. It’s failing. The bad part is, as I powered everything off to remove the failing drive, when I tried to power it back up it wouldn’t power up. So is the drive failing or the power supply? And a couple other drives in similar enclosures also made the same noise upon turning them back on, but they aren’t repeatedly making the noise. This doesn’t bode well. I’ll be calling Other World Computing this morning to talk about it and figure out a solution.

What with the flooding in Asia and how it’s affected the hard drive market, I was so hoping to avoid buying a hard drive for the next couple years until things recovered. Alas…

Oh what fun.

 

Apple iOS 5 Tech Talk – Austin

/ hsoi

Yesterday I attended Apple’s iOS 5 Tech Talk here in Austin. I wrote about it at my company blog.

Rather inspired and excited am I about prospects ahead.

 

That’s very cool, Moneydance

/ hsoi / 4 Comments

Readers may remember about a year ago I was looking for a new personal finance program and decided upon Moneydance.

Still using Moneydance. Still happy with my purchase. Sure, some of the Java GUI quirks bug me, but the reality is it does all I want and does it pretty well. I’m pleased.

But here’s something really cool they recently did: they revamped their license. They made it a lot simpler. For personal users, it’s one license per household. Doesn’t matter how many people use it, how many computers it’s installed on, how many different OS’s it’s installed under, how many data files, whatever. And “As always, we promise that we’ll never sunset our software, remove functionality from older versions, or force you to upgrade.” Which is pretty nice. Speaking as a software developer, that’s a tough promise to keep, so I applaud their efforts.

What I also dug was their reasoning:

Why are we making this change? First, because we recognize that the world is changing:  households may now be multi-generational, include blended families or housemates, or have other structures which are more complex than two adults sharing finances. Second, because we wanted to even further simplify our licensing agreement. Finally, we want to make it easier for kids, teens, and students to learn about personal finance, and we believe that Moneydance can be a great tool for the entire family.

Very cool.