Computers

More things I’ve learned about Jenkins

/ hsoi / 1 Comment

At the day job, I’m still working with Jenkins and our build system. I’ve written about some things I learned at the onset, and now I’d like to add some things I’ve learned since then.

Plugins

Be mindful of them. There’s a great many out there that can help you solve things, but they seem to also bring much risk. For example, Jenkins was going belly-up at least a couple of times a day. I’d look in the system console and see some plugin that I wasn’t even using (just had installed) was somehow causing memory failures, throwing an exception, and Jenkins went stupid and had to be restarted. Once you figure out what plugins you need, uninstall every other plugin. Keep your system as minimal and tight as possible to avoid introducing risk or uncertainty.

Backups

I had to write a lot of custom scripts (some bash, a lot of ruby). These files I can keep in our version control system. But Jenkins files themselves are harder to do. Make sure you have a backup strategy. Make sure you run it often while you work. Working on a Mac and using Time Machine is useful… make a change, screw up, easy to revert.

Updates

Don’t apply an update unless the changelog shows it directly affects you. For example, the git plugin was just updated to fix a regression that prevented using env parameters in the branch name — I need that, so I upgraded that plugin. This goes back to the plugin issues I mentioned above… and if you screw up, it goes back to my backup issue I mention above. 🙂

Planning

Do your best to plan your jobs and pipeline. Figure out the abstract ideal that you want. Then figure what Jenkins can offer to get you there. You might find plugins, but you might also have to write your own scripts and other management.

Then, don’t be afraid to revise and revise again, even starting over if you have to.

DIY

I wanted to use a lot of plugins, because they’re supposed to make it easy, right? Well, it depends. The Xcode plugin is supposed to make it easier to do xcodebuild-based, builds. And in theory it does, but our needs are different and so I just have to get dirty with our own scripts.

The email-ext plugin is really cool, but I couldn’t get it to understand and bring in a bunch of env vars and do other bits of logic processing that we needed. So again, scripting to the rescue. Net::SMTP to the rescue.

CodeSign problems

I noticed from time to time that I’d start a build and eventually xcodebuild or ‘xcrun PackageApplication’ would fail:

“There are no valid public/private key pairs in the default keychain”

It seems this would happen after rebooting the machine (and Jenkins started up via the launchd stuff). I could manually use launchctl to unload then (re)load and things might be back to normal, but that’s annoying.

I found this post at stackoverflow.com, and adding the “SessionCreate = true” key/value to the launchd plist seems to do the trick.

But then further down things failed out. Despite the codesign command line tool being granted special Keychain access, it still hated life:

<path to build output app>: User interaction is not allowed.

Uh…

So I found this and added a:

security unlock-keychain -p "" login.keychain

in the custom ruby script before invoking xcodebuild and PackageApplication.

BTW, I don’t have the passwords encoded into the script. Remember the use of the .netrc file? Use that.

Speaking of .netrc… it seems curl doesn’t like a .netrc where newlines separate entries… it wants everything on one line.

Passing Data to Downstream Jobs

Using Jenkins’ parameterized build mechanism, I was able to pass parameters around. I’d have the main build job, which would then use the promoted builds plugin to allow me to move the build through a pipeline, like to promote to QA, then promote to the App Store, or whatever. But there’s a lot of information from the main build job that I want the downstream jobs to have so the jobs can be properly named, emails can be properly formatted, version control tags set up, whatever. I found the easist way to do this was at the end of the main build job to use a little ruby scripting, create a Hash with all the things I cared to preserve, use YAML to preserve it to file, make sure the build job archived and fingerprinted that .yml file, and moved that “config.yml” file along with the rest of my archives and promotions. Then it’s simple enough to load and look up key/values out of the config.yml file in the later job scripts.

Tagging Git

One of the downstream jobs is a “deploy to the Apple app store” job. Certainly when we do that we want to tag our github-hosted repository with info about the build so we can know what was deployed. Trouble is, at that phase of the build pipeline we don’t have source code. So we can’t just “git tag” then “git push”. What to do? Use the github v3 API. In that config.yml file I was able to preserve the SHA hash of the code we used, so that’s all we need.

At first I thought to use the Tags API, but as I played with that it didn’t work. Even if I could create a tag, it wouldn’t show up in the Tags area of my git GUI app SourceTree. In fact, it started to give errors about refs. So I started to play with the References API and tried things like:

$ curl -XPOST --netrc https://api.github.com/repos/:user/:repos/git/refs \
-d '{"ref":"refs/tags/MyTestTag","sha":"be43262431c7a4b9db67a23d37f51e7901b9845c"}'

and lo… that seemed to work.

Is that enough? Is that correct? I don’t know. I’m not an expert on the low-level plumbing of git. I have contacted github support, but as of this writing have yet to hear back.

The Journey Continues

That’s all that I have for now.

It’s a lot of work to set up a build system, but it’s rather satisfying to do it. I still have a long ways to go. Right now we just have the basics for building and deploying to help our general workflow. It still needs lots of work for validations, testing, continuous integration, and other good stuff like that. One step at a time.

Let’s be consistent

/ hsoi

From Unc I read how a man was fired from his job for “liking” a Facebook post.

Daniel Ray Carter Jr. logged on to Facebook and did what millions do each day: He “liked” a page by clicking the site’s thumbs up icon. The problem was that the page was for a candidate who was challenging his boss, the sheriff of Hampton, Va.

That simple mouse click, Carter says, caused the sheriff to fire him from his job as a deputy and put him at the center of an emerging First Amendment debate over the ubiquitous digital seal of approval: Is liking something on Facebook protected free speech?

I think most people would agree that yes it is free speech, it should be protected. You are expressing your opinion. To “click Like” is merely a shortcut/shorthand for saying “I like this” or “I agree with this” or some other statement of agreement and affirmation. It’s just a more efficient (lazy?) way to do it. Are we saying that if someone typed a comment under the posting “I like this” that that wouldn’t be protected? or if I wrote it on a piece of paper, or spoke it aloud in a public venue for others to hear? So why wouldn’t clicking “like” be offered the same protection under 1A?

But apparently not:

The interest was sparked by a lower court’s ruling that “liking” a page does not warrant protection because it does not involve “actual statements.” If the ruling is upheld, the ACLU and others worry, a host of Web-based, mouse-click actions, such as re-tweeting (hitting a button to post someone else’s tweet on your Twitter account), won’t be protected as free speech.

Methinks someone in the lower court doesn’t quite understand technology advancements.

“We think it’s important as new technologies emerge . . . that the First Amendment is interpreted to protect those new ways of communicating,” said Rebecca K. Glenberg, legal director of the ACLU of Virginia. “Pressing a ‘like’ button is analogous to other forms of speech, such as putting a button on your shirt with a candidate’s name on it.”

So isn’t that interesting? Our Founding Fathers never could have imagined this thing called the Internet. They could never have imagined Facebook or Twitter or iPhone’s. They could never have imagined the act of pressing with your finger could act as a proxy for expressing your liking something. But just like they understood technology advancements like clay tablets, papyrus paper, quill pens, moveable type, printing presses, pony express, and so on… they probably understood that technology would continue to advance. I’m sure they wanted speech to be protected regardless of the technology used to convey it. Certainly that’s what it seems the WaPo, the ACLU, and others put forth. I know many people will be outraged if these advances in technology would not be upheld as protected under that 200-year old document written by men that (some day) had no clue.

So why isn’t this same standard held to the Second Amendment?

Setting up Jenkins… things I’ve learned so far

/ hsoi / 3 Comments

At the day job I’ve been tasked with setting up an automated build system. This is a Good Thing™ and I actually volunteered to do this task. I’m interested in doing what can help us make our code better, deliver our products faster, and make our lives easier in terms of automating things we don’t necessarily need to be involved in. So a Mac Mini was obtained to be a build machine, and I’ve been working to set things up.

I should note that I’m not a dedicated buildmeister. I know at larger companies they can have a person (or persons) whose full-time job is dealing with builds, but I’ve never been one of those people. I’ve done some build hacking in the past, but it was always homebrewed scripts and systems. This time around, let’s use an established system instead of homebrew.

It seems the modern hotness is Jenkins. Yes there are other options, but all signs via Google searching point to it. As well, it seemed (note the verb tense) like the more Mac-friendly solution. If nothing else, it had a nice Mac OS X installer. 🙂

I’m still far from having our complete build system, and I reckon as I learn more things on this list will change. But I’ve already learned a few useful things and I felt like sharing and adding to the greater knowledge-base out there that, through Google-Fu, helped me get this far. Maybe with my additions, someone else can be helped and maybe with a little less frustration and time sink.

The Project

A little context about the project being built.

Working on Mac OS X, developing iOS apps. Thus we’re using Mac OS X 10.7 Lion as our dev runtime environment. We’re using Xcode 4.3.3. Jenkins is version 1.474. We use git and a mix of private and public repositories on github.com.

First Time Problems

Started with a brand new, fresh from the box Mac Mini. Of course, before attempting any of this CI-specific work, the box was brought up to date with OS updates, Mac App Store updates, and so on. Note! Jenkins is a Java app and Java is NOT installed by default. So after you run the Jenkins installer and it tries to start Jenkins, things will probably fail. The OS will prompt you to install Java, so you’ll have to do that, but then Jenkins should end up running. Not a big hurdle, but it’s there.

Make sure to launch Xcode.app (the GUI app) and get it properly happy. This is mostly ensuring the various downloadable components get downloaded and installed, like command line tools and such.

You will be using command line tools, thus you will have to run xcode-select. But being as this is Xcode 4, things are different.

$ sudo /usr/bin/xcode-select -switch /Applications/Xcode.app/Contents/Developer

This will ensure xcodebuild can be found. And note, the first time you run xcodebuild, it won’t run! While you agreed to the license agreement when you first launched the GUI Xcode.app, that’s not good enough for xcodebuild; you have to agree to it all over again via the command line. Just be aware. It’s not a big deal, it’s a first-time-only occurrence, but it’s there to deal with.

The User (login) Matters

This was the source of much fun and frustration for me.

The first time I installed Jenkins I had no idea what to expect. That it had a proper Mac OS X Installer .pkg was cool, but it was also hiding a secret. Any time there’s a “Customize” button I like to click it and see what options there are. I noticed it provided two options:

  • Start at boot as “daemon”
  • Start at boot as “jenkins”

I had no idea what the relevance of these two options were. The default was “daemon” and “jenkins” was unchecked. I just figured to trust the default installer settings. Ha ha ha…  This actually caused me the most trouble and pain. I won’t recount the many hours spent dealing with this, but I will explain the issues.

I opted to go with the installer’s default of “daemon” but that creates a problem because “daemon” is a special user. When Jenkins needs to do things, it’s going to look in non-typical places, like /var/root for ssh keys or the like. Basically, it’s going to cause you a lot of headache.

When you search around for information about this, everyone starts to talk about using the “dscl” command line tool to create a hidden secret “jenkins” user and run Jenkins that way. This makes sense because it creates a user of restricted ability so it helps to keep the system secure and minimize chance of damage should someone gain access to the system via the Jenkins system or login. But in practice, this turned out to be a big pain in the ass because of what we’re doing. I’m sure there are some projects where this won’t be that problematic. But writing iOS apps brings issues. Apple makes great products, but you find they are great as long as you color within the lines; the lines might be really wide and vast, but still you must color within them. Trying to deal with this secret “jenkins” user created various issues.

For Xcode (or xcodebuild) to do code signing, it needs access to private keys and certificates and other such things. Thus the Keychain is involved, but this hidden “jenkins” user doesn’t have one. Again, more searching will turn up possible solutions, but they are not ideal solutions. In fact, one solution of putting these into the “system” keychain really defeats the purpose doesn’t it? Then there are the .mobileprovision files needed during the signing process, and those must exist somewhere in the “jenkins” user structure. In my case, the provisioning files may be updated fairly often due to the addition of device UDID’s, and setting up the initial .mobileprovision files by downloading to the logged in user then copying it all into the hidden jenkins user locations… it was just turning into a massive pain in the ass. And you’ll have to do this at least once a year, when you renew your iOS account with Apple.

So I fumbled with various permutations of these user setups and it was all just frustrating to me.

In the end, I backed everything out and started over (for the umpteenth time). I created a “jenkins” user via the System Preferences. That is a full-on login-able user named “jenkins”, admin user. I then logged in as the “jenkins” user. I ran the Jenkins installer, customize, and selected the “start at boot as ‘jenkins'” option (and deselected the “start at boot as ‘daemon'” option). I ran the installer. That did not immediately succeed, but the resolution was simple. Stop the daemon (using launchctl). Edit the .plist for the Jenkins launchd job and changed the location of JENKINS_HOME to /Users/Jenkins. Then restarted the machine. Life was good.

I think today I’m going to try doing it “yet again” tho… deleting the Jenkins user and creating it again, then installing again, and this time making a subfolder, like /Users/Jenkins/Jenkins_CI or something like that. That’ll contain all the stuff in one folder and not litter the actual home directory.

Is this the right thing to do? Well, from a pure security standpoint, no. It’s now an admin user that can log in and have all sorts of fun. But my feeling is this machine will be behind our firewall. If this box gets compromised, we have bigger problems anyways.  Could it mean bigger problems for me down the line with Jenkins-CI itself? Maybe. We’ll see. I’m not necessarily advocating taking this approach, but it’s just the one I’ve presently taken. It appears that it will create less long-term hell for me, especially in dealing with Xcode and the OS.

Authentication

Another source of pain was dealing with authentication issues.

Since we use private repositories on github, we need to authenticate. This proved to be more difficult than it should have been.

When I first tried a basic build of our project, I installed the various git plugins within Jenkins, created the new job, set it to obtain the source via the http protocol, and off we go. Well, that failed. The job just hung forever, I reckon waiting for authentication. So how can we authenticate? I saw no configuration options for it. So I started playing with ssh.

ssh would actually be a good thing in many ways, but with all the aforementioned user/login issues, it was becoming a massive pain. The way Mac OS X seems to handle ssh is by launching the ssh-agent on demand. Well… it can’t do that for the secret jenkins user. I tried all sorts of things, but I just could NOT get it working. *sigh*   Another vote for making a fully-realized jenkins users instead of this secret hacked one.

The one thing I had to then do was edit my ssh keys to not have a passphrase, and that would work (no need for ssh-agent, no need for interactive issues). But lacking a passphrase ain’t so hot. But it seemed to work… well, at first.

The next problem? Our git project has submodules. In the .gitmodules list, the “url” for each submodule used http as the protocol. And so, when the Jenkins-CI git plugin tried to obtain the submodules, we were back to the original problem. FML.

I tried fiddling with the repository URL as specified in the “git” section of the Jenkins-CI job to be something like:

https://:@github.com/path/to/repo.git

and while that worked for the top-level repository, the submodules still tripped me up. Plus, I did not like the idea of storing a password in plain text and sight like that.

I was able to solve it tho… through the magic of the .netrc file.

I created the following file in /Users/jenkins/.netrc

machine github.com
login <github username>
password <github password>

Saved it. Did a “chmod 600” on it. And lo…. everything worked. Huzzah!

I think the git plugin should have a way of doing authentication, but I’m still too much of a Jenkins n00b to know really where the fault lies.

More Ahead

At this point, I’m able to clone our git repository and build the app. It’s very basic at this point, with much configuration and fiddling yet to come. But I spent a good deal of time on these matters this past week and tried to sift through a bunch of search engine results to wind up here. So much of the information out there is Linux or Windows based, which is fine for the general but some of the little details aren’t there. And I can’t imagine we’re the first people to need to build in such a way, but I just couldn’t find anything specific to the particulars of our setup. So here’s hoping this helps contribute back.

As well, if you the reader have any information, insight, comment, etc. please share. I’m still learning, and constructive contribution is appreciated.

Trying Trello

/ hsoi / 3 Comments

The new hotness in software development is “agile“. At my prior day job, it was waterfall. Sure they tried to adopt agile processes, but it really wasn’t going to happen. Due to the nature of the products and process, it just can’t be agile, tho they could try to adopt a few things and make some improvements. But at my new day job? Agile can make a LOT of sense. Take home: you can’t impart the process merely because it’s the new hotness or you think if you just adopt X process it’ll solve problems. Like any problem to be solved, you have to understand your problem fully and then apply the right tools to solve it, and that includes what processes you use.

However, it’s tough getting folks on board, so I’ve desire to try to sneak in agile stuff as we can. It’s nothing more than a commentary on human condition — we tend to resist change. If all this change is dropped on folks all at once, we’ll balk. I would too. Massive sudden change, especially when you’ve still got daily chaos and stress to manage, well… the change will be rejected. But if we can make little changes here, little changes there, over time we get there.

One thing I can tell is we all need a way to see the whole picture. We’ve got so many things going on at once, and it changes on a daily and sometimes sub-daily basis. I find myself often making lists and (re)telling my todo lists to my teammates merely to help ensure 1. I know what I’m doing and am on track, 2. that we’re all on the same page. By sharing with them I hope that if there’s a mismatch, they’ll speak up and correct. I’d rather be perceived as over-communicating than under-communicating.

But all this talk doesn’t solve the problem for everyone. The dev team is one thing, testing another, production another, sales another, marketing another… there’s so many things. Sure, we could use our issue tracking system, and there’s a lot of sense in that approach. But the issue tracker doesn’t work as well for non-dev folk PLUS it’s harder to get a 1000′ view. Yes, pictures/diagrams can make a big difference.

One thing from the agile/scrum world is having the daily stand-up meetings at “the wall”. Let’s set aside the meeting for now (again, baby steps), and just focus on the wall. I’ve suggested the wall, because I think that would be useful. Pick a wall in the office, divide it up in whatever way makes sense for us, then start populating it with sticky notes to represent all the tasks to be done. I think that’ll be useful at keeping a somewhat permanent record of the state of things (unlike a whiteboard, which will be erased eventually). Plus it allows anyone to just look at the wall at any time of the day to see where things are. CEO wants to know what’s up with Customer X? Just look at the wall. Did we ship Product Y yet? Just look at the wall.

Alas, one shortcoming is 1. we don’t have a lot of free wall space left in the office, 2. the wall is restricted to the physical. I’ve been searching for a digital wall solution and haven’t found much that thrills me. Something that we could access from any computer via a web browser (or even a platform-based app), and it would look good. But then we could also access from iOS or Android devices. In the office. Out of the office. At 3 AM, during office hours. From my desk, in a meeting on the shared screen. Whatever. Something with power to do what we need, but flexibility to work for anyone, not just the geeks.

So… I recently read this article from Joel Spolsky about “Software Inventory”. While I read it, it sounded like it was talking directly to me and our situation. I went looking at their solution:

Trello

I’ve only just started to play with it, but it seems like it could be the answer to my problems. I could see using this in my personal life, for Hsoi Enterprises, and for the day job stuff. Even if the rest of the office doesn’t buy into it, it could be useful for my own management of my tasks and issues to deal with. Being able to SEE everything instead of sorting through a bunch of text notes and to-do lists is sometimes much more useful, even if it ends up being redundant and a little more maintenance to keep 2 data stores in sync.

It’s still preliminary, but it’s promising.

DR Performance Practice Deck for iOS version 1.1 Now Available

/ hsoi

The DR Performance Practice Deck for iOS version 1.1 is now available.

Lots of good things in this update, the biggest news being score tracking is now available.

Please spread the good word!

Thank you for your support.

Apple TV or not?

/ hsoi / 13 Comments

Any of you have experience with the Apple TV?

I’ve been thinking about ditching cable, and the Apple TV seems rather compelling (Apple lifer that I am). For $99 I reckon there’s no harm in just buying one and trying it (first world problem, I know). Still, it’d be nice to get some input from folks that may have one.

Your input requested. 🙂

 

Fun with SleepWatcher

/ hsoi

I do the majority of my computing work on a MacBook Pro — a laptop computer. The primary reason for a laptop is portability. Sure most of my work gets done at my desk, but sometimes I have to move or want to move. But moving isn’t always so quick and easy because I have lots of things hooked up to the MacBook Pro, like an external FireWire hard drive so I can run Time Machine backups. Trouble is, you can’t just yank out the cable for the disk, you must eject it, then you can remove the cable, else you could risk corruption of the data on the drive. Of course when you’re in a hurry to get out, or just consider we all get absent-minded from time to time gee… this is a pain in the neck and wouldn’t it be nice if we could automate it so we didn’t have to worry about it! Isn’t that what computers are useful for?

As I searched about for a solution, I found a little app called autoEJECT. It’s not bad, but after using it for a few days it just didn’t work like I needed it to. It did unmount things fine, but it failed to mount my FireWire drive. It has an option to “Force Mount”, which I then enabled but when I did so it mounted everything including the secret hidden partitions Apple has for recovery and booting. This is not desired as that 1. clutters things up, 2. exposes volumes that you do NOT want to risk to change. Even then it was a little quirky in how it would remember volumes to mount and unmount and just got in the way more than it was being useful. I figured there had to be a better way.

I found something called SleepWatcher.

SleepWatcher is a system daemon that watches for various events on the system, like sleep and wake, and allows you to execute scripts in response to those events. The app isn’t for novices, and the author actually goes out of his way to make it a bit complicated to use because it’s not a tool for the masses to use. But if you know what you’re doing, it can be useful. So I figured to give it a shot.

Of course, SleepWatcher itself isn’t what does the voodoo. You need scripts.

This website listed a couple scripts.

His sleep script was:

#!/bin/sh
osascript -e 'tell application "Finder" to eject disks'

and his wake script was:

#!/bin/sh
/usr/sbin/diskutil list | \
 awk '/Apple_HFS/ {print $NF}' | \
 xargs -I{} /usr/sbin/diskutil mount {}

Those are reasonable starting scripts. I thought about changing the wake script to just look for my one volume, because as it is now is ends up mounting all my partitions. Since they’re already mounted it really makes no difference, but it’d just be more efficient to scan for my lone volume. Still tho, this is useful for being complete.

The sleep script I modified slightly based upon what I found here. The change I made was simple:

#!/bin/sh
osascript -e 'tell application "Finder" to eject (every disk whose ejectable is true)'

which works a bit better in my situation. Granted, this will kick out optical discs, but as I rarely use them it’s not a big deal to me. The author also notes how this will not work if the Finder isn’t running, but I use the Finder and it should always be running (else there’s a problem).

Along the same lines as these is this hint from Mac OS X Hints, but he takes some vastly different approaches in what is selected for unmounting and what is selected for mounting. Not exactly what I want, but it shows what else you can do.

Anyways, SleepWatcher is pretty nifty and reading the code seems to be pretty well written too. Cheers, Bernhard Baeher!