coding in the rain

Homebrew for OS X

2010-02-22T22:10:58-08:00

Most Unix-based platforms have at least one package management system. Some are like Debian where the package system is one with the operating system. Others are bolted onto the system.

Mac OS X has had two package systems for many years. Fink is a port of the Debian apt system. Macports is primarily a port of the FreeBSD ports system. Unlike Linux, both of these are sandboxes that you run on top of the Apple supplied system. In many ways, this is not unlike Cygwin for Windows.

In my opinion, both Fink and Macports suffer from the fact that neither makes an effort to integrate with the core operating system. I already have a tested version of Perl or Python on my system. I’d rather the package system stuck with this as much as possible.

Now there is a new package manager on the scene called Homebrew. I think it is very promising. It is similar in approach to Macports in that you build recipes in a high level scripting language. Macports uses TCL and Homebrew uses my favorite language Ruby. Both pull down a source package, configure it, and build the code. Both also manage dependencies and attempt to get other supporting libraries.

There are three major differences. The first is that Homebrew is trying to augment the existing Mac OS X distribution from Apple. It uses what is already installed as much as it can. The other difference is that recipe development is very simple using Ruby. The last major improvement is that Homebrew is developed on GitHub so forking and submitting recipes is trivial.

Changes to Default Install

I’ve spent some time getting familiar with Homebrew the past week or so. I like what I’ve seen but I don’t completely agree with the recommended installation. I’m offering my approach as an alternate.

There are two things I don’t agree with. The first is installing Homebrew in /usr/local. Although Mac OS X surprisingly doesn’t ship with a default /usr/local directory, this path has such strong meaning to anyone who uses other Unix systems like Linux. I absolutely expect to download the source to /usr/local/src and build it by hand.

The second recommendation I don’t agree with is to make the Homebrew directory user owned instead of root. This is probably more a matter of preference but I’d rather mimic the typical behavior of root owning software available system wide. The reasons the Homebrew team give are valid but I guess I’m stuck in my ways.

Setting up Homebrew

The author of Homebrew made a very good decision to allow it to be installed anywhere you want. So the first thing to do is pick where you want your Homebrew install to live. The approach is to put the software into a special directory called HOMEBREW_PATH/Cellar. Any binaries are symlinked in HOMEBREW/bin.

I recommend emulating what Macports does and creating a standalone location for Homebrew. I picked /opt/homebrew for my install but you can chose whatever you want. I also use root as the owner so my commands will be using sudo.

sudo mkdir -p /opt/homebrew

Next, bootstrap an initial install of Homebrew. You can either git clone the repository directory or pull down a tarball of the master branch:

sudo -s
cd /opt
curl -L http://github.com/mxcl/homebrew/tarball/master | tar xz --strip 1 -C homebrew

Since I’ve chosen to install Homebrew into its own directory, I need to make the paths available. Mac OS X uses path_helper as a way of defining the PATH and MANPATH settings. The path_helper will read /etc/paths and all entries in /etc/paths.d. The files are nothing more than a path to add separated by newline. There is a similar arrangement of /etc/manpaths and /etc/manpaths.d. If you prefer, you can just add these to your PATH and MANPATH variable in your profile:

sudo -s
echo "/opt/homebrew/bin" > /etc/paths.d/homebrew
echo "/opt/homebrew/share/man" > /etc/manpaths.d/homebrew

Start up a new shell and test out brew:

sudo brew list

You should see nothing installed. Now try installing a package:

sudo brew install wget

When it is complete, you should have a /opt/homebrew/bin/wget symlinked to the binary. There are the typical package options (search, install, uninstall, list). Commands are in the form of brew action formula.

The recipes files are simply Ruby scripts in the HOMEBREW/Library/Formula directory. These files can be updated. To convert your Homebrew to a git clone:

sudo brew update

Your Homebrew directory is now a git clone of the project. This means you can do all the normal cool git activities you want. Create a new branch to test out formula changes. Track other user’s formulas by adding a git remote. Merge as you please. It is extremely flexible and elegant.

Homebrew is relatively new. The package list is much smaller than Fink or Macports. But there is a lot of activity around the project and I’ve already had great interactions with other users creating packages.

I think the advantage of my layout is that Homebrew is isolated cleanly and I can selectively use it to add packages. I can still install and build software to /usr/local with no conflicts. For example, I prefer the binary MySQL and Git installs over using the packages since I rely on these packages and don’t want to deal with any issues building them. For other software like ImageMagick or ffmpeg, Homebrew is a great way to easily add it and keep it up to date.

Capistrano Boss

2010-01-27T22:51:00-08:00

Capistrano is a very useful deployment management tool widely used for Ruby on Rails production managment. I’ve used it for over 2 years to manage multiple environments including production, staging, and testing. It is easy to get started with and also easy to extend.

It’s a tool that is so flexible that the initial reaction is to write a task for everything. The running joke on my team has been that for every question “can I …?”, the answer is “there’s a task for that.” Many times, it was true. Recently, we moved our application from Mongrel to Passenger. I took the opportunity to re-evaluate what tasks we needed. I eliminated all the dangerous stuff (like directly touching the database or loading up script/console via cap). I then extracted the most useful tasks.

Originally, the extensions were implemented in the lib folder or in a deployment plugin. Since plugins are not the way to go anymore for Rails, I decided to turn all of that into a gem. Using the fantastic new Gemcutter tool, I created Capistrano-boss.

There are recipes split up in four namespaces: deploy, apache, passenger, and rails. Most are simple but there are a couple that I find very useful. First, apache and rails both support downloading logs (log:fetch) and watching logs in real time (log:watch). The realtime log feature is one we use very frequently. I can’t even remember where I found the tip originally but it does a tail -f and marks up the stream with the host and timestamp. It makes it very easy to watch activity on all your servers at once. On a busy server, it can get overwhelming. I intend to put in some filter options for this task at some point.

The other useful recipes are for managing the database.yml file. The recipe rails:config:database creates a configuration in #{shared_path}/config/database.yml. It will prompt for the database password and write it out. This allows you to checkin a clean database.yml without a password into source control. You can easily set it just once in the shared configuation location. When deploy is run, the recipe rails:deploy:config will symlink #{shared_path}/config/database.yml to #{current_path}/config/database.yml.

One other addition is an extension to the way the Subversion repository is determined. The readme explains it but the idea is that you can set :branch or :tag and have the repository path adjusted. We used it to set which tag to deploy on the cap command line:

cap deploy -s tag='myrelease'

It assumes the standard subversion layout (branches/tags/trunk) and defaults to using trunk. As of this week though, we have moved to a pure git source control system and (thankfully) will no longer need the subversion support.

I’m hoping people will fork and submit back their favorite deployment related recipes. My goal is for light weight tasks, not full on configuration management. I headed down that path and it wasn’t good. Puppet or Chef is probably a beter solution for that. But many Rails applications don’t need much configuration help and that’s what I want in capistrano-boss. The source is on Github and the gem is published on Gemcutter.

Finding Installed Packages On Ubuntu

2010-01-15T21:44:29-08:00

Ubuntu uses the Debian package system. There are multiple applications for managing the packages on your system including Apt, aptitude, dpkg, and Synaptic. Each of these tools interacts with the deb format package files.

Synaptic is a great tool for managing packages in a GUI. But what about the command line? I run an older Dell PowerEdge 400SC as a server literally in a closet in my house. It’s got power and a network cable so I manage it over SSH from my laptop or my Ubuntu desktop. For whatever reason, I never could figure out the right way to list packages already installed. I found two methods to list the packages.

dpkg-query

This tool lists the installed packages as well as some other information. It accepts a pattern for finding matching packages. In addition, it can show package status and all the files included in the package.

dpkg-query actions
-l, --list package-name-pattern...
	List packages matching given pattern.
-s, --status package-name...
	Report status of specified package.
-L, --listfiles package-name...
	List files installed to your system from package-name.
-S, --search filename-search-pattern...
	Search for a filename from installed packages.
-p, --print-avail package-name...
	Display details about package-name, as found in
	/var/lib/dpkg/available. Users of APT-based frontends
	should use apt-cache show package-name instead.

The following is an example for the ruby package:

$ dpkg-query --list ruby1.8
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Cfg-files/Unpacked/Failed-cfg/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name    Version            Description
+++-=======-==================-======================================================
ii  ruby1.8 1.8.7.174-1ubuntu1 Interpreter of object-oriented scripting language Ruby

apt-show-versions

Another useful tool for listing both the installed packages and information on out-of-date packages is apt-show-versions. It is not part of the base Ubuntu install but can easily be added:

$ sudo apt-get install apt-show-versions

The tool has many useful query options. It can show specific packages, match by a regex pattern, and list upgradeable packages:

Apt-Show-Versions v.0.16 (c) Christoph Martin

Usage:
 apt-show-versions         shows available versions of installed packages.

Options:
 -stf|--status-file=<file>  Use <file> as the dpkg status file instead
                            of /var/lib/dpkg/status
 -ld|list-dir=<directory>   Use <directory> as path to apt's list files instead
                            of /var/state/apt/lists/ or /var/lib/apt/lists/
 -p|--package=<package>     Print versions for <package>.
 -r|--regex                 Read package with -p as regex
 -R|--regex-all             Like --regex, but also show not installed packages.
 -u|--upgradeable           Print only upgradeable packages
 -a|--allversions           Print all available versions.
 -b|--brief                 Short output.
 -nh|--nohold               Don't treat holded packages.
 -i|--initialize            Initialize or update package cache only (as root).
 -v|--verbose               Verbose messages.
 -h|--help                  Print this help.

Examples:

$ apt-show-versions ruby1.8
ruby1.8/karmic-updates uptodate 1.8.7.174-1ubuntu1

$ apt-show-versions -a ruby1.8
ruby1.8 1.8.7.174-1ubuntu1 install ok installed
ruby1.8 1.8.7.174-1        karmic         us.archive.ubuntu.com
ruby1.8 1.8.7.174-1ubuntu1 karmic-updates us.archive.ubuntu.com
No stable version
ruby1.8/karmic-updates uptodate 1.8.7.174-1ubuntu1

Apple Tablet Wishlist

2010-01-10T21:35:37-08:00

All around the tech world, heavy speculation is underway as anticipation grows for the next thing from Apple at the end of January 2010. CES has come and gone. It seems to me Apple managed to overshadow the whole thing by simply saying they are launching something. Talk about influence.

So let’s assume there is a tablet coming. What does that mean? Why is it useful? Why would I want one?

I have a wishlist for such a device. Today, I use a MacBook Pro 15”, iPhone 3GS, a Sony Reader, a Sony Playstation 3, a Sony PSPgo, and a Dell Optiplex running Ubuntu Linux. That’s a lot of devices that do a lot of things. What could the tablet do that all those devices can’t?

What I Want

Replace Other Devices

To add another device like this, I want it to replace others. In particular, I’d want the tablet to replace a dedicated ebook reader (my Sony 505 in this case). I’d also want it to negate anything I’d want a netbook to do. I had a Dell Mini 9 for a couple months but found it too much of a compromise.

Don’t Replace Every Device

Conversely, I don’t want it to replace all my other devices. I have so many because they all do something well. The tablet should be no different. It should not aim to replace everything. It won’t replace my MacBook or my desktop machine. Not my phone either. And probably not my PSP. Unlike most people, I don’t find the iPhone that good for games and like dedicated game consoles.

ebooks

The biggest reason I would want a tablet is as an ebook reader. I’ve started to transition away from print this past year. It has now become very rare for me to buy technical books any other way than electronic. I’m also trying to buy other books as ebooks as well.

My Sony 505 is a nice piece of hardware. The form factor is much better than the Kindle in my opinion. However, the major problem is not the display or the controls. It’s the size. It can’t show a large form technical book like the O’Reilly or Pragmatic Programmer books I typically buy. It does support PDF but not very well.

The tablet for me would need a screen size that can easily scale a large format book with minimal loss of fidelity. I would also gladly move my magazine subscriptions to digital as well. I buy 95% of my music digitally. I still buy some Blu-Ray movies but I use Hulu or Netflix much more frequently. It only makes sense for the same reasons to move all my reading to digital.

iPod HD

The iPhone has a pretty good iPod embedded. I like it in most cases like commuting or traveling. But at home, I could easily see enjoying a larger format personal media device. My wife and I watch very different content in different ways. She watches shows recorded on our Tivo on a flat screen TV. I watch shows off my MacBook or PS3. A personal device that has high definition video support (and can stream it) would be great. I think online streaming services like Netflix and Hulu are required too.

The other exciting thing a table could offer is a return to albums for music. I’m old enough to remember getting vinyl records (then later cassettes and CD’s) and listening to the whole album while poring over the liner notes and lyrics. The new iTunes albums could flourish on the tablet.

WiFi

It goes without saying but the tablet must have full WiFi support. I think 802.11n has to be supported to enable it to stream HD reliably. Besides, Apple already sells a whole line of 802.11n wireless gear.

Extremely Sleek Form Factor

Apple are masters at creating amazing hardware designs. I’m confident the tablet won’t disappoint here. I think it needs to be about the thickness of the iPhone but much easier to grip and hold. It should follow the trend of the iPhone and have the bare minimum of hard buttons. I think the Kindle DX is a clear example of how not to do it.

Transparent Integration

Other than the ebook capabilities, I think enabling the tablet to seamlessly integrate with your other devices is what will set it apart. Some examples would be allowing it to function as an additional screen when you dock it with your MacBook. It should be perfectly sized to act as a status window or document viewer monitor. Other companies already sell dedicated monitors in the size that the tablet is rumored to be. Think about how cool it would be to have your MacBook hooked up to your 24” Cinema display and the tablet docked next to that. You could put your IM windows or a web page on it. Then when you are ready to leave, just pull the tablet and go.

Another integration possibility is to be the universal remote for the other entertainment devices in your home media center. Or how about hooking the tablet up via HDMI to your plasma TV and streaming Hulu and Netflix.

Yet another is for live events. You could be watching a soccer match on ESPN while a synchronized stream of stats and other data could be on the tablet. There are so many possibilities to using a device like this as an extender.

What I Don’t Want

Cellular 3G

I’ve got an iPhone (and everyone has some sort of mobile phone). The tablet will not fit in my pocket (if it does, it’s a failure). I would rather see it tether to the iPhone. Or carry a portable 3G WiFi bridge instead.

Keyboard/Mouse

There is absolutely no advantage to the tablet over a slim laptop if you start carrying around a keyboard or mouse. Integrate as an extender and have a touch screen keyboard.

Bigger iPhone

If the tablet is simply a large iPhone or iPod Touch, it’s doomed. The iPhone works because the UI is designed for the iPhone and only the iPhone. Tablet needs to do the same.

Smaller Mac

If the tablet is just a less capable MacBook, it is again doomed. That’s what Microsoft is building. And it is worthless. You can buy a cheaper netbook or a spend a little more for a real laptop.

So that’s my wishlist. No idea how accurate it is but I get excited thinking about this kind of device.

I’m a lifelong Apple fan. Apple literally defined the personal computer for me. My parents were completely unaware that the Apple IIc they bought for me in 1984 would shape the course of my life. I’m a programmer today because of that machine. So I do believe that Apple has the power to change everything. I can’t say for sure that the Apple tablet is going to be that kind of device but I won’t be the least bit surprised if it is.

Rails Development on Ubuntu 9.10

2010-01-04T23:03:49-08:00

I use both Mac OS X and Ubuntu Linux for Ruby on Rails devleopment. In both cases, setting up an environment is relatively easy but not without some pitfalls. I’ve been using Rails on Mac OS X since Tiger 10.5 and on Ubuntu since 7.10. I’ve kept Backpack pages with my recipes for how to set it up. This post will explain how to setup Ruby on Rails for Ubuntu 9.10. It’s certainly not the only way and it’s obviously biased by my preferences. But I’ve used this recipe successfully over several ubuntu releases.

So with that out of the way, let’s get Ruby on Rails up on Ubuntu.

Assumptions

The steps are done assuming a new clean install of Ubuntu 9.10. I use 64-bit Alternate typically for my installs but it should be nearly identical both for 32-bit and/or Desktop installs. Also, I’ll use apt-* commands. Use synaptic or dpkg if you wish.

Pre-requisites

By default, you will have minimal development tools available. The first thing to do is install the primary tool chain. This is mostly gcc and common development libraries. In Ubuntu, many packages have a lib*-dev version that includes headers and the libraries needed for linking. Fortunately, it’s very easy to get the default gcc installed:

sudo apt-get install build-essential

The package build-essential is a metapackage that includes gcc and all the supporting libraries and tools. It’s very much like installing Xcode on a Mac. You now will have all the compilers and are ready to build most source that is packaged via autoconf.

Ruby

Ubuntu 9.10 includes both Ruby 1.8 and 1.9 in the package system. At this time, Ruby 1.8 is still more widely used for Rails development. Hopefully that will change soon since Ruby 1.9 has many great improvements.

I generally have used the packaged ruby installs for Ubuntu. There aren’t many configuration options you would tweak when building ruby so it’s only a question of the ruby version that would make you want to install from source in my opinion.

There are a couple other libraries you should install from packages as well. These include native libraries for SSL and zlib. I also install RDoc, RI and irb from packages. You also should install ruby-dev to be able to compile native ruby gems later.

Here’s the install for the ruby stack:

sudo apt-get install ruby rdoc ri irb libyaml-ruby libzlib-ruby libopenssl-ruby ruby-dev

Another optional install is the first edition of the Ruby Book. It might be handy but you do have a newer version than that don’t you?

sudo apt-get install rubybook

It will be installed to:

/usr/share/doc/rubybook/html/index.html

Rubygems

So far, setup has been easy since everything comes from the package system. Now we start to deviate. Rubygems has been a source of contention between Debian and the Rubygems developers. The core problem is that both are package systems. Rubygems can (and arguably should) manage all gem packages. On Debian systems, dpkg should own all the packages installed in the system. This could include gems as well. This is where the friction starts.

No matter which side of the argument you might find yourself, the good news is that installing rubygems from source is easy and it puts it in the system location anyway since that is where Ruby already is installed.

My rule for gems: Only use apt to install supporting C-libraries to allow native gems to build. Install gems themselves via a source install for rubygems.

For locally built source that I install (i.e. to /usr/local), I prefer to put the source in /usr/local/src. But feel free to install anywhere you like.

To install rubygems, you need to download the latest tarball. As of this writing, it is 1.3.5. Then extract the archive and install:

cd /usr/local/src
wget http://rubyforge.org/frs/download.php/60718/rubygems-1.3.5.tgz
tar xzvf rubygems-1.3.5.tgz
cd rubygems-1.3.5
sudo ruby setup.rb

The gem binary is now installed:

/usr/bin/gem1.8

If you want to have gem point to the 1.8 binary:

sudo update-alternatives --install /usr/bin/gem gem /usr/bin/gem1.8 1

Next, run a gem system update to be sure you are on the latest version:

sudo gem update --system

Rubygems should now be installed and ready to go.

Rails and Other Gems

Rubygems installs dependencies automatically. Installing the Rails gem which will bring in ActionPack, ActiveRecord, and other supporting gems. Additionally, you might want to install a few other useful gems for development:

sudo gem install rails SystemTimer ruby-debug ruby-debug-ide

SQLite

SQLite is now the default database a Rails application will use. It is easy to use since the database is kept in a local file and doesn’t require any server setup. We’ll setup MySQL later in this recipe so if you need that as well, you’ll be covered. Note that SQLite is not a good choice if you need to do concurrency development of any kind. SQLite is not a server and sharing the database on multiple instances of your Rails app isn’t going to go well.

SQLite is fully supported in the Ubuntu package system. Install the SQLite binaries and libraries, then install the native SQLite gem:

sudo apt-get install libsqlite3-dev sqlite3 sqlite3-doc
sudo gem install sqlite3-ruby

Other useful libraries

There are some optional libraries that many people use for Rails. Skip either of these if you don’t need them.

MySQL

Unlike on the Mac, MySQL is very easy to setup on Ubuntu. For 9.10, MySQL 5.1 is the default:

sudo apt-get install libmysqlclient-dev mysql-server
sudo gem install mysql

ImageMagick/RMagick

Again, unlike on the Mac, ImageMagick and RMagick are very easy to install:

sudo apt-get install imagemagick libmagickcore-dev
sudo gem install rmagick

Final Thoughts

One thing that is good to learn is how Ruby interacts with the underlying system as well as how to separate who installs what. Usually, most code is pure ruby and is very easy to install via gems.

I’ll follow up this post with a recipe for getting Passenger up and running on Ubuntu. Passenger is a fantastic way to do Rails development. It is likely going to mirror how you deploy code in production as well. I’m a big fan of running development as close to production as you can to minimize surprises.

ClickToFlash for Safari

2010-01-04T21:54:42-08:00

Not sure where I first saw this, but the WebKit plugin ClickToFlash is a required for any Safari user. Flash has numerous bad properties. It’s a resource hog and many sites use it for annoying things like ads that start playing once you navigate to a site. ClickToFlash intercepts all that and gives you a simple button in place of the Flash content. If you do want to view it, click the button.

There is also a nice whitelist feature you can use to enable entire sites to run Flash without permission. Additionally, it is easy to turn off ClickToFlash if you really need to.

Rumor is that this was originally written by someone in Adobe but no idea if that is true. Jonathan ‘Wolf’ Rentzsch is the current maintainer of it now. Code is available on GitHub. And it’s not an InputManager hack so it looks like a real solution.

Gem Server launchd Agent

2010-01-01T14:05:00-08:00

Ruby Gems provides an RDoc server for the installed gems on a system. It is trivial to get it up and running. In your console, use the following command:

gem server

A local http server will now be running on port 8808 that serves all the installed RDoc documentation for your gems. It’s a great way to look at the exact documentation that matches the code your have installed.

Starting with Mac OS X 10.4, managing daemons and other services has been performed by launchd. It’s an elegant and modern replacement for traditional startup systems like init and rc. But it wasn’t entirely obvious to me how to create my own custom startup script.

There are two places you could chose to setup your daemon. One is at the system level for any user on the machine. Another is in your local user account. If you put it at the system level (in /Library/LaunchAgents), you will have the server up and running anytime the machine boots up. If you use your user account, it will launch on login.

On my MacBook Pro laptop, I chose to use a launch agent for my account. On a shared machine or server, you might want to use the system level approach.

To define a launch agent, create a file for the launchd definition called:

~/Library/LaunchAgents/org.rubyforge.rubygems.server.plist

Then, use the following code to define a launchd agent that will startup the gem server instance:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN"
 "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
	<key>Label</key>
	<string>org.rubyforge.rubygems.server</string>
	<key>ProgramArguments</key>
	<array>
		<string>/usr/bin/gem</string>
		<string>server</string>
		<string>--daemon</string>
	</array>
	<key>RunAtLoad</key>
	<true/>
	<key>KeepAlive</key>
	<true/>
</dict>
</plist>

You can either log out then log back in or load and start it from the command line:

launchctl load ~/Library/LaunchAgents/org.rubyforge.rubygems.server.plist
launchctl start org.rubyforge.rubygems.server

Enter the following in your browser:

http://localhost:8808/

From now on when you login, you will have a running gem server with local copies of the documentation for all versions of the gems installed on your machine.

Reboot

2009-12-31T22:05:00-08:00

It’s been a long time coming but I’ve finally done a reboot of my web site. The old blog is gone. Some of the content might find its way back in here but probably not.

I plan to write about what I know - software, Rails in particular.

I have a long list of posts that I’ve wanted to write for some time. I used the excuse of the website redesign to put it off. With that out of the way, I’ll begin to publish those regularly over the next few weeks.

I have a couple notes on building this site. I’ll expand on this much more later but it is built using Jekyll. This is an excellent static site generation tool written in Ruby. When you think about it, this is the way personal web publishing should be done. I don’t need a complicated web application for the relatively few pages I want to publish.

Jekyll is a breath of fresh air compared with bloated software like Wordpress. The Jekyll project wiki has everything you need to get started. Or read Adam’s post on how he built his site using Jekyll. I’ll be putting the source for this website up on my GitHub account in the next few days.