What Happened to Programming

Last week, Mike Taylor posed the question “whatever happened to programming?” (also see his followup). I think part of the answer is simply “it’s grown up.”

Programming, either as a hobby or a career, has only been around for a little less than 70 years. (The first was the Colossus, built and used by the British during World War II to read encrypted German messages.) This is very young when compared to other scientific fields: agriculture, construction, various engineering disciplines, etc.

I won’t disagree that building the very fundamental parts of a new application from scratch can be very enjoyable. Perhaps not printf(3), but certainly other parts. The problem is bug rates, though: any seasoned software developer knows that writing less code means less bugs. If high-quality libraries are available that provide 80% of the functionality of your new application, you will likely end up with a better product, in less time, than if you chose to write the entire thing yourself.

Consider the sorry state we’d be in if:

All farmers experimented with their own methods of weed and pest control (organic or not), instead of using techniques proven to work in the past. Our agriculture industry probably couldn’t meet the population’s demand for food.

Architects and builders “winged it” when building new homes and office buildings. Bugs in that industry would likely mean structural failure. Who would want to live in those houses?

How about automakers? It’s one thing to build a kit car as a hobby, it’s quite another to tinker with your designs when you’re producing cars on a massive scale for general use (just ask Toyota).

It may not be as glamorous, but if our collective bug count goes down and the software industry ships better products that don’t require constant patching, I can’t help but consider this a very good thing.

malloc Debugging

Bill Bumgarner posted a nice tutorial on using malloc to debug memory misuse in Cocoa. I’ve run across these before when reading the malloc(1) man page, but it’s nice to have a tutorial that shows how to use them in practice.

Bash Substitutes for “seq” and Simple “sed” Expressions

There are two things I find myself doing from the shell just infrequently enough that I can’t remember the exact syntax. The first is generating a sequence and the second is repeated string transformation.

“seq” is a little GNU utility that generates sequences of numbers. I used to find this installed automatically by many Linux distributions, but lately (on Debian and Ubuntu, at least) it’s missing and I haven’t been sufficiently motivated to find it. No need: Bash has something good enough built in.

To generate the numbers from 1 to 10, inclusive:

$ echo {1..10}
1 2 3 4 5 6 7 8 9 10

The second thing is repeated string transformations, for example renaming a set of files to use a different file extension. This used to mean re-reading the sed(1) man page yet again and doing the transformation in a backtick expression. Again, no need: Bash can do this itself:

$ for f in a.foo b.foo c.foo; do echo $f ${f/%foo/bar}; done
a.foo a.bar
b.foo b.bar
c.foo c.bar

The syntax is ${parameter/pattern/replacement}. pattern only replaces the first match, unless it starts with /, in which case it replaces all matches. It can also start with # to force a match at the beginning of the string or % to match at the end.

Take a look at “Parameter Expansion” in the Bash manual for several other useful operations.

On the Pain of Developing for Facebook

I’m in the last stages of building a site that integrates with Facebook. It includes a Facebook Connect component and a separate Facebook iframe application. In the hopes that some of my experience will help someone else out there, I will share some of my suffering and how I worked through it.

This is a Rails 2.3 application, using the Facebooker gem, version 1.0.54.

Safari, iframes, and cookies.

Chances are high you’ve found this page via search, and that you are desperate to find a solution to allow your Rails sessions to carry over from page #1 of your iframe application to page #2 and beyond. Will Henderson has a solution when your Rails application uses the ActiveRecord session store, but new Rails applications use the cookie store. There is no session to retrieve from the database; passing a row ID with links doesn’t help.

I am fortunate that my iframe application has a splash screen, with only one useful link on it, to start the “real” application. I append every fb_sig parameter from the current request to the link. Once the user clicks that link, the iframe no longer falls under Safari’s 3rd party cookie policy, and so the next page can set cookies. Because all of the fb_sig parameters are there, Facebooker gives precedence to them, recreating the Facebook session without consulting cookies (which either don’t exist or are stale).

<%= link_to 'Start the app', params.merge(:action => :app) %>

Now, your app doesn’t always get these. When the application is first installed, you won’t get the fb_sig parameters. In this case, I force a redirect:

unless params[:fb_sig_ss]
  redirect_to "http://#{Facebooker.canvas_server_base}/#{Facebooker.facebooker_config['canvas_page_name']}/"
  return
end

However…

Facebooker’s Rack middleware trashes your params.

Facebooker includes a piece of Rack middleware that automatically validates the signature on fb_sig parameters, and aborts the request if the signature isn’t valid. This works great on the initial request, but fails on the second request, with exactly the same parameters.

The culprit is near the bottom of Facebook#call, in rack/facebook.rb, where it calls convert_parameters!. This method is trying to be helpful, turning things like Unix time_t into Ruby Time instances. If you are trying to pass these parameters along in a link, Time.to_s is not a Unix time_t, and so the signature validation will fail, and you’re screwed.

Pre-Rails 2.3, there is no Rack middleware, and signature validation happens in another part of the Facebooker gem, which does not trash the original params. It does the same conversion, but on a copy. As near as I can tell, calling convert_parameters! is unnecessary, and commenting it out solves my problem. When Safari loads page #2, Facebooker creates its session, which can be stored in the Rails session, using the cookie store.

Reconstituting a Facebook session in Flash.

The bulk of this iframe application is actually written in Flash. Every Facebook application has an App ID (essentially a public key) and a secret key. The secret key must never be allowed outside your organization or people can fake API requests from your application. Embedding your secret key in a Flash SWF is therefore not an option, because a SWF can be decompiled. Facebook’s solution to this is the session secret, passed as fb_sig_ss.

According to Adobe’s ActionScript 3.0 Client Library for Facebook API reference, one can use the FacebookSessionUtil class to recreate a session using only the public key and the session secret. This is not true. I have to pass in all of the fb_sig parameters to Flash in order to make it work. However you start your Flash content, FlashVars must include:

<%= params.collect { |k,v| "#{k}=#{v}" if k.starts_with?("fb_sig")}.compact.join("&") %>

A refreshed page lags a “logged-in” state change until refreshed a second time.

When a user logs out of Facebook and refreshes your Facebook Connect web site, it doesn’t reflect the user’s “logged-in” state change until another refresh. You can automate this and have the page refresh itself when necessary:

<% init_fb_connect 'XFBML', :app_settings => '{"reloadIfSessionStateChanged":true}' do %>
  FB.ensureInit(function() {
    FB.Connect.get_status().waitUntilReady(function(status) {
      switch (status) {
        case FB.ConnectState.connected:
          $$('.fbConnectLoggedIn').invoke('show');
          $$('.fbConnectLoggedOut').invoke('hide');
          break;
        case FB.ConnectState.appNotAuthorized:
        case FB.ConnectState.userNotLoggedIn:
          $$('.fbConnectLoggedIn').invoke('hide');
          $$('.fbConnectLoggedOut').invoke('show');
          break;
      }
    });
  });
<% end %>

There are actually two parts to this trick. First, :app_settings => '{"reloadIfSessionStateChanged":true}' triggers the automatic refresh.

Second, if you have some common part of your layout that switches on a user’s Facebook status, this precludes your use of page caching. For an otherwise static page, this is a real sacrifice. To work around this, I include both sets of mark-up, classed with either “fbConnectLoggedIn” or “fbConnectLoggedOut”, and invoke Prototype’s show() or hide() on them as part of the Facebook Connect initialization. As long as you are careful to use XFBML tags that don’t require actual values from a Facebook session (e.g. <fb:name uid="loggedinuser" ...>), this works very well.

New conditionally_cache Method for Fragment Caching

I just pushed a couple of updates to Banker. The boring one adds the missing “should_not” variants of the existing Shoulda macros.

The more interesting one is a conditionally_cache method for fragment caching. If you use pagination to split content across pages, this might be very useful for you. Pages 2 and later are rarely loaded, falling very quickly to the tiniest sliver of people that see page 1.

conditionally_cache adds a new first argument to the regular cache method. This argument is evaluated in a boolean context. When true, control is passed to the cache method as usual, performing fragment caching. When false, though, the block is captured and output as if the caching code were not there at all.

So instead of having no caching on your paginated results or wrapping your cache call in an ugly if statement, instead you can do this:

<% conditionally_cache params[:page].blank? || params[:page] == '1', 'page1' do %>
  ... usual HTML markup goes here ...
  ... this will be cached on page 1, but not elsewhere ...
<% end %>

Since pages 2+ are not cached, you need only expire the ‘page1′ fragment, and nothing else.

Rails Bug with render :text => proc in Tests

There is a bug in the current stable release of Rails, 2.3.4, when using render :text => proc { ... }. The Proc object is never called by the test framework, which limits what you can test. For example, the following two techniques will both fail:

  • Using assert_select or similar methods to check the content of the response.
  • Setting expectations with a mocking framework on methods that are called within the Proc body.

Lighthouse bug #2423 includes my reworked patch to fix this issue. A couple more people verifying the patch will help get it committed.

Xcode: iPhone Project Dependency on Mac OS X Project

This week, while building an iPhone application for a client, I wanted to run a custom Mac OS X command-line utility during the build phase of the iPhone project. I set up the command-line utility as a dependent project of the iPhone project, but it won’t build, telling me:

target specifies product type ‘com.apple.product-type.tool’, but there’s no such product type for the ‘iphonesimulator’ platform

There is no perfect solution to this problem, but here’s my workaround: rather than let Xcode build the dependent project for me, I simply added a call to xcodebuild as part of my custom script phase. As long as the input and output files for this phase are set up properly, the entire phase is skipped when the output file is up-to-date, and xcodebuild is pretty fast when the target binary is already built, too.

My final script phase looks like this:

(cd SubProject; xcodebuild -configuration Release)
SubProject/build/Release/SubProjectCommand

The Details of Parallel Processing Are the Hardest Part

Mike Ash has an absolutely excellent post up today discussing the importance of the details when designing an application for parallel processing.

His post focuses on using Grand Central Dispatch (part of Mac OS X 10.6 “Snow Leopard”), but the core point is valid for any approach to multiprocessing: it’s not just a simple matter of queueing jobs. You have to consider the impact of those jobs on the system as a whole.

Running Sweepers from a Model

Oh, the pain. Over the last 24 hours I have fought an exhausting battle with Rails and the testing environment to do a couple of seemingly simple things:

  1. Expire cached pages and fragments from outside the context of a normal HTTP request
  2. Test it.

Testing, in particular, is particularly difficult because Rails does not offer any built-in mechanism to test an application’s caching, and I’ve had problems in the past with the only plug-in that does it (cache_test) it on Rails 2.x. I’ve released a new plug-in, called Banker, that provides assertions and the necessary support to test caching, including Shoulda macros.

The first item has come up for me more than once. Complex applications often have scheduled jobs that make changes to the database. If the application does any caching, there is a good chance that these jobs will affect content that is cached. The problem is that expiring caches from outside the context of a controller + request is a pain. Here is the solution:

Create your sweepers as you normally would. Then, either within test code or your script/runner code:

def setup_cache_sweepers(*sweepers)
  sweepers = sweepers.flatten
 
  ActiveRecord::Base.observers = sweepers
  ActiveRecord::Base.instantiate_observers
 
  returning ActionController::Base.new do |controller|
    controller.request = ActionController::TestRequest.new
    controller.request.host = URL_HOST
    controller.instance_eval do
      @url = ActionController::UrlRewriter.new(request, {})
    end
 
    sweepers.each do |sweeper|
      sweeper.instance.controller = controller
    end
  end
end

URL_HOST is a constant, defined in each environment file, with the host:port part of a URL. It is needed in order to generate URLs, and more importantly, fragment cache keys, outside the context of an HTTP request.

The method returns a controller. For unit test code, hang on to it, because it gives you access to named routes:

@controller.send(:users_path)

For code run from script/runner, nothing else is needed. You’ve already instantiated your sweepers and given them the controller instance, so anything you do to a model instance that generates a callback to the sweeper will use that controller, including calling cache expiration methods.

Manage vendor/rails with Git on a Subversion Project

Here is something I’ve been experimenting with over the last week or so, and it’s working out very nicely so far:

Use Git to manage vendor/rails when your project is using Subversion.

Here’s how:

First time freezing? Easy:

  1. cd vendor
  2. git clone git://github.com/rails/rails.git
  3. svn add -N rails; svn ps svn:ignore .git rails
  4. cd rails; git checkout v2.3.3.1 (or whatever version you want)
  5. svn add *

Switching to a different version of Rails is now as simple as:

  1. cd vendor/rails
  2. git checkout master
  3. git pull origin master
  4. git checkout whatever
  5. svn add `svn st | grep ^\? | cut -f7 -d" "`
  6. svn rm `svn st | grep ^\! | cut -f7 -d" "`
  7. svn commit

If you already have a vendor/rails and you want to try this technique out, you should first clone the Git repository to a temporary directory, git checkout the version that matches what is already in your vendor/rails, then copy (or move) the .git directory into vendor/rails. Add the svn:ignore property as above and you’re all set.

Here’s why I’m doing this: when a new Rails release comes out, some files are changed, some added and others deleted. Because Subversion litters a checked-out project with its .svn directories, you can’t just delete the entire thing and re-freeze without losing local patches (yes, I’ve had to do this) and history (which isn’t terribly important, but can be nice). Even ignoring those two reasons, completely deleting and adding vendor/rails will cause your Subversion repository to grow more than necessary (Rails 2.3.3 checks in at 35 MB).

Git, by putting everything in a top-level .git directory, makes itself easy for Subversion to ignore. Checking out a tag to switch to a different release is simple, and Git deletes files, unlike applying a patch, which truncates deleted files to 0 bytes, requiring a find to actually remove them.

Contact us: info (at) lightyearsoftware.com | Press