Bash Substitutes for “seq” and Simple “sed” Expressions

There are two things I find myself doing from the shell just infrequently enough that I can’t remember the exact syntax. The first is generating a sequence and the second is repeated string transformation.

“seq” is a little GNU utility that generates sequences of numbers. I used to find this installed automatically by many Linux distributions, but lately (on Debian and Ubuntu, at least) it’s missing and I haven’t been sufficiently motivated to find it. No need: Bash has something good enough built in.

To generate the numbers from 1 to 10, inclusive:

$ echo {1..10}
1 2 3 4 5 6 7 8 9 10

The second thing is repeated string transformations, for example renaming a set of files to use a different file extension. This used to mean re-reading the sed(1) man page yet again and doing the transformation in a backtick expression. Again, no need: Bash can do this itself:

$ for f in a.foo b.foo c.foo; do echo $f ${f/%foo/bar}; done
a.foo a.bar
b.foo b.bar
c.foo c.bar

The syntax is ${parameter/pattern/replacement}. pattern only replaces the first match, unless it starts with /, in which case it replaces all matches. It can also start with # to force a match at the beginning of the string or % to match at the end.

Take a look at “Parameter Expansion” in the Bash manual for several other useful operations.

On the Pain of Developing for Facebook

I’m in the last stages of building a site that integrates with Facebook. It includes a Facebook Connect component and a separate Facebook iframe application. In the hopes that some of my experience will help someone else out there, I will share some of my suffering and how I worked through it.

This is a Rails 2.3 application, using the Facebooker gem, version 1.0.54.

Safari, iframes, and cookies.

Chances are high you’ve found this page via search, and that you are desperate to find a solution to allow your Rails sessions to carry over from page #1 of your iframe application to page #2 and beyond. Will Henderson has a solution when your Rails application uses the ActiveRecord session store, but new Rails applications use the cookie store. There is no session to retrieve from the database; passing a row ID with links doesn’t help.

I am fortunate that my iframe application has a splash screen, with only one useful link on it, to start the “real” application. I append every fb_sig parameter from the current request to the link. Once the user clicks that link, the iframe no longer falls under Safari’s 3rd party cookie policy, and so the next page can set cookies. Because all of the fb_sig parameters are there, Facebooker gives precedence to them, recreating the Facebook session without consulting cookies (which either don’t exist or are stale).

<%= link_to 'Start the app', params.merge(:action => :app) %>

Now, your app doesn’t always get these. When the application is first installed, you won’t get the fb_sig parameters. In this case, I force a redirect:

unless params[:fb_sig_ss]
  redirect_to "http://#{Facebooker.canvas_server_base}/#{Facebooker.facebooker_config['canvas_page_name']}/"
  return
end

However…

Facebooker’s Rack middleware trashes your params.

Facebooker includes a piece of Rack middleware that automatically validates the signature on fb_sig parameters, and aborts the request if the signature isn’t valid. This works great on the initial request, but fails on the second request, with exactly the same parameters.

The culprit is near the bottom of Facebook#call, in rack/facebook.rb, where it calls convert_parameters!. This method is trying to be helpful, turning things like Unix time_t into Ruby Time instances. If you are trying to pass these parameters along in a link, Time.to_s is not a Unix time_t, and so the signature validation will fail, and you’re screwed.

Pre-Rails 2.3, there is no Rack middleware, and signature validation happens in another part of the Facebooker gem, which does not trash the original params. It does the same conversion, but on a copy. As near as I can tell, calling convert_parameters! is unnecessary, and commenting it out solves my problem. When Safari loads page #2, Facebooker creates its session, which can be stored in the Rails session, using the cookie store.

Reconstituting a Facebook session in Flash.

The bulk of this iframe application is actually written in Flash. Every Facebook application has an App ID (essentially a public key) and a secret key. The secret key must never be allowed outside your organization or people can fake API requests from your application. Embedding your secret key in a Flash SWF is therefore not an option, because a SWF can be decompiled. Facebook’s solution to this is the session secret, passed as fb_sig_ss.

According to Adobe’s ActionScript 3.0 Client Library for Facebook API reference, one can use the FacebookSessionUtil class to recreate a session using only the public key and the session secret. This is not true. I have to pass in all of the fb_sig parameters to Flash in order to make it work. However you start your Flash content, FlashVars must include:

<%= params.collect { |k,v| "#{k}=#{v}" if k.starts_with?("fb_sig")}.compact.join("&") %>

A refreshed page lags a “logged-in” state change until refreshed a second time.

When a user logs out of Facebook and refreshes your Facebook Connect web site, it doesn’t reflect the user’s “logged-in” state change until another refresh. You can automate this and have the page refresh itself when necessary:

<% init_fb_connect 'XFBML', :app_settings => '{"reloadIfSessionStateChanged":true}' do %>
  FB.ensureInit(function() {
    FB.Connect.get_status().waitUntilReady(function(status) {
      switch (status) {
        case FB.ConnectState.connected:
          $$('.fbConnectLoggedIn').invoke('show');
          $$('.fbConnectLoggedOut').invoke('hide');
          break;
        case FB.ConnectState.appNotAuthorized:
        case FB.ConnectState.userNotLoggedIn:
          $$('.fbConnectLoggedIn').invoke('hide');
          $$('.fbConnectLoggedOut').invoke('show');
          break;
      }
    });
  });
<% end %>

There are actually two parts to this trick. First, :app_settings => '{"reloadIfSessionStateChanged":true}' triggers the automatic refresh.

Second, if you have some common part of your layout that switches on a user’s Facebook status, this precludes your use of page caching. For an otherwise static page, this is a real sacrifice. To work around this, I include both sets of mark-up, classed with either “fbConnectLoggedIn” or “fbConnectLoggedOut”, and invoke Prototype’s show() or hide() on them as part of the Facebook Connect initialization. As long as you are careful to use XFBML tags that don’t require actual values from a Facebook session (e.g. <fb:name uid="loggedinuser" ...>), this works very well.

New conditionally_cache Method for Fragment Caching

I just pushed a couple of updates to Banker. The boring one adds the missing “should_not” variants of the existing Shoulda macros.

The more interesting one is a conditionally_cache method for fragment caching. If you use pagination to split content across pages, this might be very useful for you. Pages 2 and later are rarely loaded, falling very quickly to the tiniest sliver of people that see page 1.

conditionally_cache adds a new first argument to the regular cache method. This argument is evaluated in a boolean context. When true, control is passed to the cache method as usual, performing fragment caching. When false, though, the block is captured and output as if the caching code were not there at all.

So instead of having no caching on your paginated results or wrapping your cache call in an ugly if statement, instead you can do this:

<% conditionally_cache params[:page].blank? || params[:page] == '1', 'page1' do %>
  ... usual HTML markup goes here ...
  ... this will be cached on page 1, but not elsewhere ...
<% end %>

Since pages 2+ are not cached, you need only expire the ‘page1’ fragment, and nothing else.

Rails Bug with render :text => proc in Tests

There is a bug in the current stable release of Rails, 2.3.4, when using render :text => proc { ... }. The Proc object is never called by the test framework, which limits what you can test. For example, the following two techniques will both fail:

  • Using assert_select or similar methods to check the content of the response.
  • Setting expectations with a mocking framework on methods that are called within the Proc body.

Lighthouse bug #2423 includes my reworked patch to fix this issue. A couple more people verifying the patch will help get it committed.

Xcode: iPhone Project Dependency on Mac OS X Project

This week, while building an iPhone application for a client, I wanted to run a custom Mac OS X command-line utility during the build phase of the iPhone project. I set up the command-line utility as a dependent project of the iPhone project, but it won’t build, telling me:

target specifies product type ‘com.apple.product-type.tool’, but there’s no such product type for the ‘iphonesimulator’ platform

There is no perfect solution to this problem, but here’s my workaround: rather than let Xcode build the dependent project for me, I simply added a call to xcodebuild as part of my custom script phase. As long as the input and output files for this phase are set up properly, the entire phase is skipped when the output file is up-to-date, and xcodebuild is pretty fast when the target binary is already built, too.

My final script phase looks like this:

(cd SubProject; xcodebuild -configuration Release)
SubProject/build/Release/SubProjectCommand
The Details of Parallel Processing Are the Hardest Part

Mike Ash has an absolutely excellent post up today discussing the importance of the details when designing an application for parallel processing.

His post focuses on using Grand Central Dispatch (part of Mac OS X 10.6 “Snow Leopard”), but the core point is valid for any approach to multiprocessing: it’s not just a simple matter of queueing jobs. You have to consider the impact of those jobs on the system as a whole.

Running Sweepers from a Model

Oh, the pain. Over the last 24 hours I have fought an exhausting battle with Rails and the testing environment to do a couple of seemingly simple things:

  1. Expire cached pages and fragments from outside the context of a normal HTTP request
  2. Test it.

Testing, in particular, is particularly difficult because Rails does not offer any built-in mechanism to test an application’s caching, and I’ve had problems in the past with the only plug-in that does it (cache_test) it on Rails 2.x. I’ve released a new plug-in, called Banker, that provides assertions and the necessary support to test caching, including Shoulda macros.

The first item has come up for me more than once. Complex applications often have scheduled jobs that make changes to the database. If the application does any caching, there is a good chance that these jobs will affect content that is cached. The problem is that expiring caches from outside the context of a controller + request is a pain. Here is the solution:

Create your sweepers as you normally would. Then, either within test code or your script/runner code:

def setup_cache_sweepers(*sweepers)
  sweepers = sweepers.flatten
 
  ActiveRecord::Base.observers = sweepers
  ActiveRecord::Base.instantiate_observers
 
  returning ActionController::Base.new do |controller|
    controller.request = ActionController::TestRequest.new
    controller.request.host = URL_HOST
    controller.instance_eval do
      @url = ActionController::UrlRewriter.new(request, {})
    end
 
    sweepers.each do |sweeper|
      sweeper.instance.controller = controller
    end
  end
end

URL_HOST is a constant, defined in each environment file, with the host:port part of a URL. It is needed in order to generate URLs, and more importantly, fragment cache keys, outside the context of an HTTP request.

The method returns a controller. For unit test code, hang on to it, because it gives you access to named routes:

@controller.send(:users_path)

For code run from script/runner, nothing else is needed. You’ve already instantiated your sweepers and given them the controller instance, so anything you do to a model instance that generates a callback to the sweeper will use that controller, including calling cache expiration methods.

Manage vendor/rails with Git on a Subversion Project

Here is something I’ve been experimenting with over the last week or so, and it’s working out very nicely so far:

Use Git to manage vendor/rails when your project is using Subversion.

Here’s how:

First time freezing? Easy:

  1. cd vendor
  2. git clone git://github.com/rails/rails.git
  3. svn add -N rails; svn ps svn:ignore .git rails
  4. cd rails; git checkout v2.3.3.1 (or whatever version you want)
  5. svn add *

Switching to a different version of Rails is now as simple as:

  1. cd vendor/rails
  2. git checkout master
  3. git pull origin master
  4. git checkout whatever
  5. svn add `svn st | grep ^\? | cut -f7 -d" "`
  6. svn rm `svn st | grep ^\! | cut -f7 -d" "`
  7. svn commit

If you already have a vendor/rails and you want to try this technique out, you should first clone the Git repository to a temporary directory, git checkout the version that matches what is already in your vendor/rails, then copy (or move) the .git directory into vendor/rails. Add the svn:ignore property as above and you’re all set.

Here’s why I’m doing this: when a new Rails release comes out, some files are changed, some added and others deleted. Because Subversion litters a checked-out project with its .svn directories, you can’t just delete the entire thing and re-freeze without losing local patches (yes, I’ve had to do this) and history (which isn’t terribly important, but can be nice). Even ignoring those two reasons, completely deleting and adding vendor/rails will cause your Subversion repository to grow more than necessary (Rails 2.3.3 checks in at 35 MB).

Git, by putting everything in a top-level .git directory, makes itself easy for Subversion to ignore. Checking out a tag to switch to a different release is simple, and Git deletes files, unlike applying a patch, which truncates deleted files to 0 bytes, requiring a find to actually remove them.

ssh-agent on Mac OS X 10.5

For as long as I can remember, I’ve been using a tool called SSHKeychain on Mac OS X to manage ssh-agent and my identities, to make logging into remote servers secure, yet password-free.

Lately, however, something has changed and SSHKeychain isn’t able to keep track of my keys. The result is that instead of rarely typing my passphrases, I’m doing it constantly. I think it started around the time I updated to 10.5.8.

Turns out that Leopard has much better support for ssh-agent built-in and SSHKeychain isn’t necessary. Dave Dribin’s blog lays it all out: ssh-agent on Mac OS X 10.5 and, for the security conscious, Securing ssh-agent on Mac OS X 10.5.

A couple of things to watch out for:

  • If you are switching from SSHKeychain, remove the environment override for SSH_AUTH_SOCK from ~/.MacOSX/environment.plist.
  • To get the GUI passphrase dialog and the option to save the passphrase in your keychain, you must use the system ssh, not one from Fink or MacPorts.
CakePHP Quick Start Guide for Experts

I’ve started a new project and, to my disappointment, the hosting environment won’t support Ruby. It is the client’s own facility, so using another provider is not an option.

I looked at the web frameworks page at Wikipedia and picked out six for a deeper look: Akelos, CakePHP, CodeIgniter, Kohana, Symfony and Zend. My major requirements were:

  • PHP (<= 5.1)
  • MVC
  • ORM
  • Test framework
  • Some support for database migration
  • Caching
  • Maturity: stable code and a healthy community

In a nutshell, something approximating Rails. I settled on CakePHP. Much of what makes Rails great can be traced directly back to Ruby, and you can’t approach that level of expressiveness in PHP, so an approximation is the best we can do.

Coming up to speed on a new framework is always a stop-and-go affair, but Matt Curry’s Super Awesome Advanced CakePHP Tips (free PDF e-book) is terrific. Highly recommended if you already know what you’re doing and just need answers to “how do I… in CakePHP?”