The CrunchyFrog Blog - Home

Updating PostGIS Databases

2010-11-01T17:56:00Z

I recently tried upgrading a PostGIS database to a newer version, going from 1.4.0 to 1.5.2. Unfortunately the upgrade scripts failed with the error

ERROR:  type "box3d_extent" is only a shell

The only thing that worked was to follow the advice given by Paul Ramsey, in his blog post PostGIS Back-up / Restore, and Steve Woodbridge, in a thread on the postgis-users list.

The only gotcha I see here is that pg_dump doesn’t include the database’s search path setting. So, I’ll have to remember to set the search path for the database when it’s restored. That doesn’t happen very often — usually only with major version upgrades of PostgreSQL — but it’s infrequent enough that it’s something that has the potential to get overlooked if it’s not documented.

Response Order in Rails Controller Actions

2010-10-28T06:07:00Z

When checking test coverage on a Rails application, there was one code path that was missing, even though a test existed for it.

Here’s a stripped-down version of the controller spec:

describe BooksController do
  it "should get a book via an AJAX request" do
    xhr :get, :show, :id => 9
    response.should be_success
    response.should have_text(/\bBook 9\b/)
  end

  it "should get a book via a non-AJAX request" do
    get :show, :id => 9
    response.should be_success
    response.should have_text(/\bBook 9\b/)
  end
end

And here’s the relevant parts of the controller:

class BooksController < ApplicationController
  def show
    respond_to do |format|
      format.js do
        # Do stuff for AJAX requests
      end
      format.html do
        # Do stuff for non-AJAX requests
      end
    end
  end
end

But in both cases, only the first response block was being run. Of course, if the expected response text had taken into account the differences between the two responses, and not just one piece of text, the non-AJAX spec would have failed and the problem would have been found sooner. But it wasn’t.

The problem was with the HTTP Accept header or, in this case, the lack of one. The get call did not generate an Accept header so the first response block handled the request. The xhr call, on the other hand, generated an Accept header containing text/javascript, text/html, application/xml, text/xml, */* so the correct response block would handle the request regardless of the order.

The problem never showed up in browsers because an HTTP Accept header is always being passed; Firefox, for example, sends a HTTP Accept header with a value of text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8. Only command-line tools like curl and wget don’t include text/html as part of the Accept header unless told to do so (for example, by using the --header option). I don’t know what search engines and other robots specify for an Accept header, but it might be safe to assume that they act more like curl and wget than Firefox.

So the easy fix in this case was to reverse the order of the response blocks, and remember to make sure that response blocks are always ordered so that the default response is first.

GIS Experiments

2010-10-26T06:00:00Z

Despite the painful problem I encountered in my previous post, I did manage to recover and get some maps made and thought I’d share one of them. Over on the right side of the page, you’ll find a link to a little project I put together as a basis for learning the Google Maps API.

The database currently lists about 4,200 breweries. While still incomplete, especially in Asia, Africa, Central and South America, and parts of Europe, I was still curious about how the worldwide distribution looked. Unfortunately, trying to plot them on a map with the Google Maps API is not feasible because of the large number of markers; supposedlly more than a couple hundred markers are a problem, but I’ve not had any serious problem showing up to about 1,000 markers, but 4,000+ definitely won’t work.

Enter Quantum GIS. Since the data is stored in a PostGIS database, it was relatively easy to generate a world map showing all of the brewery locations that have been entered in the database. Admitedly, the map could use some refinement, but I’ve only recently started working with Quantum GIS and have limited experience with its capabilities at the moment.

System Overload

2010-10-26T05:20:00Z

This is probably not what you want to see on a typical UNIX system:

$ uptime
 15:07:27 up 500 days,  4:06,  3 users,  load average: 163.05, 170.70, 156.37

There might well be some powerful systems that can handle that type of load, but the one I was working on sure wasn’t coping very well. I can’t recall ever seeing such a high load average before on any UNIX system, and I almost didn’t get to see that because I was very close to power cycling the system to regain control before I was able to recover.

The problem started with an out-of-control process that began sucking up memory. The system began to get slower and slower, and I had a good idea which process was at fault. The Linux OOM (out-of-memory) killer didn’t kick in so I had to, but it was becoming harder and harder to get any response from a terminal, but I did manage to get a ps command entered to identify the likely culprit, watching the characters echo back at about the speed of a 110 baud modem connection on a noisy phone line, and eventually got results:

$ ps -C qgis u
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
dcato    20596 15.1 76.0 6185492 3088452 pts/27 D   13:42  10:54 qgis

This on a system with 4GB RAM and 6GB swap, and a few other running processes that weren’t shy about using memory either. Thankfully I caught the system at just the right time and was able to enter kill -9 20596 and kill the process without any of the same painful delay that the ps command took. I immediately typed uptime and got the results shown above.

Offline application gotcha in Firefox

2010-05-19T03:30:00Z

I’ve been working on an HTML5-based offline web application but ran into trouble when updating files listed in the cache manifest.

Even though the files were downloaded and cached when the page was initially loaded, if a change was made to one (or more) of the files listed on the server, the browser detected the updated manifest file and began pulling down the updated file(s). However, the update would halt with an error part way through the process, but only in Firefox; the update process worked fine in Safari and Chrome.

I’ve been working on an HTML5-based offline web application but ran into trouble when updating files listed in the cache manifest.

There does not appear to be any way to determine which file was being downloaded when the error event is fired and the server logs only showed status codes of 200 and 304 (as expected) but no 4xx or 5xx status codes. As it turned out, the server never received a request for the file that caused the error anyway.

After a lot of frustration and futile debugging steps, I began hacking entries out of the manifest file and eventually narrowed the problem down to a single file. Although the file existed it was empty and, evidently, Firefox 3.6.3 does not like having empty files included in the cache manifest. The behavior was the same on both Mac OS X and Linux. (My guess is that the problem exists on other platforms too, but I have not verified that it does.) As I mentioned above, the problem did not occur with the current versions of Safari or Chrome; nor does the problem exist with the most recent Firefox 3.7 nightly build.

The obvious fix is to avoid including any empty files in the cache manifest. And for good measure, don’t reference any empty files anywhere else in the web app to avoid accessing files that are not included in the offline cache when the user is working offline.

ActiveScaffold problems when using acts_as_audited

2009-06-29T17:36:00Z

While beefing up the admin backend of a Rails app that uses ActiveScaffold for its admin UI, I encountered an error that only happened in production and the stack trace wasn’t much help in identifying the cause.

After failing to recreate the problem with a (relatively) small test case, I took the opposite approach and began ripping code out of the original application until the failure went away.

Background

The problem began when I tried to access a nested scaffold from its parent scaffold. The relevant models are as follows:

123	class Country < ActiveRecord::Base has_many :regionsend

123	class Region < ActiveRecord::Base belongs_to :countryend

The Problem

When viewing the list of countries, I should have been able to click the regions column and get a nested list of regions in the country like so.

Except, this is what I got.

And the log file had the following stack trace.

NoMethodError (undefined method `controller_name' for nil:NilClass):
    /usr/lib64/ruby/gems/1.8/gems/actionpack-2.2.2/lib/action_controller/caching/sweeping.rb:83:in `callback'
    /usr/lib64/ruby/gems/1.8/gems/actionpack-2.2.2/lib/action_controller/caching/sweeping.rb:65:in `after'
    /usr/lib64/ruby/gems/1.8/gems/actionpack-2.2.2/lib/action_controller/filters.rb:208:in `around_proc'
    /usr/lib64/ruby/gems/1.8/gems/activesupport-2.2.2/lib/active_support/callbacks.rb:182:in `call'
    /usr/lib64/ruby/gems/1.8/gems/activesupport-2.2.2/lib/active_support/callbacks.rb:182:in `evaluate_method'
    /usr/lib64/ruby/gems/1.8/gems/actionpack-2.2.2/lib/action_controller/filters.rb:184:in `call'
    /usr/lib64/ruby/gems/1.8/gems/actionpack-2.2.2/lib/action_controller/filters.rb:635:in `run_before_filters'
    /usr/lib64/ruby/gems/1.8/gems/actionpack-2.2.2/lib/action_controller/filters.rb:615:in `call_filters'
    /usr/lib64/ruby/gems/1.8/gems/actionpack-2.2.2/lib/action_controller/filters.rb:610:in `perform_action_without_benchmark'
    /usr/lib64/ruby/gems/1.8/gems/actionpack-2.2.2/lib/action_controller/benchmarking.rb:68:in `perform_action_without_rescue'
    /usr/lib64/ruby/1.8/benchmark.rb:293:in `measure'
    /usr/lib64/ruby/gems/1.8/gems/actionpack-2.2.2/lib/action_controller/benchmarking.rb:68:in `perform_action_without_rescue'
    /usr/lib64/ruby/gems/1.8/gems/actionpack-2.2.2/lib/action_controller/rescue.rb:136:in `perform_action_without_caching'
    /usr/lib64/ruby/gems/1.8/gems/actionpack-2.2.2/lib/action_controller/caching/sql_cache.rb:13:in `perform_action'
    /usr/lib64/ruby/gems/1.8/gems/activerecord-2.2.2/lib/active_record/connection_adapters/abstract/query_cache.rb:34:in `cache'
    /usr/lib64/ruby/gems/1.8/gems/activerecord-2.2.2/lib/active_record/query_cache.rb:8:in `cache'
    /usr/lib64/ruby/gems/1.8/gems/actionpack-2.2.2/lib/action_controller/caching/sql_cache.rb:12:in `perform_action'
    /usr/lib64/ruby/gems/1.8/gems/actionpack-2.2.2/lib/action_controller/base.rb:524:in `send'
    /usr/lib64/ruby/gems/1.8/gems/actionpack-2.2.2/lib/action_controller/base.rb:524:in `process_without_filters'
    /usr/lib64/ruby/gems/1.8/gems/actionpack-2.2.2/lib/action_controller/filters.rb:606:in `process_without_session_management_support'
    /usr/lib64/ruby/gems/1.8/gems/actionpack-2.2.2/lib/action_controller/session_management.rb:134:in `process'
    /usr/lib64/ruby/gems/1.8/gems/actionpack-2.2.2/lib/action_controller/base.rb:392:in `process'
    /usr/lib64/ruby/gems/1.8/gems/actionpack-2.2.2/lib/action_controller/dispatcher.rb:183:in `handle_request' 
    ...

So the stack trace is pointing to a caching problem, which explains why the problem did not occur in development mode. But that’s about all the stack trace revealed to me.

Another issue is that this same code worked fine in a couple other Rails applications. However, none of the applications used identical sets of gems and plugins so there was likely a problem introduced by some combination of external dependencies.

The Fix

After having failed to recreate the problem in a small test case, it was time to start ripping the code apart.

As I pulled more and more functionality out of the application, I finally got it working when I ripped out the acts_as_audited plugin. A quick google took me to this post that offered the solution. After applying that fix, the nested scaffolds worked as expected, and after reverting all the other changes no other problems were raised.

Rotating Ad Banners for Radiant

2009-06-15T00:08:00Z

As part of on ongoing project with Radiant CMS, I needed to display random ad banners on a page. The banner_rotator extension didn’t quite fit my needs so I rolled my own. It’s available at http://github.com/davec/radiant-ad_banners-extension.

What’s different from the banner_rotator extension?

Simplified usage
Just include <r:ad_banner/> to generate the appropriate HTML on the page. If you want a specific banner, use <r:ad_banner name="Banner Name"/>. Include as many banners on the page as you want, though if you include more banners than you have defined, the excess ad_banner tags will be ignored. A banner will only be displayed once on a page unless you specifically force it by specifying the name attribute.
Uses the paperclipped extension for asset management. Drag-and-drop an image into the form when creating or editing a banner definition.
Banners are global, not tied to a specific page or set of pages. Depending on your use case, this could be considered a strike against the ad_banners extension.

Events Calendar Extension for Radiant

2009-06-06T02:38:00Z

I’ve recently been working with Radiant CMS for a project and needed a calendar of events. None of the existing calendar-related extensions fit the bill so I created one that did.

Find it at http://github.com/davec/radiant-events-calendar-extension.

I’ve recently been working with Radiant CMS for a project and needed a calendar of events. None of the existing calendar-related extensions fit the bill so I created one that did.

Find it at http://github.com/davec/radiant-events-calendar-extension.

The Public Calendar

The public face of the website gets a small monthly calendar that can be navigated month-by-month using the previous and next month links at the top of the calendar (the calendar is updated via an AJAX request). To display a calendar, simply add the radius tag <r:calendar/> to the page (you might want to add a section header of some sort — the Calendar title shown here is not included in the output of the <r:calendar/> tag).

The calendar cell of the current day, as known by the server, is highlighted; days that have events associated with them are shown in a bold red font. (Tweak the CSS to your own preferences if the defaults are not to your liking.)

When the mouse pointer is within a calendar cell that has events, a tooltip is displayed with some basic information about the events for the day. To see the details on the day’s events, do as the tooltip says and click on the date. Doing so will render a separate page that contains the details of the day’s events.

The tooltip is normally aligned a few pixels above and to the left of the lower left corner of the calendar cell it’s associated with. However, if the calendar is too close to the top of the browser’s viewport, the JavaScript code that displays the tooltip is intelligent enough to adjust the location of the tooltip to keep it within the viewport.

“Ah, ha!” you say. “What if I place my calendar on the left side of the page? Surely there isn’t room to display the tooltip.”

Fear not, just as with the tooltip positioning adjustments made when the calendar it too close to the top of the page, if the tooltip cannot be fully displayed to the left of its associated calendar cell, it gets displayed to the right.

There’s a potential for trouble if the browser window it too narrow. The JavaScript code does not currently consider the situation where there is insufficient room to display the tooltip on the left and the right sides. If there’s insufficient room on the left, the tooltip is displayed on the right with the potential of having the right side of the tooltip clipped.

The Admin Side

The public calendar, and associated events page, is pulling data from a simple table of events. The admin UI gets an Events tab that displays the list of current and future events.

Events that have already happened are not accessible — you’ll have to dive into the database directly if you want to rewrite history.

The UI for creating new events and editing existing events is straight forward.

Events must have a name and a date, but everything else is optional.

Caveats

This extension works for my purposes, but still has some rough edges. I’m currently using it with Radiant 0.8RC1 but I’ve also done some testing with 0.7.1. I haven’t tried it with any earlier versions of Radiant. If you do, let me know if it works.

Use the master branch for 0.8 and the radiant-0.7.1 branch for 0.7.1.

The calendar display has been tested with Firefox 3.0.10, Safari 3.2.3, IE6, IE7, IE8, and Opera 9.64.

Acknowledgements

This extension makes use of Tim Harper’s calendar_date_select plugin, Geoffrey Grosenbach’s calendar_helper plugin, and Tony Amoyal’s simple_time_select plugin. ¹

All of these plugins are included with the events calendar extension.

¹ The simple_time_select plugin uses a 12 hour clock. If your users expect a 24 hour clock then you’ll need to do some hacking.

Spam on the rebound

2009-06-01T04:13:00Z

Well, it looks like the spam levels this week have not only returned to their pre-McColo shutdown levels, but they’re beginning to surpass them.

Just before the McColo shutdown, my mail server was receiving roughly 75K emails per day during the week, somewhat less on the weekends. Afterwards, the level dropped to less than 20K per day, which is where it stayed through the end of the year. Since the beginning of the year the amount of mail hitting my mail server has been rising, hitting about 60K messages per day two weeks ago.

Well, it looks like the spam levels this week have not only returned to their pre-McColo shutdown levels, but they’re beginning to surpass them.

This past Monday, May 25, 74,372 messages hit the mail server. The next day, 80,312 — a new record for me (not that I’m trying for any records). Activity slacked off slightly after that with only 79,909 messages on Wednesday and 79,040 on Thursday before taking a dive on Friday when only 70,739 messages came knocking.

Not surprisingly, the amount of legitimate mail coming through has remained relatively stable, so all this increase is nothing but spam, most of which gets blocked outright at the server or shuffled off to the spam graveyard by the various spam filters I have in place.

You might be a spammer if …

A look at why the spam gets binned shows just how stupid the spammers are.

By far the biggest reason spam gets caught is because the spammer forges the email address to appear as if the sender is from the domain to which the spam is being sent. How many mail servers actually accept such email these days without requiring authentication first? The only way a local user of my systems is sending mail is by going through the internal mail server, not by connecting from a foreign IP address. This type of spam accounts for between 30% and 50% of all mail hitting my mail server.

A specialized variation of the above is impersonating the user being spammed. I get several of these a day via an email address I have forwarded from another domain, so blocking them at the SMTP server doesn’t do any good but bogofilter catches them just fine. Really, who would send a message to themselves with a subject of “Open or your a racial epithet” or “FW: Hi user@example.com visit me”? (Of course, I have to wonder about the intelligence of people who would actually bother to open such a message in the first place unless they’re analyzing their spam.)

Coming in at number two on the spam block meter is connections from IP addresses in the Spamhaus SBL and XBL lists. These addresses account for about 25% of all inbound SMTP connections.

Close behind in third place are bogus HELO hostnames, mainly localhost (sometimes fully-qualified as localhost.localdomain) and wireless_broadband_router. Less often the spammer uses the domain name, host name, or IP address of the host they’re connecting to. Bzzt, wrong.

The sender-SMTP MUST ensure that the <domain> parameter in a HELO command is a valid principal host domain name for the client host.

If anyone knows of a legitimate SMTP server that doesn’t provide the correct hostname in the HELO (or EHLO) command, I’d like to hear about it.

Though not too common, there is the occasional mangling of the SMTP protocol or perhaps attempts to break the server. Why else would email be sent to +.-_|@example.com or {%TO%}. Similarly, mail purporting to be from {$FROM} or {%FNAME%RND_NUM@%RND_DOM}.

Weekend cat blogging

2009-05-31T03:29:00Z

It’s the mystery cat. Yes, there is a cat under there.

This blog has been a bit like the cat under those bed covers, and like the cat this blog eventually emerges from the darkness.

Curiously, she just recently began burrowing under the bed covers this week as the weather has been warming up. Nothing like a 88°F (31°C) day to seek out more warmth. OK, so it was a bit cooler inside, getting up to almost 80°F (27°C) early this evening before beginning to cool down.

Firefox display problems on Linux

2009-03-30T22:54:00Z

A week or so after upgrading Firefox to 3.0.7 on my Linux system, I began running across an occasional web page that wouldn’t render correctly. At first I attributed it to poor page coding — perhaps one of those old-timers that still hasn’t learned that there’s more to web browsers than IE, or someone who mucked up their CSS and wound up with white text on a white background — but since I wasn’t necessarily in need of the information at the site, considering the Google search that took me to the site provided many more possible links, I just moved on to the next site without bothering to figure out why the page didn’t render.

Unfortunately, the frequency of pages not rendering properly began to increase to the point that I began experiencing the problem with web sites that I did need to access. Opera could render them fine, and Firefox on my Mac OS X system had no problem, but Firefox on Linux was a basket case. Even worse, Galeon and Seamonkey crashed when I tried to visit those same sites.

OK, let’s start Firefox from the command line and visit one of the problematic sites.

 (firefox-bin:25838): Pango-WARNING **: failed to create cairo scaled font, expect ugly output. the offending font is 'Helvetica 8.19921875'

 (firefox-bin:25838): Pango-WARNING **: font_font status is: out of memory

 (firefox-bin:25838): Pango-WARNING **: scaled_font status is: out of memory

The Great Gazoogle wasn’t being too helpful, leading me to several sites where a similar problem had been reported, usually with an early beta build of Firefox 3, but no obvious solution. I eventually came across an Ubuntu bug report that lead to the apparent solution. I’m not running Ubuntu, so the dpkg-reconfigure and update-pangox-aliases commands were not helpful, but the fc-cache was.

So, I ran

 sudo fc-cache -fs

and waited for it to plow through all the installed fonts on the system. For good measure I did the same on my user-installed fonts too.

 fc-cache -f

Restart Firefox and visit a failing site, and … no errors, no rendering problems, everything seems to be back to normal.

Just for completeness, I visited the same site in Galeon and Seamonkey and, instead of crashing, they now render the sites correctly too.

I have no idea how the font cache got hosed — I haven’t installed any fonts recently — and I haven’t (knowingly) done anything that would have modified the font cache directories. Nevertheless, something got screwed up and a regeneration of the font cache fixed it.

Problems with Mephisto and Passenger

2009-03-22T22:27:00Z

My previous post contained multiple code fragments contained in macro:code blocks, but they were not formatted correctly. There was an excessive amount of space at the beginning of each code block and a smaller amount of excess space at the end, as well as extra lines between each line of code.

Looking at the source code revealed that every line of the code block was terminated with a BR tag. The XHTML code for the line number columns looked like this:

<pre>1<tt><br />
</tt>2<tt><br />
</tt>3<tt><br />
</tt>4<tt><br />
</tt><strong>5</strong><tt><br />
</tt>6<tt><br />
</tt></pre>

Clearly, not what is expected.

I tried the same post in a mephisto development environment but could not replicate the problem. After ruling out browser and OS differences, it was clear that the problem was confined to the production server.

Another curiosity was that the problem only happened when there were multiple code snippets or text followed a code snippet. If only one code snippet appeared in a post and nothing followed it, the BR tags were not inserted.

The differences between the development and production systems are the web server (mongrel in development and Apache + Phusion Passenger in production) and the Ruby interpreter (plain ruby in development and Ruby Enterprise Edition in production).

I quickly narrowed the problem down to Passenger. Passenger 2.1.2 was recently released but the production server was still running version 2.0.6. A quick sudo gem update passenger, an edit of the Apache config, and a restart of Apache, and the extra BR tags have been eliminated.

I can’t tell from the Passenger release notes what might have caused the problem in the first place but, whatever it was, it affected the CodeRay parsing and is no longer an issue.

Now if I can only sort out the CSS issues that are causing some minor formatting glitches with the code snippets in Opera and Firefox 2.

Rails functional tests and 406 errors

2009-03-22T05:57:00Z

So I have an controller method that responds to multiple formats, like so.

def index
  respond_to do |format|
    format.html
    format.pdf
  end
end

Simple enough. So what do the tests look like?

def test_default_index
  get :index
  assert_response :success
end

def test_pdf_index
  get :index, :format => :pdf
  assert_response :success
end

Again, nothing fancy. Let’s run the test and …


  1) Failure:
test_pdf_index(FoosControllerTest)

    [stacktrace omitted for brevity]

Expected response to be a <:success>, but was <406>

2 tests, 2 assertions, 1 failures, 0 errors

Ruh-roh. How could such a simple test fail, and what’s a 406 response? Well, it’s the numeric code for a Not Acceptable HTTP response, so it looks like the HTTP request is borked. Perhaps the log file will reveal something, and indeed it does.


Processing FoosController#index to pdf (for 0.0.0.0 at 2009-03-21 22:57:10) [GET]
  Parameters: {"format"=>:pdf, "action"=>"index", "controller"=>"foos"}

Looks like we can’t use a symbol for the format in the test. Sure enough, a quick change to the failing test,

1234	def test_pdf_index get :index, :format => 'pdf' assert_response :successend

and the results are as expected.


2 tests, 2 assertions, 0 failures, 0 errors

Update: Looks like the code formatting is fubar.

Update II: The code formatting is now fixed. See the explanation if you’re interested in the details.

Two milestones, two years apart

2009-03-21T00:36:00Z

Today finds me celebrating (in a rather low-key manner) two milestones. The first milestone is finally getting this blog off the ground, and the second is the second anniversary of the first commit of my first serious Rails project. The initial site went live two weeks later, replacing a crufty old PHP-based codebase that I had grown weary of maintaining. Naturally, several new features have been added since the original launch and both Ruby and Rails have made developing the site more enjoyable than any of the languages that had been used previously.

For much of the year the site doesn’t look like anything that anyone would bother using Rails, or any other application framework, to create. Why not just throw up a bunch of static HTML pages?

Well, the site exists to support the Dixie Cup homebrew competition that’s held every year in October. The public side of the site handles the online registration process while an administrative backend handles the running of the competition. For about 10 months of the year the site sits quietly with infrequent updates and then springs to life for a couple of months of serious activity.

Since I’ve had several inquiries about the code from other clubs who want to use it for their own competitions, I’ve begun working to extract the guts into a separate project. I found myself needing to look at the revision history of some files and, at one point, decided to look all the way back to the beginning of the revision history for the entire project and noticed the date of the first commit was March 20, 2007.