Updating PostGIS Databases

Posted by david
on 01 Nov 2010 at 10:56

I recently tried upgrading a PostGIS database to a newer version, going from 1.4.0 to 1.5.2. Unfortunately the upgrade scripts failed with the error

ERROR:  type "box3d_extent" is only a shell

The only thing that worked was to follow the advice given by Paul Ramsey, in his blog post PostGIS Back-up / Restore, and Steve Woodbridge, in a thread on the postgis-users list.

The only gotcha I see here is that pg_dump doesn’t include the database’s search path setting. So, I’ll have to remember to set the search path for the database when it’s restored. That doesn’t happen very often — usually only with major version upgrades of PostgreSQL — but it’s infrequent enough that it’s something that has the potential to get overlooked if it’s not documented.

Response Order in Rails Controller Actions

Posted by david
on 27 Oct 2010 at 23:07

When checking test coverage on a Rails application, there was one code path that was missing, even though a test existed for it.

Here’s a stripped-down version of the controller spec:

1
2
3
4
5
6
7
8
9
10
11
12
13
describe BooksController do
  it "should get a book via an AJAX request" do
    xhr :get, :show, :id => 9
    response.should be_success
    response.should have_text(/\bBook 9\b/)
  end

  it "should get a book via a non-AJAX request" do
    get :show, :id => 9
    response.should be_success
    response.should have_text(/\bBook 9\b/)
  end
end

And here’s the relevant parts of the controller:

1
2
3
4
5
6
7
8
9
10
11
12
class BooksController < ApplicationController
  def show
    respond_to do |format|
      format.js do
        # Do stuff for AJAX requests
      end
      format.html do
        # Do stuff for non-AJAX requests
      end
    end
  end
end

But in both cases, only the first response block was being run. Of course, if the expected response text had taken into account the differences between the two responses, and not just one piece of text, the non-AJAX spec would have failed and the problem would have been found sooner. But it wasn’t.

The problem was with the HTTP Accept header or, in this case, the lack of one. The get call did not generate an Accept header so the first response block handled the request. The xhr call, on the other hand, generated an Accept header containing text/javascript, text/html, application/xml, text/xml, */* so the correct response block would handle the request regardless of the order.

The problem never showed up in browsers because an HTTP Accept header is always being passed; Firefox, for example, sends a HTTP Accept header with a value of text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8. Only command-line tools like curl and wget don’t include text/html as part of the Accept header unless told to do so (for example, by using the --header option). I don’t know what search engines and other robots specify for an Accept header, but it might be safe to assume that they act more like curl and wget than Firefox.

So the easy fix in this case was to reverse the order of the response blocks, and remember to make sure that response blocks are always ordered so that the default response is first.

GIS Experiments

Posted by david
on 25 Oct 2010 at 23:00

Despite the painful problem I encountered in my previous post, I did manage to recover and get some maps made and thought I’d share one of them. Over on the right side of the page, you’ll find a link to a little project I put together as a basis for learning the Google Maps API.

The database currently lists about 4,200 breweries. While still incomplete, especially in Asia, Africa, Central and South America, and parts of Europe, I was still curious about how the worldwide distribution looked. Unfortunately, trying to plot them on a map with the Google Maps API is not feasible because of the large number of markers; supposedlly more than a couple hundred markers are a problem, but I’ve not had any serious problem showing up to about 1,000 markers, but 4,000+ definitely won’t work.

Enter Quantum GIS. Since the data is stored in a PostGIS database, it was relatively easy to generate a world map showing all of the brewery locations that have been entered in the database. Admitedly, the map could use some refinement, but I’ve only recently started working with Quantum GIS and have limited experience with its capabilities at the moment.

System Overload

Posted by david
on 25 Oct 2010 at 22:20

This is probably not what you want to see on a typical UNIX system:

$ uptime
 15:07:27 up 500 days,  4:06,  3 users,  load average: 163.05, 170.70, 156.37

There might well be some powerful systems that can handle that type of load, but the one I was working on sure wasn’t coping very well. I can’t recall ever seeing such a high load average before on any UNIX system, and I almost didn’t get to see that because I was very close to power cycling the system to regain control before I was able to recover.

The problem started with an out-of-control process that began sucking up memory. The system began to get slower and slower, and I had a good idea which process was at fault. The Linux OOM (out-of-memory) killer didn’t kick in so I had to, but it was becoming harder and harder to get any response from a terminal, but I did manage to get a ps command entered to identify the likely culprit, watching the characters echo back at about the speed of a 110 baud modem connection on a noisy phone line, and eventually got results:

$ ps -C qgis u
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
dcato    20596 15.1 76.0 6185492 3088452 pts/27 D   13:42  10:54 qgis

This on a system with 4GB RAM and 6GB swap, and a few other running processes that weren’t shy about using memory either. Thankfully I caught the system at just the right time and was able to enter kill -9 20596 and kill the process without any of the same painful delay that the ps command took. I immediately typed uptime and got the results shown above.

Offline application gotcha in Firefox

Posted by david
on 18 May 2010 at 20:30

I’ve been working on an HTML5-based offline web application but ran into trouble when updating files listed in the cache manifest.

Even though the files were downloaded and cached when the page was initially loaded, if a change was made to one (or more) of the files listed on the server, the browser detected the updated manifest file and began pulling down the updated file(s). However, the update would halt with an error part way through the process, but only in Firefox; the update process worked fine in Safari and Chrome.

ActiveScaffold problems when using acts_as_audited

Posted by david
on 29 Jun 2009 at 10:36

While beefing up the admin backend of a Rails app that uses ActiveScaffold for its admin UI, I encountered an error that only happened in production and the stack trace wasn’t much help in identifying the cause.

After failing to recreate the problem with a (relatively) small test case, I took the opposite approach and began ripping code out of the original application until the failure went away.

Rotating Ad Banners for Radiant

Posted by david
on 14 Jun 2009 at 17:08

As part of on ongoing project with Radiant CMS, I needed to display random ad banners on a page. The banner_rotator extension didn’t quite fit my needs so I rolled my own. It’s available at http://github.com/davec/radiant-ad_banners-extension.

What’s different from the banner_rotator extension?

  • Simplified usage
    Just include <r:ad_banner/> to generate the appropriate HTML on the page. If you want a specific banner, use <r:ad_banner name="Banner Name"/>. Include as many banners on the page as you want, though if you include more banners than you have defined, the excess ad_banner tags will be ignored. A banner will only be displayed once on a page unless you specifically force it by specifying the name attribute.
  • Uses the paperclipped extension for asset management. Drag-and-drop an image into the form when creating or editing a banner definition.
  • Banners are global, not tied to a specific page or set of pages. Depending on your use case, this could be considered a strike against the ad_banners extension.

Events Calendar Extension for Radiant

Posted by david
on 05 Jun 2009 at 19:38

I’ve recently been working with Radiant CMS for a project and needed a calendar of events. None of the existing calendar-related extensions fit the bill so I created one that did.

Find it at http://github.com/davec/radiant-events-calendar-extension.

Spam on the rebound

Posted by david
on 31 May 2009 at 21:13

Well, it looks like the spam levels this week have not only returned to their pre-McColo shutdown levels, but they’re beginning to surpass them.

Just before the McColo shutdown, my mail server was receiving roughly 75K emails per day during the week, somewhat less on the weekends. Afterwards, the level dropped to less than 20K per day, which is where it stayed through the end of the year. Since the beginning of the year the amount of mail hitting my mail server has been rising, hitting about 60K messages per day two weeks ago.

Weekend cat blogging

Posted by david
on 30 May 2009 at 20:29

It’s the mystery cat. Yes, there is a cat under there.

This blog has been a bit like the cat under those bed covers, and like the cat this blog eventually emerges from the darkness.

Curiously, she just recently began burrowing under the bed covers this week as the weather has been warming up. Nothing like a 88°F (31°C) day to seek out more warmth. OK, so it was a bit cooler inside, getting up to almost 80°F (27°C) early this evening before beginning to cool down.