graphics - Why does a heavy background slow LocatorPane from updating, even if only the locators are Dynamic?

I'm having some trouble getting a dynamic plot to operate efficiently, and I think I've tracked down at least part of the problem to some pretty weird behaviour on the part of LocatorPane.

Take two different image backgrounds, which can be quite similar except for their size on disk:

image1 = ContourPlot[x^2-y^2, {x, -1, 1}, {y, -1, 1}, ImageSize -> 250];

image2 = ContourPlot[x^2-y^2, {x, -1, 1}, {y, -1, 1}, ImageSize -> 250, PlotPoints -> 200];

where the ByteCount of image2 is, in this case, about twenty times bigger than that of image1, and now use these images as the background for a LocatorPane:

point = {0.4, 0};
Row[{

  LocatorPane[
   Dynamic[point],
   image2
   ],
  Dynamic[point]
  }]

which produces something like this

As you can see, the version with the heavier background is a huge lot more sluggish than the lighter one (though this is much more evident if you run it yourself). Why is this? Both occurrences of point are inside a Dynamic command but the background is not, so I don't really see a reason for the background of the pane to be re-loaded every time the locators are changed. More importantly, how can this be avoided? I have a byte-heavy background which I would like to only calculate once and then display without it slowing me down.

I should also note that raster-based solutions will most likely prove unsatisfactory, particularly because I want to deploy a fair bit of information in the tooltips associated with the background contours.