plotting - Animating a growing ListPlot

I stumbled upon this (rather contrived but interesting) integer sequence. As it exhibits quite different behaviour at different scales, I would like to generate an animated ListPlot of it, where I'm changing the domain and range at an exponential rate to zoom through the orders of magnitude.

I've got something working but it's pretty glitchy and I'm having trouble getting the size of the points to shrink at a good proportional rate.

seq[n_] := seq@n = Module[{b = 2, l},
   While[! IntegerQ[l = Sqrt@Length@IntegerDigits[n, b]], ++b];
   Flatten[Transpose@(IntegerDigits[n, b]~Partition~l)]~FromDigits~b
   ]

plot[n_] := 
 ListPlot[
   seq /@ Range[n], 
   PlotRange -> {{0, n + 1}, {0, n + 1}}, 
   AspectRatio -> 1, 
   ImageSize -> 400, 
   PlotStyle -> PointSize[0.2/n]
 ]
frames = Rasterize[#, "Image"] & /@ Table[plot[10^(n/25)], {n, 25, 100}];
ListAnimate[frames]

or ultimately

Export["sequence.gif", frames]

which gives

As you can see, despite the fixed aspect ratio and the fixed image size, the actual frames seem to have different sizes, leading to artefacts at the bottom. The axes are glitching all over the place as well, probably due to the changing tick labels.

Ideally I also want to increase the maximum n by another one or two orders of magnitude. I've also considered making the lower bound grow as well (so that I'm always showing a fixed ratio between nMax and nMin, since the interesting features of the plot are concentrated around the seq[n]==n line).

How could I improve the animation to get rid of the glitches, get a consistently useful PointSize and just generally make it look smoother?

---

Edit: I've incorporated some of the great suggestions in the comments. Here is my current version of the plot function, this time with fixed PointSize to show the issue of the increasing density. I've also adapted the code to only plot one order of magnitude at a time, because lower-left 10th of the domain isn't really interesting at any given scale:

plot[n_] := 
 ListPlot[
   {#, seq@#} & /@ Range[Round[n/10], Round@n], 
   PlotRange -> {{n/10 - 1, n + 1}, {n/10 - 1, n + 1}}, 
   AspectRatio -> 1, 
   ImageSize -> 400, 
   PlotStyle -> PointSize[0.01], 
   ImagePadding -> {{10, 10}, {10, 10}}, 
   TicksStyle -> Directive[FontOpacity -> 0, FontSize -> 0]

 ]

And this is the corresponding animation:

However, that still has issues with the point size and the efficiency of the solution. As I said, I'd like to continue up to one or two more orders of magnitude, and the problem only gets worse, of course.

I also wonder if it would be possible to generate the full ListPlot only once (probably without any axes at all) and then generate an animation of a growing window into that ListPlot. I feel like that should be significantly faster to process, especially if I want to increase the framerate. Of course, that wouldn't make the PointSize issue any easier.