If you have high enough resolution on your camera, you could set the camera far enough back to get the entire panned area in one frame. Then use software to do a digital zoom and pan.
I assume pressing the shutter button on your camera is still the primary cause of your camera bumps, even with using the delay timer.
There was so much animation in nearly every shot (primary and background action), kudos for tackling so much detail. It must have been quite a challenge to keep track of everything that was going on in each shot as you animated, and I imagine pretty slow to film. Nice work.