That's normal. It's becuase DOS Lemmings only has a resolution of 320x200. So, a move by in-game coordinates of 1 pixel will be equivalent to, on a modern screen - let's say 1080p - a move of about 5 pixels.
The only workaround is to use a clone engine instead of the official game. Lemmix (the regular one, not NeoLemmix) tries to be as close as possible to DOS in terms of game physics, but it does not have the jittery mouse issue. It does have most of the same fine-control features NeoLemmix has, but of course, no one's forcing you to use them.