Good luck!
When you consider the rewrite serious enough, we can make a board here for it.
I warmly recommend to keep the new formats human-readable: Text, common image formats, grouped resources are directories (or zip files in a pinch, but plain directories are nicer).
Consider widespread text-based serialization formats such as Json, SDLang, Toml, ... When they have good library support, you and others can parse them more easily and write new tooling. Some things, in particular nested structures, are annoying to retrofit into hand-rolled serialization formats that don't support it, and one starts to borrow ideas from the more widespread formats anyway. >_>;;
Maybe consider a good new name for your project? Lemmix, or Vanilla Lemmix, stands for DOS-bug-compatible physics.
-- Simon