I think a lot of people use the mouse to select skills in NL though; the learning curve for the hotkeys is fairly steep because of how much separation there is between where you can look up the hotkeys and where you can actually apply them. In Lix, if you don't remember the hotkey, you can just look at the button and then press the button it says. In NL you have to completely stop playing to go to the options menu and check, which probably means going back to the main menu (I forget if you can access it or not during gameplay, but while we're on the topic of forgetting what the hotkeys were, it doesn't really matter if it's accessible in-game or not if you gotta go back to the main menu to be able to produce the options menu without a hotkey). That's why it's much faster to click with the mouse in NL: technically if you knew the skill hotkeys, it would be faster, but the lookup process is insanely long.
The funny thing is that I grew up with the Dos version of Lemmings. I occasionally used the F function keys to select skills, but most of the time it was via the mouse. Instead, the only function key I usually used was the one for pause (F11), as well as the builder (F7) and bomber (F5). I think the one for builder is because it's the most used skill in the game and hence I took to memorizing the function key for it. However, I think in both cases it was just the function key itself, being somewhat iconic themselves.
Now, in NL I only use the P key for pausing, even though F11 can still be used for pausing too. Thus, when I finally came back to checking out something on the Dos port of Lemmings earlier this year, I'm so used to hitting P for pause that I forgot that you cannot do so via that key on Dos! Needless to say, using F11 for pause is probably going to be the one very hard thing for me to come back to doing, if I ever, say, LP that port of the game, similar to what Colorful Arty did for V-Day for his stream of the game.
No, you can't access the main menu during gameplay. Instead, you need to exit out first to the post-results screen and then again to get to the main menu. So, it does indeed take a while to access it in-game when you're in a level.
Regarding notifying the player how to cancel replay mode, I guess another option could be to have a toggable button for it in the skill panel. Similar to how there's one for CPM if you don't use the compact skill panel and you can activate it either by hotkey (T for the traditional layout) or click the button for it, the same thing can happen with the replay cancel button. However, the problem here is that this would mean another change to the UI, and I don't know if the result would still look fine even with just an extra button added. Just a thought really, but I thought I could just throw it out there just in case.