NeoLemmix will first load the music.nxmi file in a pack's directory (note that on a technical level, a rank counts as just a sub-pack of the main pack - so each rank can even have its own music.nxmi file, if I remember correctly. If there's none for the individual rank, it'll use the one for the pack as a whole). If there isn't one, it instead uses the one in the "data" directory. This is assuming the individual level doesn't specify a music; if it does, that overrides everything else, although it will fall back on the rotation if the level's specified music is not found. Theoretically if the rotation is an empty list or the music from that isn't found either, it should play no music, but currently there's a bug where it instead uses the music from the last level you played.
Music filenames in music.nxmi (or a level) can contain path info (bug: the editor and pack toolkit do not preserve path names, so this needs to be done by hand in the raw files; NeoLemmix itself handles it fine though) relative to "music". For example, see Lemmings Plus III, where all musics are in a "Lemmings_Plus_III" subfolder of music, so it'd be specified as eg "Lemmings_Plus_III/bam4". However, they should not contain an extension - ie: "track_01" is correct, "track_01.ogg" or "track_01.it" is not (currently NeoLemmix will just ignore the extension specified anyway, but proper use is to not include it at all). NeoLemmix automatically tries all supported extensions. I don't remember the exact order, but I do remember that OGG is highest priority, and all sample-based formats (OGG, WAV, etc) are higher-priority than module-based formats (IT, MOD, XM, etc). Note that while most common formats are supported, two common ones that are not are M4A (which might be what you mean by iTunes format?) and MIDI. Additionally, while supported, MP3 is not recommended - I always recommended, and I suspect this still holds, using either OGG or a module-based format.