[?][CE] Running CE and NL 12.14 from the same directory

[WillLem]

Yes all works good if the CE is put in its own directory.

OK, I can confirm that running CE from a NL 12.14 directory results in checkmarks not being displayed in the level select menu. It turns out that it's because CE uses larger images (24 x 24 rather than 16 x 16) due to the treeview itself being larger (and having larger clickable areas). I forgot that I'd made this change, and it does result in the checkmarks not being visible if the assets don't match what's expected.

the text scrolling bar on the main at the bottom of the main page not displaying properly.

And yeah, this is due to the data\title.nxmi file being set up slightly differently to accomodate the CE menu graphics.

All in all, not sure if this idea of running CE from NL is that good. It looks like it leads to some conflicts and gives you some troubles.

I'd probably have to agree at this point. CE has been created with the sole purpose of being a UI upgrade, but it clearly needs to be able to load its own assets in order to achieve this goal.

It seems that the best thing to do would be to ask CE to check for a subdirectory (we could call it "assets-ce" or something) which contains all the CE-specific graphics, data files, etc required to run CE correctly. Mutual assets (levels, replays, music, etc) and NL-only graphics can be placed in the root directory as usual.

Then, when loading something, CE can check for "assets-ce" before checking the root directory. Anything that's found to be missing (Sleeper sprite) or different (title.nxmi, the checkmark .pngs) can be placed into "assets-ce". (Of course, running CE from its own directory will not require the CE assets folder).

That's something I can certainly look at for the release of 1.1 proper. It may be worth doing an RC-2 to check that it's definitely playing ball, if we decide to go ahead with this.

[WillLem]

OK, this should be fixed once and for all.

CE will now ship with an "assets-ce" folder which will contain all of the CE-specific graphics, data files, and overrides. CE-only users will actually not need this*, but anyone wanting to run CE and NL side-by-side can do so simply by placing the .exe and the assets-ce folder into the NL root directory.

It works a treat, and is set up for future-proofing should we need to add or update any more assets in CE.

Implemented across commits 3e70709 - a32d957.

*On second thoughts, the root folder should be identical to NL 12.14 to keep things as simple as possible. "assets-ce", then, will contain everything CE needs to run, whether in its own directory or an NL one (which will be identical anyway!).

[namida]

Yeah, I'll make sure that CE can read from "sound" as well as "sounds" in 1.1 proper.

Scratch that, I've already done it! In commit 5d091f3, full cross-compatibility with legacy NL directories has been added. This was done after the release of 1.1 RC-1. I don't even remember doing it, but it's there and it works! EDIT: Thinking about it, it was probably done in response to this topic. I've made directory cross-compatibility a fairly high priority in general, since it does make a lot of sense that people would want to run both NL and CE from the same directory.

One thing to note is that some of the sounds themselves may have different names or may use different sounds for the same thing. It's worth combining all sounds from CE and NL into the same directory (whichever one you use) to ensure that both programs can access the sound files that they need. You should only need to do this once, and thereafter add any sound that may later be added to CE (this would need to be done for regular NL anyway).

I would go a step further and say - discontinue support for the alternate "sounds" folder. Otherwise, it could cause problems if people use NLCE when creating their packs, and it works fine there but (due to the different folder name) doesn't work on standard NL.

There's no reason this should ever have been changed in NLCE in the first place. That's an SLX-type change; there's no reason for NLCE to be changing file/folder names or adding alternative options for them (it should only, where necessary, add entirely *new* files, not rename - even as optional things - existing ones), given the goal is data and physics compatibility with 12.14. It's too late to not do it in the first place; best to nuke it now while it's still early days (and might cause issues *once* for people upgrading from an older CE version) than have it become a long-term lingering problem.

[WillLem]

I would go a step further and say - discontinue support for the alternate "sounds" folder. Otherwise, it could cause problems if people use NLCE when creating their packs, and it works fine there but (due to the different folder name) doesn't work on standard NL.

The directories were renamed because every other directory in the tree is named "----s".

"Replay" (with its capital letter - why?) and "sound" seem like they shouldn't be there. It's purely aesthetic and, sure, doesn't really matter, but with CE I felt there was an opportunity to sort that out. But you're probably right, it's likely cause more problems than it's worth in the future, so by all means lets revert those folders back to "sound" and "Replay". CE will internally continue to support the use of "sounds" and "replays" folders for those who wish to name their directories this way (I see no reason not to allow this), but we will ship it with "sound" and "Replay" as the default to minimise issues going forward.

The solution here (regarding CE/NL data compatibility in general) is probably to keep the root directory of CE identical to NL 12.14, and add anything new to the aforementioned "assets-ce" folder (see this post, above). This then becomes a necessary component of CE development which will allow us to update existing graphics and data files without risk of conflicts. CE will prefer this directory when looking for resources, so the rest of the tree can be identical to NL. This will ensure that anyone making content for CE that is then played in NL will add items to the correct relative folders (i.e. nothing should ever need to be added to assets-ce by an author unless it's to mod their own copy).

[WillLem]

Here's a transcripted Discord conversation between myself and Dullstar regarding NL/CE directory organisation:

Discord conversation

Will — Yesterday at 12:04 PM
@namida Regarding CE/NL directories, how difficult is it to embed resources into the .exe itself? After giving it some thought, maybe the best option would be to make CE completely compatible with a 12.14 directory, and instead of shipping it with an additional assets folder, just bake those into the .exe and make it fully portable
...
Dullstar — Yesterday at 8:04 PM
I would personally advise against it, as it makes it difficult for users to modify them.
Better to just make sure the CE resources have unique names.
...
Will  — Yesterday at 8:15 PM
Hmm, this is a good idea actually 👍
...
Instead of an "assets-ce" folder, which could get messy, look for a "-ce" suffix on the item itself
Dullstar — Yesterday at 8:17 PM
Personally I'd suggest prefixing
...
Reason: sort by name will put same-prefix together
Will — Yesterday at 8:18 PM
Ah
Yes, that's good
...
The only thing with separate and unique data files is that users then have to hunt through the directories to grab the stuff they need (that is, if they're running CE from NL)
The thinking behind a separate "assets-ce" folder is that they could move everything in one go
...
It just makes it that bit more instantly portable
Dullstar — Yesterday at 8:22 PM
As long as anything that's different between a CE and vanilla NL installation has a unique name, they'll merge cleanly -- the OS might ask about overwriting files, but if the shared files are identical it doesn't matter which ones they keep
Will — Yesterday at 8:22 PM
It's also easier to keep track of what has been updated or added
With a single "assets" folder, that is
Dullstar — Yesterday at 8:23 PM
Particularly if you have a lot of them
Will — Yesterday at 8:24 PM
Hm, it's difficult to decide now. Unique names for individual files should probably happen anyway, but then it's do we just mix it in with NL stuff or keep it in its own directory...?
Dullstar — Yesterday at 8:24 PM
Another option might be a ce subdirectory
...
So, instead of having an "assets" and an "assets-ce" (both in the root folder) you'd have a "ce" folder inside of "assets"
Will — Yesterday at 8:26 PM
That could work, sure
We're adding more folders though, then
Dullstar — Yesterday at 8:26 PM
well both options add a folder 🙂 but the advantage of a subdirectory is that it doesn't add a folder to the root
Will — Yesterday at 8:27 PM
Yes, but NL has many subdirectories
Dullstar — Yesterday at 8:27 PM
There's nothing wrong with that though
Will  — Yesterday at 8:28 PM
So, if we update a menu image, do we simply add the menu image with a "ce" suffix, add it to a "ce" folder inside of "gfx/menu", or add it to a "ce" folder in the root
...
All have advantages and disadvantages I suppose
What I essentially want is for it to be as easy as possible for someone to drop CE into a NL directory and it just work
But, we do need to be able to update assets or there's not really much point in continuing development
Dullstar — Yesterday at 8:30 PM
Here's a question: by this, do you mean dropping CE into a NL directory to do an in-place upgrade, or do you mean running NL and CE from the same directory?
(i.e. should the vanilla NL installation remain intact?)
Will  — Yesterday at 8:31 PM
So, Guigui on the Forums has started a topic about running both from the same directory. I think this should definitely be possible
...
Dullstar — Yesterday at 8:33 PM
In that case, the only important thing is ensuring that all changed files have unique paths; how they're organized between them doesn't matter too much (though I suggest against adding new folders to the root to avoid visual clutter).
In order to install it, you can pretty much just drag and drop, and the OS should handle merging everything properly.
Will — Yesterday at 8:35 PM
The advantage with a single folder in the root is that everything ce needs can go into that folder. Portable, tidy, easy to maintain
Any time anyone's having issues I can simply pass them a copy of that folder with all the updated assets
It's likely to never be more than a few MB
Dullstar — Yesterday at 8:36 PM
If it's mostly gfx I'd say put it in that folder.
(ce folder with everything is fine)
Will  — Yesterday at 8:37 PM
It's a mix of text files, images and sounds
Dullstar — Yesterday at 8:37 PM
Now the big "don't do this" is "Oh hey, someone's running me on Linux! I'm going to make a hidden folder in the user's home directory!" (alongside 3,000 other applications that had the same idea)
Will  — Yesterday at 8:38 PM
Yeah I won't be doing that. Cba with trying to manage that mess
For me it's either: everything in one folder if possible, or bake everything into the exe
We could even have a combination of the two, where people who aren't bothered about mods can just use the exe, and those who want to mod stuff can add the assets folder
...
Dullstar — Yesterday at 8:40 PM
I feel like the assets folder is mostly stuff a user might reasonably try to reskin.
Will — Yesterday at 8:41 PM
The more we talk, the more I'm realising that's how I'd want to do it. Bake everything ce-specific into the exe, but also check for an optional "ce-mods" or "ce-assets" folder which overwrites the default
Dullstar  — Yesterday at 8:41 PM
Embedding defaults can be reasonable.
Will  — Yesterday at 8:42 PM
That way, anyone who isn't bothered about modding everything can just drop the exe into an NL directory and they're good to go
...
Dullstar — Yesterday at 8:44 PM
Of course, do keep in mind that "select all the stuff in the zip, and drop that in the NL directory" isn't any more complicated either.

Following this conversation, it's been decided that CE-specific assets should be uniquely-named, especially if they also exist in NL. Allowing CE to modify/rename any files/folders also used by NL is clearly a bad idea, so the next version (1.1) will at the very least have all CE-specific files and folders marked with a "ce-" prefix. Anything that can be used by both NL and CE (such as levels, sounds, music, etc) or that hasn't been modified for NL will be stored and named exactly as it is in NL 12.14.

From here, we then need to decide how CE-specific files should be stored. From the conversation above, I can figure 3 clear options:

1) Only rename the files, and store them in the same directories as NL. So, for example, the "data/title.nxmi" file has been modified for CE. We'd keep it in the data folder as normal, but rename it to "ce-title.nxmi".

Advantages: File system will look identical to NL | Internal loading is identical to NL, all that needs to happen is renaming the files.
Disadvantages: Harder to keep track of what has been changed/updated | Users might only copy NeoLemmixCE.exe into their NL 12.14 directory and find that things don't work because they haven't copied across the other files they need (so, we're relying on users merging directories correctly whenever there's an update). This could get messy.

2) Provide a single "ce-assets" folder which contains everything that CE needs to run. This is placed into the root directory of NL 12.14 along with the .exe, and can be modified if desired.

Advantages: Very easy to migrate CE stuff to the NL directory | Very easy to keep track of what has been updated | Very easy to debug.
Disadvantages: A bit of extra code (more possibility for loading issues) | User still has to manually copy additional stuff from CE to the NL directory (but at least it's all in one folder).

3) Bake all CE-specific assets into the .exe itself, don't provide any additional folders at all. All the user has to do to run CE from their NL directory is drop the .exe into the root folder. That's it!

Advantages: By far the simplest and easiest solution for the end user.
Disadvantages: Significantly more code (and, I'd have to learn how to embed assets in the .exe) | User cannot mod the CE assets.

4) A mix of options 1 and 3. Bake the CE assets into the .exe, but also provide the individual assets (with "ce-" prefix) in their usual folders so that users can mod files, graphics, etc. Then, only the CE.exe is necessary for allowing CE to run from the NL directory. Users would only need the individual files if they wanted to modify something.

Advantages: Best of both worlds in terms of portability and modifiability.
Disadvantages: The most code, so the most chance for things to go wrong program-side. I'd probably create a new resource management class and run everything through that when loading assets. That should help to at least make debugging easier.

Thoughts?

(Your silence = I'll probably go with option 4).

[Guigui]

It looks like I launched this whole idea of booting CE from the same directory as NL, so I'm giving my vague 2 cents here : either solution 2) or 4) sound fine to me.

1) could indeed turn CE things hard to locate, and 3) does not allow modifications (which I already did for some tiny things in the CE).

Please note that I am not a programmer at all so I may not understand all arguments when they become complicated like here, I'm just an end user who likes to keep things kind of tidy on my machine.

[WillLem]

It looks like I launched this whole idea of booting CE from the same directory as NL, so I'm giving my vague 2 cents here : either solution 2) or 4) sound fine to me.

Agreed. I'll probably go with 4. It's a bit more work for a lot more benefit.

I'm just an end user who likes to keep things kind of tidy on my machine.

This actually makes you the perfect person to provide feedback, and I'm very grateful for it!