Lemmings Forums

Lix => Lix Main => Topic started by: Forestidia86 on January 28, 2018, 07:43:09 AM

Title: Stop maps from crashing (OutOfMemoryError)
Post by: Forestidia86 on January 28, 2018, 07:43:09 AM
I've tried to force it in singleplayer (map: "What could possibly go wrong? (3p)") and got the OutOfMemoryError.
But even with debug build it only gives a very short message. (Probably not different from release build?) I've just attached it for completeness. (Old laptop, Win 7, 4 GB RAM, ca. 1,7 GB for graphics (shared memory, only 64MB dedicated on graphics card))
The taskmanager said that I've used approx 3 out of 4 GB available for me. (Lix ate up about 1,8 GB at this point. That is really much for this kind of game though not generally. But my laptop uses shared memory, so that graphics memory is to a big part handled via RAM as well.)

I will try to run against Allegro debug libraries but can't promise anything
Title: Re: OutOfMemoryError
Post by: Simon on January 28, 2018, 09:11:15 AM
Thanks for forcing this. It's uncanny how Lix uses 1.8 GB on Windows.

After the a couple crashes in yesterday's mulitplayer session, I openend top (command-line program) and watched memory throughout the remaining 2 hours of the session. Lix used 2.x % of my 16 GB, never any more than 3 % = 480 MB.

That's still more than I thought, but not exceedingly much. Will look into it.

D is garbage-collected and I force collections every couple seconds. I could force them even more aggressively, and migrate more of the fundamental data to reference counting.

-- Simon
Title: Re: OutOfMemoryError
Post by: Forestidia86 on January 28, 2018, 09:28:08 AM
I've tried it with Allegro debug libraries but the crash seems to come from Lix not Allegro, at least I got the Lix crash message and not the MVC++-RTL message. The allegro.log gets huge very fast (in about 30 min about 800 MB(!)).
Title: Re: OutOfMemoryError
Post by: Simon on January 28, 2018, 09:31:24 AM
Very good, then I call reasonable libraries and am merely greedy myself.

-- Simon
Title: Re: OutOfMemoryError
Post by: Forestidia86 on January 28, 2018, 10:00:10 AM
I've created an allegro.log for just opening a big map, letting Lix spawn, doing some actions and changing to a big map and wait a bit there. There was no crash, but it shows the inner workings of the handling Allegro does. The file is about 9 MB big, shall I provide it?
Title: Re: OutOfMemoryError
Post by: Simon on January 28, 2018, 10:03:55 AM
Yep, please attach. It's likely of no use to debug out-of-RAM, but I'm curious.

-- Simon
Title: Re: OutOfMemoryError
Post by: Forestidia86 on January 28, 2018, 10:11:42 AM
Attached the log for the short session. It has more than 70000 lines though many of them have similiar content (dealing with bitmaps).
Title: Re: OutOfMemoryError
Post by: Forestidia86 on January 28, 2018, 10:53:38 AM
It's probably a stupid question but the error message mentions:
@src\core\exception.d

I don't find such in my Lix-src folder, not even a core-subfolder. I'm a bit confused.
Title: Re: OutOfMemoryError
Post by: Simon on January 28, 2018, 11:48:40 AM
If it's not in Lix, then it can be in the D runtime or in Phobos, the D standard library. You can search for this (truncated) path within the D installation directory.

In this case, it's D/dmd2/src/druntime/src/core/exception.d.

With out-of-RAM, there is no single statement that always crashes. Even if we had a stack trace and would find a fat allocation that crashes with out-of-RAM, there is no obvious bug in that allocation. The real problem is that I waste memory throughout the program, and that allocation happens to be the first that was too much.

-- Simon
Title: Re: OutOfMemoryError
Post by: Forestidia86 on January 28, 2018, 12:19:34 PM
If I have a big map open the RAM rises for some time until a certain value. If it has reached a certain high point and I change to another huge map, crash seems more likely (RAM rises beyond this point in some cases then I think).
So it really looks like a certain point of RAM-usage is stepped over, although I've still RAM left. (Nevertheless the RAM-usage is scarily high for my old laptop.)
Though it's still not consistent and I can't see a real pattern in the workings. I have the impression that it's different how low the RAM-usages sinks when closing the map.
Title: Re: OutOfMemoryError
Post by: Simon on January 28, 2018, 04:26:36 PM
Thanks, that's a good baseline. I agree how that RAM usage is scary.

I've spent the better part of today in the code, working on the replay/physics caching. Now, I call the garbage collector more often and return memory to the OS every time old states could be dropped from the cache.

Please pull from unstable, and build debug-replay (commit 3153eb07) and test:
-- Simon
Title: Re: OutOfMemoryError
Post by: Forestidia86 on January 28, 2018, 05:31:49 PM
  • Is this fixed? #294 Manual savestate, sporadic replay desync (https://github.com/SimonN/LixD/issues/294)
  • Is this fixed? #295 Savestate shouldn't preserve future when (don't keep replay after ◀▮) (https://github.com/SimonN/LixD/issues/295)

It doesn't seem to desync anymore with the method I found.
To preserving future: The case mentioned in the issue shouldn't work this way anyways since rewinding before an action cancels if you don't keep replay (, unless you've meant A,C instead of A,B in the end?). But you've mentioned another case in the thread, where you load instead of framestep back. Well loading cuts now all out what doesn't belong to the savestate (even if you have keep replay on and it's after the savestate), so with that the future replay is discarded as well.

  • Does Lix eat as much RAM as you tested earlier today?

Unfortunately RAM still rises high especially in the course of changing maps from time to time. Still got that error eventually.
I looked at the ressource monitor to have a more precise number for Lix' consumption and the highest with crash was ca. 1,4 GB. It really brought the RAM to the edge for my laptop but it was always more allocated for the game then used up as far as I could see. Although the computer had to care constantly for adjusting and increasing the allocation if necessary.
Title: Re: OutOfMemoryError
Post by: Simon on January 28, 2018, 05:42:16 PM
Re 2 fixed replay bugs: Cool, thanks for confirming! Yeah, you're correct that #294's example is invalid. And you're correct that the second example (in other LF thread) is valid and fixed by the fix of #294.

Re RAM: That's a sad result, but certainly very valuable feedback. I can work on more manual memory management from the bottom upwards, but that will take lots of work, and makes the code more complicated.

Ideally, before shaving this yak, I'd get my hands on a Windows machine, see the RAM ballooning myself, and toying around with some debugging. Made issue on github: #296 RAM balloons to over 1.5 GB, Windows (https://github.com/SimonN/LixD/issues/296)

-- Simon
Title: Re: OutOfMemoryError
Post by: Forestidia86 on January 29, 2018, 05:48:59 AM
I've tested with the newer laptop that has 8GB RAM.
I compared the behavior of 0.9.8 with your fix in the unstable.
And it looks like the fix has at least improved the situation:

With fix: I had at most 1,1 GB RAM used with changing maps a couple of times. I didn't seem to have this insane rising generally and seemed to be more stable.

0.9.8: Even on the first map rising to 1,5 GB RAM without doing anything, just waiting. Then nuke and change to "Infinitus (8p)" resulted in immediate crash. I had plenty of RAM still free, it seemed to have been more allocated to the game then used up by the game => nevertheless crash.
Title: Re: OutOfMemoryError
Post by: Forestidia86 on January 31, 2018, 01:32:31 PM
Attached video that shows Ressource Monitor, tab RAM, to show memory consumption of Lix 0.9.8 in huge multiplayer maps up to crash.
Title: Re: OutOfMemoryError
Post by: Forestidia86 on January 31, 2018, 02:22:08 PM
Attached a video that shows 0.9.10 without crash.
Title: Re: OutOfMemoryError
Post by: Simon on February 01, 2018, 02:40:08 PM
Thanks, helpful videos. The single most important figure is the ballpark guess of how much RAM the app eats over time, and the vids show that clearly.

Allocating 40 MB per second and almost never shrinking is a no-go, even during turbo-fast-forward on a large map.

I have observed similar behavior of my Windows builds in Wine, that's great. I'll slowly gnaw on this problem over the next weeks.

-- Simon
Title: Re: OutOfMemoryError
Post by: Simon on February 02, 2018, 04:51:46 PM
Looks like it's the automatic savestates.

If I disable automatic savestating, Lix will keep only 2 gamestates: The zero state and the active state. Lix will not bookkeep with up to 8 savestates. Together with GC'ing more often in 0.9.10, Windows Lix stays at about 300 MB in large levels. That's getting acceptable, even though still big.

-- Simon
Title: Re: OutOfMemoryError
Post by: Simon on February 02, 2018, 10:06:17 PM
Findings from Linux! Exciting story! :lix-cool: Boring technobabble! :8():

Normal allocation. When I allocate a bitmap of 1,000 x 1,000 pixels and paint them all yellow (to ensure that the lazy library/gfx card really allocates something), VRAM usage goes up by 4 MB. Makes perfect sense, it's 1,000,000 pixels with 32-bit colors.

Maybe the card allocates chunks only powers of two and it really allocates 1,024 x 1,024 pixels, but such details don't matter. The 4 MB is perfectly what we would expect. I've measured this with nvidia-msi, command-line tool from my Linux nvidia driver, but an even broader ballpark guess was available in Lix if you check "show FPS" in the options.

RAM usage doesn't increase when I allocate such VRAM bitmaps, at least as long as the graphics card has free VRAM. I can have 100 MB in VRAM and RAM would merely increase by a couple kB, too little to dent any estimate.

Swapping. When you allocate VRAM and the graphics card grows full, the newer allocations still go through without error -- but the graphics card will swap some bitmaps from VRAM into RAM.

We remember how the bitmap of 1,000 x 1,000 pixels occupied 4 MB of VRAM. Now, when the graphics card swaps VRAM into RAM, suddenly such a swapped bitmap occupies 8 MB.

This still doesn't sum to the 2 GB that I observed on Windows. Certainly, the 10 savestates during networking games aren't 100 MB to 200 MB each -- the biggest ingredients are a physics matrix, a land bitmap, and the arrays of lixes. The swapped land is probably the fattest ingredient.

Still, let's keep in mind for now that the graphics card can swap, and the bitmaps surely won't become any leaner or faster in the RAM.

Deallocation. Allegro VRAM bitmaps are manually managed; I have to call destroy explicitly on them, but then the VRAM is immediately available again. I'm reasonably sure that I have no VRAM leaks, but certainly allocate several fat bitmaps.

Leftovers from un-swapping. Now comes the fun part: A freshly started Lix occupies maybe 50 MB in RAM and 11 MB in VRAM. I begin allocating VRAM test bitmaps. Eventually these get swapped to RAM, as descibed above. Let's continue allocating VRAM until 2 GB of RAM is filled with swapped bitmaps.

Now, we deallocate these test bitmaps one-by-one, until none of them is left. The VRAM of Lix has dropped to around 5 MB, slightly less than the 11 MB from a fresh start. But we still have 1.5 GB of RAM filled! This leftover swap-RAM, or whatever it is, doesn't seem to be freed anytime soon. It's not part of the garbage-collected D heap -- I can call the garbage collection manually as often as I want, with no effect.

Looks like it's private memory allocated by the video driver that is called via Allegro inside Lix. In top, these 1.5 GB are still ascribed to Lix.

-- Simon
Title: Re: OutOfMemoryError
Post by: Forestidia86 on February 03, 2018, 06:32:37 AM
Swapping. When you allocate VRAM and the graphics card grows full, the newer allocations still go through without error -- but the graphics card will swap some bitmaps from VRAM into RAM.

That actually fits to what I generally have read about that: If no VRAM is free the RAM takes over, which is said to be less performant then.
(As I indicated laptops often work with that even by design (graphics card uses a part of RAM) (shared memory (graphics cards)) since they often have underpowered graphics cards.)

Do you have any ideas how to clear the RAM from the leftover bitmaps?
Title: Stop maps from crashing
Post by: Colorful Arty on February 14, 2018, 07:23:42 PM
After playing the last few multiplayer sessions, I've noticed a few maps are prone to crashing peoples' machines. Simon says this is caused by the maps in question being too large, and caused people to have OutOfRAM and/or OutOfVRAM crashes. I think something should be done about this. There are a few options:

1: Lower the maximum size of custom levels. My level "The Corridors of Time" played last session was as large as possible, but with very little terrain. This still crashed, which makes me wonder why people are allowed to make levels that big in the first place. The downsides are this may cause problems with existing maps that are already that large.

2: Make the multiplayer code more efficient spatially-efficient, preventing future crashes. I don't know how easy or feasible this is though.

3: Everyone installs more RAM on their computers. ;P
Title: Re: Stop maps from crashing
Post by: Forestidia86 on February 14, 2018, 08:02:35 PM
After playing the last few multiplayer sessions, I've noticed a few maps are prone to crashing peoples' machines. Simon says this is caused by the maps in question being too large, and caused people to have OutOfRAM and/or OutOfVRAM crashes. I think something should be done about this. There are a few options:

1: Lower the maximum size of custom levels. My level "The Corridors of Time" played last session was as large as possible, but with very little terrain. This still crashed, which makes me wonder why people are allowed to make levels that big in the first place. The downsides are this may cause problems with existing maps that are already that large.

2: Make the multiplayer code more efficient spatially-efficient, preventing future crashes. I don't know how easy or feasible this is though.

3: Everyone installs more RAM on their computers. ;P

Option 1 sadly seems to be the most realistic in my eyes at the moment, although it would be only a workaround. Improving the code would be of course the best.
I'm not sure if more RAM would help since the game seems to have a memory leak and crashes even if still free RAM is left. Maybe more VRAM would help. Simon has explained what happens here (https://www.lemmingsforums.net/index.php?topic=3701.msg69288#msg69288):
The game makes autosavestates of the bitmaps and if not enough VRAM is available it goes into RAM. Whereas VRAM is cleared by the game from time to time, the RAM sadly is not.

Maybe restarting the game before playing a huge map and restarting afterwards could alleviate that.
Title: Re: Stop maps from crashing
Post by: Simon on February 14, 2018, 08:10:17 PM
Yeah, this is the nastiest problem in the game.

Solution 1 (limit size) is hackish, but a reasonable workaround because everything else is hard. I have to guess how widespread graphics cards behave with what sizes of image. It's hard and inconsistent. (https://www.lemmingsforums.net/index.php?topic=3701.msg69288#msg69288) (Forestidia posted this same link.)

Then enforce a max size that doesn't crash anybody's machine. Map size limit is purely client-side; the server relays levels, but doesn't interpret the level file.

Quote
game seems to have a memory leak and crashes even if still free RAM

Game is largely garbage-collected (0.9.10 and newer run the GC more often than before), and some parts are reference-counted (these don't seem to leak). The leak comes mostly from the graphics card that swaps to RAM, but doesn't free this RAM before end of application. I call the graphics hardware via Allegro 5 and don't see a way to tell the card to free such swapped RAM.

What maps did crash so far? Infinitus 8p crashed, and Arty's clock map at maximum size crashed.



Solution 2 (efficient storage) is the best, but needs the deepest reworking. The Clones solution split the map into 128x128 chunks, and save only the chunks that change over time.

-- Simon
Title: Re: Stop maps from crashing
Post by: Simon on February 21, 2018, 06:41:25 AM
Here are the plans for 0.9.12.

Warn on large maps: On all previews of maps with more than 3 million pixels, I'll print a warning that these maps can crash Lix.

Still allow everybody to play these large maps, nonetheless. You can ignore the warning and press ready for multiplayer, play the map in singleplayer, or load it in the editor. This way, 0.9.12 won't be a downgrade, and it leaves player in control.

Example of large maps where the warning will show:
These 3 maps have crashed in the past. If you experience crashes on any other maps, please tell me, and I'll revise in-game warning. These crashes will generate a message box that says either "OutOfMemory: Memory allocation failed" or "OutOfVramException".

I've merely guessed a map size (3 million pixels) and hope that it works as a catch-all. Ideally, we'll find a better solution than such a warning in the long run.

-- Simon
Title: Re: Stop maps from crashing (OutOfMemoryError)
Post by: Forestidia86 on November 09, 2018, 03:06:48 PM
The problem is probably related to the weak 32bit garbage collection (cf. here (https://forum.dlang.org/thread/pmunyfdozzumkbqgxptd@forum.dlang.org)).
The current Win Lix release is 32bit, whereas on Linux it is commonly 64bit since most have 64bit systems.
That would explain why people on Win crash but the ones on Linux not.

A 64bit build should at least lessen RAM usuage, which could maybe solve this problem.
The problem for a 64bit build was in earlier time that Visual Studio was needed for linking (cf. here (https://www.lemmingsforums.net/index.php?topic=3801.0)).

But now the compiler LDC has released a beta (https://github.com/ldc-developers/ldc/releases/tag/v1.13.0-beta1) that is self-sufficient, that doesn't need Visual Studio. Only the Visual C++ 2015 runtime (https://www.microsoft.com/de-de/download/details.aspx?id=48145) to run the game could be needed still, but there are quite a few games that install that with it, so it is likely that people have it already.

So there are good chances that in future big maps can be played again.

At the moment there are still quite a few problems (https://github.com/SimonN/LixD/issues/366) open, though.