Author Topic: OutOfMemoryError  (Read 729 times)

0 Members and 1 Guest are viewing this topic.

Offline Forestidia86

  • Posts: 433
    • View Profile
OutOfMemoryError
« on: January 28, 2018, 07:43:09 am »
I've tried to force it in singleplayer (map: "What could possibly go wrong? (3p)") and got the OutOfMemoryError.
But even with debug build it only gives a very short message. (Probably not different from release build?) I've just attached it for completeness. (Old laptop, Win 7, 4 GB RAM, ca. 1,7 GB for graphics (shared memory, only 64MB dedicated on graphics card))
The taskmanager said that I've used approx 3 out of 4 GB available for me. (Lix ate up about 1,8 GB at this point. That is really much for this kind of game though not generally. But my laptop uses shared memory, so that graphics memory is to a big part handled via RAM as well.)

I will try to run against Allegro debug libraries but can't promise anything

Offline Simon

  • Administrator
  • Posts: 2364
    • View Profile
    • Lix
Re: OutOfMemoryError
« Reply #1 on: January 28, 2018, 09:11:15 am »
Thanks for forcing this. It's uncanny how Lix uses 1.8 GB on Windows.

After the a couple crashes in yesterday's mulitplayer session, I openend top (command-line program) and watched memory throughout the remaining 2 hours of the session. Lix used 2.x % of my 16 GB, never any more than 3 % = 480 MB.

That's still more than I thought, but not exceedingly much. Will look into it.

D is garbage-collected and I force collections every couple seconds. I could force them even more aggressively, and migrate more of the fundamental data to reference counting.

-- Simon

Offline Forestidia86

  • Posts: 433
    • View Profile
Re: OutOfMemoryError
« Reply #2 on: January 28, 2018, 09:28:08 am »
I've tried it with Allegro debug libraries but the crash seems to come from Lix not Allegro, at least I got the Lix crash message and not the MVC++-RTL message. The allegro.log gets huge very fast (in about 30 min about 800 MB(!)).

Offline Simon

  • Administrator
  • Posts: 2364
    • View Profile
    • Lix
Re: OutOfMemoryError
« Reply #3 on: January 28, 2018, 09:31:24 am »
Very good, then I call reasonable libraries and am merely greedy myself.

-- Simon

Offline Forestidia86

  • Posts: 433
    • View Profile
Re: OutOfMemoryError
« Reply #4 on: January 28, 2018, 10:00:10 am »
I've created an allegro.log for just opening a big map, letting Lix spawn, doing some actions and changing to a big map and wait a bit there. There was no crash, but it shows the inner workings of the handling Allegro does. The file is about 9 MB big, shall I provide it?

Offline Simon

  • Administrator
  • Posts: 2364
    • View Profile
    • Lix
Re: OutOfMemoryError
« Reply #5 on: January 28, 2018, 10:03:55 am »
Yep, please attach. It's likely of no use to debug out-of-RAM, but I'm curious.

-- Simon

Offline Forestidia86

  • Posts: 433
    • View Profile
Re: OutOfMemoryError
« Reply #6 on: January 28, 2018, 10:11:42 am »
Attached the log for the short session. It has more than 70000 lines though many of them have similiar content (dealing with bitmaps).

Offline Forestidia86

  • Posts: 433
    • View Profile
Re: OutOfMemoryError
« Reply #7 on: January 28, 2018, 10:53:38 am »
It's probably a stupid question but the error message mentions:
@src\core\exception.d

I don't find such in my Lix-src folder, not even a core-subfolder. I'm a bit confused.

Offline Simon

  • Administrator
  • Posts: 2364
    • View Profile
    • Lix
Re: OutOfMemoryError
« Reply #8 on: January 28, 2018, 11:48:40 am »
If it's not in Lix, then it can be in the D runtime or in Phobos, the D standard library. You can search for this (truncated) path within the D installation directory.

In this case, it's D/dmd2/src/druntime/src/core/exception.d.

With out-of-RAM, there is no single statement that always crashes. Even if we had a stack trace and would find a fat allocation that crashes with out-of-RAM, there is no obvious bug in that allocation. The real problem is that I waste memory throughout the program, and that allocation happens to be the first that was too much.

-- Simon

Offline Forestidia86

  • Posts: 433
    • View Profile
Re: OutOfMemoryError
« Reply #9 on: January 28, 2018, 12:19:34 pm »
If I have a big map open the RAM rises for some time until a certain value. If it has reached a certain high point and I change to another huge map, crash seems more likely (RAM rises beyond this point in some cases then I think).
So it really looks like a certain point of RAM-usage is stepped over, although I've still RAM left. (Nevertheless the RAM-usage is scarily high for my old laptop.)
Though it's still not consistent and I can't see a real pattern in the workings. I have the impression that it's different how low the RAM-usages sinks when closing the map.

Offline Simon

  • Administrator
  • Posts: 2364
    • View Profile
    • Lix
Re: OutOfMemoryError
« Reply #10 on: January 28, 2018, 04:26:36 pm »
Thanks, that's a good baseline. I agree how that RAM usage is scary.

I've spent the better part of today in the code, working on the replay/physics caching. Now, I call the garbage collector more often and return memory to the OS every time old states could be dropped from the cache.

Please pull from unstable, and build debug-replay (commit 3153eb07) and test:
-- Simon
« Last Edit: January 28, 2018, 05:25:02 pm by Simon »

Offline Forestidia86

  • Posts: 433
    • View Profile
Re: OutOfMemoryError
« Reply #11 on: January 28, 2018, 05:31:49 pm »

It doesn't seem to desync anymore with the method I found.
To preserving future: The case mentioned in the issue shouldn't work this way anyways since rewinding before an action cancels if you don't keep replay (, unless you've meant A,C instead of A,B in the end?). But you've mentioned another case in the thread, where you load instead of framestep back. Well loading cuts now all out what doesn't belong to the savestate (even if you have keep replay on and it's after the savestate), so with that the future replay is discarded as well.

  • Does Lix eat as much RAM as you tested earlier today?

Unfortunately RAM still rises high especially in the course of changing maps from time to time. Still got that error eventually.
I looked at the ressource monitor to have a more precise number for Lix' consumption and the highest with crash was ca. 1,4 GB. It really brought the RAM to the edge for my laptop but it was always more allocated for the game then used up as far as I could see. Although the computer had to care constantly for adjusting and increasing the allocation if necessary.
« Last Edit: January 28, 2018, 05:50:06 pm by Forestidia86 »

Offline Simon

  • Administrator
  • Posts: 2364
    • View Profile
    • Lix
Re: OutOfMemoryError
« Reply #12 on: January 28, 2018, 05:42:16 pm »
Re 2 fixed replay bugs: Cool, thanks for confirming! Yeah, you're correct that #294's example is invalid. And you're correct that the second example (in other LF thread) is valid and fixed by the fix of #294.

Re RAM: That's a sad result, but certainly very valuable feedback. I can work on more manual memory management from the bottom upwards, but that will take lots of work, and makes the code more complicated.

Ideally, before shaving this yak, I'd get my hands on a Windows machine, see the RAM ballooning myself, and toying around with some debugging. Made issue on github: #296 RAM balloons to over 1.5 GB, Windows

-- Simon
« Last Edit: January 28, 2018, 06:13:18 pm by Simon »

Offline Forestidia86

  • Posts: 433
    • View Profile
Re: OutOfMemoryError
« Reply #13 on: January 29, 2018, 05:48:59 am »
I've tested with the newer laptop that has 8GB RAM.
I compared the behavior of 0.9.8 with your fix in the unstable.
And it looks like the fix has at least improved the situation:

With fix: I had at most 1,1 GB RAM used with changing maps a couple of times. I didn't seem to have this insane rising generally and seemed to be more stable.

0.9.8: Even on the first map rising to 1,5 GB RAM without doing anything, just waiting. Then nuke and change to "Infinitus (8p)" resulted in immediate crash. I had plenty of RAM still free, it seemed to have been more allocated to the game then used up by the game => nevertheless crash.
« Last Edit: January 29, 2018, 06:08:50 am by Forestidia86 »

Offline Forestidia86

  • Posts: 433
    • View Profile
Re: OutOfMemoryError
« Reply #14 on: January 31, 2018, 01:32:31 pm »
Attached video that shows Ressource Monitor, tab RAM, to show memory consumption of Lix 0.9.8 in huge multiplayer maps up to crash.

Offline Forestidia86

  • Posts: 433
    • View Profile
Re: OutOfMemoryError
« Reply #15 on: January 31, 2018, 02:22:08 pm »
Attached a video that shows 0.9.10 without crash.

Offline Simon

  • Administrator
  • Posts: 2364
    • View Profile
    • Lix
Re: OutOfMemoryError
« Reply #16 on: February 01, 2018, 02:40:08 pm »
Thanks, helpful videos. The single most important figure is the ballpark guess of how much RAM the app eats over time, and the vids show that clearly.

Allocating 40 MB per second and almost never shrinking is a no-go, even during turbo-fast-forward on a large map.

I have observed similar behavior of my Windows builds in Wine, that's great. I'll slowly gnaw on this problem over the next weeks.

-- Simon

Offline Simon

  • Administrator
  • Posts: 2364
    • View Profile
    • Lix
Re: OutOfMemoryError
« Reply #17 on: February 02, 2018, 04:51:46 pm »
Looks like it's the automatic savestates.

If I disable automatic savestating, Lix will keep only 2 gamestates: The zero state and the active state. Lix will not bookkeep with up to 8 savestates. Together with GC'ing more often in 0.9.10, Windows Lix stays at about 300 MB in large levels. That's getting acceptable, even though still big.

-- Simon

Offline Simon

  • Administrator
  • Posts: 2364
    • View Profile
    • Lix
Re: OutOfMemoryError
« Reply #18 on: February 02, 2018, 10:06:17 pm »
Findings from Linux! Exciting story! :lix-cool: Boring technobabble! :8():

Normal allocation. When I allocate a bitmap of 1,000 x 1,000 pixels and paint them all yellow (to ensure that the lazy library/gfx card really allocates something), VRAM usage goes up by 4 MB. Makes perfect sense, it's 1,000,000 pixels with 32-bit colors.

Maybe the card allocates chunks only powers of two and it really allocates 1,024 x 1,024 pixels, but such details don't matter. The 4 MB is perfectly what we would expect. I've measured this with nvidia-msi, command-line tool from my Linux nvidia driver, but an even broader ballpark guess was available in Lix if you check "show FPS" in the options.

RAM usage doesn't increase when I allocate such VRAM bitmaps, at least as long as the graphics card has free VRAM. I can have 100 MB in VRAM and RAM would merely increase by a couple kB, too little to dent any estimate.

Swapping. When you allocate VRAM and the graphics card grows full, the newer allocations still go through without error -- but the graphics card will swap some bitmaps from VRAM into RAM.

We remember how the bitmap of 1,000 x 1,000 pixels occupied 4 MB of VRAM. Now, when the graphics card swaps VRAM into RAM, suddenly such a swapped bitmap occupies 8 MB.

This still doesn't sum to the 2 GB that I observed on Windows. Certainly, the 10 savestates during networking games aren't 100 MB to 200 MB each -- the biggest ingredients are a physics matrix, a land bitmap, and the arrays of lixes. The swapped land is probably the fattest ingredient.

Still, let's keep in mind for now that the graphics card can swap, and the bitmaps surely won't become any leaner or faster in the RAM.

Deallocation. Allegro VRAM bitmaps are manually managed; I have to call destroy explicitly on them, but then the VRAM is immediately available again. I'm reasonably sure that I have no VRAM leaks, but certainly allocate several fat bitmaps.

Leftovers from un-swapping. Now comes the fun part: A freshly started Lix occupies maybe 50 MB in RAM and 11 MB in VRAM. I begin allocating VRAM test bitmaps. Eventually these get swapped to RAM, as descibed above. Let's continue allocating VRAM until 2 GB of RAM is filled with swapped bitmaps.

Now, we deallocate these test bitmaps one-by-one, until none of them is left. The VRAM of Lix has dropped to around 5 MB, slightly less than the 11 MB from a fresh start. But we still have 1.5 GB of RAM filled! This leftover swap-RAM, or whatever it is, doesn't seem to be freed anytime soon. It's not part of the garbage-collected D heap -- I can call the garbage collection manually as often as I want, with no effect.

Looks like it's private memory allocated by the video driver that is called via Allegro inside Lix. In top, these 1.5 GB are still ascribed to Lix.

-- Simon

Offline Forestidia86

  • Posts: 433
    • View Profile
Re: OutOfMemoryError
« Reply #19 on: February 03, 2018, 06:32:37 am »
Swapping. When you allocate VRAM and the graphics card grows full, the newer allocations still go through without error -- but the graphics card will swap some bitmaps from VRAM into RAM.

That actually fits to what I generally have read about that: If no VRAM is free the RAM takes over, which is said to be less performant then.
(As I indicated laptops often work with that even by design (graphics card uses a part of RAM) (shared memory (graphics cards)) since they often have underpowered graphics cards.)

Do you have any ideas how to clear the RAM from the leftover bitmaps?
« Last Edit: February 03, 2018, 09:17:57 am by Forestidia86 »