Stop maps from crashing (OutOfMemoryError)

[Forestidia86]

Attached a video that shows 0.9.10 without crash.

[Simon]

Thanks, helpful videos. The single most important figure is the ballpark guess of how much RAM the app eats over time, and the vids show that clearly.

Allocating 40 MB per second and almost never shrinking is a no-go, even during turbo-fast-forward on a large map.

I have observed similar behavior of my Windows builds in Wine, that's great. I'll slowly gnaw on this problem over the next weeks.

-- Simon

[Simon]

Looks like it's the automatic savestates.

If I disable automatic savestating, Lix will keep only 2 gamestates: The zero state and the active state. Lix will not bookkeep with up to 8 savestates. Together with GC'ing more often in 0.9.10, Windows Lix stays at about 300 MB in large levels. That's getting acceptable, even though still big.

-- Simon

[Simon]

Findings from Linux! Exciting story! Boring technobabble!

Normal allocation. When I allocate a bitmap of 1,000 x 1,000 pixels and paint them all yellow (to ensure that the lazy library/gfx card really allocates something), VRAM usage goes up by 4 MB. Makes perfect sense, it's 1,000,000 pixels with 32-bit colors.

Maybe the card allocates chunks only powers of two and it really allocates 1,024 x 1,024 pixels, but such details don't matter. The 4 MB is perfectly what we would expect. I've measured this with nvidia-msi, command-line tool from my Linux nvidia driver, but an even broader ballpark guess was available in Lix if you check "show FPS" in the options.

RAM usage doesn't increase when I allocate such VRAM bitmaps, at least as long as the graphics card has free VRAM. I can have 100 MB in VRAM and RAM would merely increase by a couple kB, too little to dent any estimate.

Swapping. When you allocate VRAM and the graphics card grows full, the newer allocations still go through without error -- but the graphics card will swap some bitmaps from VRAM into RAM.

We remember how the bitmap of 1,000 x 1,000 pixels occupied 4 MB of VRAM. Now, when the graphics card swaps VRAM into RAM, suddenly such a swapped bitmap occupies 8 MB.

This still doesn't sum to the 2 GB that I observed on Windows. Certainly, the 10 savestates during networking games aren't 100 MB to 200 MB each -- the biggest ingredients are a physics matrix, a land bitmap, and the arrays of lixes. The swapped land is probably the fattest ingredient.

Still, let's keep in mind for now that the graphics card can swap, and the bitmaps surely won't become any leaner or faster in the RAM.

Deallocation. Allegro VRAM bitmaps are manually managed; I have to call destroy explicitly on them, but then the VRAM is immediately available again. I'm reasonably sure that I have no VRAM leaks, but certainly allocate several fat bitmaps.

Leftovers from un-swapping. Now comes the fun part: A freshly started Lix occupies maybe 50 MB in RAM and 11 MB in VRAM. I begin allocating VRAM test bitmaps. Eventually these get swapped to RAM, as descibed above. Let's continue allocating VRAM until 2 GB of RAM is filled with swapped bitmaps.

Now, we deallocate these test bitmaps one-by-one, until none of them is left. The VRAM of Lix has dropped to around 5 MB, slightly less than the 11 MB from a fresh start. But we still have 1.5 GB of RAM filled! This leftover swap-RAM, or whatever it is, doesn't seem to be freed anytime soon. It's not part of the garbage-collected D heap -- I can call the garbage collection manually as often as I want, with no effect.

Looks like it's private memory allocated by the video driver that is called via Allegro inside Lix. In top, these 1.5 GB are still ascribed to Lix.

-- Simon

[Forestidia86]

Swapping. When you allocate VRAM and the graphics card grows full, the newer allocations still go through without error -- but the graphics card will swap some bitmaps from VRAM into RAM.

That actually fits to what I generally have read about that: If no VRAM is free the RAM takes over, which is said to be less performant then.
(As I indicated laptops often work with that even by design (graphics card uses a part of RAM) (shared memory (graphics cards)) since they often have underpowered graphics cards.)

Do you have any ideas how to clear the RAM from the leftover bitmaps?

[Colorful Arty]

After playing the last few multiplayer sessions, I've noticed a few maps are prone to crashing peoples' machines. Simon says this is caused by the maps in question being too large, and caused people to have OutOfRAM and/or OutOfVRAM crashes. I think something should be done about this. There are a few options:

1: Lower the maximum size of custom levels. My level "The Corridors of Time" played last session was as large as possible, but with very little terrain. This still crashed, which makes me wonder why people are allowed to make levels that big in the first place. The downsides are this may cause problems with existing maps that are already that large.

2: Make the multiplayer code more efficient spatially-efficient, preventing future crashes. I don't know how easy or feasible this is though.

3: Everyone installs more RAM on their computers.

[Forestidia86]

After playing the last few multiplayer sessions, I've noticed a few maps are prone to crashing peoples' machines. Simon says this is caused by the maps in question being too large, and caused people to have OutOfRAM and/or OutOfVRAM crashes. I think something should be done about this. There are a few options:

1: Lower the maximum size of custom levels. My level "The Corridors of Time" played last session was as large as possible, but with very little terrain. This still crashed, which makes me wonder why people are allowed to make levels that big in the first place. The downsides are this may cause problems with existing maps that are already that large.

2: Make the multiplayer code more efficient spatially-efficient, preventing future crashes. I don't know how easy or feasible this is though.

3: Everyone installs more RAM on their computers.

Option 1 sadly seems to be the most realistic in my eyes at the moment, although it would be only a workaround. Improving the code would be of course the best.
I'm not sure if more RAM would help since the game seems to have a memory leak and crashes even if still free RAM is left. Maybe more VRAM would help. Simon has explained what happens here:
The game makes autosavestates of the bitmaps and if not enough VRAM is available it goes into RAM. Whereas VRAM is cleared by the game from time to time, the RAM sadly is not.

Maybe restarting the game before playing a huge map and restarting afterwards could alleviate that.

[Simon]

Yeah, this is the nastiest problem in the game.

Solution 1 (limit size) is hackish, but a reasonable workaround because everything else is hard. I have to guess how widespread graphics cards behave with what sizes of image. It's hard and inconsistent. (Forestidia posted this same link.)

Then enforce a max size that doesn't crash anybody's machine. Map size limit is purely client-side; the server relays levels, but doesn't interpret the level file.

game seems to have a memory leak and crashes even if still free RAM

Game is largely garbage-collected (0.9.10 and newer run the GC more often than before), and some parts are reference-counted (these don't seem to leak). The leak comes mostly from the graphics card that swaps to RAM, but doesn't free this RAM before end of application. I call the graphics hardware via Allegro 5 and don't see a way to tell the card to free such swapped RAM.

What maps did crash so far? Infinitus 8p crashed, and Arty's clock map at maximum size crashed.


Solution 2 (efficient storage) is the best, but needs the deepest reworking. The Clones solution split the map into 128x128 chunks, and save only the chunks that change over time.

-- Simon

[Simon]

Here are the plans for 0.9.12.

Warn on large maps: On all previews of maps with more than 3 million pixels, I'll print a warning that these maps can crash Lix.

Still allow everybody to play these large maps, nonetheless. You can ignore the warning and press ready for multiplayer, play the map in singleplayer, or load it in the editor. This way, 0.9.12 won't be a downgrade, and it leaves player in control.

Example of large maps where the warning will show:

• IchoTolot's 3-player What Could Possibly Go Wrong at size 1,792 x 1,824 = 3.3 million pixels
  
• Rubix's 8-player Infinitus at size 2,560 x 1,280 = 3.3 million pixels
  
• Arty's 8-player Corridors of Time at size 3,200 x 2,000 = 6.4 million pixels (the maximum allowed)
  

These 3 maps have crashed in the past. If you experience crashes on any other maps, please tell me, and I'll revise in-game warning. These crashes will generate a message box that says either "OutOfMemory: Memory allocation failed" or "OutOfVramException".

I've merely guessed a map size (3 million pixels) and hope that it works as a catch-all. Ideally, we'll find a better solution than such a warning in the long run.

-- Simon

[Forestidia86]

The problem is probably related to the weak 32bit garbage collection (cf. here).
The current Win Lix release is 32bit, whereas on Linux it is commonly 64bit since most have 64bit systems.
That would explain why people on Win crash but the ones on Linux not.

A 64bit build should at least lessen RAM usuage, which could maybe solve this problem.
The problem for a 64bit build was in earlier time that Visual Studio was needed for linking (cf. here).

But now the compiler LDC has released a beta that is self-sufficient, that doesn't need Visual Studio. Only the Visual C++ 2015 runtime to run the game could be needed still, but there are quite a few games that install that with it, so it is likely that people have it already.

So there are good chances that in future big maps can be played again.

At the moment there are still quite a few problems open, though.