2.4.0.0 - Crash

[thexa4]

On windows I've had 5 crashes that I wasn't able to reproduce afterwards. They happened over multiple saves, most of the time they trigger when skipping a lot of months.

My assumption was that it was related to saving but it also happened on the map mode. The map texture seemed to not have been loaded before the crash, it was all black:
   

I've tried to capture a minidump but that didn't succeed. This is a screenshot of the stack trace and exception:
   

The memory usage graph over time (top right) does look weird, my machine has 64gb memory installed and it crashes exactly once the game uses that amount of memory.

[thexa4]

It looks related to the turn computation, I reliably get 400mb extra memory use after every turn, also when asking the game to skip multiple turns.

[thexa4]

I just got the crash again while the process only used 13gb so the crash might not be related to the memory usage.

[thexa4]

I've managed to reproduce a crash twice by leaving the map open on the Build a New Factory window and switching focus to a different application.

Attached is the minidump with only the stack trace. The dump with the heap is 200mb while compressed and won't fit here.

[Eric.B]

Are you using OpenGL or DirectX?

[thexa4]

The renderer was set to DirectX

[thexa4]

When the renderer is set to OpenGL I cannot reproduce the crash on the map

[Eric.B]

Sounds like known issues with the DX9 renderer caused by our upgrades to the libraries. Ogre, who handles our rendering engines removed a lot of hack fixes to the DX9 system to improve DX11 performance, since no one should being using DX9 at this point. It's not getting very much complex use or testing upstream.

I probably should set GC:2G to default to OpenGL2 until we get render API updates.

[thexa4]

The memory leak / increase per turn is also present when using the OpenGL2 renderer.

[thexa4]

I just experienced a crash with another access violation when alt+tabbing when on the R&D Research Budget screen. This is with the OpenGL2 renderer.

[Eric.B]

The memory leak / increase per turn is also present when using the OpenGL2 renderer.

I'll check it out with Valgrind when I am able to, but the ram usage should increase every turn because of the increased size of the database every turn. The save games are stored in ram.

I frequently play on low ram systems, and haven't ran into any issues, so we'll see.

[Eric.B]

Just ran 5 years of your latest uploaded save game and did not go above 3.59GB of total system ram on this machine. It did not display any symptoms of memory leaks. Granted, this is not a Valgrind test and I am not using Windows. I will check both when I can get on a workstation.

With respect to these memory leaks and crashing, be sure that you have the latest video drivers from your gpu chipset manufacturer (AMD, Intel, nVidia), especially if you're using Windows 11. OpenGL is heavily driver dependent on Windows (since it's implemented solely by drivers) and should be pretty rock solid if the drivers are good. Likewise, 2.4 shouldn't have any more memory leak issues than 2.3 had, as I can't think of any new code in this version that would be significantly at risk of memory issues.

[thexa4]

I have updated the video drivers to the latest version and am running on Windows 10 using the OpenGL2 renderer in windowed mode.
This time I used the Task Manager to look at the memory usage and it seems like the Dedicated GPU memory usage is steadily climbing every tick. Normal memory usage stays constant.

The same test in windowed mode using DirectX9 does not result in high memory use.

[Eric.B]

I have updated the video drivers to the latest version and am running on Windows 10 using the OpenGL2 renderer in windowed mode.
This time I used the Task Manager to look at the memory usage and it seems like the Dedicated GPU memory usage is steadily climbing every tick. Normal memory usage stays constant.

The same test in windowed mode using DirectX9 does not result in high memory use.

That's really odd because the only assets you should be loading on end turn is the office backgrounds.

Are you sure it's actually crashing because of this? If so, please provide the log file for me. That will tell me what resources are being loaded and it should give me a reason for the crash if it's related to the GPU.

I will try to duplicate it when I can use Windows tomorrow, but I am pretty sure I don't have this issue. And there really isn't much I can do about it on my end except manually call to clear Ogre's cached resources, except that will slow down everyone's else machines who don't have this issue.

[thexa4]

I have tried to reproduce the issues on a different machine:

Windows 11, up to date drivers and DirectX renderer
 - Map in background doesn't crash
 - Don't see GPU memory increase per turn

Windows 11, up to date drivers and OpenGL2 renderer
 - Map in background doesn't crash
 - There is an increase in GPU memory every turn. It seems more subtle on this machine, maybe because it's not running in 4k resolution here.

I am not able to wait long enough to make it crash today but I will try to upload the logs an out of memory tomorrow.

[thexa4]

I had to get it to use 180gb memory before it crashed but I got a log. Should probably disable the swapfile if I want to do this again.

The game crashed while automatically skipping turns. In the end when it had trouble allocating memory it showed the office scene between each turn. It crashed while showing the office scene.

[Eric.B]

I see a database error which could cause a memory leak. But I am not seeing anything related to GPU issues that could cause vram leak.

You had 180GB of video ram? Or was this 180GB of system ram/swap before it crashed?

I'm hoping to be able to pop into the office tonight and see what it does on my Windows machine, I'm having no such issues in Linux with systems that are significantly weaker.

[Eric.B]

Also, if you could, bounce over to 2.3 and see if you can duplicate the same issue with OpenGL.

Thanks

[Eric.B]

I can confirm there is a memory leak issue with Windows OpenGL2 in the latest versions of Ogre. Unfortunately, this is an upstream issue and nothing I can address due to lack of tooling in Windows.

Checking the game in Linux, it does have a couple really tiny leaks, most of which are in upstream libraries. The Unix builds suffer from a Vorbis leak when using OGG files (it also crashes when you close the game), and CEF also leaks a block per load page load. Both of these are upstream issues and the former a known issue for over a decade. The AI can also trigger a minor leak, and by minor, I mean about 600kb over 360 turns. I already put in a fix for that. Total leakage, 3mb over 360 turns. But there are no VRAM or major system ram leaks.

Anyway, I have planned for a while to implement FreeImage codecs, which we used for all other versions of the game, in hopes that it improves DX9 stability and fixes the fuzzy issue. Perhaps the issue with OpenGL is related, but it is unlikely. I also planned on making a cut down version of the engine to pass along upstream to have some of my issues duplicated and addressed. But both of these plans were foiled by how long it took to complete 2.4. With the up coming pause to the FBS system after Milestone #5 is funded, I'll likely get both done around then. Since I plan on halting the FBS for a good 6 months or so.

[Eric.B]

Note I haven't found the crash issue yet. I am in the process of making a cut down version of the game engine so I can open source it and share it with upstream libraries in the hopes they can find the issue. When that's done, I'll play version wack-a-mole to find out what changes they made that caused the OpenGL leak and see if I can fix it.

That said, in your log file there was a bug with the infinite money setting. I fixed that, and changes will be in the next update.