discussion Ren'Py + VM = Memory Leaks?

[LewisCarroll]

Currently in the beginning stages of developing a VN, but am running into an issue I cannot find an answer to. I sandbox all my games in a Windows 10 VM and once I do Ren'Py begins to chew through memory at about 1GB every ten seconds until it maxes out (I've increased RAM to 32GB). In a non-VM Windows 11 machine most games only use ~1GB max. Games with more DAZ animations seem to crash/use up memory faster while 2D cartoonish games usually are OK. Is there a way to manage this in my own games? Does anyone know why this is happening?

 

[SegmentCoalescing]

What kind of VM is this? I know that Windows is incredible buggy with RAM ballooning in KVM so if you are using that, disable it.
It had the same behavior where it acted like a memory leak and just used more and more RAM without releasing any.

 

[LewisCarroll]

I'm using VMware Workstaion Player, latest version. When VMWare updated to 17.6 all kinds of issues started popping up. VM was freezing and graphics were 'boxy' until I added mks.enableDX12Presentation=FALSE then everything smoothed out. I'm more of a networkperson, so KVM to me are the controllers for multiple servers in a rack lol. But, it makes sense. VMWare runs ontop of the current OS so you think it has something to do with hypervisor memory ballooning? How would one disable it on VMWare VM's?

 

[SegmentCoalescing]

I'm using VMware Workstaion Player, latest version. When VMWare updated to 17.6 all kinds of issues started popping up. VM was freezing and graphics were 'boxy' until I added mks.enableDX12Presentation=FALSE then everything smoothed out. I'm more of a networkperson, so KVM to me are the controllers for multiple servers in a rack lol. But, it makes sense. VMWare runs ontop of the current OS so you think it has something to do with hypervisor memory ballooning? How would one disable it on VMWare VM's?

I haven't really used VMware that much so I'm of limited help when it comes to that, sorry.
I looked it up a bit and it seems that it should have the necessary drivers installed by default through VMware tools (Unless you somehow don't have them?) but in the case I mentioned above, even the virtio (KVM paravirtualization) drivers did not solve the issue with Windows specifically. So try to find a way to disable it, it's worth a try.

Have you looked at the task manager to see what specifically is using the RAM? In the guest of course. Anything besides Renpy? Does it go away when you close the VN?

I'm currently installing a Windows 10 VM, I'll take a look and see if I notice anything similar.

 

[Vulky]

You should try Oracle Virtualbox, it's free and I never had such problems (I usually do Win 10 and 11 host, Linux VMs).

 

[LewisCarroll]

I haven't really used VMware that much so I'm of limited help when it comes to that, sorry.
I looked it up a bit and it seems that it should have the necessary drivers installed by default through VMware tools (Unless you somehow don't have them?) but in the case I mentioned above, even the virtio (KVM paravirtualization) drivers did not solve the issue with Windows specifically. So try to find a way to disable it, it's worth a try.

Have you looked at the task manager to see what specifically is using the RAM? In the guest of course. Anything besides Renpy? Does it go away when you close the VN?

I'm currently installing a Windows 10 VM, I'll take a look and see if I notice anything similar.

I watched the task manager and I can watch Ren'Py eating away. Literally ticks up 1GB...2GB...3GB...5GB....7GB....9GB... until it crashes (no error code) and the memory is freed up and the VM becomes stable again. Games with a lot of 3D animations crash super fast while Ren'Py games with 2D with limited animations don't crash, but they still use 2-3GB of memory. I just installed a fresh Windows 10 VM with nothing on it and same thing happened. I'd be really interested to see what happens to your install. I have 3D acceleration activated.

 

[LewisCarroll]

You should try Oracle Virtualbox, it's free and I never had such problems (I usually do Win 10 and 11 host, Linux VMs).

I used to use it years ago, but it just wasn't as smooth as VMWare Fusion (back then I was using Mac). Player is free and very snappy. Maybe I'll give it another try and see if it has improved.

 

[SegmentCoalescing]

I used to use it years ago, but it just wasn't as smooth as VMWare Fusion (back then I was using Mac). Player is free and very snappy. Maybe I'll give it another try and see if it has improved.

VMware Workstation Pro is actually free now too, ever since Broadcom bought VMware.
Anyway, I did install Windows 10 in a VM using VMware Workstation Pro and it was pretty laggy because it was hammering the CPU by just starting the test game which is Desert Stalker. It did not have the same issue as you with the RAM though.
I did however see the RAM usage slowly go up, not by gigabytes though.

However, installing the VMware tools/drivers package did fix that particular CPU issue.
I even tried playing a very compressed (96KB!) 3D FPS game named .kkrieger to make sure that 3D acceleration worked.. and it worked fine.

The test VM was speced like this:

Windows 10 Pro x64 UEFI/Secure Boot
4 cores/VCPUs
16GB RAM

 

[LewisCarroll]

VMware Workstation Pro is actually free now too, ever since Broadcom bought VMware.
Anyway, I did install Windows 10 in a VM using VMware Workstation Pro and it was pretty laggy because it was hammering the CPU by just starting the test game which is Desert Stalker. It did not have the same issue as you with the RAM though.
I did however see the RAM usage slowly go up, not by gigabytes though.

However, installing the VMware tools/drivers package did fix that particular CPU issue.
I even tried playing a very compressed (96KB!) 3D FPS game named .kkrieger to make sure that 3D acceleration worked.. and it worked fine.

The test VM was speced like this:

Windows 10 Pro x64 UEFI/Secure Boot
4 cores/VCPUs
16GB RAM

I'll DL the Pro version but I have to get on a plane in a few minutes. Could you try Ina's Christmas [2024.12] [Cinnamonbread].? I spoke to to the creator and he said he'll look into it, but haven't heard back yet. Just tried PGI-257 and it ran smooth with no memory issues. Sitter crashed on hitting Start. SUCH A SHARP PAIN [V0.11R] [KETZAL] & Libertine-0.69 both ran smooth as butter (and that games is pretty big). I don't have Desert Stalker though. Malignant hovers around 8GB memory usage, but seems to stop there and play fine.

Event viewer says librenpython.dll was the module that crashes Inas-Xmas, but that's likely just because memory runs out for it to use.

My install is the same as yours. My base machine is an Intel i9-17900 (bios is current with fixes) with 64GB and dual 4070Ti Super's. I don't see any real CPU hits. I can play Unity and Unreal games without much issue. Maybe yours was updating in the background?

Anyway, thanks for the help with this. It's really bugging me.

 

[SegmentCoalescing]

I know why it's happening now, kinda. It is not due to ballooning.
Ina's Christmas does clearly show the issue and it does have more animations/videos than most other VNs.

It's basically due to an issue with 3D acceleration, if you disable that (mks.enable3d = "FALSE") you get rid of the memory leak but you get laggy playback instead. Finding a fix for this will most likely need a lot of testing with the VM settings.

Minimizing the game seems to release the memory, or at least it did once.

FOUND A FIX! At least for Ren'Py games I guess..

Press SHIFT+G in game
Select Force ANGLE2 Renderer
Restart the game.

 

[LewisCarroll]

I know why it's happening now, kinda. It is not due to ballooning.
Ina's Christmas does clearly show the issue and it does have more animations/videos than most other VNs.

It's basically due to an issue with 3D acceleration, if you disable that (mks.enable3d = "FALSE") you get rid of the memory leak but you get laggy playback instead. Finding a fix for this will most likely need a lot of testing with the VM settings.

Minimizing the game seems to release the memory, or at least it did once.

FOUND A FIX! At least for Ren'Py games I guess..

Press SHIFT+G in game
Select Force ANGLE2 Renderer
Restart the game.

That did it! No more frustration! I'm learning Daz3D and Twine (for scripting and flow charting scripts) and haven't gotten to Ren'Py programing yet. Didn't even know that option was available so I learned two things! The CPU is being hammered, but it's running pretty smooth now and memory isn't going over 500Mb now. Thank you very much @SegmentCoalescing, you're a god send.