Never been one to let the carrier drop

So, I’ve been maintaining several very large projects in perl – I can enumerate them at a later date – and one of them in perl and XS. Interestingly, the one in perl and XS is the first to have significant memory leaks – but they’re not all from XS!

I’ve identified three types of memory leaks

#1) Scalers leaving XS without being made mortal and given a reasonable refcount. A useful tool for looking for these is Devel::Leak
#2) Circular references that are not weakened and therefore can never be reaped. A useful tool for looking for these is Devel::Cycle
#3) Scalers gaining a refcount that is not being cleared. A useful tool for looking for these is also Devel::Leak

I now need to go through all the major subsystems and make sure that they aren’t leaking one of these ways. If I find new ways to leak I will add them to this document. Comment below if you’d like a more comprehensive document on how to find these things.

So, one of the big problems I’m having with kt3 is memory leaks. This is probably no surprise to anyone who has done XS programming before.

I found one particularly pernicious one by using Devel::Leak.. this is in the log handler.

Originally I had this, which I’m fairly sure I got from the internet somewhere:


static void _kt_log_callback(int iLogLevel, char *log_subsystem, char *msg)
{
dTHX;
dSP;

call_pv("KittenTrader::KittenBrain::Log", G_DISCARD);
}


This worked well enough – it created a hash ref and passed it to the function – but the hash ref kept leaking.

Eventually – after much digging through the documentation – I figured out that what I needed to do on the XPUSHs was this:


XPUSHs(sv_2mortal(newRV((SV *)rv)));


If anyone is curious how I debugged.. I used this relatively simple function to invoke the callback:


my $count = Devel::Leak::NoteSV($handle);
for($i=0;$i<1000;$i++) { KittenTrader::KittenBrain::testLog(4, "THIS IS A MESSAGE"); } my $count2 = Devel::Leak::NoteSV($handle); Devel::Leak::CheckSV($handle); print "Count: $count\n"; print "Count2: $count2\n";

This gave me a count of SVs, which I could clearly see my leaking SVs in.

Then I added returns everywhere along the path until I could isolate the leaking SV to a few lines of code. In particular I could definitely see that as soon as I created the hash, I was leaking a SV, but not if I destroyed it before passing it in. I realized the problem was that the callee didn't realize that the SV was mortal - I had made the hash mortal but not the *reference* to the hash. ANd of course the hash could never be destroyed until the reference to it was.

As you can see, the genetic algorithm experiments are coming to a head. I have absurd amounts of computing power in my garage, and I’m starting to think about the data center.