Never been one to let the carrier drop

So, I’ve been maintaining several very large projects in perl – I can enumerate them at a later date – and one of them in perl and XS. Interestingly, the one in perl and XS is the first to have significant memory leaks – but they’re not all from XS!

I’ve identified three types of memory leaks

#1) Scalers leaving XS without being made mortal and given a reasonable refcount. A useful tool for looking for these is Devel::Leak
#2) Circular references that are not weakened and therefore can never be reaped. A useful tool for looking for these is Devel::Cycle
#3) Scalers gaining a refcount that is not being cleared. A useful tool for looking for these is also Devel::Leak

I now need to go through all the major subsystems and make sure that they aren’t leaking one of these ways. If I find new ways to leak I will add them to this document. Comment below if you’d like a more comprehensive document on how to find these things.

So, one of the big problems I’m having with kt3 is memory leaks. This is probably no surprise to anyone who has done XS programming before.

I found one particularly pernicious one by using Devel::Leak.. this is in the log handler.

Originally I had this, which I’m fairly sure I got from the internet somewhere:


static void _kt_log_callback(int iLogLevel, char *log_subsystem, char *msg)
{
dTHX;
dSP;

call_pv("KittenTrader::KittenBrain::Log", G_DISCARD);
}


This worked well enough – it created a hash ref and passed it to the function – but the hash ref kept leaking.

Eventually – after much digging through the documentation – I figured out that what I needed to do on the XPUSHs was this:


XPUSHs(sv_2mortal(newRV((SV *)rv)));


If anyone is curious how I debugged.. I used this relatively simple function to invoke the callback:


my $count = Devel::Leak::NoteSV($handle);
for($i=0;$i<1000;$i++) { KittenTrader::KittenBrain::testLog(4, "THIS IS A MESSAGE"); } my $count2 = Devel::Leak::NoteSV($handle); Devel::Leak::CheckSV($handle); print "Count: $count\n"; print "Count2: $count2\n";

This gave me a count of SVs, which I could clearly see my leaking SVs in.

Then I added returns everywhere along the path until I could isolate the leaking SV to a few lines of code. In particular I could definitely see that as soon as I created the hash, I was leaking a SV, but not if I destroyed it before passing it in. I realized the problem was that the callee didn't realize that the SV was mortal - I had made the hash mortal but not the *reference* to the hash. ANd of course the hash could never be destroyed until the reference to it was.

This is a perl script that can be used to pull all the files from a DA-6400 and automatically divide them into folders based on their timestamps

#!/usr/bin/perl

$|=1;

$targetbase = “~/DownloadLocation”;

$host = “tascam”;
use Net::FTP;

use Time::Local qw ( timelocal );
use Data::Dumper;
print “Creating FTP object\n”;
$ftp = Net::FTP->new($host, Debug => 0) || die “Can’t connect to tascam”;

print “Logging in\n”;
$ftp->login(“DA-6400″,”DA-6400”) || die “Can’t login ” , $ftp->message;

print “CWD\n”;

$ftp->cwd(“/ssd/DA Files”) || die “Cannot CWD: ” . $ftp->message;

print “BIN\n”;

$ftp->binary() || die “Cannot set to bin mode: ” . $ftp->message;
my $list = $ftp->ls();
my $maxtime = 0;

my $timestamps = {};
# determine newest date

foreach $file (@{$list}) {

        next if($file eq “.”);

        next if($file eq “..”);

        next if(!($file =~ /.*.wav/));
        my $unixtime = getFileTimestamp($file);

        my $ts = getTimestamp($unixtime);
        print “file: [$file] ts: $ts\n”;

        $maxtime = $unixtime if($unixtime > $maxtime);

        $timestamps->{$unixtime} = 1;

}

foreach $ts (keys %{$timestamps}) {
        $targetstub = getTargetDir($ts);

        $targetdir = $targetbase . ‘/’ . $targetstub;

        if(! -d $targetdir) {

                mkdir $targetdir;

        } else {

                next;

        }
        $targetdir .= “/ftp”;

        if(! -d $targetdir) {

                mkdir $targetdir;

        }
        chdir $targetdir;

        print “$ts – Writing to $targetdir\n”;
        foreach $file (@{$list}) {

                next if($file eq “.”);

                next if($file eq “..”);

                next if(!($file =~ /.*.wav/));
                my $unixtime = getFileTimestamp($file);

                if($unixtime == $ts) {

                        print “Fetching $file ..”;

                        $ftp->get($file);

                        print (-s $file);

                        print “\n”;

                }

        }

}
sub getTargetDir

{

        my $time = shift;

        my ($sec,$min,$hour,$mday,$mon,$year,$wday,$yday,$isdst) = localtime($time);

        return sprintf(“%02d%02d%04d”,$mon+1,$mday,$year+1900);

}
sub getTimestamp

{

        my $time = shift || time();
        my ($sec,$min,$hour,$mday,$mon,$year,$wday,$yday,$isdst) = localtime($time);

        return sprintf(“%02d/%02d/%04d-%02d:%02d:%02d”,$mon+1,$mday,$year+1900,$hour,$min,$sec);

}
sub getFileTimestamp

{

        my $file = shift;

        my ($prefix, $datetime, $take, $channel, $name) = split(/_/, $file);
        print “Datetime: [$datetime]\n” if($main::debug);
        my ($date, $time) = split(/-/, $datetime);

        print “Date: [$date]\n” if($main::debug);
        my ($yy, $MM, $dd) = $date =~ /(\d{4})(\d{2})(\d{2})/;

        my ($hh, $mm, $ss) = $time =~ /(\d{2})(\d{2})(\d{2})/;
        print “yy: [$yy] mm: $mm dd: $dd\n” if($main::debug);
        $MM -= 1;

        $yy -= 1900;
        my $unixtime = timelocal($ss, $mm, $hh, $dd, $MM, $yy);
        return $unixtime;

}

So, I found a major lack of documentation for Device::Hue, and chatGPT had some information that I would describe as whimsically wrong.

Here’s what I ended up doing to get a working perl setup controlling my hue lights:

1) Put a valid URL in /etc/environment for the key HUE_BRIDGE
2) Put a key in /etc/environment for the key HUE_KEY
3) The following code sets lights 17 and 19 to red:

my $bridge = Device::Hue->new();

$bridge->config();

foreach $light_id (17, 19) {

   my $light = $bridge->light($light_id);

   $light->set_state( { hue => 0, sat => 254 });

   $light->commit();

}

As you can see, the genetic algorithm experiments are coming to a head. I have absurd amounts of computing power in my garage, and I’m starting to think about the data center.

Just a heads up that hopefully google will index and save someone else from the several hours of digging I’ve just had to do. the perl module IPC::Shareable sets SIG{CHLD} to ignore any time you include it. You can set it back to the default behavior, but until you do, waitpid() is going to return -1 and $? will contain -1 for any pid-waiting you do.

1) If you haven’t tested it, it doesn’t work.
2) If what it’s doing appears impossible, it’s probably two or more bugs working together
3) Intermittent faults are sensitive to Murphy’s law, and will be triggered by the time when the most damage will be done
4) As a rational reaction to #3, always have a backup plan.
5) Focus your optimization efforts on the parts of the code that run the most
6) Add at least two hours to your time estimate for testing

I had a couple of notes on using Net::Bluetooth from a raspberry pi to talk to a Bluetooth serial port because I couldn’t get the example code given with Net::Bluetooth to work.

The following will connect to a mac address and send and receive data from it (in this particular example, a ESP32):

#!/usr/bin/perl

use Net::Bluetooth;

use Data::Dumper;

use IO::Handle;

my $obj = Net::Bluetooth->newsocket(“RFCOMM”);
$addr = ‘C4:4F:33:58:B6:FB’;

$port = 1;
if($obj->connect($addr, $port) != 0) {

     die “connect error: $!\n”;

}
my $fh = $obj->perlfh();

$fh->autoflush(1);
sleep(1);
print “sending \n”;
print $fh “V\n”;

print “receiving\n”;
$buf = readline($fh);

print “Fetchhost: $buf\n”;

Just sticking this here in the hopes that it will get listed in google and maybe save some folks a little time.

If you are trying to use rabbitMQ’s websocket STOMP connector, as discussed on https://www.rabbitmq.com/stomp.html, be aware that the URL given in the documentation on this page is wrong (or perhaps only applies to newer versions of rabbitmq).

The correct way to successfully create a websocket to RabbitMQ 3.5.7 (which ships with Ubuntu 16.04) is as follows:

var ws = new WebSocket(‘ws://yourhostname:15674/stomp/websocket’);

var client = Stomp.over(ws);

Later note: The documentation is correct for newer versions of RabbitMQ.

So, I’ve become sufficiently cynical to suspect that the actual running of the pipeline recently shut down by ransomware was done by industrial controllers, and the operators shut it down because they’re very conservative and they hoped they could make the Biden administration et al look bad. IT’s difficult to know, although I do hope this will push massive EV adoption. It would be nice if in their attempt to hold a gun to our head Big Oil committed suicide instead.

What I do know is that the hack occured becuase Microsoft abandons old windows versions. Now, this should be illegal – similar to how car manufacturers are required to provide repair parts for any mass produced automobile, Microsoft should be forced to release the source code for any operating system they abandon so volunteers can continue to maintain it. Linux proves that open-source software is more stable and secure than closed-source anyway.

Now, many people would say “No, you should be forced to regularly “upgrade” (in the case of win 10, downgrade) so Microsoft can continue making a profit. I don’t understand why these people want to live in a dystopia. Being forced to adapt to a change when you’re busy doing something else is clearly dystopian, and there’s also almost no chance that all the makers of (perfectly functional, useful, sometimes very complex and expensive) computer-interfaced hardware are going to manage to get drivers out for each new version Microsoft chooses to create.

Someone was trying to convince me we should nationalize Facebook because they’re upset Trump got kicked off it. No, that’s a horrible idea – my final argument that I would describe as a crushing blow was to ask if we should nationalize all communication infrastructure – should Fox News be nationalized? How is this different from Ganz Deutschland hört den Furher im Volksempfanger? No, I don’t think so. But we probably do need to have greater government oversight of Microsoft given the position operating systems play in our world – and as I said, they should be *forced* – if they want to stay in business in the USA – to release the source code of anything they abandon so it can continue to be maintained.

Let’s *try* to aim for *less* of a dystopia rather than more?

Side note – being forced to ride the upgrade train may be another example of how capitalism encourages us to make really bad decisions. It raises GDP, and our government seems to think a perpetually climbing GDP is a good thing, but it reduces effectiveness and wastes man-hours – it wastes real world resources to make paper ones. Sounds like another great example of the tool using us instead of us using the tool!