Big module sizes

I measured the size of some big modules, with B::Stats and B::C, statically compiled.
This amounts to about to the same size as dynamic usage via perl, with all options and modules used at once. I.e. the worst-case scenario.

Interestingly Module::Build is much harder to compile - you'd need 6GB RAM at least -
and much bigger in the end-result than Moose, which compiles/compresses really fine.

Modules

Module::Build

perl -e'use Module::Build: print q(k)'

ops68K
files48
src lines18K
compiled c lines1.5M
perl compiler memory360M
gcc compiler memory3.4G
executable size37M

Moose

ops91K
files109
src lines28K
compiled c lines889K
perl compiler memory218M
gcc compiler memory1.6G
executable size23M

DateTime::TimeZone

ops2.3K
files6
src lines2.2K
compiled c lines130K
perl compiler memory19M
gcc compiler memory415M
executable size32M

Typical Apps

Biggest cPanel app

The recipe for less than 2MB RAM and gcc < 4.5 is:
cc_harness -E -o$@-cpp.c $@.c
cc_harness -O0 -fno-var-tracking -fno-stack-protector \
-fno-tree-loop-optimize \
-fpreprocessed -fno-exceptions $@-cpp.c

ops467K
files539
src lines95K
compiled c lines1.5M
perl compiler memory352M
gcc compiler memory (32bit -O0)2.0G
executable size61M

Typical cPanel app

ops120K
files156
src lines28K
compiled c lines1M
perl compiler memory120M
gcc compiler memory (32bit -O0)734M
executable size42M

dzil --help

ops167K
files214
src lines48K
compiled c lines1.4M
perl compiler memory485M
gcc compiler memory4.5G
executable size46M
B::C time182s
cc time348s

Minimal Web apps

A simple two-state form like this without Android.

Plack

ops78K
files98
src lines29K
compiled c lines388K
perl compiler memory95M
gcc compiler memory1.0G
executable size11M
B::C time11s
cc time72s

Dancer

ops123K
files143
src lines40K
compiled c lines517K
perl compiler memory159M
gcc compiler memory1.1G
executable size14M
B::C time20s
cc time97s

Mojo (daemon)

ops151K
files150
src lines46K
compiled c lines622K
perl compiler memory153M
gcc compiler memory1.4G
executable size17M
B::C time29s
cc time114s

Catalyst

catalyst.pl Catalyst::Test; script/catalyst_test_create.pl. Same numbers with the server

ops199K
files214
src lines57K
compiled c lines941K
perl compiler memory267M
gcc compiler memory1.7G
executable size26M
B::C time44s
cc time169s

Minimal OO frameworks

Moose

ops91K
files109
src lines28K
compiled c lines889K
perl compiler memory218M
gcc compiler memory1.6G
executable size23M

Mouse

ops20K
files23
src lines6.5K
compiled c lines120K
perl compiler memory19M
gcc compiler memory427M
executable size3.3M

Moo

ops6.4K
files16
src lines3.7K
compiled c lines96K
perl compiler memory17M
gcc compiler memoryerror (326M)
executable sizeerror (2.7M)

Mo

ops3.7K
files8
src lines2.6K
compiled c lines67K
perl compiler memory- (too tiny, not measurable)
gcc compiler memory346M
executable size1.8M

Summary

If you compile an app like Module::Build, i.e dzil (which means in the convoluted java-like style like Module::Build) you need a really big build machine, and developers will not be able to compile it by themselves, because they have not enough RAM. Errors will only appear daily (from the build machine), not immediately.

Moose alone is okay to use, dzil for sure not.
A minimal web app with Plack, Dancer, Mojo and even Catalyst look all okay.
SW metrics to explain the style, why the numbers are so different in the next blog post.

So if you plan to do "modern perl" you'll either take care, like using Mouse or Moo, or do something like prefork or FCGI and do not care.

Measured numbers are for x86_64 perl5.14.2-nt, gcc -Os -msse4.2 -march=corei7

*gcccexe
subs*mem[M]linessize
Dist::Zilla 4309* 45001320K46M
Module::Build 4017* 34001500K37M
CPAN 4142*33001319K33M
Catalyst 4508* 1700941K26M
ExtUtils::CBuilder 1633*1552 595K15M
Moose 2523*1600889K23M
Mojo 2559* 1400622K17M
Dancer 2368* 1100517K14M
Plack 1945* 1000 388K11M
ExtUtils::MakeMaker1233*957 335K8.4M
DateTime 1277* 840 293K6.8M
DateTime::TimeZone542* 415 130K3.1M
DateTime::Locale471* 464138K3.4M
Mouse 743* 427120K3.3M
Moo 572* 32696K2.7M
Mo 388* 34667K1.8M

Makefile tricks

If you want to check in your Makefile how much memory is available and
disable some gcc optimizations accordingly you can something like this:

Makefile snippet

AVAILMEM=$(shell availmem.sh)
bigapp: $@.pl
        if [ ${AVAILMEM} -lt 2000000 ]; then \
                echo "less than 2GB free RAM, disable some gcc optimizations"; \
                /bin/rm -f $@ $@-cpp.c $@.c; \
                perl -c $@.pl; \
                perl -MO=C,-O3,-o$@.c $@.pl; \
                cc_harness -E -o$@-cpp.c $@.c; \
                cc_harness -O0 -fno-var-tracking -fno-stack-protector \
                         -fno-tree-loop-optimize -fpreprocessed \
                         -fno-exceptions $@-cpp.c -o$@; \
                /bin/rm -f $@-cpp.c $@.c; \
        else \
                perlcc $(PERLCC_OPTS) $@.pl -o $@; \
        fi

My availmem.sh on linux:

#!/bin/sh
memtotal=$(perl -ane'print $F[1] if /^MemTotal:/' /proc/meminfo)
memused=$(perl -ane'print $F[1] if /^Active:/' /proc/meminfo)
echo $(($memtotal - $memused))

12 Comments

This is a surprising result, Module::Build is not that large of a codebase, why should the compiled version be so large? Do you have any ideas?

Can you clarify what you are analyzing? Are you saying to to use something like Module::Build you are needing 3.5gig of memory? And that using Moose requires 1.6gig of memory?

Fascinating stuff. Please give awstats a try if you have some time. :)

Hmmm. You realize 8GiB (dual-channel) is

Also, for debug builds, try -O0, that'll make gcc use a lot less memory.

(Trying again, it takes HTML tags. Oops. And no edit… this time I'm using preview1)

Hmmm. You realize 8GiB (dual-channel) is <$50, and 12 GiB (tri-channel) is <$70?

Also, for debug builds, try -O0, that'll make gcc use a lot less memory.

@anthony: Even though RAM is cheap, finding a nice laptop/netbook which supports 16-32GB or larger (and preferably an SSD drive) is still a challenge.

*Extremely* surprising. I have no idea why M::B would be so huge under this metric. Can you tell whether it's M::B itself, or its dependencies?

Leave a comment

About Reini Urban

user-pic Working at cPanel on B::C (the perl-compiler), p2, types, parrot, B::Generate, cygwin perl and more guts (LLVM, jit, optimizations).