The unexpected case of -Mblib
I'm in constant worry of unnecessary bloat because our perls have to run fast. "Bloat" means added dependencies, loading new files at startup, needing more time.
Generally little benefit for more cost.
With the perl compiler I can analyze code and dependencies at compile-time and strip unneeded packages.
My recent concerns have been:
1. Carp being used in the DynaLoader AUTOLOAD fallback, when a XS is not loaded give a full stacktrace.
2. Carp checking for B being loaded and checking CALLER_OVERRIDE_CHECK_OK. This outsmarts the compilers which pulls in B at compile-time, because that anything which *can* be done at run-time has to be included in a compiled executable.
3. warnings requiring now warnings::register which are huge hashes, seldom used.
4. utf8 now requiring the huge ToFold tables when doing lc, uc or m//i. There is no simple only_ascii check for the string in question beforehand. And besides utf8_heavy really should load their tables not into big perl hashes, it should pre-compile them like Encode does and load the .so's on demand. A pure functional interface to the unicode tables.
So I improved B::Stash which tells all me all the compile-time dependencies. I got rid of B, which is huge, but I still have to use DynaLoader to load the XS part, which in turn might include Carp, which might include B and so on.
I did it to check my own requirements, the requirements of B::C.
The results are surprising.
B::C stash footprint: mainly caused by blib, warnings, and Carp loaded with DynaLoader
1.
perl5.15.7d-nt -MO=C,-o/dev/null -MO=Stash -e0
-umain,-ure,-umro,-ustrict,-uAnyDBM_File,-uFcntl,-uRegexp,-uoverload,-uErrno,-uExporter,-uExporter::Heavy,-uConfig,-uwarnings,-uwarnings::register,-uDB,-unext,-umaybe,-umaybe::next,-uFileHandle,-ufields,-uvars,-uAutoLoader,-uCarp,-uSymbol,-uPerlIO,-uPerlIO::scalar,-uSelectSaver,-uExtUtils,-uExtUtils::Constant,-uExtUtils::Constant::ProxySubs,-uthreads,-ubase
2.
perl5.15.7d-nt -MErrno -MO=Stash -e0
-umain,-ure,-umro,-ustrict,-uRegexp,-uoverload,-uErrno,-uExporter,-uExporter::Heavy,-uwarnings,-uwarnings::register,-uConfig,-uDB,-uvars,-uCarp,-uPerlIO,-uthreads
3.
perl5.15.7d-nt -Mblib -MO=Stash -e0
-umain,-ure,-umro,-ustrict,-uCwd,-uRegexp,-uoverload,-uFile,-uFile::Spec,-uFile::Spec::Unix,-uDos,-uExporter,-uExporter::Heavy,-uConfig,-uwarnings,-uwarnings::register,-uDB,-uEPOC,-ublib,-uScalar,-uScalar::Util,-uvars,-uCarp,-uVMS,-uVMS::Filespec,-uVMS::Feature,-uWin32,-uPerlIO,-uthreads
4.
perl5.15.7d-nt -MO=Stash -e0
-umain,-uTie,-uTie::Hash,-ure,-umro,-ustrict,-uRegexp,-uoverload,-uExporter,-uExporter::Heavy,-uwarnings,-uDB,-uCarp,-uPerlIO,-uthreads
5.
perl5.15.7d-nt -MB::Stash -e0
-umain,-ure,-umro,-uRegexp,-uPerlIO,-uExporter,-uDB
Okay, so pure perl has:
main re mro Regexp PerlIO Exporter DB
O added:
Tie Tie::Hash strict overload Exporter::Heavy warnings Carp threads
blib added:
Cwd File File::Spec File::Spec::Unix Dos Config warnings::register EPOC blib Scalar Scalar::Util vars VMS VMS::Filespec VMS::Feature Win32
In contrast Errno only adds to O:
Errno warnings::register Config vars
The B::C compiler itself adds to Errno:
AnyDBM_File Fcntl next maybe maybe::next FileHandle fields AutoLoader Symbol PerlIO::scalar SelectSaver ExtUtils ExtUtils::Constant ExtUtils::Constant::ProxySubs base
Big win for O over blib!
At first I changed all my aliases of
pb='perl -Mblib'
to
pb='perl -Iblib/arch -Iblib/lib'
This is what blib does, with checks for VMS and MSWin32. A smaller blib would get rid of all of this footprint because we know the $^O and pathdelim without any module.
You will not like stripping modules but I do care because I need it in order to decide which modules have to be included compiled and which not. It makes a difference to have 3 MB or 15MB executables, to have 0.1ms or 0.01 ms startup-time.
If you don't trust B::Stash or B::Stats, just try this, safe for home:
$ perl -Dt -Mblib -e0 2>&1 | perl -lne'm/^\((.+?):\d+\)\s+(.*)/ && print $1;'|sort -u
-e
/usr/local/lib/perl5/5.15.7/blib.pm
/usr/local/lib/perl5/5.15.7/Exporter.pm
/usr/local/lib/perl5/5.15.7/strict.pm
/usr/local/lib/perl5/5.15.7/vars.pm
/usr/local/lib/perl5/5.15.7/warnings.pm
/usr/local/lib/perl5/5.15.7/warnings/register.pm
/usr/local/lib/perl5/5.15.7/x86_64-linux-thread-multi-debug/Cwd.pm
/usr/local/lib/perl5/5.15.7/x86_64-linux-thread-multi-debug/File/Spec.pm
/usr/local/lib/perl5/5.15.7/x86_64-linux-thread-multi-debug/File/Spec/Unix.pm
/usr/local/lib/perl5/5.15.7/XSLoader.pm
vs:
$ perl -Dt -Iblib/arch -Iblib/lib -e0 2>&1 | perl -lne'm/^\((.+?):\d+\)\s+(.*)/ && print $1;'|sort -u
-e
This way you van also check code size compile-time
(ops before EXECUTING.., or just use perl -Dt -c)
vs run-time (ops after EXECUTING...), which is the same what B::Stats does.
$ perl -c -Dt -Mblib -e0 2>&1 | wc -l
4180
4180 ops footprint for -Mblib compared to 0 with -Iblib/arch -Iblib/lib
The biggest case here is warnings, and then Cwd which requires Dynaloader. That's why cygwin perl 5.14 will contain Cwd as static_ext.
$ perl -Dt -Mblib -e0 2>&1 | perl -ne'm/^\((.+?):\d+\)\s+(.*)/; $f{$1}++;END{for (keys %f){print "$_\t$f{$_}\n";}}'
/usr/local/lib/perl5/5.15.7/x86_64-linux-thread-multi-debug/Cwd.pm 425
/usr/local/lib/perl5/5.15.7/blib.pm 146
/usr/local/lib/perl5/5.15.7/warnings/register.pm 42
/usr/local/lib/perl5/5.15.7/vars.pm 511
/usr/local/lib/perl5/5.15.7/XSLoader.pm 210
/usr/local/lib/perl5/5.15.7/x86_64-linux-thread-multi-debug/File/Spec/Unix.pm 308
/usr/local/lib/perl5/5.15.7/Exporter.pm 351
/usr/local/lib/perl5/5.15.7/x86_64-linux-thread-multi-debug/File/Spec.pm 80
-e 16
/usr/local/lib/perl5/5.15.7/strict.pm 540
/usr/local/lib/perl5/5.15.7/warnings.pm 1553
By "our" do you mean $work? I thought for cPanel the most important thing for its perl is code obfuscation and perhaps memory footprint for the daemon?
What I think you're seeing is the impact of File::Spec and Cwd, and it would be a huge win to see their impact reduced perhaps by making them load modules on demand... but something doesn't ring true about your module list.
> Cwd File File::Spec File::Spec::Unix Dos Config
> warnings::register EPOC blib Scalar
> Scalar::Util vars VMS VMS::Filespec
> VMS::Feature Win32
Nothing should be loading the VMS stuff, nor Win32 stuff, and certainly not EPOC and DOS! Unless 5.15 really messed up PathTools, all that is loaded on demand. I think what you're measuring is the packages mentioned in the code and NOT what it actually loads.
Steven: cPanel's most important thing for its perl is security and memory size.
Schwern:
Of course there is a difference of parsed compile-time stashes to used run-time stashes and run-time %INC also.
But since I am statically compiling I do not demand-load.
I have to package everything, even if some code paths are never touched and some packages are for sure never needed. That's the ugly default. PAR has the same problem.
To check the bloat difference compile-time vs run-time there is B::Stats http://search.cpan.org/dist/B-Stats/
Ahh, I didn't fully understand the impact of compiling on what you were measuring. Sorry for the confusion.