Compiler progress with 5.16

The latest B::C package on CPAN 1.42 works stable for almost all perls until 5.14, but so far did not work good enough for the upcoming 5.16 release.

I couldn't even pinpoint to a specific perl change which caused the problems. I know that hashes need a different initialization now. Empty hashes need to declare

HvTOTALKEYS(hv) = 0

after creation, and readonly hashes must be set readonly after they were created.

DynaLoader and %INC handling is much stricter now with 5.16.

And then there is the PMOP pmoffset IV hack. We have now first-class REGEXP SVs, but not really. With threads we still store the regexp body in PL_regex_padav, not as normal body in an arena, and the parser stores the latest pmoffset (I think) behind the PV in PL_regex_pad[0]. Thing is, the compiler does not use head and body arenas yet, it uses static arrays. But when the first body arena for some dynamic SV is initialized, the PL_regex_padav is reset.

All these changes forced me to rewrite the tricky parts of the compiler, the recursive walker to detect used packages and objects.

In my desperation I added checkers to detect the name of objects for method calls, to check bless for used object names, to associate blessed and new scalars with a method call, and to check bareword require for dynamically included packages. The ISA search is now recursive and tries a lof of candidates to find unknown objects for method calls, with proper AUTOLOAD and UNIVERSAL fallback.

This increased the compile time dramatically, but it must be correct and should include all possible used packages. Or if not, at least the %INC hash must be correct to allow dynamically added packages to be loaded properly at run-time. Possibly with DynaLoader.

Still, 5.16 did not pass the most tricky tests and production code, which all worked fine with 5.14 back in November.

Lately I fought deep recursion troubles when compiling some recursive functions in tricky package inclusion scenarios. Moose and recursive Pod::Simple functions mainly. Either the compiler missed some functions, stored some packages only halfway, like Encode (UTF8 did not work), or the compiler went into deep recursion issues in the walker. So I went the old-school way and removed too recursive functions from the walker.

First I disabled recursing into op->first, because the B function walkoptree already steps into op->first.

Second I disabled walking into newly found packages immediately and just mark them as new. The main walker loops now over all packages again and again until the list of newly found packages in each loop is empty. This adds costly compiler passes, and certainly increases the size of the produced binaries, but better load such code at once compiled, then defer it to later.

https://github.com/rurban/perl-compiler/compare/recurse

2 Comments

Thanks a lot for your work, it is very interesting! I always appreciate to read you.

Max.

Hi Reini

Yeah, I'd like to add my thanx. I suppose in theory if I studied this stuff long enough I might be able to do what you're doing, but I have to say I just won't be doing that! So, I do appreciate the efforts of those who do actually do this sort of work.

Cheers
Ron

Leave a comment

About Reini Urban

user-pic Working at cPanel on B::C (the perl-compiler), p2, types, parrot, B::Generate, cygwin perl and more guts (LLVM, jit, optimizations).