The Incredible Journey

Last January there was a post on reddit which claimed that my module JSON::Parse was not only failing some of the JSON Test Suite tests, but also crashing on one of them. Anyway I should have got around to doing something about it sooner, but here are my conclusions.

First of all there was a crash on one of the files, which went something like this: [{"":[{"":[{"", repeated about 100,000 times. (The actual file is here if you really want to see it.) Investigating it using a LInode, I found that after 80,000 open brackets the stack was overflowing, causing the crash to occur. If I added a printf in the midst of my code the printf would cause the stack overflow, so it wasn't actually due to my code but just because the stack size seems to be quite small on Linux.

There are various things one could do to tackle this, but it does seem a bit unlikely that anyone would want to have that many open brackets, so what I did as a strategy was to add a "max_depth" of parsing after which it would stop. I thought 10,000 open { and [ would be enough for anyone, and it would satisfy the people who want to run the JSON Test Suite tests, but I also added an option for the user to alter the max depth and get the max depth as well.

After finishing that I started to wonder what other modules were doing that they had passed this test. A quick glance at the Cpanel::JSON::XS documentation and it seems that those people had the same idea, but their maximum depth is a measly 512 as opposed to my whopping 10,000. I'm not sure which is better or worse. 😕

There were a couple of other tests in the JSON Test Suite which I failed, which were tests about Unicode; the JSON Test Suite thinks that it is compulsory for JSON parsers to pass through Unicode non-characters, UTF-8 representing surrogate pairs, and so on. I'm not really sure why they think so, there is a bit of a discussion on the github but that doesn't strike me as being very convincing. Frankly I think JSON::Parse is much more useful tool for users if it rejects the non-characters and other crap, 💩 so I would need to see much stronger justification than I have done to alter it in the way that JSON Test Suite suggests.

In other news, I've also made it possible for the module to be installed from github, so it should now be possible to start using continuous integration. I've already got Travis working, and I hope to add some more useful things soon.