One Korean title, three different lengths: testing Unicode keys in Perl
A notebook entry for the series 푸른길 makes a useful test fixture. The title looks like three characters. How many should a Perl script count? On decoded text, there are three answers worth keeping separate: code points, grapheme clusters, and the bytes you will eventually write. Here is a small regression test that needs no network, subtitle files, or extra data. Save the script as UTF-8.
use strict;
use warnings;
use utf8; # decode this source file, not STDIN
use Unicode::Normalize qw(NFC NFD);
use Encode qw(encode);
use Test::More;
my $title = '푸른길';
my $other = NFD($title);
isnt($title, $other, 'different code-point sequences');
is(length($title), 3, 'three precomposed syllables');
is(length($other), 8, 'eight decomposed Hangul jamo');
my @clusters = $other =~ /\X/g;
is(scalar @clusters, 3, 'three grapheme clusters in either spelling');
is(length(encode('UTF-8', $title)), 9, 'nine UTF-8 bytes for NFC');
my %notes = (NFC($title) => 'opening credits');
is($notes{NFC($other)}, 'opening credits', 'same lookup key');
done_testing;
Why eight? 푸 separates into two jamo, an initial consonant and a vowel. 른 and 길 each have a final consonant as well, so each splits into three. That gives 2 + 3 + 3 code points without adding a visible syllable. The nine in the next test measures something else entirely: UTF-8 uses three bytes for each precomposed syllable here.
The hash example is why this matters beyond typography. Perl keys compare strings as supplied. If a title arrives from two catalogs in different normalization forms, normalize both when inserting and when searching. Keep the display string as received if the original spelling matters; use a separate NFC key for matching. NFC will not transliterate a Korean title, fix a typo or decide whether two shows with the same title are the same work.
One easy mistake is to add only use utf8 when switching from the literal above to a file. That pragma describes the source code. For a UTF-8 text file, open it with an explicit <:encoding(UTF-8) layer so the normalization function receives decoded characters. If the source encoding is unknown, find out what it is first: normalizing the wrong decoding does not repair it.
Leave a comment