<?xml version="1.0" encoding="utf-8"?>
<feed xmlns="http://www.w3.org/2005/Atom">
    <title>rillbyteonion57</title>
    <link rel="alternate" type="text/html" href="https://blogs.perl.org/users/rillbyteonion57/" />
    <link rel="self" type="application/atom+xml" href="https://blogs.perl.org/users/rillbyteonion57/atom.xml" />
    <id>tag:blogs.perl.org,2009-11-03:/users/rillbyteonion57//4913</id>
    <updated>2026-11-22T13:13:53Z</updated>
    <subtitle>Perl field notes on small scripts and the details that make them work.</subtitle>
    <generator uri="http://www.sixapart.com/movabletype/">Movable Type Pro 4.38</generator>

<entry>
    <title>One Korean title, three different lengths: testing Unicode keys in Perl</title>
    <link rel="alternate" type="text/html" href="https://blogs.perl.org/users/rillbyteonion57/2026/11/one-korean-title-three-different-lengths-testing-unicode-keys-in-perl.html" />
    <id>tag:blogs.perl.org,2026:/users/rillbyteonion57//4913.12169</id>

    <published>2026-11-22T13:12:39Z</published>
    <updated>2026-11-22T13:13:53Z</updated>

    <summary>A notebook entry for the series 푸른길 makes a useful test fixture. The title looks like three characters. How many should a Perl script count? On decoded text, there are three answers worth keeping separate: code points, grapheme clusters, and...</summary>
    <author>
        <name>rillbyteonion57</name>
        
    </author>
    
    
    <content type="html" xml:lang="en" xml:base="https://blogs.perl.org/users/rillbyteonion57/">
        <![CDATA[<p>A notebook entry for the series 푸른길 makes a useful test fixture. The title looks like three characters. How many should a Perl script count? On decoded text, there are three answers worth keeping separate: code points, grapheme clusters, and the bytes you will eventually write. Here is a small regression test that needs no network, subtitle files, or extra data. Save the script as UTF-8.</p>

<pre><code>use strict;
use warnings;
use utf8;                    # decode this source file, not STDIN
use Unicode::Normalize qw(NFC NFD);
use Encode qw(encode);
use Test::More;

my $title = '푸른길';
my $other = NFD($title);
isnt($title, $other, 'different code-point sequences');
is(length($title), 3, 'three precomposed syllables');
is(length($other), 8, 'eight decomposed Hangul jamo');
my @clusters = $other =~ /\X/g;
is(scalar @clusters, 3, 'three grapheme clusters in either spelling');
is(length(encode('UTF-8', $title)), 9, 'nine UTF-8 bytes for NFC');
my %notes = (NFC($title) =&gt; 'opening credits');
is($notes{NFC($other)}, 'opening credits', 'same lookup key');
done_testing;
</code></pre>

<p>Why eight? 푸 separates into two jamo, an initial consonant and a vowel. 른 and 길 each have a final consonant as well, so each splits into three. That gives 2 + 3 + 3 code points without adding a visible syllable. The nine in the next test measures something else entirely: UTF-8 uses three bytes for each precomposed syllable here.</p>

<p>The hash example is why this matters beyond typography. Perl keys compare strings as supplied. If a title arrives from two catalogs in different normalization forms, normalize both when inserting <strong>and</strong> when searching. Keep the display string as received if the original spelling matters; use a separate NFC key for matching. NFC will not transliterate a Korean title, fix a typo or decide whether two shows with the same title are the same work.</p>

<p>One easy mistake is to add only <code>use utf8</code> when switching from the literal above to a file. That pragma describes the source code. For a UTF-8 text file, open it with an explicit <code>&lt;:encoding(UTF-8)</code> layer so the normalization function receives decoded characters. If the source encoding is unknown, find out what it is first: normalizing the wrong decoding does not repair it.</p>
]]>
        

    </content>
</entry>

</feed>
