domm

Hmm?

---------------

#!/home/ben/software/install/bin/perl
use warnings;
use strict;
use LWP::UserAgent;
use utf8;
my $utf8 = 'さる';
my $ua = LWP::UserAgent->new ();
my $response =
$ua->post('http://mikan/b/cgi/cgi.cgi',
'Content-type' => 'application/json;charset=utf-8',
Content => $utf8,
);
print $response->decoded_content ();

----------

$ ./test-lwp.pl
HTTP::Message content must be bytes at /home/ben/software/install/lib/perl5/site_perl/5.10.0/HTTP/Request/Common.pm line 91

> But sometimes you just have to be very explicit, especially when dealing with picky modules like HTTP::Request.

It's not "sometimes" - you should always decode in the input (when reading files, databases or web request data) and encode in the output (writing files, databases or web response). It's that simple.

HTTP::Request, in your use case, is supposedly the output, so you have to explicitly encode the strings into utf-8.

However, as the other commenter say, if your JSON data is a decoded string you should be geting an error, but it seems you don't. That means your JSON data is already encoded (like you say in the post), so you're sending doubly encoded UTF-8 to the server. I wonder if the server side decoder makes evil things or do something /too smart/ to hide that bug.

> which in this case are actually all

Er, something is cut off: i meant 'which in this case are actually all less than 255 (aka in latin-1 range)'

It's not "sort of broken", the contents of the string $y in the case

use utf8;
my $x = 'さる';
my $y = encode_utf8 $x;

are the same as in $z in

no utf8;
my $z = 'さる';

Try it:

#!/home/ben/software/install/bin/perl
use warnings;
use strict;
use Encode;
use utf8;
my $x = 'さる';
my $y = encode_utf8 $x;
no utf8;
my $z = 'さる';
if ($y eq $z) {
    print "same\n";
}
else {
    print "different\n";
}

[ben@mikan] ~ 527 $ ./moo.pl 
same

Anyway, what Tatsuhiko Miyagawa said is correct:

It's not "sometimes" - you should always decode in the input (when reading files, databases or web request data) and encode in the output (writing files, databases or web response). It's that simple.

You should probably consult Moose developers about the problem you have. JSON is a data, so it should be encoded into utf-8 bytes instead of strings.

This gotcha often happens in the web API programming, since creating URI and HTTP requests require some *data*, which should explicitly be encoded into utf-8 or other encodings.

I have a similar problem. I use the JSON module from CPAN along with LWP. It was a bit confusing, because it works if i use latin1 in JSON but the problems seens to be that i must tell the perl parser with the utf8 pragma that i want to use utf8 in my source code...

This works (maybe it helps someone...)

use utf8;

use JSON;

use use LWP::UserAgent;

$requestValues->{jsonrpc} = "2.0";
$requestValues->{params}{text} = "ÄÜÖ"
...
$ua = LWP::UserAgent->new;
...
$req->content_type('application/json; charset=utf-8');
$req->content(encode_json($requestValues));
...
or
...
$req->content(JSON->new->utf8->encode($requestValues));
...