xiaoayfeng

I wouldn't read so much into this. This month they added a large number of extra search engines as statistical sources, and a whole bunch of their stats jumped. Look at what happened to C and Objective-C!

If you're doing Unicode, Perl is your best tool.

And, remember that TIOBE is deeply flawed. Google that. They are completely useless rankings for any purpose.

I admit, TIOBE is worthless, worthless as a ranking index, but unfortunately it's at least bad marketing.
It's all about marketing, so how can we place perl on a better rank?

Catalyst, Poet/Mason, Dancer, Mojolicious.

@brian d foy, @Nick Patch
Could you write specific Unicode examples where you'd have trouble in Ruby, Python or PHP but works in Perl?

How about this?

#!/usr/bin/env perl
use v5.14;
use utf8;
my $string = "café";
substr($string, -1, 1) = "e";
say $string;
# output: "cafe"

Versus:

<?php
$string = "café";
$string = substr_replace($string, "e", -1, 1);
print $string . "\n";
# output: "caf\x{c3}e"

PHP 5 strings are strictly byte strings; all the standard string manipulation functions (and thus, all the third-party libraries that use those built-in functions) assume that one character = one byte.

There are a separate set of functions for character strings in the "mbstring" extension, but this extension is not enabled by default, and not widely used. (Whatsmore it only offers multibyte equivalents for some of PHP's built-in string functions; not all of them.)

Indeed I always hear that perl has best unicode support, but I did not see evidences yet.
Ok, there is one for PHP above. But what about Ruby 1.8/1.9/Python 2/3?

Ruby seem to have different paradigm - all strings are byte strings with known encoding. This can be treated as performance advantage - you don't need to convert data to unicode "internal representation". Or it can produce mess (a matter of taste imho).

@nick

I hear you on Perl and UTF8, however I do think that the subject is esoteric enough that even though Perl is ++ in regards to it, it does not shine as brightly as it should. I've seen several of your talks and we actually worked together and I think I would still have a hard time showing why this is a great thing about Perl :(

And one downside here is that it tends to play to the Perl stereotype a bit, that its great for text processing and related analysis.

Maybe you can convince someone at Shutterstock to toss some of that sweet post IPO money into sponsoring having you write some sort of Enterprise UTF8 with Perl5 book? I think starting a book like this and manage it on Github, similar to the Modern Perl book, would be fantastic. I'd be willing to help a bit, if that helps.

John

Regarding web frameworks, I guess I would say Perl web development does not aim to be as full stack as RoR. For example with Catalyst one uses that as a base and then builds as needed, using CPAN on the way. I do agree that for newcomers it is not easy enough to figure out what pieces to use where, but it is the general paradigm Perl programmers prefer.

@Nick, thanks!

@Nick, it indeed looks like special area of natural language processing.

imho lot's of "typical" applications just need something from that list:

1) store/retrieve correctly to/from external storage.
2) display it correctly
3) determine length correctly
4) foldercase it correctly.
5) don't produce broken utf when truncating

6) optional: uppercase/lowercase it correctly.
7) optional: sort with respect to collations.
8) optional: deal with filesystem encodings issues
9) optional: search text correctly.

and they sometimes fail to do even that.

btw I believe your example above can be solved for most European and Cyrillic languages by normalizing to NFC, so 1 character=1 graphem

Just a small note:

The TIOBE algorithm is to search for " programming".

Now Perl has a moniker that is unique to the language. There is no other item in the world that has this name and is in common use. Meanwhile Ruby and Python are named after a gemstone and a snake (or a comedy troupe).

This means that anyone talking about Perl will have no reason to say "Perl programming language", while users of the other language actually need to say "Ruby programming language" or "Python programming language".

Now i invite you to do a little experiment. TIOBE's third-ranked search engine is wikipedia. Go on there, and search for:

* perl: http://en.wikipedia.org/w/index.php?search=perl&title=Special%3ASearch&fulltext=1
* perl programming: http://en.wikipedia.org/w/index.php?title=Special%3ASearch&profile=default&search=%22perl+programming%22&fulltext=Search
* python programming: http://en.wikipedia.org/w/index.php?title=Special%3ASearch&profile=default&search=%22python+programming%22&fulltext=Search
* python: http://en.wikipedia.org/w/index.php?title=Special%3ASearch&profile=default&search=python&fulltext=Search

You'll find that there are actually considerably less entries about Python on Wikipedia than Perl, but due to a small difference in their names, Perl shows up much less in TIOBE's searches. I'm fairly sure you'll be able to observe the same in other search engines they use.

@Mithaldu,
+1

@Nick,
у́ and и́ in "aку́т" or "окси́я" are not parts or Russian alphabet. It's just accents, can be used in any language probably. Same chance russians will use accents in the text, that
they will use characters from foreign alphabets (unless they are writing dictionary article). Same probably applies to "n̈" (it's not on keyboard according to Wiki).
CRLF imho should be processed in different layer, unless you writing word processor.

It's of course very interesting and indeed should be used in applications that should process unicode 100% correct, but if I report the bugs like "text truncation
function will truncate accent from "у́", to any company I worked in the past, they will probably ignore it or put in the bottom of bug queue.

btw if dictionary article truncated in the middle of word with accents, accent is probably useless and can be dropped.

I'm not understand problem with unicode.

Still on perl version v5.8.8:

echo "café" | perl -MEncode -lpe'
$_=Encode::decode("utf8",$_);
substr($_, -1, 1) = "e";
Encode::encode("utf8",$_)'

# output: cafe

In Ruby or Python this is not possible? I'm under table.

Ruby 1.9x - possible

$echo café|ruby 5.rb
cafe

$cat 5.rb
# encoding: utf-8
puts STDIN.read.gsub(/é/, 'e')

> I'm not understand problem with unicode.

see above comments from @Nick about