The transposition code should still be considered experimental at this stage, although it is passing all the tests from its parent module, Text::Levenshtein::Damerau::XS.
Text::Fuzzy is applicable to the general case of misspellings, and comes with two example scripts, a spell-checker based on /usr/dict/words, and a cgi script which does something like mod_speling.
Text::Fuzzy also offers edit distance based searches across Unicode strings, treating each character as a single entity. The Unicode searches are not quite as souped up as the ASCII searches, because some of the tricks are more difficult to apply.
Update
I've added another example, a Unicode-based fuzzy search over a dictionary. I also found an error in the documentation of Text::Fuzzy, which is now corrected with version 0.09.
Update 2
On closer examination, the speed of the new and old modules is actually roughly the same, so I have removed the statements about speed differences. I didn't remember adding anything to Text::Fuzzy beyond what was in CPAN::Nearest, so the differences were probably due to a configuration error.
The code is basically a very big set of regular expressions. I have improved the internals of the code using qr// (quotes for regular expression) and /x (the extended regular expression flag) to make the regular expressions more readable. There is also one bug fix.
]]>The module includes a script "mat2json" which can be used to convert a MAT-File into JSON. People wishing to evaluate the module for their use may find this a convenient aid:
mat2json "some.mat" | json_xs
]]>Edit:
This module should probably be mothballed and if anyone wants to make an equivalent module, to create one which is specific to the particular Imgur API. For example,
WWW::Imgur::API3
or
WWW::Imgur3
etc.
]]>http://testanything.org/wiki/index.php/Special:Recentchanges
This has been going on for about a year I think. Almost all of the pages on that wiki are spam pages, as you will find out if you try the "random page" feature:
http://testanything.org/wiki/index.php/Special:Random
If this is not a concern, then never mind, but just in case I am the only person who has noticed this, I would like to draw attention to it.
]]>
git: 'cofig' is not a git command. See 'git --help'.Did you mean this?
config
$ ./nearest-module CGI::Compress:Gzip
Closest to 'CGI::Compress:Gzip' is 'CGI::Compress::Gzip'.
$ ./nearest-module Lingua::Stopwords
Closest to 'Lingua::Stopwords' is 'Lingua::StopWords'.
$ time ./nearest-module Lingua::Stopwords
Closest to 'Lingua::Stopwords' is 'Lingua::StopWords'.real 0m0.803s
user 0m0.771s
sys 0m0.012s
Perhaps if the speed can be improved, this module might be useful for people writing module installation scripts. Even better, maybe someone can work out how to make it run faster, perhaps by using a better algorithm, or perhaps by caching the results of common spelling mistakes.
The module is on github as well as CPAN, so please have a go.
Update:
An improvement on the Levenshtein edit distance search is to have a "cutoff" in the algorithm so that it gives up searching once it finds that the edit distance is bigger than a maximum. Using this improvement resulted in about doubling of the speed of the search. I also instituted a "maximum possible distance" cutoff of 10 edits' distance, which is the furthest I think it could go, and if there is nothing the module returns an undef. The speed of version 0.03 is as follows:
$ time ./nearest-module Lingua::Stopwords
Closest to 'Lingua::Stopwords' is 'Lingua::StopWords'.real 0m0.393s
user 0m0.392s
sys 0m0.001s
Update part two: a reduced version of the edit distance calculating matrix
The original algorithm used a (length string 1) x (length string 2) matrix, but it is possible to reduce this to a 2 x (length string 2) sized matrix. I didn't think this would make much difference to the speed, but it does. Here is the speed for version 0.04 of CPAN::Nearest:
$ time ./nearest-module Lingua::Stopwords
Closest to 'Lingua::Stopwords' is 'Lingua::StopWords'.real 0m0.186s
user 0m0.177s
sys 0m0.008s
Update 3: why is the uncompression so slow?
gzip -d on the file only takes 0.15 seconds:
$ time gzip -d --keep 02packages.details.txt.gzreal 0m0.152s
user 0m0.067s
sys 0m0.022s
Update 4: culprit is gzgets
The culprit for the extremely slow performance was the zlib routine "gzgets", which is incredibly slow. Replacing gzgets with gzread, reading the 02packages file takes only about 0.07 seconds more than reading from an uncompressed file. The following is reading from the compressed file:
$ time ./nearest-module Lingua::Stopwords
Closest to 'Lingua::Stopwords' is 'Lingua::StopWords'.real 0m0.259s
user 0m0.258s
sys 0m0.002s
There is more information about gzgets' performance problems on this discussion on stackoverflow.com and on the zlib mailing list. The problem seems to have gone away in recent versions of zlib, but since end-users of the module may not have updated, CPAN::Nearest will stick with not using gzgets.
Update 5
I found another speedup, which reduces the time for a search by about half for a realistic search:
Search term: 'Lingo::Jingo::Mojo'
New time: 0.14 Old time: 0.35 Speedup factor: 2.5Search term: 'Lingua::Stopwords'
New time: 0.17 Old time: 0.24 Speedup factor: 1.4118Search term: 'Bust::A::Move'
New time: 0.13 Old time: 0.27 Speedup factor: 2.0769Search term: 'thequickbrownfoxjumpedoverthelazydogTHEQUICKBROWNFOXJUMPEDOVERTHELAZYDOG:'
New time: 0.64 Old time: 0.64 Speedup factor: 1Search term: 'thequickbrownfoxjumpedoverthelazydogTHEQUICKBROWNFOX'
New time: 0.43 Old time: 0.54 Speedup factor: 1.2558Search term: 'thequickbrownfoxjumpedoverthelazydog'
New time: 0.29 Old time: 0.47 Speedup factor: 1.6207
By experimenting, I found that there is a cutoff point of about forty-five unique characters in the string, where the alphabet method doesn't provide any speedup, and in fact it starts to slow the algorithm down, so the method cuts off if there are more than a certain number of unique characters in the search term. I haven't counted what CPAN module has the maximum number of characters, but anyway this module may be generalized to other edit distances in future.
Update 6
Since the above time is quite largely the decompression time, let's compare the speeds searching the uncompressed file:
For the realistic search terms, the speedup obtained using the alphabet filter may be as much as four times as much, and the total search time is now less than a tenth of a second. ]]>
Search term: 'Lingo::Jingo::Mojo'
New time: 0.07 Old time: 0.28 Speedup factor: 4Search term: 'Lingua::Stopwords'
New time: 0.09 Old time: 0.17 Speedup factor: 1.8889Search term: 'Bust::A::Move'
New time: 0.05 Old time: 0.19 Speedup factor: 3.8Search term: 'thequickbrownfoxjumpedoverthelazydogTHEQUICKBROWNFOXJUMPEDOVERTHELAZYDOG:'
New time: 0.57 Old time: 0.57 Speedup factor: 1Search term: 'thequickbrownfoxjumpedoverthelazydogTHEQUICKBROWNFOX'
New time: 0.35 Old time: 0.47 Speedup factor: 1.3429Search term: 'thequickbrownfoxjumpedoverthelazydog'
New time: 0.22 Old time: 0.39 Speedup factor: 1.7727