Mike B

In my previous post I explained how many utf-8 related issues are now fixed in the latest DBD::mysql development release, and I asked for feedback and testing from our users.

Even better utf-8 handling

I'm happy to inform you that, thanks to your feedback, we've now released yet another development version, with even more UTF8 issues fixed. Tanabe Yoshinori reported that column names and database warnings were not properly encoded and prolific contributor Pali Rohár fixed that issue, and much more.

Serialization issue under taint mode

One other issue which was reported by amavisd users has to do with how perl stores values internally under taint mode, which is the default way amavisd is run. This is now fixed, see for more information https://github.com/perl5-dbi/DBD-mysql/issues/78.

Extreme CI testing setup

And Pali also modified our Travis setup so we now do continuous integration testing not only on many different perl versions, but also using many different MySQL and MariaDB versions. This uncovered many smaller and larger issues and we can compile again all the way back with MySQL against version 4 if you would want that (and we had multiple people wanting that and filing bug reports for this in the past).

This also lead to Pali discovering a use-after-free security issue (CVE-2017-3302) in libmysqlclient which was fixed in MySQL 5.6 and up but still present in 5.5 and also in MariaDB. See for more info this thread on oss-security.

Your feedback is welcome!

Find the full change log below; when all is well on Wednesday 8th of March we'll release the stable version 4.042, including all these changes, as well as the changes from the previous post.

You can leave your feedback via the DBI-users mailing list, or using our GitHub page.

2017-02-28 change log of version 4.041_2)

• Statement handle now also has mysql_sock attribute, just as database handle. (by Pali)
• Fix type conversions for magic types. Issue reported by Dmitriy Yatsenko and Giovanni Bechis, fix by Pali. https://lists.amavis.org/pipermail/amavis-users/2016-December/004674.html, https://github.com/perl5-dbi/DBD-mysql/issues/78
• Fix UTF8-encoding of table names, field names, warnings and error messages. Reported by Tanabe Yoshinori, fix by Pali. https://rt.cpan.org/Public/Bug/Display.html?id=120141
• Fix mysql_auto_reconnect when using mysql_server_prepare (pali). Reported by Vladimir Marek. https://github.com/perl5-dbi/DBD-mysql/pull/95
• Improve regex for removing database from dsn (pali) https://rt.cpan.org/Public/Bug/Display.html?id=118837
• Locate MySQL libs using Devel::CheckLib (pali)
• Support async on Windows (pali)
• Fix test suite on range of older and newer MySQL and MariaDB versions https://github.com/perl5-dbi/DBD-mysql/pull/87
• Fix compilation on MySQL 4.1 (pali)
• Do not leak dangling pointer to mysql result (pali)
• Fix logic when assigning to variable bind_comment_placeholders (pali)
• mysql_fd() still returned file descriptor after closing connection. Reported by Larry Leszczynski, fixed by Pali Rohár. https://rt.cpan.org/Public/Bug/Display.html?id=110983
• Fix parsing configure libs from mysql_config --libs output in Makefile.PL Libraries in mysql_config --libs output can be specified by library name with the -l prefix or by absolute path to library name without any prefix. Parameters must start with a hyphen, so treat all options without leading hyphen in mysql_config --libs output as libraries with full path. Partially fixes bug https://rt.cpan.org/Public/Bug/Display.html?id=100898 Fix by Pali Rohár.
• Fix support for magic scalars (pali) https://github.com/perl5-dbi/DBD-mysql/pull/76

After a couple of years of more or less "maintenance mode" on DBD::mysql - we had a hand full of people contributing occasional fixes and a whole slew of drive-by contributors - we now have a prolific contributor again: Pali Rohár.

It's great to see some more long-standing issues taken care of!

This time around, in the new development release 4.041_01 that is on CPAN now, there are some important fixes for some Unicode-related issues that I would like to point out. The sections below I have distilled based on the descriptions made by Pali.

Automatically converting to UTF-8 for bind parameters

Before this release perl scalars (statements or bind parameters) without UTF8 status flag were not encoded to UTF-8 even if mysql_enable_utf8 was enabled. This caused perl scalars with internal Latin1 encoding to be sent to the mysql server as Latin1 even if mysql_enable_utf8 was enabled.

Now all statements and bind parameters which are not a DBI binary type (SQL_BIT, SQL_BLOB, SQL_BINARY, SQL_VARBINARY or SQL_LONGVARBINARY) are automatically encoded to UTF-8 when mysql_enable_utf8 is enabled.

If mysql_enable_utf8 is not enabled and your statement or bind parameter contains a wide Unicode character then DBD::mysql shows a warning. If a binary parameter contains a wide Unicode character then DBD::mysql shows a warning too, similar like function print without using a :utf8 perlio layer. ("Wide character in...")

Perl's SvPV() returns char* from a perl scalar and the following SvUTF8() call for that scalar returns true if SvPV returned the data in UTF-8 or Latin1.

Decoding of UTF-8 fields when mysqlenableutf8 is enabled

For each fetched field mysql server tells us its charset id. Before this release when mysqlenableutf8 was enabled DBD::mysql UTF-8 decoded all fields with a charset id different than 63 (which means binary).

Now DBD::mysql UTF-8 decodes only those fields which have their charset set to utf8 or utf8mb4. By default mysql server sends data in encoding specified by the SET NAMES command, which is by default Latin1. So any received Latin1 data is not UTF-8 decoded anymore.

The mysql server sends a charset id, not a charset name. Each combination of charset name and collation pairs has its own charset id. A new function charsetnr_is_utf8() has hardcoded all utf8 and utf8mb4 charset ids from mysql (up to 8.0.0) and mariadb (up to 10.2.2) from their source code. So far it looks like those ids are not changing since old mysql 5.0, only new ones are added.

Conclusion

We hope these changes make DBD::mysql a lot more consistent for you. Since the changes are rather big, we'd urge you to test the development release 4.041_01 which is on CPAN and give feedback NOW; this allows us to make changes if needed before we create an actual stable release with these features.

And of course, if you test it with your software and all is good, we'd like to hear that as well!

You can leave your feedback via the DBI-users mailing list, or using our GitHub page.

DBD::mysql is the perl DBI driver for MySQL and the primary way Perl applications and scripts access MySQL and MariaDB databases. The source repository is at https://github.com/perl5-dbi/DBD-mysql.

A vulnerability was discovered that can lead to a use after free when using prepared statements. This vulnerability is present in all releases at least back to versions 3.0 of the driver, which were released in 2005.

The CVE identifier for this vulnerability is CVE-2016-1251.

Version 4.041, including the fix for this vulnerability, is available on CPAN at https://metacpan.org/pod/DBD::mysql

Users of DBD::mysql with prepared statements are advised to patch their installations as soon as possible.

Many thanks to Pali Rohár for discovering and fixing the vulnerability.

The DBD::mysql maintainers, Patrick Galbraith Michiel Beijen

DBD::mysql is the perl DBI driver for MySQL and the primary way Perl applications and scripts access MySQL and MariaDB databases. The source repository is at https://github.com/perl5-dbi/DBD-mysql.

A vulnerability was discovered that can lead to a buffer overflow, possibly triggered by user supplied data. This vulnerability is present in all releases at least back to versions 3.0 of the driver, which were released in 2005.

The CVE identifier for this vulnerability is CVE-2016-1246.

Version 4.037, including the fix for this vulnerability, is available on CPAN at https://metacpan.org/pod/DBD::mysql

Users of DBD::mysql are advised to patch their installations as soon as possible.

We have already made a pre-announcement for this security release at the distros security mailing list. People using DBD::mysql installed from their (linux) distributions can expect to receive an updated version soon.

Many thanks to Pali Rohár for discovering and fixing the vulnerability.

The DBD::mysql maintainers, Patrick Galbraith Michiel Beijen

Dear Perl and MySQL community,

We're pleased to announce the release of DBD::mysql 4.033_01, the Perl DBI driver for MySQL and MariaDB databases. This is not a 'stable' release but merely for testing and feedback. We'll put out a stable 4.034 release soon; probably before christmas.

Linking against SSL by default?

Apart from that, I'd like to announce that we might want to link to SSL by default. MySQL 5.7 makes SSL connections to databases more common; right now in DBD::mysql you'd need to pass an option to Makefile.PL (--ssl) in order to enable linking to libssl. Of course, many people (and linux distributions!) don't do this by default. On the expense of the added dependency to libssl we'd want to default to compiling against libssl. We'd introduce a --nossl flag for the cases where you'd explicitly NOT want to link to SSL. When DBD::mysql is compiled against libssl you can still make connections to not-sslified servers.

Any feedback (+1's, remarks or objections) would be appreciated!

Changelog for this development release

2015-12-15 Patrick Galbraith, Michiel Beijen, DBI/DBD community (4.033_01)

• Raise minimum DBI version to 1.609 (from 2009!) in order to make tests pass on RHEL5.
• Add explicit documentation for how to enable SSL at build time, provided by genio.
• Improve test suite to make it more robust using mixed versions of client and server.
• Fix use after free error in my_login, provided by hannob.
• Add explicit instructions stating brackets are required around ipv6-addresses in connection strings, provided by Kenny Gryp.

Release on CPAN: https://metacpan.org/release/MICHIELB/DBD-mysql-4.033_01

UPDATE, UPDATE, read all about it!

After feedback we've decided to make the jump, which was probably long overdue, and link to SSL by default as proposed above. There's a new development release with these changes on CPAN.

2015-12-18 Patrick Galbraith, Michiel Beijen, DBI/DBD community (4.033_02)

• Compile against libssl by default. This allows to connect against remote MySQL servers using SSL. Previously this was only achieved with an explicit switch provided to Makefile.PL - if for some reason you can't or don't wantto link against libssl, you can use the new --nossl switch to Makefile.PL.
• Made tests more robust after CPAN Testers failures.

Release on CPAN: https://metacpan.org/release/MICHIELB/DBD-mysql-4.033_02

Github repo: https://github.com/perl5-dbi/DBD-mysql

Regards,

Patrick and Michiel