[Phpmyadmin-devel] UTF8 and connection encoding
Hi All, if someone changes connection setting in main.php from UTF-8 to something else, all content from DB will be delivered with this encoding to users Browser, but we say the document is UTF-8 and strings coming from us (language files) are in UTF-8 so we end up with a mixed encoding in the HTML output there are two options: - setting PHP output encoding http://php.net/manual/en/mbstring.http.php do not know if this takes care of already mixed encodings - do not allow switching connection charset why is this required anyway? i would vote for dropping, as is see no requirement for this, any modern browser should be UTF-8 capable any comments suggestions? -- Sebastian Mendel
On 08.02.2009 16:02, Sebastian Mendel wrote:
Hi All,
if someone changes connection setting in main.php from UTF-8 to something else, all content from DB will be delivered with this encoding to users Browser, but we say the document is UTF-8 and strings coming from us (language files) are in UTF-8
so we end up with a mixed encoding in the HTML output
there are two options:
- setting PHP output encoding
http://php.net/manual/en/mbstring.http.php
do not know if this takes care of already mixed encodings
- do not allow switching connection charset
why is this required anyway?
i would vote for dropping, as is see no requirement for this, any modern browser should be UTF-8 capable
any comments suggestions?
also see bug #2565948 https://sourceforge.net/tracker2/?func=detail&aid=2565948&group_id=23067&ati... -- Sebastian Mendel
Hi Sebastian and list, Sebastian Mendel schrieb:
if someone changes connection setting in main.php from UTF-8 to something else, all content from DB will be delivered with this encoding to users Browser, but we say the document is UTF-8 and strings coming from us (language files) are in UTF-8
Who changed this anyway? In my original MySQL 4.1 compatibility code, the option to change the connection charset was intentionally not given to the user - only for dump files, afaik.
so we end up with a mixed encoding in the HTML output
Right, we would need to either recode the language files or on the fly any string that comes from MySQL. But I guess, we would destroy the "intention" of that feature - see my explanation below.
there are two options:
- setting PHP output encoding
http://php.net/manual/en/mbstring.http.php
do not know if this takes care of already mixed encodings
It doesn't, afaik.
- do not allow switching connection charset
+++ from my side
why is this required anyway?
Well, if you use some old MySQL application that was coded against MySQL 3.23 and thus is not aware of all that charset stuff, it certainly will happen that this application writes in a different charset than MySQL assumes. For instance, phpBB 2.x internally uses UTF-8 (which MySQL prior 4.1 did non support anyway), but does not send "SET NAMES utf8". MySQL by default assumes latin1 (which was the default charset in MySQL 3.23). The result is that you have kind of double encoded data in your tables. If you access these tables with phpMyAdmin, it looks like they are displayed wrong (while in fact, phpMyAdmin is right). Setting the connection charset to latin1 but treating incoming data as if it was utf8 is exactly what phpBB 2 and other applications do, so the data displays seemingly correctly. Anyway, when I wrote the code for MySQL 4.1, my opinion was that this should not be the problem of phpMyAdmin. If you decide to keep that "feature", it is imho to prominent where it is right now and you should add warnings with tons of exclamation marks against using it. Regards, Alexander
On 09.02.2009 10:47, cand. inf. Alexander M. Turek wrote:
Hi Sebastian and list,
Sebastian Mendel schrieb:
if someone changes connection setting in main.php from UTF-8 to something else, all content from DB will be delivered with this encoding to users Browser, but we say the document is UTF-8 and strings coming from us (language files) are in UTF-8
Who changed this anyway? In my original MySQL 4.1 compatibility code, the option to change the connection charset was intentionally not given to the user - only for dump files, afaik.
so we end up with a mixed encoding in the HTML output
Right, we would need to either recode the language files or on the fly any string that comes from MySQL. But I guess, we would destroy the "intention" of that feature - see my explanation below.
there are two options:
- setting PHP output encoding
http://php.net/manual/en/mbstring.http.php
do not know if this takes care of already mixed encodings
It doesn't, afaik.
- do not allow switching connection charset
+++ from my side
why is this required anyway?
Well, if you use some old MySQL application that was coded against MySQL 3.23 and thus is not aware of all that charset stuff, it certainly will happen that this application writes in a different charset than MySQL assumes. For instance, phpBB 2.x internally uses UTF-8 (which MySQL prior 4.1 did non support anyway), but does not send "SET NAMES utf8". MySQL by default assumes latin1 (which was the default charset in MySQL 3.23). The result is that you have kind of double encoded data in your tables. If you access these tables with phpMyAdmin, it looks like they are displayed wrong (while in fact, phpMyAdmin is right).
Setting the connection charset to latin1 but treating incoming data as if it was utf8 is exactly what phpBB 2 and other applications do, so the data displays seemingly correctly.
Anyway, when I wrote the code for MySQL 4.1, my opinion was that this should not be the problem of phpMyAdmin.
If you decide to keep that "feature", it is imho to prominent where it is right now and you should add warnings with tons of exclamation marks against using it.
in short: drop this in phpMyAdmin 3? -- Sebastian Mendel
Sebastian Mendel a écrit :
On 09.02.2009 10:47, cand. inf. Alexander M. Turek wrote:
Hi Sebastian and list,
If you decide to keep that "feature", it is imho to prominent where it is right now and you should add warnings with tons of exclamation marks against using it.
in short: drop this in phpMyAdmin 3?
Let's target 3.2 and either: - add a configuration parameter that controls this feature and disables it by default, or - add a message on the main page when the connection collation has been changed, to explain the possible consequences Marc
cand. inf. Alexander M. Turek a écrit :
Hi Sebastian and list,
Sebastian Mendel schrieb:
if someone changes connection setting in main.php from UTF-8 to something else, all content from DB will be delivered with this encoding to users Browser, but we say the document is UTF-8 and strings coming from us (language files) are in UTF-8
Who changed this anyway? In my original MySQL 4.1 compatibility code, the option to change the connection charset was intentionally not given to the user - only for dump files, afaik.
Alexander, you did this: http://phpmyadmin.svn.sourceforge.net/viewvc/phpmyadmin/tags/RELEASE_2_6_0PL... for this bug: https://sourceforge.net/tracker2/index.php?func=detail&aid=984912&group_id=2... where you said: "My suggestion is to let the user choose his connection character set on main.php. In addition, we should imho force the users to convert their pma DBs to UTF-8. As these tables contain metadata and should be able to store strings in any language, this would be the best solution anyway." So it was useful back then, but nowadays we could remove this feature or at least make it configurable and not display it by default.
so we end up with a mixed encoding in the HTML output
Right, we would need to either recode the language files or on the fly any string that comes from MySQL. But I guess, we would destroy the "intention" of that feature - see my explanation below.
there are two options:
- setting PHP output encoding
http://php.net/manual/en/mbstring.http.php
do not know if this takes care of already mixed encodings
It doesn't, afaik.
- do not allow switching connection charset
+++ from my side
why is this required anyway?
Well, if you use some old MySQL application that was coded against MySQL 3.23 and thus is not aware of all that charset stuff, it certainly will happen that this application writes in a different charset than MySQL assumes. For instance, phpBB 2.x internally uses UTF-8 (which MySQL prior 4.1 did non support anyway), but does not send "SET NAMES utf8". MySQL by default assumes latin1 (which was the default charset in MySQL 3.23). The result is that you have kind of double encoded data in your tables. If you access these tables with phpMyAdmin, it looks like they are displayed wrong (while in fact, phpMyAdmin is right).
Setting the connection charset to latin1 but treating incoming data as if it was utf8 is exactly what phpBB 2 and other applications do, so the data displays seemingly correctly.
Anyway, when I wrote the code for MySQL 4.1, my opinion was that this should not be the problem of phpMyAdmin.
If you decide to keep that "feature", it is imho to prominent where it is right now and you should add warnings with tons of exclamation marks against using it.
Regards,
Alexander
------------------------------------------------------------------------
------------------------------------------------------------------------------ Create and Deploy Rich Internet Apps outside the browser with Adobe(R)AIR(TM) software. With Adobe AIR, Ajax developers can use existing skills and code to build responsive, highly engaging applications that combine the power of local resources and data with the reach of the web. Download the Adobe AIR SDK and Ajax docs to start building applications today-http://p.sf.net/sfu/adobe-com
------------------------------------------------------------------------
_______________________________________________ Phpmyadmin-devel mailing list Phpmyadmin-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/phpmyadmin-devel
Hi Marc & list, Marc Delisle schrieb:
cand. inf. Alexander M. Turek a écrit :
Who changed this anyway? In my original MySQL 4.1 compatibility code, the option to change the connection charset was intentionally not given to the user - only for dump files, afaik.
Alexander, you did this:
Just read the docs to refresh my memory. Changing the connection charset is fine. It is used for comparisions and should not affect the results. In the current PMA, the client charset is changed instead (and this is the evil thing I was talking about). I'm having a look at this, hang on. Regards, Alexander
Hi again, cand. inf. Alexander M. Turek schrieb:
Just read the docs to refresh my memory. Changing the connection charset is fine. It is used for comparisions and should not affect the results. In the current PMA, the client charset is changed instead (and this is the evil thing I was talking about). I'm having a look at this, hang on.
The evil change has happened here: http://phpmyadmin.svn.sourceforge.net/viewvc/phpmyadmin/trunk/phpMyAdmin/lib... Shame on you, cybot_tm! ;-) The bug should be fixed in SVN, please confirm. Regards, Alexander
On 09.02.2009 16:27, cand. inf. Alexander M. Turek wrote:
Hi again,
cand. inf. Alexander M. Turek schrieb:
Just read the docs to refresh my memory. Changing the connection charset is fine. It is used for comparisions and should not affect the results. In the current PMA, the client charset is changed instead (and this is the evil thing I was talking about). I'm having a look at this, hang on.
The evil change has happened here:
http://phpmyadmin.svn.sourceforge.net/viewvc/phpmyadmin/trunk/phpMyAdmin/lib...
Shame on you, cybot_tm! ;-)
uh, someone must have hacked into my account ... -- Sebastian Mendel
participants (3)
-
cand. inf. Alexander M. Turek -
Marc Delisle -
Sebastian Mendel