Hi, I could not find a showstopper bug for the upcoming 4.1.0-rc1.
Date: Tue, 19 Nov 2013 07:28:00 -0500 From: marc@infomarc.info To: phpmyadmin-devel@lists.sourceforge.net Subject: [Phpmyadmin-devel] ready for 4.1.0-rc1 ?
Hi, I could not find a showstopper bug for the upcoming 4.1.0-rc1.
-- Marc Delisle http://infomarc.info
Hi,
I found this bug: https://sourceforge.net/p/phpmyadmin/bugs/4156/
Can we have the author of the causing commit fix that for us?
Best, JM
J.M. . a écrit :
Date: Tue, 19 Nov 2013 07:28:00 -0500 From: marc@infomarc.info To: phpmyadmin-devel@lists.sourceforge.net Subject: [Phpmyadmin-devel] ready for 4.1.0-rc1 ?
Hi, I could not find a showstopper bug for the upcoming 4.1.0-rc1.
-- Marc Delisle http://infomarc.info
Hi,
I found this bug: https://sourceforge.net/p/phpmyadmin/bugs/4156/
Can we have the author of the causing commit fix that for us?
Indeed, this must be fixed before 4.1.0-rc1.
Hi,
Personally I do not think it is advisable to move forward until certain issues have been addressed. I am talking about https://sourceforge.net/p/phpmyadmin/bugs/3944 and related bug reports.
The alleged solution for 4.1 is requiring MySQL 5.5 as a minimum version. This is not going to solve the problems that people have reported when using the 4.x series with large systems, since while 5.5 is better than 5.1 and below, a full directory scan can still be triggered by any number of combinations of variables as described here: http://dev.mysql.com/doc/refman/5.6/en/information-schema-optimization.html
I explained this in that bug report where i showed PMAs default usage of IS for the navigation on a *5.6* system (with innodb_stats_on_metadata switched off). While that was just a 2 second example, I have seen that get up to 60 seconds before query killers kick in. Reports of users on shared hosting accounts, where this kind of behavior can get users accounts terminated has also already been reported ( https://sourceforge.net/p/phpmyadmin/bugs/3945, second page "Here is what one provider sent as a warning before *deleting my account*")
If you remove the use of disableIS and related functionality, it will not solve any issues. Take the SCHEMATA queries for example for getting database lists, this still effectively does a full directory scan, as documented: http://dev.mysql.com/doc/refman/5.6/en/schemata-table.html In this case for example, switching to IS to avoid using the slow performing show databases is redundant since they basically do the same thing. As a basic example, take a directory, with tens of thousands of files, and attempt to do an ls in it, even unsorted (-U) it will take a long time, simply because traversal of the filesystem takes time. Requiring higher versions isnt going to miraculously speed up the iteration of directories.
On a similar note, when I referenced *innodb_stats_on_metadata *in earlier mentioned bug report, I was using it as an example of why relying on IS can be as bad, or worse than directly doing a show tables, since it can potentially end up doing additional work. I am pretty worried that modifying the variable was considered as a potential solution. The description of that very setting describes how it affects how the system handles IS queries. An administrator should be able to trust that installing something like PMA is not going to change any system variables that could otherwise affect the operation of the system
Thank you for your consideration.
Regards, Martyn
On Tue, Nov 19, 2013 at 4:28 AM, Marc Delisle marc@infomarc.info wrote:
Hi, I could not find a showstopper bug for the upcoming 4.1.0-rc1.
-- Marc Delisle http://infomarc.info
Shape the Mobile Experience: Free Subscription Software experts and developers: Be at the forefront of tech innovation. Intel(R) Software Adrenaline delivers strategic insight and game-changing conversations that shape the rapidly evolving mobile landscape. Sign up now. http://pubads.g.doubleclick.net/gampad/clk?id=63431311&iu=/4140/ostg.clk... _______________________________________________ Phpmyadmin-devel mailing list Phpmyadmin-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/phpmyadmin-devel
Martyn Dale a écrit :
Hi,
Personally I do not think it is advisable to move forward until certain issues have been addressed. I am talking about https://sourceforge.net/p/phpmyadmin/bugs/3944 and related bug reports.
The alleged solution for 4.1 is requiring MySQL 5.5 as a minimum version. This is not going to solve the problems that people have reported when using the 4.x series with large systems, since while 5.5 is better than 5.1 and below, a full directory scan can still be triggered by any number of combinations of variables as described here: http://dev.mysql.com/doc/refman/5.6/en/information-schema-optimization.html
I explained this in that bug report where i showed PMAs default usage of IS for the navigation on a *5.6* system (with innodb_stats_on_metadata switched off). While that was just a 2 second example, I have seen that get up to 60 seconds before query killers kick in. Reports of users on shared hosting accounts, where this kind of behavior can get users accounts terminated has also already been reported ( https://sourceforge.net/p/phpmyadmin/bugs/3945, second page "Here is what one provider sent as a warning before *deleting my account*")
If you remove the use of disableIS and related functionality, it will not solve any issues. Take the SCHEMATA queries for example for getting database lists, this still effectively does a full directory scan, as documented: http://dev.mysql.com/doc/refman/5.6/en/schemata-table.html In this case for example, switching to IS to avoid using the slow performing show databases is redundant since they basically do the same thing. As a basic example, take a directory, with tens of thousands of files, and attempt to do an ls in it, even unsorted (-U) it will take a long time, simply because traversal of the filesystem takes time. Requiring higher versions isnt going to miraculously speed up the iteration of directories.
On a similar note, when I referenced *innodb_stats_on_metadata *in earlier mentioned bug report, I was using it as an example of why relying on IS can be as bad, or worse than directly doing a show tables, since it can potentially end up doing additional work. I am pretty worried that modifying the variable was considered as a potential solution. The description of that very setting describes how it affects how the system handles IS queries. An administrator should be able to trust that installing something like PMA is not going to change any system variables that could otherwise affect the operation of the system
Thank you for your consideration.
Regards, Martyn
Thanks Martyn for your report. We'll discuss this issue before taking a decision about 4.1.0-rc1.
Sadly, 4.1.0 will be released with some bugs in it, but we should avoid regressions. So, in your opinion, is the current state of 4.1.0-beta worse than 4.0.9, regarding this IS issue?
Hi Marc,
4.1.0 is no worse in this regard than 4.0.x, however given that the performance issues appeared to be the primary reason for the dropping of 5.1 in 4.1 (at least that was the implications from postings in the discussions of the show databases command and disableIS issues), I thought it important to express that, not just so the underlying issue can be resolved for MySQL versions 5.5 and above, but also because a number of distros still ship MySQL 5.1 as its default version, as do multiple shared hosting providers and that keeping 5.1 support would probably benefit a reasonable percentage of your userbase.
Unfortunately due to time constraints, offering a fully working solution is not something I am able to do at present, however I can provide my (admittedly horrifically hacky) solution that solves the most severe issue experienced.
I added code to getData (libraries/navigation/Nodes/Node.class.php) which is provided at the close of this message. The purpose of this was to simply return a query object that contained the static information based on the only_db contents. This way, the related node building functions did not also need to be edited. For the start of a more complete solution, I would add a check in there to make sure only_db is actually set before building it, otherwise having it fall back over to the non database code, as the code provided will fail if only_db is not set.
The only downside to this base approach is that if an only_db database does not exist, it will still display on the navigation (whereas currently it would not), however its children of course wont as they still use the classic checks. It might however be an opportunity to display that fact, so users who use only_db are informed they have non existent databases in their list.
Following is probably some of the most horrible code to hit the mailing list, but hopefully it gives someone who is more familiar with the PMA codebase an idea. I shall also keep an eye on the mailing list in case I get the time to offer something a little more substantial.
if ($type == 'databases') { $db_list = $GLOBALS['cfg']['Server']['only_db']; $query = "SELECT * FROM ( SELECT '";
if (is_string($db_list)) { $db_list = array($db_list); }
if (count($db_list)) { $query .= implode("' UNION ALL SELECT '",$db_list); $query .= "' "; } return $GLOBALS['dbi']->fetchResult($query . ") as alias"); } else { // ORIGINAL getData CODE }
On Wed, Nov 20, 2013 at 4:42 AM, Marc Delisle marc@infomarc.info wrote:
Martyn Dale a écrit :
Hi,
Personally I do not think it is advisable to move forward until certain issues have been addressed. I am talking about https://sourceforge.net/p/phpmyadmin/bugs/3944 and related bug reports.
The alleged solution for 4.1 is requiring MySQL 5.5 as a minimum version. This is not going to solve the problems that people have reported when using the 4.x series with large systems, since while 5.5 is better than
5.1
and below, a full directory scan can still be triggered by any number of combinations of variables as described here:
http://dev.mysql.com/doc/refman/5.6/en/information-schema-optimization.html
I explained this in that bug report where i showed PMAs default usage of
IS
for the navigation on a *5.6* system (with innodb_stats_on_metadata switched off). While that was just a 2 second example, I have seen that
get
up to 60 seconds before query killers kick in. Reports of users on shared hosting accounts, where this kind of behavior can get users accounts terminated has also already been reported ( https://sourceforge.net/p/phpmyadmin/bugs/3945, second page "Here is
what
one provider sent as a warning before *deleting my account*")
If you remove the use of disableIS and related functionality, it will not solve any issues. Take the SCHEMATA queries for example for getting database lists, this still effectively does a full directory scan, as documented: http://dev.mysql.com/doc/refman/5.6/en/schemata-table.html In this case for example, switching to IS to avoid using the slow performing show databases is redundant since they basically do the same thing. As a basic example, take a directory, with tens of thousands of files, and attempt to do an ls in it, even unsorted (-U) it will take a long time, simply because traversal of the filesystem takes time.
Requiring
higher versions isnt going to miraculously speed up the iteration of directories.
On a similar note, when I referenced *innodb_stats_on_metadata *in
earlier
mentioned bug report, I was using it as an example of why relying on IS
can
be as bad, or worse than directly doing a show tables, since it can potentially end up doing additional work. I am pretty worried that modifying the variable was considered as a potential solution. The description of that very setting describes how it affects how the system handles IS queries. An administrator should be
able
to trust that installing something like PMA is not going to change any system variables that could otherwise affect the operation of the system
Thank you for your consideration.
Regards, Martyn
Thanks Martyn for your report. We'll discuss this issue before taking a decision about 4.1.0-rc1.
Sadly, 4.1.0 will be released with some bugs in it, but we should avoid regressions. So, in your opinion, is the current state of 4.1.0-beta worse than 4.0.9, regarding this IS issue?
-- Marc Delisle http://infomarc.info
Shape the Mobile Experience: Free Subscription Software experts and developers: Be at the forefront of tech innovation. Intel(R) Software Adrenaline delivers strategic insight and game-changing conversations that shape the rapidly evolving mobile landscape. Sign up now. http://pubads.g.doubleclick.net/gampad/clk?id=63431311&iu=/4140/ostg.clk... _______________________________________________ Phpmyadmin-devel mailing list Phpmyadmin-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/phpmyadmin-devel
Martyn Dale a écrit :
Hi Marc,
4.1.0 is no worse in this regard than 4.0.x, however given that the performance issues appeared to be the primary reason for the dropping of 5.1 in 4.1 (at least that was the implications from postings in the discussions of the show databases command and disableIS issues), I thought it important to express that, not just so the underlying issue can be resolved for MySQL versions 5.5 and above, but also because a number of distros still ship MySQL 5.1 as its default version, as do multiple shared hosting providers and that keeping 5.1 support would probably benefit a reasonable percentage of your userbase.
Unfortunately due to time constraints, offering a fully working solution is not something I am able to do at present, however I can provide my (admittedly horrifically hacky) solution that solves the most severe issue experienced.
I added code to getData (libraries/navigation/Nodes/Node.class.php) which is provided at the close of this message. The purpose of this was to simply return a query object that contained the static information based on the only_db contents. This way, the related node building functions did not also need to be edited. For the start of a more complete solution, I would add a check in there to make sure only_db is actually set before building it, otherwise having it fall back over to the non database code, as the code provided will fail if only_db is not set.
The only downside to this base approach is that if an only_db database does not exist, it will still display on the navigation (whereas currently it would not), however its children of course wont as they still use the classic checks. It might however be an opportunity to display that fact, so users who use only_db are informed they have non existent databases in their list.
Following is probably some of the most horrible code to hit the mailing list, but hopefully it gives someone who is more familiar with the PMA codebase an idea. I shall also keep an eye on the mailing list in case I get the time to offer something a little more substantial.
Thanks Martyn, I have committed this code in your name: https://github.com/phpmyadmin/phpmyadmin/commit/30d9af4789fafdf766ea2c3eb0ea...
adding a check for the existence of only_db as you suggested. This way we can all test the behavior of your suggestion.
About the minimum MySQL version supported (5.5), as discussed on this list, "Considering that MySQL 5.5 has reached the "Generally Available" status on 2010-12-03 with version 5.5.8" we now mention that this is the minimum version supported. This MySQL was released three years ago.
Note that the code does not block your using phpMyAdmin 4.1.x with MySQL 5.1, but it's not supported (meaning no tests, no fixes for 5.1-specific issues, etc).
I would probably add a to-do of sorts to improve on my method, I just implemented it in that way for completion speed and not needing to affect surrounding code. Sending a query to MySQL which just returns a static list is probably too hacky to be considered production worthy. Some similar code would probably be best placed in the higher levels of the navigation tree builder which does not even attempt to call this function and instead return the array at that point. I hope that time will permit me to optimize my own hacks but it shouldn't be something to be relied on.
Thank you.
On Thu, Nov 21, 2013 at 10:12 AM, Marc Delisle marc@infomarc.info wrote:
Martyn Dale a écrit :
Hi Marc,
4.1.0 is no worse in this regard than 4.0.x, however given that the performance issues appeared to be the primary reason for the dropping of 5.1 in 4.1 (at least that was the implications from postings in the discussions of the show databases command and disableIS issues), I
thought
it important to express that, not just so the underlying issue can be resolved for MySQL versions 5.5 and above, but also because a number of distros still ship MySQL 5.1 as its default version, as do multiple
shared
hosting providers and that keeping 5.1 support would probably benefit a reasonable percentage of your userbase.
Unfortunately due to time constraints, offering a fully working solution
is
not something I am able to do at present, however I can provide my (admittedly horrifically hacky) solution that solves the most severe
issue
experienced.
I added code to getData (libraries/navigation/Nodes/Node.class.php) which is provided at the close of this message. The purpose of this was to
simply
return a query object that contained the static information based on the only_db contents. This way, the related node building functions did not also need to be edited. For the start of a more complete solution, I
would
add a check in there to make sure only_db is actually set before building it, otherwise having it fall back over to the non database code, as the code provided will fail if only_db is not set.
The only downside to this base approach is that if an only_db database
does
not exist, it will still display on the navigation (whereas currently it would not), however its children of course wont as they still use the classic checks. It might however be an opportunity to display that fact,
so
users who use only_db are informed they have non existent databases in their list.
Following is probably some of the most horrible code to hit the mailing list, but hopefully it gives someone who is more familiar with the PMA codebase an idea. I shall also keep an eye on the mailing list in case I get the time to offer something a little more substantial.
Thanks Martyn, I have committed this code in your name:
https://github.com/phpmyadmin/phpmyadmin/commit/30d9af4789fafdf766ea2c3eb0ea...
adding a check for the existence of only_db as you suggested. This way we can all test the behavior of your suggestion.
About the minimum MySQL version supported (5.5), as discussed on this list, "Considering that MySQL 5.5 has reached the "Generally Available" status on 2010-12-03 with version 5.5.8" we now mention that this is the minimum version supported. This MySQL was released three years ago.
Note that the code does not block your using phpMyAdmin 4.1.x with MySQL 5.1, but it's not supported (meaning no tests, no fixes for 5.1-specific issues, etc).
-- Marc Delisle http://infomarc.info
Shape the Mobile Experience: Free Subscription Software experts and developers: Be at the forefront of tech innovation. Intel(R) Software Adrenaline delivers strategic insight and game-changing conversations that shape the rapidly evolving mobile landscape. Sign up now. http://pubads.g.doubleclick.net/gampad/clk?id=63431311&iu=/4140/ostg.clk... _______________________________________________ Phpmyadmin-devel mailing list Phpmyadmin-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/phpmyadmin-devel
-
J.M. . -
Marc Delisle -
Martyn Dale