[Phpmyadmin-devel] #4536 - master: import problem (PMA_String)
Hi everyone, It seems that since that commit [1], the master is quite slow to import data. This seems to be linked to multi-bytes functions that are really slower compared to standard string functions. I tried several variants of this, but without a good result: instead of always using PMA_StringMB, try to detect encoding and so use PMA_StringNative when possible. To improve this, I cached the encoding, to avoid to detect it each time. See [2] (this is not totally safe, here, for the tests, I removed the mb_* detection). But the result is not as good as before… I also tried something more experimental: convert all the strings used in PMA_String to UTF-8 string and so, mb_* functions won't convert anymore. But it is not effective either. Do you have any idea about the way to improve it please? I thought about generalising the strpos to a strallpos/strposall (for redundant strpos) with an explicit encoding, not to let PHP convert each time. Thanks for your help, [1] https://github.com/phpmyadmin/phpmyadmin/commit/9b77d746aba [2] https://github.com/Tithugues/phpmyadmin/commit/ab6f493449d90e58bd4caa15740d8... Hugues.
Le 2014-09-20 14:16, Hugues Peccatte a écrit :
Hi everyone,
It seems that since that commit [1], the master is quite slow to import data. This seems to be linked to multi-bytes functions that are really slower compared to standard string functions.
I tried several variants of this, but without a good result: instead of always using PMA_StringMB, try to detect encoding and so use PMA_StringNative when possible. To improve this, I cached the encoding, to avoid to detect it each time. See [2] (this is not totally safe, here, for the tests, I removed the mb_* detection). But the result is not as good as before… I also tried something more experimental: convert all the strings used in PMA_String to UTF-8 string and so, mb_* functions won't convert anymore. But it is not effective either.
Do you have any idea about the way to improve it please? I thought about generalising the strpos to a strallpos/strposall (for redundant strpos) with an explicit encoding, not to let PHP convert each time.
Thanks for your help,
[1] https://github.com/phpmyadmin/phpmyadmin/commit/9b77d746aba [2] https://github.com/Tithugues/phpmyadmin/commit/ab6f493449d90e58bd4caa15740d8...
Hugues.
Hi Hugues, I have not looked deep into this logic, so it seems that you've become the expert here in these matters. Taking into account that the current master is not acceptable for a 4.3.0-alpha release, I see a few choices: - remove the mb modifications from the import logic - remove the current parser from the import logic, therefore removing support for things like a custom delimiter and probably other things (import of compressed files?) - delay 4.3.0 until we find the correct solution with mb -- Marc Delisle | phpMyAdmin
Le 2014-09-20 21:05, Marc Delisle a écrit :
Le 2014-09-20 14:16, Hugues Peccatte a écrit :
Hi everyone,
It seems that since that commit [1], the master is quite slow to import data. This seems to be linked to multi-bytes functions that are really slower compared to standard string functions.
I tried several variants of this, but without a good result: instead of always using PMA_StringMB, try to detect encoding and so use PMA_StringNative when possible. To improve this, I cached the encoding, to avoid to detect it each time. See [2] (this is not totally safe, here, for the tests, I removed the mb_* detection). But the result is not as good as before… I also tried something more experimental: convert all the strings used in PMA_String to UTF-8 string and so, mb_* functions won't convert anymore. But it is not effective either.
Do you have any idea about the way to improve it please? I thought about generalising the strpos to a strallpos/strposall (for redundant strpos) with an explicit encoding, not to let PHP convert each time.
Thanks for your help,
[1] https://github.com/phpmyadmin/phpmyadmin/commit/9b77d746aba [2] https://github.com/Tithugues/phpmyadmin/commit/ab6f493449d90e58bd4caa15740d8...
Hugues.
Hi Hugues,
I have not looked deep into this logic, so it seems that you've become the expert here in these matters.
Taking into account that the current master is not acceptable for a 4.3.0-alpha release, I see a few choices:
- remove the mb modifications from the import logic
- remove the current parser from the import logic, therefore removing support for things like a custom delimiter and probably other things (import of compressed files?)
- delay 4.3.0 until we find the correct solution with mb
We could also add another (!) custom option in the import dialog: multi-byte or not. The multi-byte way is the more correct one for importing files with multi-byte characters, but simply does not work for big files (10 to 15 times slower). So by default, the option could be set to not use the multi-byte way. A user with a big multi-byte file would have a problem, unless she is allowed to set the PHP execution time limit to huge values (which not many sysadmins will allow unless they want their shared server to perform badly). -- Marc Delisle | phpMyAdmin
2014-09-21 12:20 GMT+02:00 Marc Delisle <marc@infomarc.info>:
Le 2014-09-20 21:05, Marc Delisle a écrit :
Hi Hugues,
I have not looked deep into this logic, so it seems that you've become the expert here in these matters.
Taking into account that the current master is not acceptable for a 4.3.0-alpha release, I see a few choices:
- remove the mb modifications from the import logic
This one would be the easiest.
- remove the current parser from the import logic, therefore removing support for things like a custom delimiter and probably other things (import of compressed files?)
The issue isn't about compressed files. I tried to import the SQL into your zip file: it was too long…
- delay 4.3.0 until we find the correct solution with mb
I don't know the delay to find a solution, so not sure that would be enough.
We could also add another (!) custom option in the import dialog: multi-byte or not.
This might be possible… I'll do some tests about choosing the PMA_String* to use.
The multi-byte way is the more correct one for importing files with multi-byte characters, but simply does not work for big files (10 to 15 times slower). So by default, the option could be set to not use the multi-byte way.
A user with a big multi-byte file would have a problem, unless she is allowed to set the PHP execution time limit to huge values (which not many sysadmins will allow unless they want their shared server to perform badly).
-- Marc Delisle | phpMyAdmin
Thanks for your feedback. Hugues.
2014-09-21 16:30 GMT+02:00 Hugues Peccatte <hugues.peccatte@gmail.com>:
2014-09-21 12:20 GMT+02:00 Marc Delisle <marc@infomarc.info>:
Le 2014-09-20 21:05, Marc Delisle a écrit :
Hi Hugues,
I have not looked deep into this logic, so it seems that you've become the expert here in these matters.
Taking into account that the current master is not acceptable for a 4.3.0-alpha release, I see a few choices:
- remove the mb modifications from the import logic
This one would be the easiest.
This one isn't possible. Even with undoing all modifications in this function, the execution is too long.
- remove the current parser from the import logic, therefore removing support for things like a custom delimiter and probably other things (import of compressed files?)
The issue isn't about compressed files. I tried to import the SQL into your zip file: it was too long…
- delay 4.3.0 until we find the correct solution with mb
I don't know the delay to find a solution, so not sure that would be enough.
We could also add another (!) custom option in the import dialog: multi-byte or not.
This might be possible… I'll do some tests about choosing the PMA_String* to use.
So, this is not possible either.
The multi-byte way is the more correct one for importing files with multi-byte characters, but simply does not work for big files (10 to 15 times slower). So by default, the option could be set to not use the multi-byte way.
A user with a big multi-byte file would have a problem, unless she is allowed to set the PHP execution time limit to huge values (which not many sysadmins will allow unless they want their shared server to perform badly).
-- Marc Delisle | phpMyAdmin
Thanks for your feedback.
Hugues.
I still try to find the greedy part of the script. Hugues.
Hugues Peccatte a écrit :
2014-09-21 16:30 GMT+02:00 Hugues Peccatte <hugues.peccatte@gmail.com>:
2014-09-21 12:20 GMT+02:00 Marc Delisle <marc@infomarc.info>:
Le 2014-09-20 21:05, Marc Delisle a écrit :
Hi Hugues,
I have not looked deep into this logic, so it seems that you've become the expert here in these matters.
Taking into account that the current master is not acceptable for a 4.3.0-alpha release, I see a few choices:
- remove the mb modifications from the import logic This one would be the easiest.
This one isn't possible. Even with undoing all modifications in this function, the execution is too long.
Hugues, do you still have the branch where each commit was separate, before squashing?
- remove the current parser from the import logic, therefore removing support for things like a custom delimiter and probably other things (import of compressed files?) The issue isn't about compressed files. I tried to import the SQL into your zip file: it was too long…
- delay 4.3.0 until we find the correct solution with mb I don't know the delay to find a solution, so not sure that would be enough.
We could also add another (!) custom option in the import dialog: multi-byte or not.
This might be possible… I'll do some tests about choosing the PMA_String* to use.
So, this is not possible either.
The multi-byte way is the more correct one for importing files with multi-byte characters, but simply does not work for big files (10 to 15 times slower). So by default, the option could be set to not use the multi-byte way.
A user with a big multi-byte file would have a problem, unless she is allowed to set the PHP execution time limit to huge values (which not many sysadmins will allow unless they want their shared server to perform badly).
-- Marc Delisle | phpMyAdmin
Thanks for your feedback.
Hugues.
I still try to find the greedy part of the script.
Hugues.
------------------------------------------------------------------------
------------------------------------------------------------------------------ Meet PCI DSS 3.0 Compliance Requirements with EventLog Analyzer Achieve PCI DSS 3.0 Compliant Status with Out-of-the-box PCI DSS Reports Are you Audit-Ready for PCI DSS 3.0 Compliance? Download White paper Comply to PCI DSS 3.0 Requirement 10 and 11.5 with EventLog Analyzer http://pubads.g.doubleclick.net/gampad/clk?id=154622311&iu=/4140/ostg.clktrk
------------------------------------------------------------------------
_______________________________________________ Phpmyadmin-devel mailing list Phpmyadmin-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/phpmyadmin-devel
-- Marc Delisle (phpMyAdmin)
2014-09-22 18:44 GMT+02:00 Marc Delisle <marc@infomarc.info>:
Hugues, do you still have the branch where each commit was separate, before squashing? -- Marc Delisle (phpMyAdmin)
With "git reflog", I've been able to get it back: git checkout -b memory 972e038 What do you have in mind? Hugues.
2014-09-22 21:08 GMT+02:00 Hugues Peccatte <hugues.peccatte@gmail.com>:
2014-09-22 18:44 GMT+02:00 Marc Delisle <marc@infomarc.info>:
Hugues, do you still have the branch where each commit was separate, before squashing? -- Marc Delisle (phpMyAdmin)
With "git reflog", I've been able to get it back: git checkout -b memory 972e038
What do you have in mind?
Hugues.
I did a simple test: * run substr 100000 times * run $obj->substr 100000 times (where $obj->substr simply calls substr) The second one is 3 times longer than the first one. The simple fact to use an object is too long. I tried something else: overwrite the functions. It's better: 2 times longer that the first one, but still slower… That means that importing your file would take ~24 seconds instead of ~12 seconds. Should we think about defining "pmaSubstr", "pmaStrlen", etc functions and use it everywhere? Hugues.
Hugues Peccatte a écrit :
2014-09-22 21:08 GMT+02:00 Hugues Peccatte <hugues.peccatte@gmail.com>:
2014-09-22 18:44 GMT+02:00 Marc Delisle <marc@infomarc.info>:
Hugues, do you still have the branch where each commit was separate, before squashing? -- Marc Delisle (phpMyAdmin)
With "git reflog", I've been able to get it back: git checkout -b memory 972e038
What do you have in mind?
Hugues.
I did a simple test: * run substr 100000 times * run $obj->substr 100000 times (where $obj->substr simply calls substr)
The second one is 3 times longer than the first one. The simple fact to use an object is too long. I tried something else: overwrite the functions. It's better: 2 times longer that the first one, but still slower…
That means that importing your file would take ~24 seconds instead of ~12 seconds.
Should we think about defining "pmaSubstr", "pmaStrlen", etc functions and use it everywhere?
Do you mean using functions without objects? -- Marc Delisle (phpMyAdmin)
2014-09-23 16:23 GMT+02:00 Marc Delisle <marc@infomarc.info>:
Hugues Peccatte a écrit :
2014-09-22 21:08 GMT+02:00 Hugues Peccatte <hugues.peccatte@gmail.com>:
2014-09-22 18:44 GMT+02:00 Marc Delisle <marc@infomarc.info>:
Hugues, do you still have the branch where each commit was separate, before squashing? -- Marc Delisle (phpMyAdmin)
With "git reflog", I've been able to get it back: git checkout -b memory 972e038
What do you have in mind?
Hugues.
I did a simple test: * run substr 100000 times * run $obj->substr 100000 times (where $obj->substr simply calls substr)
The second one is 3 times longer than the first one. The simple fact to use an object is too long. I tried something else: overwrite the functions. It's better: 2 times longer that the first one, but still slower…
That means that importing your file would take ~24 seconds instead of ~12 seconds.
Should we think about defining "pmaSubstr", "pmaStrlen", etc functions and use it everywhere?
Do you mean using functions without objects?
-- Marc Delisle (phpMyAdmin)
That's it. Instead of using $pmaString->strlen or strlen, we could imagine to use pmaStrlen and this function would call strlen or mb_strlen if mb_* functions exist. Another way could be to overwrite strlen with mb_strlen, but we wouldn't be able to use the original strlen if needed. So I think that we should define new functions. I'll try to do a mass replacement and check if the execution time is better. Hugues.
2014-09-23 18:49 GMT+02:00 Hugues Peccatte <hugues.peccatte@gmail.com>:
2014-09-23 16:23 GMT+02:00 Marc Delisle <marc@infomarc.info>:
Hugues Peccatte a écrit :
2014-09-22 21:08 GMT+02:00 Hugues Peccatte <hugues.peccatte@gmail.com>:
2014-09-22 18:44 GMT+02:00 Marc Delisle <marc@infomarc.info>:
Hugues, do you still have the branch where each commit was separate, before squashing? -- Marc Delisle (phpMyAdmin)
With "git reflog", I've been able to get it back: git checkout -b memory 972e038
What do you have in mind?
Hugues.
I did a simple test: * run substr 100000 times * run $obj->substr 100000 times (where $obj->substr simply calls substr)
The second one is 3 times longer than the first one. The simple fact to use an object is too long. I tried something else: overwrite the functions. It's better: 2 times longer that the first one, but still slower…
That means that importing your file would take ~24 seconds instead of ~12 seconds.
Should we think about defining "pmaSubstr", "pmaStrlen", etc functions and use it everywhere?
Do you mean using functions without objects?
-- Marc Delisle (phpMyAdmin)
That's it. Instead of using $pmaString->strlen or strlen, we could imagine to use pmaStrlen and this function would call strlen or mb_strlen if mb_* functions exist. Another way could be to overwrite strlen with mb_strlen, but we wouldn't be able to use the original strlen if needed. So I think that we should define new functions.
I'll try to do a mass replacement and check if the execution time is better.
Hugues.
I had a better execution time with functions instead of objects [1]. My server is configured to have a maximum execution time of 300 seconds/5 minutes. And I wasn't able to import all the Marc's SQL file. But now, I can import the whole file into 3.5 minutes. We're still far of the ~12 seconds, but this is better… Two other ideas to improve it again: * Use runkit [2], but it would add a dependency, so I'm not really for this solution * In PHP ≥ 5.6, we'll be able to use this feature [3] So maybe I'll try to look for the slower part and try to improve it… I'll also try to find the commit which slows the script, as suggested by Marc. But this could be hard, because the commits between the first one and the last one are not really stables… If anyone has another idea… Thanks, [1] https://github.com/Tithugues/phpmyadmin/tree/stringFunctions_master [2] http://php.net/manual/en/function.runkit-function-copy.php [3] http://php.net/manual/en/language.namespaces.importing.php Hugues.
Hugues Peccatte a écrit :
2014-09-23 16:23 GMT+02:00 Marc Delisle <marc@infomarc.info>:
Hugues Peccatte a écrit :
2014-09-22 21:08 GMT+02:00 Hugues Peccatte <hugues.peccatte@gmail.com>:
2014-09-22 18:44 GMT+02:00 Marc Delisle <marc@infomarc.info>:
Hugues, do you still have the branch where each commit was separate, before squashing? -- Marc Delisle (phpMyAdmin)
With "git reflog", I've been able to get it back: git checkout -b memory 972e038
What do you have in mind?
Hugues.
I did a simple test: * run substr 100000 times * run $obj->substr 100000 times (where $obj->substr simply calls substr)
The second one is 3 times longer than the first one. The simple fact to use an object is too long. I tried something else: overwrite the functions. It's better: 2 times longer that the first one, but still slower…
That means that importing your file would take ~24 seconds instead of ~12 seconds.
Should we think about defining "pmaSubstr", "pmaStrlen", etc functions and use it everywhere? Do you mean using functions without objects?
-- Marc Delisle (phpMyAdmin)
That's it. Instead of using $pmaString->strlen or strlen, we could imagine to use pmaStrlen and this function would call strlen or mb_strlen if mb_* functions exist. Another way could be to overwrite strlen with mb_strlen, but we wouldn't be able to use the original strlen if needed. So I think that we should define new functions.
What I don't like is that we would have two different syntaxes to call string functions. We would need comments in the calling code to explain why we are using a way and not another way.
I'll try to do a mass replacement and check if the execution time is better.
Hugues.
-- Marc Delisle (phpMyAdmin)
2014-09-23 20:04 GMT+02:00 Marc Delisle <marc@infomarc.info>:
Hugues Peccatte a écrit :
2014-09-23 16:23 GMT+02:00 Marc Delisle <marc@infomarc.info>:
That's it. Instead of using $pmaString->strlen or strlen, we could imagine to use pmaStrlen and this function would call strlen or mb_strlen if mb_* functions exist. Another way could be to overwrite strlen with mb_strlen, but we wouldn't be able to use the original strlen if needed. So I think that we should define new functions.
What I don't like is that we would have two different syntaxes to call string functions. We would need comments in the calling code to explain why we are using a way and not another way.
I'll try to do a mass replacement and check if the execution time is
better.
Hugues.
-- Marc Delisle (phpMyAdmin)
If by "two ways", you think about PMA_String and pmaStr* functions, there should be a misunderstanding. I don't want to keep PMA_String (at least, not for string functions), I'd like to migrate it into functions. Here, I kept it not to break anything, just to check performance. If you think about pmaStr* functions and standard PHP functions, I don't think this is a real issue, because we use PMA string object almost everywhere today, so it would be replaced by function calls. Hugues.
Hugues Peccatte a écrit :
2014-09-23 20:04 GMT+02:00 Marc Delisle <marc@infomarc.info>:
Hugues Peccatte a écrit :
2014-09-23 16:23 GMT+02:00 Marc Delisle <marc@infomarc.info>:
That's it. Instead of using $pmaString->strlen or strlen, we could imagine to use pmaStrlen and this function would call strlen or mb_strlen if mb_* functions exist. Another way could be to overwrite strlen with mb_strlen, but we wouldn't be able to use the original strlen if needed. So I think that we should define new functions. What I don't like is that we would have two different syntaxes to call string functions. We would need comments in the calling code to explain why we are using a way and not another way.
I'll try to do a mass replacement and check if the execution time is better. Hugues. -- Marc Delisle (phpMyAdmin)
If by "two ways", you think about PMA_String and pmaStr* functions, there should be a misunderstanding. I don't want to keep PMA_String (at least, not for string functions), I'd like to migrate it into functions. Here, I kept it not to break anything, just to check performance. If you think about pmaStr* functions and standard PHP functions, I don't think this is a real issue, because we use PMA string object almost everywhere today, so it would be replaced by function calls.
Hugues.
I believe it's a good idea to use just functions, but other developers might think otherwise. -- Marc Delisle (phpMyAdmin)
Hi, On Wed, Sep 24, 2014 at 5:13 PM, Marc Delisle <marc@infomarc.info> wrote:
Hugues Peccatte a écrit :
2014-09-23 20:04 GMT+02:00 Marc Delisle <marc@infomarc.info>:
Hugues Peccatte a écrit :
2014-09-23 16:23 GMT+02:00 Marc Delisle <marc@infomarc.info>:
That's it. Instead of using $pmaString->strlen or strlen, we could imagine to use pmaStrlen and this function would call strlen or mb_strlen if mb_* functions exist. Another way could be to overwrite strlen with mb_strlen, but we wouldn't be able to use the original strlen if needed. So I think that we should define new functions. What I don't like is that we would have two different syntaxes to call string functions. We would need comments in the calling code to explain why we are using a way and not another way.
I'll try to do a mass replacement and check if the execution time is better. Hugues. -- Marc Delisle (phpMyAdmin)
If by "two ways", you think about PMA_String and pmaStr* functions, there should be a misunderstanding. I don't want to keep PMA_String (at least, not for string functions), I'd like to migrate it into functions. Here, I kept it not to break anything, just to check performance. If you think about pmaStr* functions and standard PHP functions, I don't think this is a real issue, because we use PMA string object almost everywhere today, so it would be replaced by function calls.
Hugues.
I believe it's a good idea to use just functions, but other developers might think otherwise.
In that case, how will function decide whether to use mb_* function or simple one? What if we use static methods?
-- Marc Delisle (phpMyAdmin)
-- Regards, Chirayu Chiripal https://chirayuchiripal.wordpress.com/
Chirayu Chiripal a écrit :
Hi,
On Wed, Sep 24, 2014 at 5:13 PM, Marc Delisle <marc@infomarc.info> wrote:
Hugues Peccatte a écrit :
2014-09-23 20:04 GMT+02:00 Marc Delisle <marc@infomarc.info>:
Hugues Peccatte a écrit :
2014-09-23 16:23 GMT+02:00 Marc Delisle <marc@infomarc.info>:
That's it. Instead of using $pmaString->strlen or strlen, we could imagine to use pmaStrlen and this function would call strlen or mb_strlen if mb_* functions exist. Another way could be to overwrite strlen with mb_strlen, but we wouldn't be able to use the original strlen if needed. So I think that we should define new functions. What I don't like is that we would have two different syntaxes to call string functions. We would need comments in the calling code to explain why we are using a way and not another way.
I'll try to do a mass replacement and check if the execution time is better. Hugues. -- Marc Delisle (phpMyAdmin)
If by "two ways", you think about PMA_String and pmaStr* functions, there should be a misunderstanding. I don't want to keep PMA_String (at least, not for string functions), I'd like to migrate it into functions. Here, I kept it not to break anything, just to check performance. If you think about pmaStr* functions and standard PHP functions, I don't think this is a real issue, because we use PMA string object almost everywhere today, so it would be replaced by function calls.
Hugues. I believe it's a good idea to use just functions, but other developers might think otherwise.
In that case, how will function decide whether to use mb_* function or simple one? What if we use static methods?
In the current implementation, the mb_* functions are preferred; so we would use the same priority, to define functions like PMA_substr(). -- Marc Delisle (phpMyAdmin)
On Wed, Sep 24, 2014 at 5:35 PM, Marc Delisle <marc@infomarc.info> wrote:
Chirayu Chiripal a écrit :
Hi,
On Wed, Sep 24, 2014 at 5:13 PM, Marc Delisle <marc@infomarc.info> wrote:
Hugues Peccatte a écrit :
2014-09-23 20:04 GMT+02:00 Marc Delisle <marc@infomarc.info>:
Hugues Peccatte a écrit :
2014-09-23 16:23 GMT+02:00 Marc Delisle <marc@infomarc.info>:
That's it. Instead of using $pmaString->strlen or strlen, we could imagine to use pmaStrlen and this function would call strlen or mb_strlen if mb_* functions exist. Another way could be to overwrite strlen with mb_strlen, but we wouldn't be able to use the original strlen if needed. So I think that we should define new functions. What I don't like is that we would have two different syntaxes to call string functions. We would need comments in the calling code to explain why we are using a way and not another way.
I'll try to do a mass replacement and check if the execution time is better. Hugues. -- Marc Delisle (phpMyAdmin)
If by "two ways", you think about PMA_String and pmaStr* functions, there should be a misunderstanding. I don't want to keep PMA_String (at least, not for string functions), I'd like to migrate it into functions. Here, I kept it not to break anything, just to check performance. If you think about pmaStr* functions and standard PHP functions, I don't think this is a real issue, because we use PMA string object almost everywhere today, so it would be replaced by function calls.
Hugues. I believe it's a good idea to use just functions, but other developers might think otherwise.
In that case, how will function decide whether to use mb_* function or simple one? What if we use static methods?
In the current implementation, the mb_* functions are preferred; so we would use the same priority, to define functions like PMA_substr().
For what it's worth I did some profiling of the import functionality and here is what I observed. If we consider the substring functionality Using the native functions wrapped in OOP function calls, php::substr took 0.34 micro secs on average (2.7ms for 8308 calls) PMA_StringNative->substr took 13.26 micro secs on average (110ms for 8295 calls) PMS_String->substr took 28.93 micro secs on average (244ms for 8294 calls) Using the mb functions wrapped in OOP function calls, php::mb_substr took 22 micro secs on average (220ms for 9700 calls) PMA_StringMB->substr took 120.54 micro secs on average (1170ms for 9706 calls) PMS_String->substr took 115.50 micro secs on average (958ms for 8294 calls) So it seems like both factors (using mb functions as well as object orientation) are responsible for the slowness. See the attached screenshots too. -- Thanks and Regards, Madhura Jayaratne
On Wed, Sep 24, 2014 at 5:54 PM, Madhura Jayaratne <madhura.cj@gmail.com> wrote:
On Wed, Sep 24, 2014 at 5:35 PM, Marc Delisle <marc@infomarc.info> wrote:
Chirayu Chiripal a écrit :
Hi,
On Wed, Sep 24, 2014 at 5:13 PM, Marc Delisle <marc@infomarc.info> wrote:
Hugues Peccatte a écrit :
2014-09-23 20:04 GMT+02:00 Marc Delisle <marc@infomarc.info>:
Hugues Peccatte a écrit : > 2014-09-23 16:23 GMT+02:00 Marc Delisle <marc@infomarc.info>: > > That's it. Instead of using $pmaString->strlen or strlen, we could imagine > to use pmaStrlen and this function would call strlen or mb_strlen if mb_* > functions exist. > Another way could be to overwrite strlen with mb_strlen, but we wouldn't be > able to use the original strlen if needed. So I think that we should define > new functions. What I don't like is that we would have two different syntaxes to call string functions. We would need comments in the calling code to explain why we are using a way and not another way.
> I'll try to do a mass replacement and check if the execution time is better. > Hugues. -- Marc Delisle (phpMyAdmin)
If by "two ways", you think about PMA_String and pmaStr* functions, there should be a misunderstanding. I don't want to keep PMA_String (at least, not for string functions), I'd like to migrate it into functions. Here, I kept it not to break anything, just to check performance. If you think about pmaStr* functions and standard PHP functions, I don't think this is a real issue, because we use PMA string object almost everywhere today, so it would be replaced by function calls.
Hugues. I believe it's a good idea to use just functions, but other developers might think otherwise.
In that case, how will function decide whether to use mb_* function or simple one? What if we use static methods?
In the current implementation, the mb_* functions are preferred; so we would use the same priority, to define functions like PMA_substr().
For what it's worth I did some profiling of the import functionality and here is what I observed.
If we consider the substring functionality
Using the native functions wrapped in OOP function calls, php::substr took 0.34 micro secs on average (2.7ms for 8308 calls) PMA_StringNative->substr took 13.26 micro secs on average (110ms for 8295 calls) PMS_String->substr took 28.93 micro secs on average (244ms for 8294 calls)
Using the mb functions wrapped in OOP function calls, php::mb_substr took 22 micro secs on average (220ms for 9700 calls) PMA_StringMB->substr took 120.54 micro secs on average (1170ms for 9706 calls) PMS_String->substr took 115.50 micro secs on average (958ms for 8294 calls)
So it seems like both factors (using mb functions as well as object orientation) are responsible for the slowness.
See the attached screenshots too.
I was just wondering what will be the effect if we make current methods as static in those classes?
-- Thanks and Regards,
Madhura Jayaratne
-- Regards, Chirayu Chiripal https://chirayuchiripal.wordpress.com/
2014-09-24 14:42 GMT+02:00 Chirayu Chiripal <chirayu.chiripal@gmail.com>:
On Wed, Sep 24, 2014 at 5:54 PM, Madhura Jayaratne <madhura.cj@gmail.com> wrote:
On Wed, Sep 24, 2014 at 5:35 PM, Marc Delisle <marc@infomarc.info> wrote:
In the current implementation, the mb_* functions are preferred; so we would use the same priority, to define functions like PMA_substr().
Ok. I'll try to do this.
For what it's worth I did some profiling of the import functionality and here is what I observed.
It's worth it!
If we consider the substring functionality
Using the native functions wrapped in OOP function calls, php::substr took 0.34 micro secs on average (2.7ms for 8308 calls) PMA_StringNative->substr took 13.26 micro secs on average (110ms for 8295 calls) PMS_String->substr took 28.93 micro secs on average (244ms for 8294 calls)
Using the mb functions wrapped in OOP function calls, php::mb_substr took 22 micro secs on average (220ms for 9700 calls) PMA_StringMB->substr took 120.54 micro secs on average (1170ms for 9706 calls) PMS_String->substr took 115.50 micro secs on average (958ms for 8294 calls)
So it seems like both factors (using mb functions as well as object orientation) are responsible for the slowness.
Yes, we know than both makes the script slower. OOP can be avoided, but I don't think that we could avoid using mb functions. So this is our minimal constraint.
See the attached screenshots too.
I don't know this tool! What is it please? This seems to be very useful, at least for the benchmark I try to do currently.
I was just wondering what will be the effect if we make current methods as static in those classes?
Interesting question. :) I'll include it into my benchmark and give the result.
-- Thanks and Regards,
Madhura Jayaratne
-- Regards, Chirayu Chiripal https://chirayuchiripal.wordpress.com/
Thanks for all your feedbacks. Hugues.
On Wed, Sep 24, 2014 at 9:10 PM, Hugues Peccatte <hugues.peccatte@gmail.com> wrote:
2014-09-24 14:42 GMT+02:00 Chirayu Chiripal <chirayu.chiripal@gmail.com>:
On Wed, Sep 24, 2014 at 5:54 PM, Madhura Jayaratne <madhura.cj@gmail.com> wrote:
On Wed, Sep 24, 2014 at 5:35 PM, Marc Delisle <marc@infomarc.info> wrote:
In the current implementation, the mb_* functions are preferred; so we would use the same priority, to define functions like PMA_substr().
Ok. I'll try to do this.
For what it's worth I did some profiling of the import functionality and here is what I observed.
It's worth it!
If we consider the substring functionality
Using the native functions wrapped in OOP function calls, php::substr took 0.34 micro secs on average (2.7ms for 8308 calls) PMA_StringNative->substr took 13.26 micro secs on average (110ms for 8295 calls) PMS_String->substr took 28.93 micro secs on average (244ms for 8294 calls)
Using the mb functions wrapped in OOP function calls, php::mb_substr took 22 micro secs on average (220ms for 9700 calls) PMA_StringMB->substr took 120.54 micro secs on average (1170ms for 9706 calls) PMS_String->substr took 115.50 micro secs on average (958ms for 8294 calls)
So it seems like both factors (using mb functions as well as object orientation) are responsible for the slowness.
Yes, we know than both makes the script slower. OOP can be avoided, but I don't think that we could avoid using mb functions. So this is our minimal constraint.
See the attached screenshots too.
I don't know this tool! What is it please? This seems to be very useful, at least for the benchmark I try to do currently.
I used Xdebug profiler to profile and WinCacheGrind for Windows to
analyze the profiling report. KCacheGrind can be used in Linux environments. More details can be found at [1].
[1] http://www.xdebug.org/docs/profiler -- Thanks and Regards, Madhura Jayaratne
2014-09-24 18:32 GMT+02:00 Madhura Jayaratne <madhura.cj@gmail.com>:
I used Xdebug profiler to profile and WinCacheGrind for Windows to analyze the profiling report. KCacheGrind can be used in Linux environments. More details can be found at [1].
Thanks. I'm on Windows. I'll see how to use it. Hugues.
-- Thanks and Regards,
Madhura Jayaratne
2014-09-24 14:42 GMT+02:00 Chirayu Chiripal <chirayu.chiripal@gmail.com>:
I was just wondering what will be the effect if we make current methods as static in those classes?
-- Regards, Chirayu Chiripal https://chirayuchiripal.wordpress.com/
It's almost the same thing to use a static or a non-static function. This is 2.5x slower than standard PHP function. See: Time: 1.7506999969482 //standard PHP function Time: 4.3722500801086 //object Time: 4.3722498416901 //object with static function Time: 3.6722099781036 //PMA function, even this one is 2x slower than standard PHP function… Hugues.
On Thu, Sep 25, 2014 at 12:17 AM, Hugues Peccatte <hugues.peccatte@gmail.com
wrote:
2014-09-24 14:42 GMT+02:00 Chirayu Chiripal <chirayu.chiripal@gmail.com>:
I was just wondering what will be the effect if we make current methods as static in those classes?
-- Regards, Chirayu Chiripal https://chirayuchiripal.wordpress.com/
It's almost the same thing to use a static or a non-static function. This is 2.5x slower than standard PHP function.
See: Time: 1.7506999969482 //standard PHP function Time: 4.3722500801086 //object Time: 4.3722498416901 //object with static function Time: 3.6722099781036 //PMA function, even this one is 2x slower than standard PHP function…
Are these tests performed on mb_* function or non mb_* one? One thing is clear from these results is that even wrapping a built in function in another function (wrapper function) has some significant overhead. And using OOP/Classes is like wrapping that wrapper function further into a another wrapper causing further slowness. So why not just use mb_* function directly everywhere? If we do this, then what to do in case mb_* functions are not available? Maybe we can just define a new mb_* function to use respective non mb_* function for it. For e.g. // Define a function like this function mb_strlen($str) { return strlen($str); } // or create an alias of strlen() as mb_strlen() (I heard this is possible in PHP 5.6+) By doing this, we can get speed advantage of built in mb_* function whenever it is available by avoiding any wrapper function or class. And in cases where it is not available, a wrapper function with standard non mb function can deliver equivalent performance as non mb function is inherently faster than its mb_* mod.
Hugues.
2014-09-25 20:08 GMT+02:00 Chirayu Chiripal <chirayu.chiripal@gmail.com>:
On Thu, Sep 25, 2014 at 12:17 AM, Hugues Peccatte < hugues.peccatte@gmail.com> wrote:
2014-09-24 14:42 GMT+02:00 Chirayu Chiripal <chirayu.chiripal@gmail.com>:
I was just wondering what will be the effect if we make current methods as static in those classes?
-- Regards, Chirayu Chiripal https://chirayuchiripal.wordpress.com/
It's almost the same thing to use a static or a non-static function. This is 2.5x slower than standard PHP function.
See: Time: 1.7506999969482 //standard PHP function Time: 4.3722500801086 //object Time: 4.3722498416901 //object with static function Time: 3.6722099781036 //PMA function, even this one is 2x slower than standard PHP function…
Are these tests performed on mb_* function or non mb_* one?
One thing is clear from these results is that even wrapping a built in function in another function (wrapper function) has some significant overhead. And using OOP/Classes is like wrapping that wrapper function further into a another wrapper causing further slowness. So why not just use mb_* function directly everywhere? If we do this, then what to do in case mb_* functions are not available? Maybe we can just define a new mb_* function to use respective non mb_* function for it. For e.g. // Define a function like this function mb_strlen($str) { return strlen($str); } // or create an alias of strlen() as mb_strlen() (I heard this is possible in PHP 5.6+)
By doing this, we can get speed advantage of built in mb_* function whenever it is available by avoiding any wrapper function or class. And in cases where it is not available, a wrapper function with standard non mb function can deliver equivalent performance as non mb function is inherently faster than its mb_* mod.
Hugues.
Hi, This is a very good idea! Marc, you asked to create PMA_* string functions. Do you agree with the Chirayu's proposal? So we won't have PMA_* functions, but only mb_* functions. Hugues.
Hugues Peccatte a écrit :
2014-09-25 20:08 GMT+02:00 Chirayu Chiripal <chirayu.chiripal@gmail.com>:
On Thu, Sep 25, 2014 at 12:17 AM, Hugues Peccatte < hugues.peccatte@gmail.com> wrote:
2014-09-24 14:42 GMT+02:00 Chirayu Chiripal <chirayu.chiripal@gmail.com>:
I was just wondering what will be the effect if we make current methods as static in those classes?
-- Regards, Chirayu Chiripal https://chirayuchiripal.wordpress.com/
It's almost the same thing to use a static or a non-static function. This is 2.5x slower than standard PHP function.
See: Time: 1.7506999969482 //standard PHP function Time: 4.3722500801086 //object Time: 4.3722498416901 //object with static function Time: 3.6722099781036 //PMA function, even this one is 2x slower than standard PHP function…
Are these tests performed on mb_* function or non mb_* one?
One thing is clear from these results is that even wrapping a built in function in another function (wrapper function) has some significant overhead. And using OOP/Classes is like wrapping that wrapper function further into a another wrapper causing further slowness. So why not just use mb_* function directly everywhere? If we do this, then what to do in case mb_* functions are not available? Maybe we can just define a new mb_* function to use respective non mb_* function for it. For e.g. // Define a function like this function mb_strlen($str) { return strlen($str); } // or create an alias of strlen() as mb_strlen() (I heard this is possible in PHP 5.6+)
By doing this, we can get speed advantage of built in mb_* function whenever it is available by avoiding any wrapper function or class. And in cases where it is not available, a wrapper function with standard non mb function can deliver equivalent performance as non mb function is inherently faster than its mb_* mod.
Hugues.
Hi,
This is a very good idea!
Marc, you asked to create PMA_* string functions. Do you agree with the Chirayu's proposal? So we won't have PMA_* functions, but only mb_* functions.
Hugues.
I have not followed the latest test results in this thread. If it's true that using wrappers has a big cost, I am in favor but I fear that people reading the code do not realize that the mb_* function could be a wrapper. Not sure how to document this. -- Marc Delisle (phpMyAdmin)
2014-09-25 22:02 GMT+02:00 Marc Delisle <marc@infomarc.info>:
Hugues Peccatte a écrit :
2014-09-25 20:08 GMT+02:00 Chirayu Chiripal <chirayu.chiripal@gmail.com :
On Thu, Sep 25, 2014 at 12:17 AM, Hugues Peccatte < hugues.peccatte@gmail.com> wrote:
2014-09-24 14:42 GMT+02:00 Chirayu Chiripal <
chirayu.chiripal@gmail.com>:
I was just wondering what will be the effect if we make current
methods
as static in those classes?
-- Regards, Chirayu Chiripal https://chirayuchiripal.wordpress.com/
It's almost the same thing to use a static or a non-static function. This is 2.5x slower than standard PHP function.
See: Time: 1.7506999969482 //standard PHP function Time: 4.3722500801086 //object Time: 4.3722498416901 //object with static function Time: 3.6722099781036 //PMA function, even this one is 2x slower than standard PHP function…
Are these tests performed on mb_* function or non mb_* one?
I forgot to answer to this question… So the tests are with non mb_* functions.
One thing is clear from these results is that even wrapping a built in function in another function (wrapper function) has some significant overhead. And using OOP/Classes is like wrapping that wrapper function further into a another wrapper causing further slowness. So why not just use mb_* function directly everywhere? If we do this, then what to do in case mb_* functions are not available? Maybe we can just define a new
mb_*
function to use respective non mb_* function for it. For e.g. // Define a function like this function mb_strlen($str) { return strlen($str); } // or create an alias of strlen() as mb_strlen() (I heard this is possible in PHP 5.6+)
By doing this, we can get speed advantage of built in mb_* function whenever it is available by avoiding any wrapper function or class. And in cases where it is not available, a wrapper function with standard non mb function can deliver equivalent performance as non mb function is inherently faster than its mb_* mod.
Hugues.
Hi,
This is a very good idea!
Marc, you asked to create PMA_* string functions. Do you agree with the Chirayu's proposal? So we won't have PMA_* functions, but only mb_* functions.
Hugues.
I have not followed the latest test results in this thread. If it's true that using wrappers has a big cost, I am in favor but I fear that people reading the code do not realize that the mb_* function could be a wrapper.
Not sure how to document this.
-- Marc Delisle (phpMyAdmin)
Using wrappers, the time could be doubled. So with his proposition, the wrappers would be created only when mb_* doesn't exist. If mb_* exist, we won't redefine it, but the execution would be slower because mb_* are slower. If mb_* don't exist, the non-mb_* functions would be wrapped, so will be a little bit longer than without wrap. And we would also be able to force the use of non-mb_* functions if needed. As you said, the only thing that might be difficult, would be to document the fact that mb_* functions could be pMA functions. We should choose between clarity and "performance". Hugues.
Le 2014-09-25 16:16, Hugues Peccatte a écrit :
2014-09-25 22:02 GMT+02:00 Marc Delisle <marc@infomarc.info <mailto:marc@infomarc.info>>:
Hugues Peccatte a écrit : > 2014-09-25 20:08 GMT+02:00 Chirayu Chiripal <chirayu.chiripal@gmail.com <mailto:chirayu.chiripal@gmail.com>>: > >> >> On Thu, Sep 25, 2014 at 12:17 AM, Hugues Peccatte < >> hugues.peccatte@gmail.com <mailto:hugues.peccatte@gmail.com>> wrote: >> >>> 2014-09-24 14:42 GMT+02:00 Chirayu Chiripal <chirayu.chiripal@gmail.com <mailto:chirayu.chiripal@gmail.com>>: >>>> >>>> I was just wondering what will be the effect if we make current methods >>>> as static in those classes? >>>> >>>> -- >>>> Regards, >>>> Chirayu Chiripal >>>> https://chirayuchiripal.wordpress.com/ >>>> >>> It's almost the same thing to use a static or a non-static function. This >>> is 2.5x slower than standard PHP function. >>> >>> See: >>> Time: 1.7506999969482 //standard PHP function >>> Time: 4.3722500801086 //object >>> Time: 4.3722498416901 //object with static function >>> Time: 3.6722099781036 //PMA function, even this one is 2x slower than >>> standard PHP function… >>> >> Are these tests performed on mb_* function or non mb_* one?
I forgot to answer to this question… So the tests are with non mb_* functions.
>> >> One thing is clear from these results is that even wrapping a built in >> function in another function (wrapper function) has some significant >> overhead. And using OOP/Classes is like wrapping that wrapper function >> further into a another wrapper causing further slowness. So why not just >> use mb_* function directly everywhere? If we do this, then what to do in >> case mb_* functions are not available? Maybe we can just define a new mb_* >> function to use respective non mb_* function for it. For e.g. >> // Define a function like this >> function mb_strlen($str) { >> return strlen($str); >> } >> // or create an alias of strlen() as mb_strlen() (I heard this is possible >> in PHP 5.6+) >> >> By doing this, we can get speed advantage of built in mb_* function >> whenever it is available by avoiding any wrapper function or class. And in >> cases where it is not available, a wrapper function with standard non mb >> function can deliver equivalent performance as non mb function is >> inherently faster than its mb_* mod. >> >> >>> Hugues. >>> > Hi, > > This is a very good idea! > > Marc, you asked to create PMA_* string functions. Do you agree with the > Chirayu's proposal? So we won't have PMA_* functions, but only mb_* > functions. > > Hugues.
I have not followed the latest test results in this thread. If it's true that using wrappers has a big cost, I am in favor but I fear that people reading the code do not realize that the mb_* function could be a wrapper.
Not sure how to document this.
-- Marc Delisle (phpMyAdmin)
Using wrappers, the time could be doubled. So with his proposition, the wrappers would be created only when mb_* doesn't exist. If mb_* exist, we won't redefine it, but the execution would be slower because mb_* are slower. If mb_* don't exist, the non-mb_* functions would be wrapped, so will be a little bit longer than without wrap. And we would also be able to force the use of non-mb_* functions if needed.
As you said, the only thing that might be difficult, would be to document the fact that mb_* functions could be pMA functions. We should choose between clarity and "performance".
Performance should have priority, but maybe we can have both. I assume that the changes will be done automatically, so what about introducing a small PHP comment at the same time? Something like this: (before) if ($pmaString->strlen($db)) { (after) if (/* overload? */ mb_strlen($db)) { -- Marc Delisle | phpMyAdmin
2014-09-26 6:29 GMT+02:00 Marc Delisle <marc@infomarc.info>:
Le 2014-09-25 16:16, Hugues Peccatte a écrit :
2014-09-25 22:02 GMT+02:00 Marc Delisle <marc@infomarc.info <mailto:marc@infomarc.info>>:
Hugues Peccatte a écrit : > 2014-09-25 20:08 GMT+02:00 Chirayu Chiripal <chirayu.chiripal@gmail.com <mailto:chirayu.chiripal@gmail.com>>: > >> >> On Thu, Sep 25, 2014 at 12:17 AM, Hugues Peccatte < >> hugues.peccatte@gmail.com <mailto:hugues.peccatte@gmail.com>> wrote: >> >>> 2014-09-24 14:42 GMT+02:00 Chirayu Chiripal <chirayu.chiripal@gmail.com <mailto:chirayu.chiripal@gmail.com>>: >>>> >>>> I was just wondering what will be the effect if we make current methods >>>> as static in those classes? >>>> >>>> -- >>>> Regards, >>>> Chirayu Chiripal >>>> https://chirayuchiripal.wordpress.com/ >>>> >>> It's almost the same thing to use a static or a non-static function. This >>> is 2.5x slower than standard PHP function. >>> >>> See: >>> Time: 1.7506999969482 //standard PHP function >>> Time: 4.3722500801086 //object >>> Time: 4.3722498416901 //object with static function >>> Time: 3.6722099781036 //PMA function, even this one is 2x slower than >>> standard PHP function… >>> >> Are these tests performed on mb_* function or non mb_* one?
I forgot to answer to this question… So the tests are with non mb_* functions.
>> >> One thing is clear from these results is that even wrapping a built in >> function in another function (wrapper function) has some significant >> overhead. And using OOP/Classes is like wrapping that wrapper function >> further into a another wrapper causing further slowness. So why not just >> use mb_* function directly everywhere? If we do this, then what to do in >> case mb_* functions are not available? Maybe we can just define a new mb_* >> function to use respective non mb_* function for it. For e.g. >> // Define a function like this >> function mb_strlen($str) { >> return strlen($str); >> } >> // or create an alias of strlen() as mb_strlen() (I heard this is possible >> in PHP 5.6+) >> >> By doing this, we can get speed advantage of built in mb_* function >> whenever it is available by avoiding any wrapper function or class. And in >> cases where it is not available, a wrapper function with standard non mb >> function can deliver equivalent performance as non mb function is >> inherently faster than its mb_* mod. >> >> >>> Hugues. >>> > Hi, > > This is a very good idea! > > Marc, you asked to create PMA_* string functions. Do you agree with the > Chirayu's proposal? So we won't have PMA_* functions, but only mb_* > functions. > > Hugues.
I have not followed the latest test results in this thread. If it's true that using wrappers has a big cost, I am in favor but I fear that people reading the code do not realize that the mb_* function could be a wrapper.
Not sure how to document this.
-- Marc Delisle (phpMyAdmin)
Using wrappers, the time could be doubled. So with his proposition, the wrappers would be created only when mb_* doesn't exist. If mb_* exist, we won't redefine it, but the execution would be slower because mb_* are slower. If mb_* don't exist, the non-mb_* functions would be wrapped, so will be a little bit longer than without wrap. And we would also be able to force the use of non-mb_* functions if needed.
As you said, the only thing that might be difficult, would be to document the fact that mb_* functions could be pMA functions. We should choose between clarity and "performance".
Performance should have priority, but maybe we can have both. I assume that the changes will be done automatically, so what about introducing a small PHP comment at the same time?
Something like this:
(before) if ($pmaString->strlen($db)) {
(after) if (/* overload? */ mb_strlen($db)) {
-- Marc Delisle | phpMyAdmin
Hi, I'll see how to add a comment. PHPCS will curse us for too long lines. :) I just found a limit to this method: the tests. If mb_* functions exist, we won't be able to create our mb_* functions and so to test it. Even if there is this issue, I'm currently working on this new "pattern". Hugues.
2014-09-26 23:35 GMT+02:00 Hugues Peccatte <hugues.peccatte@gmail.com>:
Hi,
I'll see how to add a comment. PHPCS will curse us for too long lines. :)
I just found a limit to this method: the tests. If mb_* functions exist, we won't be able to create our mb_* functions and so to test it.
Even if there is this issue, I'm currently working on this new "pattern".
Hugues.
Hi, I created the new mb_* functions and initialised the unit tests. As I said, we won't be able to run tests for native mb_* functions and PMA mb_* functions (aliases for standard string functions) on the same environment. Do you think that it would be possible to ask travis to run tests on a server without multibytes functions please? Thanks, H.
2014-09-28 14:17 GMT+02:00 Hugues Peccatte <hugues.peccatte@gmail.com>:
2014-09-26 23:35 GMT+02:00 Hugues Peccatte <hugues.peccatte@gmail.com>:
Hi,
I'll see how to add a comment. PHPCS will curse us for too long lines. :)
I just found a limit to this method: the tests. If mb_* functions exist, we won't be able to create our mb_* functions and so to test it.
Even if there is this issue, I'm currently working on this new "pattern".
Hugues.
Hi,
I created the new mb_* functions and initialised the unit tests.
As I said, we won't be able to run tests for native mb_* functions and PMA mb_* functions (aliases for standard string functions) on the same environment. Do you think that it would be possible to ask travis to run tests on a server without multibytes functions please?
Thanks, H.
Hi everyone, Work is not over, there are still some errors, but it's partially done. See [1]. Marc, import seems to work (at least SQL, I didn't try with compressed ones). Could you please try to import your SQL file and confirm that you can import structure and data in a reasonable time? [1] https://github.com/Tithugues/phpmyadmin/tree/stringFunctions_master Thanks, Hugues.
Le 2014-09-28 15:53, Hugues Peccatte a écrit :
2014-09-28 14:17 GMT+02:00 Hugues Peccatte <hugues.peccatte@gmail.com <mailto:hugues.peccatte@gmail.com>>:
2014-09-26 23:35 GMT+02:00 Hugues Peccatte <hugues.peccatte@gmail.com <mailto:hugues.peccatte@gmail.com>>:
Hi,
I'll see how to add a comment. PHPCS will curse us for too long lines. :)
I just found a limit to this method: the tests. If mb_* functions exist, we won't be able to create our mb_* functions and so to test it.
Even if there is this issue, I'm currently working on this new "pattern".
Hugues.
Hi,
I created the new mb_* functions and initialised the unit tests.
As I said, we won't be able to run tests for native mb_* functions and PMA mb_* functions (aliases for standard string functions) on the same environment. Do you think that it would be possible to ask travis to run tests on a server without multibytes functions please?
Thanks, H.
Hi everyone,
Work is not over, there are still some errors, but it's partially done. See [1]. Marc, import seems to work (at least SQL, I didn't try with compressed ones). Could you please try to import your SQL file and confirm that you can import structure and data in a reasonable time?
Hi Hugues, Fatal error: Call to undefined function mb_chr() in libraries/Util.class.php on line 4448
[1] https://github.com/Tithugues/phpmyadmin/tree/stringFunctions_master
Thanks, Hugues.
------------------------------------------------------------------------------ Meet PCI DSS 3.0 Compliance Requirements with EventLog Analyzer Achieve PCI DSS 3.0 Compliant Status with Out-of-the-box PCI DSS Reports Are you Audit-Ready for PCI DSS 3.0 Compliance? Download White paper Comply to PCI DSS 3.0 Requirement 10 and 11.5 with EventLog Analyzer http://pubads.g.doubleclick.net/gampad/clk?id=154622311&iu=/4140/ostg.clktrk
_______________________________________________ Phpmyadmin-devel mailing list Phpmyadmin-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/phpmyadmin-devel
-- Marc Delisle | phpMyAdmin
2014-09-28 22:35 GMT+02:00 Marc Delisle <marc@infomarc.info>:
Le 2014-09-28 15:53, Hugues Peccatte a écrit :
2014-09-28 14:17 GMT+02:00 Hugues Peccatte <hugues.peccatte@gmail.com <mailto:hugues.peccatte@gmail.com>>:
2014-09-26 23:35 GMT+02:00 Hugues Peccatte <hugues.peccatte@gmail.com <mailto:hugues.peccatte@gmail.com>>:
Hi,
I'll see how to add a comment. PHPCS will curse us for too long lines. :)
I just found a limit to this method: the tests. If mb_* functions exist, we won't be able to create our mb_* functions and so to test it.
Even if there is this issue, I'm currently working on this new "pattern".
Hugues.
Hi,
I created the new mb_* functions and initialised the unit tests.
As I said, we won't be able to run tests for native mb_* functions and PMA mb_* functions (aliases for standard string functions) on the same environment. Do you think that it would be possible to ask travis to run tests on a server without multibytes functions please?
Thanks, H.
Hi everyone,
Work is not over, there are still some errors, but it's partially done. See [1]. Marc, import seems to work (at least SQL, I didn't try with compressed ones). Could you please try to import your SQL file and confirm that you can import structure and data in a reasonable time?
Hi Hugues, Fatal error: Call to undefined function mb_chr() in libraries/Util.class.php on line 4448
[1] https://github.com/Tithugues/phpmyadmin/tree/stringFunctions_master
Thanks, Hugues.
------------------------------------------------------------------------------
Meet PCI DSS 3.0 Compliance Requirements with EventLog Analyzer Achieve PCI DSS 3.0 Compliant Status with Out-of-the-box PCI DSS Reports Are you Audit-Ready for PCI DSS 3.0 Compliance? Download White paper Comply to PCI DSS 3.0 Requirement 10 and 11.5 with EventLog Analyzer
http://pubads.g.doubleclick.net/gampad/clk?id=154622311&iu=/4140/ostg.clktrk
_______________________________________________ Phpmyadmin-devel mailing list Phpmyadmin-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/phpmyadmin-devel
-- Marc Delisle | phpMyAdmin
Marc, Do you have multibytes functions on your server please? Hugues.
Hugues Peccatte a écrit :
2014-09-28 22:35 GMT+02:00 Marc Delisle <marc@infomarc.info>:
Le 2014-09-28 15:53, Hugues Peccatte a écrit :
2014-09-28 14:17 GMT+02:00 Hugues Peccatte <hugues.peccatte@gmail.com <mailto:hugues.peccatte@gmail.com>>:
2014-09-26 23:35 GMT+02:00 Hugues Peccatte <hugues.peccatte@gmail.com <mailto:hugues.peccatte@gmail.com>>:
Hi,
I'll see how to add a comment. PHPCS will curse us for too long lines. :)
I just found a limit to this method: the tests. If mb_* functions exist, we won't be able to create our mb_* functions and so to test it.
Even if there is this issue, I'm currently working on this new "pattern".
Hugues.
Hi,
I created the new mb_* functions and initialised the unit tests.
As I said, we won't be able to run tests for native mb_* functions and PMA mb_* functions (aliases for standard string functions) on the same environment. Do you think that it would be possible to ask travis to run tests on a server without multibytes functions please?
Thanks, H.
Hi everyone,
Work is not over, there are still some errors, but it's partially done. See [1]. Marc, import seems to work (at least SQL, I didn't try with compressed ones). Could you please try to import your SQL file and confirm that you can import structure and data in a reasonable time? Hi Hugues, Fatal error: Call to undefined function mb_chr() in libraries/Util.class.php on line 4448
[1] https://github.com/Tithugues/phpmyadmin/tree/stringFunctions_master
Thanks, Hugues.
Marc,
Do you have multibytes functions on your server please?
Hugues.
Yes, I have the mbstring extension (it's PHP 5.6.0). -- Marc Delisle (phpMyAdmin)
Le 2014-09-29 08:02, Marc Delisle a écrit :
Hugues Peccatte a écrit :
2014-09-28 22:35 GMT+02:00 Marc Delisle <marc@infomarc.info>:
Le 2014-09-28 15:53, Hugues Peccatte a écrit :
2014-09-28 14:17 GMT+02:00 Hugues Peccatte <hugues.peccatte@gmail.com <mailto:hugues.peccatte@gmail.com>>:
2014-09-26 23:35 GMT+02:00 Hugues Peccatte <hugues.peccatte@gmail.com <mailto:hugues.peccatte@gmail.com>>:
Hi,
I'll see how to add a comment. PHPCS will curse us for too long lines. :)
I just found a limit to this method: the tests. If mb_* functions exist, we won't be able to create our mb_* functions and so to test it.
Even if there is this issue, I'm currently working on this new "pattern".
Hugues.
Hi,
I created the new mb_* functions and initialised the unit tests.
As I said, we won't be able to run tests for native mb_* functions and PMA mb_* functions (aliases for standard string functions) on the same environment. Do you think that it would be possible to ask travis to run tests on a server without multibytes functions please?
Thanks, H.
Hi everyone,
Work is not over, there are still some errors, but it's partially done. See [1]. Marc, import seems to work (at least SQL, I didn't try with compressed ones). Could you please try to import your SQL file and confirm that you can import structure and data in a reasonable time? Hi Hugues, Fatal error: Call to undefined function mb_chr() in libraries/Util.class.php on line 4448
[1] https://github.com/Tithugues/phpmyadmin/tree/stringFunctions_master
Thanks, Hugues.
Marc,
Do you have multibytes functions on your server please?
Hugues.
Yes, I have the mbstring extension (it's PHP 5.6.0).
Hi Hugues, I retested this morning on a laptop, importing a SQL file containing 10000 employees from the sample employees database. This is a small file (660 KB). Current master: 3 min 25 sec (and ends with JSON.parse: unexpected character) Current Tithugues/stringFunctions_master: 2 min 10 sec (same js error) Current QA_4_2: 0 min 5 sec There has been improvement, but we cannot release 4.3 with this import speed. -- Marc Delisle | phpMyAdmin
2014-10-03 12:26 GMT+02:00 Marc Delisle <marc@infomarc.info>:
Hi Hugues, I retested this morning on a laptop, importing a SQL file containing 10000 employees from the sample employees database. This is a small file (660 KB).
Current master: 3 min 25 sec (and ends with JSON.parse: unexpected character)
Current Tithugues/stringFunctions_master: 2 min 10 sec (same js error)
Current QA_4_2: 0 min 5 sec
There has been improvement, but we cannot release 4.3 with this import speed.
-- Marc Delisle | phpMyAdmin
Hi, I agree… But I'm afraid this is linked to multibytes functions… Maybe we shouldn't use the multibytes functions everywhere… I'll still try to improve performances. Hugues.
On Sat, Oct 4, 2014 at 1:24 AM, Hugues Peccatte <hugues.peccatte@gmail.com> wrote:
2014-10-03 12:26 GMT+02:00 Marc Delisle <marc@infomarc.info>:
Hi Hugues, I retested this morning on a laptop, importing a SQL file containing 10000 employees from the sample employees database. This is a small file (660 KB).
Current master: 3 min 25 sec (and ends with JSON.parse: unexpected character)
Current Tithugues/stringFunctions_master: 2 min 10 sec (same js error)
Current QA_4_2: 0 min 5 sec
There has been improvement, but we cannot release 4.3 with this import speed.
-- Marc Delisle | phpMyAdmin
Hi,
I agree… But I'm afraid this is linked to multibytes functions… Maybe we shouldn't use the multibytes functions everywhere…
I'll still try to improve performances.
Hugues.
Indeed, I also think that we should use mb_* function only when necessary and choice to use them should be made in case by case basis. -- Thanks and Regards, Madhura Jayaratne
Le 4 oct. 2014 03:22, "Madhura Jayaratne" <madhura.cj@gmail.com> a écrit :
On Sat, Oct 4, 2014 at 1:24 AM, Hugues Peccatte <hugues.peccatte@gmail.com>
wrote:
2014-10-03 12:26 GMT+02:00 Marc Delisle <marc@infomarc.info>:
Hi Hugues, I retested this morning on a laptop, importing a SQL file containing 10000 employees from the sample employees database. This is a small file (660 KB).
Current master: 3 min 25 sec (and ends with JSON.parse: unexpected character)
Current Tithugues/stringFunctions_master: 2 min 10 sec (same js error)
Current QA_4_2: 0 min 5 sec
There has been improvement, but we cannot release 4.3 with this import speed.
-- Marc Delisle | phpMyAdmin
Hi,
I agree… But I'm afraid this is linked to multibytes functions… Maybe we shouldn't use the multibytes functions everywhere…
I'll still try to improve performances.
Hugues.
Indeed, I also think that we should use mb_* function only when necessary and choice to use them should be made in case by case basis.
-- Thanks and Regards,
Madhura Jayaratne
Hi, I didn't push my commits, but that's what I've started. I replaced the mb_* calls by standard calls on configuration variables, reserved words, etc. Hugues.
2014-10-04 9:01 GMT+02:00 Hugues Peccatte <hugues.peccatte@gmail.com>:
Le 4 oct. 2014 03:22, "Madhura Jayaratne" <madhura.cj@gmail.com> a écrit :
On Sat, Oct 4, 2014 at 1:24 AM, Hugues Peccatte <
hugues.peccatte@gmail.com> wrote:
2014-10-03 12:26 GMT+02:00 Marc Delisle <marc@infomarc.info>:
Hi Hugues, I retested this morning on a laptop, importing a SQL file containing 10000 employees from the sample employees database. This is a small
file
(660 KB).
Current master: 3 min 25 sec (and ends with JSON.parse: unexpected character)
Current Tithugues/stringFunctions_master: 2 min 10 sec (same js error)
Current QA_4_2: 0 min 5 sec
There has been improvement, but we cannot release 4.3 with this import speed.
-- Marc Delisle | phpMyAdmin
Hi,
I agree… But I'm afraid this is linked to multibytes functions… Maybe we shouldn't use the multibytes functions everywhere…
I'll still try to improve performances.
Hugues.
Indeed, I also think that we should use mb_* function only when necessary and choice to use them should be made in case by case basis.
-- Thanks and Regards,
Madhura Jayaratne
Hi,
I didn't push my commits, but that's what I've started. I replaced the mb_* calls by standard calls on configuration variables, reserved words, etc.
Hugues.
Hi, Out of desperation, I try another algorithm. Instead of buffering data until SQL delimiter, I'll try to parse all lines. So, I won't parse 1000 times a buffer of 50000 characters, but less than 10 times many buffers of 500 characters. I hope this will be faster. Hugues.
2014-10-05 20:36 GMT+02:00 Hugues Peccatte <hugues.peccatte@gmail.com>:
2014-10-04 9:01 GMT+02:00 Hugues Peccatte <hugues.peccatte@gmail.com>:
Le 4 oct. 2014 03:22, "Madhura Jayaratne" <madhura.cj@gmail.com> a écrit :
On Sat, Oct 4, 2014 at 1:24 AM, Hugues Peccatte <
hugues.peccatte@gmail.com> wrote:
2014-10-03 12:26 GMT+02:00 Marc Delisle <marc@infomarc.info>:
Hi Hugues, I retested this morning on a laptop, importing a SQL file containing 10000 employees from the sample employees database. This is a small
file
(660 KB).
Current master: 3 min 25 sec (and ends with JSON.parse: unexpected character)
Current Tithugues/stringFunctions_master: 2 min 10 sec (same js error)
Current QA_4_2: 0 min 5 sec
There has been improvement, but we cannot release 4.3 with this import speed.
-- Marc Delisle | phpMyAdmin
Hi,
I agree… But I'm afraid this is linked to multibytes functions… Maybe we shouldn't use the multibytes functions everywhere…
I'll still try to improve performances.
Hugues.
Indeed, I also think that we should use mb_* function only when necessary and choice to use them should be made in case by case basis.
-- Thanks and Regards,
Madhura Jayaratne
Hi,
I didn't push my commits, but that's what I've started. I replaced the mb_* calls by standard calls on configuration variables, reserved words, etc.
Hugues.
Hi,
Out of desperation, I try another algorithm. Instead of buffering data until SQL delimiter, I'll try to parse all lines. So, I won't parse 1000 times a buffer of 50000 characters, but less than 10 times many buffers of 500 characters. I hope this will be faster.
Hugues.
Hi, The new algorithm is over. There are still some controls to add, but it is usable with the file in this ticket: [1] You can find my modifications here: [2] Marc, is it faster for you ? It seems that I won ~33% of time. We're still far from 5 seconds… Maybe I'll try to use standard PHP functions to see the difference. If the standard PHP functions are really faster, I'll try to add an option to use mb_* functions or standard PHP functions, as you said. [1] https://sourceforge.net/p/phpmyadmin/bugs/4536/ [2] https://github.com/Tithugues/phpmyadmin/tree/stringFunctions_useStandardFunc... Thanks, Hugues.
Le 2014-10-12 12:57, Hugues Peccatte a écrit :
2014-10-05 20:36 GMT+02:00 Hugues Peccatte <hugues.peccatte@gmail.com <mailto:hugues.peccatte@gmail.com>>:
2014-10-04 9:01 GMT+02:00 Hugues Peccatte <hugues.peccatte@gmail.com <mailto:hugues.peccatte@gmail.com>>:
Le 4 oct. 2014 03:22, "Madhura Jayaratne" <madhura.cj@gmail.com <mailto:madhura.cj@gmail.com>> a écrit :
> > > > On Sat, Oct 4, 2014 at 1:24 AM, Hugues Peccatte <hugues.peccatte@gmail.com <mailto:hugues.peccatte@gmail.com>> wrote: >> >> 2014-10-03 12:26 GMT+02:00 Marc Delisle <marc@infomarc.info <mailto:marc@infomarc.info>>: >>> >>> Hi Hugues, >>> I retested this morning on a laptop, importing a SQL file containing >>> 10000 employees from the sample employees database. This is a small file >>> (660 KB). >>> >>> Current master: 3 min 25 sec (and ends with JSON.parse: unexpected >>> character) >>> >>> Current Tithugues/stringFunctions_master: 2 min 10 sec (same js error) >>> >>> Current QA_4_2: 0 min 5 sec >>> >>> There has been improvement, but we cannot release 4.3 with this import >>> speed. >>> >>> -- >>> Marc Delisle | phpMyAdmin >> >> >> Hi, >> >> I agree… But I'm afraid this is linked to multibytes functions… >> Maybe we shouldn't use the multibytes functions everywhere… >> >> I'll still try to improve performances. >> >> Hugues. >> > > Indeed, I also think that we should use mb_* function only when necessary and choice to use them should be made in case by case basis. > > -- > Thanks and Regards, > > Madhura Jayaratne
Hi,
I didn't push my commits, but that's what I've started. I replaced the mb_* calls by standard calls on configuration variables, reserved words, etc.
Hugues.
Hi,
Out of desperation, I try another algorithm. Instead of buffering data until SQL delimiter, I'll try to parse all lines. So, I won't parse 1000 times a buffer of 50000 characters, but less than 10 times many buffers of 500 characters. I hope this will be faster.
Hugues.
Hi,
The new algorithm is over. There are still some controls to add, but it is usable with the file in this ticket: [1] You can find my modifications here: [2]
Marc, is it faster for you ? It seems that I won ~33% of time. We're still far from 5 seconds… Maybe I'll try to use standard PHP functions to see the difference. If the standard PHP functions are really faster, I'll try to add an option to use mb_* functions or standard PHP functions, as you said.
[1] https://sourceforge.net/p/phpmyadmin/bugs/4536/ [2] https://github.com/Tithugues/phpmyadmin/tree/stringFunctions_useStandardFunc...
Hi Hugues, yes it's faster. With the same testing conditions, the import takes 1 min 20 sec. -- Marc Delisle | phpMyAdmin
2014-10-12 21:06 GMT+02:00 Marc Delisle <marc@infomarc.info>:
Le 2014-10-12 12:57, Hugues Peccatte a écrit :
2014-10-05 20:36 GMT+02:00 Hugues Peccatte <hugues.peccatte@gmail.com <mailto:hugues.peccatte@gmail.com>>:
2014-10-04 9:01 GMT+02:00 Hugues Peccatte <hugues.peccatte@gmail.com <mailto:hugues.peccatte@gmail.com>>:
Le 4 oct. 2014 03:22, "Madhura Jayaratne" <madhura.cj@gmail.com <mailto:madhura.cj@gmail.com>> a écrit :
> > > > On Sat, Oct 4, 2014 at 1:24 AM, Hugues Peccatte <hugues.peccatte@gmail.com <mailto:hugues.peccatte@gmail.com>> wrote: >> >> 2014-10-03 12:26 GMT+02:00 Marc Delisle <marc@infomarc.info <mailto:marc@infomarc.info>>: >>> >>> Hi Hugues, >>> I retested this morning on a laptop, importing a SQL file containing >>> 10000 employees from the sample employees database. This is a small file >>> (660 KB). >>> >>> Current master: 3 min 25 sec (and ends with JSON.parse: unexpected >>> character) >>> >>> Current Tithugues/stringFunctions_master: 2 min 10 sec (same js error) >>> >>> Current QA_4_2: 0 min 5 sec >>> >>> There has been improvement, but we cannot release 4.3 with this import >>> speed. >>> >>> -- >>> Marc Delisle | phpMyAdmin >> >> >> Hi, >> >> I agree… But I'm afraid this is linked to multibytes functions… >> Maybe we shouldn't use the multibytes functions everywhere… >> >> I'll still try to improve performances. >> >> Hugues. >> > > Indeed, I also think that we should use mb_* function only when necessary and choice to use them should be made in case by case basis. > > -- > Thanks and Regards, > > Madhura Jayaratne
Hi,
I didn't push my commits, but that's what I've started. I replaced the mb_* calls by standard calls on configuration variables, reserved words, etc.
Hugues.
Hi,
Out of desperation, I try another algorithm. Instead of buffering data until SQL delimiter, I'll try to parse all lines. So, I won't parse 1000 times a buffer of 50000 characters, but less than 10 times many buffers of 500 characters. I hope this will be faster.
Hugues.
Hi,
The new algorithm is over. There are still some controls to add, but it is usable with the file in this ticket: [1] You can find my modifications here: [2]
Marc, is it faster for you ? It seems that I won ~33% of time. We're still far from 5 seconds… Maybe I'll try to use standard PHP functions to see the difference. If the standard PHP functions are really faster, I'll try to add an option to use mb_* functions or standard PHP functions, as you said.
https://github.com/Tithugues/phpmyadmin/tree/stringFunctions_useStandardFunc...
Hi Hugues, yes it's faster. With the same testing conditions, the import takes 1 min 20 sec.
-- Marc Delisle | phpMyAdmin
Thanks for your feedback. I'll try another another improvement to be faster. Note for my self: * read X characters but don't restart the search from 0 each time * search for the escaped quote with a lookbehind expression, something like `(?!<\\)(\\\\)*'` Hugues.
2014-10-13 12:21 GMT+02:00 Hugues Peccatte <hugues.peccatte@gmail.com>:
2014-10-12 21:06 GMT+02:00 Marc Delisle <marc@infomarc.info>:
Le 2014-10-12 12:57, Hugues Peccatte a écrit :
2014-10-05 20:36 GMT+02:00 Hugues Peccatte <hugues.peccatte@gmail.com <mailto:hugues.peccatte@gmail.com>>:
2014-10-04 9:01 GMT+02:00 Hugues Peccatte < hugues.peccatte@gmail.com <mailto:hugues.peccatte@gmail.com>>:
Le 4 oct. 2014 03:22, "Madhura Jayaratne" <madhura.cj@gmail.com <mailto:madhura.cj@gmail.com>> a écrit :
> > > > On Sat, Oct 4, 2014 at 1:24 AM, Hugues Peccatte <hugues.peccatte@gmail.com <mailto:hugues.peccatte@gmail.com>> wrote: >> >> 2014-10-03 12:26 GMT+02:00 Marc Delisle <marc@infomarc.info <mailto:marc@infomarc.info>>: >>> >>> Hi Hugues, >>> I retested this morning on a laptop, importing a SQL file containing >>> 10000 employees from the sample employees database. This is a small file >>> (660 KB). >>> >>> Current master: 3 min 25 sec (and ends with JSON.parse: unexpected >>> character) >>> >>> Current Tithugues/stringFunctions_master: 2 min 10 sec (same js error) >>> >>> Current QA_4_2: 0 min 5 sec >>> >>> There has been improvement, but we cannot release 4.3 with this import >>> speed. >>> >>> -- >>> Marc Delisle | phpMyAdmin >> >> >> Hi, >> >> I agree… But I'm afraid this is linked to multibytes functions… >> Maybe we shouldn't use the multibytes functions everywhere… >> >> I'll still try to improve performances. >> >> Hugues. >> > > Indeed, I also think that we should use mb_* function only when necessary and choice to use them should be made in case by case basis. > > -- > Thanks and Regards, > > Madhura Jayaratne
Hi,
I didn't push my commits, but that's what I've started. I replaced the mb_* calls by standard calls on configuration variables, reserved words, etc.
Hugues.
Hi,
Out of desperation, I try another algorithm. Instead of buffering data until SQL delimiter, I'll try to parse all lines. So, I won't parse 1000 times a buffer of 50000 characters, but less than 10 times many buffers of 500 characters. I hope this will be faster.
Hugues.
Hi,
The new algorithm is over. There are still some controls to add, but it is usable with the file in this ticket: [1] You can find my modifications here: [2]
Marc, is it faster for you ? It seems that I won ~33% of time. We're still far from 5 seconds… Maybe I'll try to use standard PHP functions to see the difference. If the standard PHP functions are really faster, I'll try to add an option to use mb_* functions or standard PHP functions, as you said.
https://github.com/Tithugues/phpmyadmin/tree/stringFunctions_useStandardFunc...
Hi Hugues, yes it's faster. With the same testing conditions, the import takes 1 min 20 sec.
-- Marc Delisle | phpMyAdmin
Thanks for your feedback. I'll try another another improvement to be faster.
Note for my self: * read X characters but don't restart the search from 0 each time * search for the escaped quote with a lookbehind expression, something like `(?!<\\)(\\\\)*'`
Hugues.
Hi, As asked by Marc, I added on option to import by reading as a multibytes string or not. The default configuration won't read as multibytes string (because it's too long…). It seems that the DnD to import doesn't use the default configuration, so what ever you define as default, it won't be use in this process. Should we create a ticket for this ? I think it's possible to get it in javascript. Hugues.
Le 2014-10-18 14:48, Hugues Peccatte a écrit :
2014-10-13 12:21 GMT+02:00 Hugues Peccatte <hugues.peccatte@gmail.com <mailto:hugues.peccatte@gmail.com>>:
2014-10-12 21:06 GMT+02:00 Marc Delisle <marc@infomarc.info <mailto:marc@infomarc.info>>:
Le 2014-10-12 12:57, Hugues Peccatte a écrit : > 2014-10-05 20:36 GMT+02:00 Hugues Peccatte <hugues.peccatte@gmail.com <mailto:hugues.peccatte@gmail.com> > <mailto:hugues.peccatte@gmail.com <mailto:hugues.peccatte@gmail.com>>>: > > 2014-10-04 9:01 GMT+02:00 Hugues Peccatte <hugues.peccatte@gmail.com <mailto:hugues.peccatte@gmail.com> > <mailto:hugues.peccatte@gmail.com <mailto:hugues.peccatte@gmail.com>>>: > > Le 4 oct. 2014 03:22, "Madhura Jayaratne" <madhura.cj@gmail.com <mailto:madhura.cj@gmail.com> > <mailto:madhura.cj@gmail.com <mailto:madhura.cj@gmail.com>>> a écrit : > > > > > > > > > > On Sat, Oct 4, 2014 at 1:24 AM, Hugues Peccatte > <hugues.peccatte@gmail.com <mailto:hugues.peccatte@gmail.com> <mailto:hugues.peccatte@gmail.com <mailto:hugues.peccatte@gmail.com>>> > wrote: > >> > >> 2014-10-03 12:26 GMT+02:00 Marc Delisle <marc@infomarc.info <mailto:marc@infomarc.info> > <mailto:marc@infomarc.info <mailto:marc@infomarc.info>>>: > >>> > >>> Hi Hugues, > >>> I retested this morning on a laptop, importing a SQL file > containing > >>> 10000 employees from the sample employees database. This is > a small file > >>> (660 KB). > >>> > >>> Current master: 3 min 25 sec (and ends with JSON.parse: > unexpected > >>> character) > >>> > >>> Current Tithugues/stringFunctions_master: 2 min 10 sec (same > js error) > >>> > >>> Current QA_4_2: 0 min 5 sec > >>> > >>> There has been improvement, but we cannot release 4.3 with > this import > >>> speed. > >>> > >>> -- > >>> Marc Delisle | phpMyAdmin > >> > >> > >> Hi, > >> > >> I agree… But I'm afraid this is linked to multibytes functions… > >> Maybe we shouldn't use the multibytes functions everywhere… > >> > >> I'll still try to improve performances. > >> > >> Hugues. > >> > > > > Indeed, I also think that we should use mb_* function only > when necessary and choice to use them should be made in case by > case basis. > > > > -- > > Thanks and Regards, > > > > Madhura Jayaratne > > Hi, > > I didn't push my commits, but that's what I've started. I > replaced the mb_* calls by standard calls on configuration > variables, reserved words, etc. > > Hugues. > > > Hi, > > Out of desperation, I try another algorithm. Instead of buffering > data until SQL delimiter, I'll try to parse all lines. > So, I won't parse 1000 times a buffer of 50000 characters, but less > than 10 times many buffers of 500 characters. I hope this will be > faster. > > Hugues. > > > Hi, > > The new algorithm is over. There are still some controls to add, but it > is usable with the file in this ticket: [1] > You can find my modifications here: [2] > > Marc, is it faster for you ? > It seems that I won ~33% of time. We're still far from 5 seconds… > Maybe I'll try to use standard PHP functions to see the difference. If > the standard PHP functions are really faster, I'll try to add an option > to use mb_* functions or standard PHP functions, as you said. > > [1] https://sourceforge.net/p/phpmyadmin/bugs/4536/ > [2] > https://github.com/Tithugues/phpmyadmin/tree/stringFunctions_useStandardFunc...
Hi Hugues, yes it's faster. With the same testing conditions, the import takes 1 min 20 sec.
-- Marc Delisle | phpMyAdmin
Thanks for your feedback. I'll try another another improvement to be faster.
Note for my self: * read X characters but don't restart the search from 0 each time * search for the escaped quote with a lookbehind expression, something like `(?!<\\)(\\\\)*'`
Hugues.
Hi,
As asked by Marc, I added on option to import by reading as a multibytes string or not. The default configuration won't read as multibytes string (because it's too long…). It seems that the DnD to import doesn't use the default configuration, so what ever you define as default, it won't be use in this process. Should we create a ticket for this ? I think it's possible to get it in javascript.
Hugues.
With the new option "Read as multibytes" left to its default disabled value, my test file imports in 5 seconds. When this option is ticked, it takes 60 seconds. Note that in both cases, I get a Javascript error at the end (the error report does not show the same error in Firefox than in Chromium). Can anyone confirm this? -- Marc Delisle | phpMyAdmin
2014-10-19 13:21 GMT+02:00 Marc Delisle <marc@infomarc.info>:
Le 2014-10-18 14:48, Hugues Peccatte a écrit :
2014-10-13 12:21 GMT+02:00 Hugues Peccatte <hugues.peccatte@gmail.com <mailto:hugues.peccatte@gmail.com>>:
2014-10-12 21:06 GMT+02:00 Marc Delisle <marc@infomarc.info <mailto:marc@infomarc.info>>:
Le 2014-10-12 12:57, Hugues Peccatte a écrit : > 2014-10-05 20:36 GMT+02:00 Hugues Peccatte < hugues.peccatte@gmail.com <mailto:hugues.peccatte@gmail.com> > <mailto:hugues.peccatte@gmail.com <mailto:hugues.peccatte@gmail.com>>>: > > 2014-10-04 9:01 GMT+02:00 Hugues Peccatte < hugues.peccatte@gmail.com <mailto:hugues.peccatte@gmail.com> > <mailto:hugues.peccatte@gmail.com <mailto:hugues.peccatte@gmail.com>>>: > > Le 4 oct. 2014 03:22, "Madhura Jayaratne" < madhura.cj@gmail.com <mailto:madhura.cj@gmail.com> > <mailto:madhura.cj@gmail.com <mailto:madhura.cj@gmail.com>>> a écrit : > > > > > > > > > > On Sat, Oct 4, 2014 at 1:24 AM, Hugues Peccatte > <hugues.peccatte@gmail.com <mailto:hugues.peccatte@gmail.com> <mailto:hugues.peccatte@gmail.com <mailto:hugues.peccatte@gmail.com>>> > wrote: > >> > >> 2014-10-03 12:26 GMT+02:00 Marc Delisle < marc@infomarc.info <mailto:marc@infomarc.info> > <mailto:marc@infomarc.info <mailto:marc@infomarc.info
: > >>> > >>> Hi Hugues, > >>> I retested this morning on a laptop, importing a SQL file > containing > >>> 10000 employees from the sample employees database. This is > a small file > >>> (660 KB). > >>> > >>> Current master: 3 min 25 sec (and ends with JSON.parse: > unexpected > >>> character) > >>> > >>> Current Tithugues/stringFunctions_master: 2 min 10 sec (same > js error) > >>> > >>> Current QA_4_2: 0 min 5 sec > >>> > >>> There has been improvement, but we cannot release 4.3 with > this import > >>> speed. > >>> > >>> -- > >>> Marc Delisle | phpMyAdmin > >> > >> > >> Hi, > >> > >> I agree… But I'm afraid this is linked to multibytes functions… > >> Maybe we shouldn't use the multibytes functions everywhere… > >> > >> I'll still try to improve performances. > >> > >> Hugues. > >> > > > > Indeed, I also think that we should use mb_* function only > when necessary and choice to use them should be made in case by > case basis. > > > > -- > > Thanks and Regards, > > > > Madhura Jayaratne > > Hi, > > I didn't push my commits, but that's what I've started. I > replaced the mb_* calls by standard calls on configuration > variables, reserved words, etc. > > Hugues. > > > Hi, > > Out of desperation, I try another algorithm. Instead of buffering > data until SQL delimiter, I'll try to parse all lines. > So, I won't parse 1000 times a buffer of 50000 characters, but less > than 10 times many buffers of 500 characters. I hope this will be > faster. > > Hugues. > > > Hi, > > The new algorithm is over. There are still some controls to add, but it > is usable with the file in this ticket: [1] > You can find my modifications here: [2] > > Marc, is it faster for you ? > It seems that I won ~33% of time. We're still far from 5 seconds… > Maybe I'll try to use standard PHP functions to see the difference. If > the standard PHP functions are really faster, I'll try to add an option > to use mb_* functions or standard PHP functions, as you said. > > [1] https://sourceforge.net/p/phpmyadmin/bugs/4536/ > [2] >
https://github.com/Tithugues/phpmyadmin/tree/stringFunctions_useStandardFunc...
Hi Hugues, yes it's faster. With the same testing conditions, the import takes 1 min 20 sec.
-- Marc Delisle | phpMyAdmin
Thanks for your feedback. I'll try another another improvement to be faster.
Note for my self: * read X characters but don't restart the search from 0 each time * search for the escaped quote with a lookbehind expression, something like `(?!<\\)(\\\\)*'`
Hugues.
Hi,
As asked by Marc, I added on option to import by reading as a multibytes string or not. The default configuration won't read as multibytes string (because it's too long…). It seems that the DnD to import doesn't use the default configuration, so what ever you define as default, it won't be use in this process. Should we create a ticket for this ? I think it's possible to get it in javascript.
Hugues.
With the new option "Read as multibytes" left to its default disabled value, my test file imports in 5 seconds.
When this option is ticked, it takes 60 seconds.
Note that in both cases, I get a Javascript error at the end (the error report does not show the same error in Firefox than in Chromium). Can anyone confirm this?
-- Marc Delisle | phpMyAdmin
Marc, Could we please have both errors? I don't have it on Chrome. I'll try on Firefox and IE. Thanks, Hugues.
Le 2014-10-19 15:01, Hugues Peccatte a écrit :
2014-10-19 13:21 GMT+02:00 Marc Delisle <marc@infomarc.info <mailto:marc@infomarc.info>>:
Le 2014-10-18 14:48, Hugues Peccatte a écrit : > 2014-10-13 12:21 GMT+02:00 Hugues Peccatte <hugues.peccatte@gmail.com <mailto:hugues.peccatte@gmail.com> > <mailto:hugues.peccatte@gmail.com <mailto:hugues.peccatte@gmail.com>>>: > > 2014-10-12 21:06 GMT+02:00 Marc Delisle <marc@infomarc.info <mailto:marc@infomarc.info> > <mailto:marc@infomarc.info <mailto:marc@infomarc.info>>>: > > Le 2014-10-12 12:57, Hugues Peccatte a écrit : > > 2014-10-05 20:36 GMT+02:00 Hugues Peccatte <hugues.peccatte@gmail.com <mailto:hugues.peccatte@gmail.com> <mailto:hugues.peccatte@gmail.com <mailto:hugues.peccatte@gmail.com>> > > <mailto:hugues.peccatte@gmail.com <mailto:hugues.peccatte@gmail.com> > <mailto:hugues.peccatte@gmail.com <mailto:hugues.peccatte@gmail.com>>>>: > > > > 2014-10-04 9:01 GMT+02:00 Hugues Peccatte <hugues.peccatte@gmail.com <mailto:hugues.peccatte@gmail.com> <mailto:hugues.peccatte@gmail.com <mailto:hugues.peccatte@gmail.com>> > > <mailto:hugues.peccatte@gmail.com <mailto:hugues.peccatte@gmail.com> > <mailto:hugues.peccatte@gmail.com <mailto:hugues.peccatte@gmail.com>>>>: > > > > Le 4 oct. 2014 03:22, "Madhura Jayaratne" <madhura.cj@gmail.com <mailto:madhura.cj@gmail.com> <mailto:madhura.cj@gmail.com <mailto:madhura.cj@gmail.com>> > > <mailto:madhura.cj@gmail.com <mailto:madhura.cj@gmail.com> > <mailto:madhura.cj@gmail.com <mailto:madhura.cj@gmail.com>>>> a écrit : > > > > > > > > > > > > > > > > On Sat, Oct 4, 2014 at 1:24 AM, Hugues Peccatte > > <hugues.peccatte@gmail.com <mailto:hugues.peccatte@gmail.com> > <mailto:hugues.peccatte@gmail.com <mailto:hugues.peccatte@gmail.com>> > <mailto:hugues.peccatte@gmail.com <mailto:hugues.peccatte@gmail.com> > <mailto:hugues.peccatte@gmail.com <mailto:hugues.peccatte@gmail.com>>>> > > wrote: > > >> > > >> 2014-10-03 12:26 GMT+02:00 Marc Delisle <marc@infomarc.info <mailto:marc@infomarc.info> <mailto:marc@infomarc.info <mailto:marc@infomarc.info>> > > <mailto:marc@infomarc.info <mailto:marc@infomarc.info> <mailto:marc@infomarc.info <mailto:marc@infomarc.info>>>>: > > >>> > > >>> Hi Hugues, > > >>> I retested this morning on a laptop, importing a > SQL file > > containing > > >>> 10000 employees from the sample employees > database. This is > > a small file > > >>> (660 KB). > > >>> > > >>> Current master: 3 min 25 sec (and ends with > JSON.parse: > > unexpected > > >>> character) > > >>> > > >>> Current Tithugues/stringFunctions_master: 2 min 10 > sec (same > > js error) > > >>> > > >>> Current QA_4_2: 0 min 5 sec > > >>> > > >>> There has been improvement, but we cannot release > 4.3 with > > this import > > >>> speed. > > >>> > > >>> -- > > >>> Marc Delisle | phpMyAdmin > > >> > > >> > > >> Hi, > > >> > > >> I agree… But I'm afraid this is linked to > multibytes functions… > > >> Maybe we shouldn't use the multibytes functions > everywhere… > > >> > > >> I'll still try to improve performances. > > >> > > >> Hugues. > > >> > > > > > > Indeed, I also think that we should use mb_* > function only > > when necessary and choice to use them should be made > in case by > > case basis. > > > > > > -- > > > Thanks and Regards, > > > > > > Madhura Jayaratne > > > > Hi, > > > > I didn't push my commits, but that's what I've started. I > > replaced the mb_* calls by standard calls on configuration > > variables, reserved words, etc. > > > > Hugues. > > > > > > Hi, > > > > Out of desperation, I try another algorithm. Instead of > buffering > > data until SQL delimiter, I'll try to parse all lines. > > So, I won't parse 1000 times a buffer of 50000 characters, > but less > > than 10 times many buffers of 500 characters. I hope this > will be > > faster. > > > > Hugues. > > > > > > Hi, > > > > The new algorithm is over. There are still some controls to > add, but it > > is usable with the file in this ticket: [1] > > You can find my modifications here: [2] > > > > Marc, is it faster for you ? > > It seems that I won ~33% of time. We're still far from 5 seconds… > > Maybe I'll try to use standard PHP functions to see the > difference. If > > the standard PHP functions are really faster, I'll try to add > an option > > to use mb_* functions or standard PHP functions, as you said. > > > > [1] https://sourceforge.net/p/phpmyadmin/bugs/4536/ > > [2] > > > https://github.com/Tithugues/phpmyadmin/tree/stringFunctions_useStandardFunc... > > Hi Hugues, > yes it's faster. With the same testing conditions, the import > takes 1 > min 20 sec. > > > -- > Marc Delisle | phpMyAdmin > > > Thanks for your feedback. > I'll try another another improvement to be faster. > > Note for my self: > * read X characters but don't restart the search from 0 each time > * search for the escaped quote with a lookbehind expression, > something like `(?!<\\)(\\\\)*'` > > Hugues. > > > Hi, > > As asked by Marc, I added on option to import by reading as a multibytes > string or not. > The default configuration won't read as multibytes string (because it's > too long…). It seems that the DnD to import doesn't use the default > configuration, so what ever you define as default, it won't be use in > this process. > Should we create a ticket for this ? I think it's possible to get it in > javascript. > > Hugues. >
With the new option "Read as multibytes" left to its default disabled value, my test file imports in 5 seconds.
When this option is ticked, it takes 60 seconds.
Note that in both cases, I get a Javascript error at the end (the error report does not show the same error in Firefox than in Chromium). Can anyone confirm this?
-- Marc Delisle | phpMyAdmin
Marc,
Could we please have both errors? I don't have it on Chrome. I'll try on Firefox and IE.
Thanks, Hugues.
From Iceweasel 24.8.1:
{ "pma_version": "4.3.0-dev", "browser_name": "FIREFOX", "browser_version": "24.0", "user_os": "Linux", "server_software": "Apache/2.2.22 (Debian)", "user_agent_string": "Mozilla/5.0 (X11; Linux x86_64; rv:24.0) Gecko/20140924 Firefox/24.0 Iceweasel/24.8.1", "locale": "en", "configuration_storage": "enabled", "php_version": "5.4.4-14+deb7u14", "exception_type": "js", "exception": { "mode": "stack", "name": "SyntaxError", "message": "JSON.parse: unexpected character", "stack": [ { "func": ".parseJSON", "args": "", "line": 2, "column": "", "context": [ "/*! jQuery v1.8.3 jquery.com | jquery.org/license */\r", "(function(e,t){function _(e){var t=M[e]={};return v.each(e.split(y),functio//...", "", "function sprintf() {", "/*", " * Copyright (c) 2013 Kevin van Zonneveld (http://kvz.io)", " * and Contributors (http://phpjs.org/authors)" ], "filename": "jquery/jquery-1.8.3.min.js" }, { "func": "PMA_console.initialize/<", "args": "", "line": 164, "column": "", "context": [ " $('#pma_console_options input[name=current_query]').prop('c//...", " PMA_console.updateConfig();", " });", "", " $(document).ajaxComplete(function (event, xhr) {", " var data = $.parseJSON(xhr.responseText);", " PMA_console.ajaxCallback(data);", " });", "", " PMA_console.isInitialized = true;", " }" ], "filename": "console.js" }, { "func": "ErrorReport.wrap_function/new_func", "args": "", "line": 262, "column": "", "context": [ " */", " wrap_function: function (func) {", " if (!func.wrapped) {", " var new_func = function () {", " try {", " return func.apply(this, arguments);", " } catch (x) {", " TraceKit.report(x);", " }", " };", " new_func.wrapped = true;" ], "filename": "error_report.js" }, { "func": "v.event.dispatch", "args": "", "line": 2, "column": "", "context": [ "/*! jQuery v1.8.3 jquery.com | jquery.org/license */\r", "(function(e,t){function _(e){var t=M[e]={};return v.each(e.split(y),functio//...", "", "function sprintf() {", "/*", " * Copyright (c) 2013 Kevin van Zonneveld (http://kvz.io)", " * and Contributors (http://phpjs.org/authors)" ], "filename": "jquery/jquery-1.8.3.min.js" }, { "func": "$event.dispatch", "args": "", "line": 372, "column": "", "context": [ "$event.dispatch = function( event ){", "\tif ( $.data( this, \"suppress.\"+ event.type ) - new Date().getTime() > 0 ){", "\t\t$.removeData( this, \"suppress.\"+ event.type );", "\t\treturn;", "\t}", "\treturn $dispatch.apply( this, arguments );", "};", "", "// event fix hooks for touch events...", "var touchHooks = ", "$event.fixHooks.touchstart = " ], "filename": "jquery/jquery.event.drag-2.2.js" }, { "func": "v.event.add/u", "args": "", "line": 2, "column": "", "context": [ "/*! jQuery v1.8.3 jquery.com | jquery.org/license */\r", "(function(e,t){function _(e){var t=M[e]={};return v.each(e.split(y),functio//...", "", "function sprintf() {", "/*", " * Copyright (c) 2013 Kevin van Zonneveld (http://kvz.io)", " * and Contributors (http://phpjs.org/authors)" ], "filename": "jquery/jquery-1.8.3.min.js" }, { "func": "v.event.trigger", "args": "", "line": 2, "column": "", "context": [ "/*! jQuery v1.8.3 jquery.com | jquery.org/license */\r", "(function(e,t){function _(e){var t=M[e]={};return v.each(e.split(y),functio//...", "", "function sprintf() {", "/*", " * Copyright (c) 2013 Kevin van Zonneveld (http://kvz.io)", " * and Contributors (http://phpjs.org/authors)" ], "filename": "jquery/jquery-1.8.3.min.js" }, { "func": "v.event.trigger", "args": "", "line": 2, "column": "", "context": [ "/*! jQuery v1.8.3 jquery.com | jquery.org/license */\r", "(function(e,t){function _(e){var t=M[e]={};return v.each(e.split(y),functio//...", "", "function sprintf() {", "/*", " * Copyright (c) 2013 Kevin van Zonneveld (http://kvz.io)", " * and Contributors (http://phpjs.org/authors)" ], "filename": "jquery/jquery-1.8.3.min.js" }, { "func": "T", "args": "", "line": 2, "column": "", "context": [ "/*! jQuery v1.8.3 jquery.com | jquery.org/license */\r", "(function(e,t){function _(e){var t=M[e]={};return v.each(e.split(y),functio//...", "", "function sprintf() {", "/*", " * Copyright (c) 2013 Kevin van Zonneveld (http://kvz.io)", " * and Contributors (http://phpjs.org/authors)" ], "filename": "jquery/jquery-1.8.3.min.js" }, { "func": ".send/r", "args": "", "line": 2, "column": "", "context": [ "/*! jQuery v1.8.3 jquery.com | jquery.org/license */\r", "(function(e,t){function _(e){var t=M[e]={};return v.each(e.split(y),functio//...", "", "function sprintf() {", "/*", " * Copyright (c) 2013 Kevin van Zonneveld (http://kvz.io)", " * and Contributors (http://phpjs.org/authors)" ], "filename": "jquery/jquery-1.8.3.min.js" } ], "useragent": "Mozilla/5.0 (X11; Linux x86_64; rv:24.0) Gecko/20140924 Firefox/24.0 Iceweasel/24.8.1", "incomplete": "false", "uri": "db_import.php?target=" }, "script_name": "db_import.php", "microhistory": { "pages": [ { "hash": "index.php?db=&table=&server=1&target=&token=c455fd8a0595a5032d18c07a64e7ab0a" }, { "hash": "db_structure.php?db=test&table=&server=1&target=&token=c455fd8a0595a5032d18c07a64e7ab0a", "params": { "opendb_url": "db_structure.php", "safari_browser": "0", "collation_connection": "utf8mb4_general_ci", "lang": "en", "server": "1", "text_dir": "ltr", "pma_text_default_tab": "Browse", "pma_text_left_default_tab": "Structure", "LimitChars": "50", "pftext": "", "confirm": "true", "LoginCookieValidity": "144000", "logged_in": "true" } }, { "hash": "index.php?db=&table=&server=1&target=&token=c455fd8a0595a5032d18c07a64e7ab0a", "params": { "opendb_url": "db_structure.php", "safari_browser": "0", "collation_connection": "utf8mb4_general_ci", "lang": "en", "server": "1", "text_dir": "ltr", "pma_text_default_tab": "Browse", "pma_text_left_default_tab": "Structure", "LimitChars": "50", "pftext": "", "confirm": "true", "LoginCookieValidity": "144000", "logged_in": "true" } }, { "hash": "db_structure.php?db=test&table=&server=1&target=&token=c455fd8a0595a5032d18c07a64e7ab0a", "params": { "opendb_url": "db_structure.php", "safari_browser": "0", "collation_connection": "utf8mb4_general_ci", "lang": "en", "server": "1", "text_dir": "ltr", "pma_text_default_tab": "Browse", "pma_text_left_default_tab": "Structure", "LimitChars": "50", "pftext": "", "confirm": "true", "LoginCookieValidity": "144000", "logged_in": "true" } }, { "hash": "db_import.php?db=test&table=&server=1&target=&token=c455fd8a0595a5032d18c07a64e7ab0a", "params": { "opendb_url": "db_structure.php", "safari_browser": "0", "collation_connection": "utf8mb4_general_ci", "lang": "en", "server": "1", "text_dir": "ltr", "pma_text_default_tab": "Browse", "pma_text_left_default_tab": "Structure", "LimitChars": "50", "pftext": "", "confirm": "true", "LoginCookieValidity": "144000", "logged_in": "true" } } ], "current_index": "5" } } -- Marc Delisle | phpMyAdmin
2014-10-19 21:53 GMT+02:00 Marc Delisle <marc@infomarc.info>:
Le 2014-10-19 15:01, Hugues Peccatte a écrit :
2014-10-19 13:21 GMT+02:00 Marc Delisle <marc@infomarc.info <mailto:marc@infomarc.info>>:
Le 2014-10-18 14:48, Hugues Peccatte a écrit : > 2014-10-13 12:21 GMT+02:00 Hugues Peccatte < hugues.peccatte@gmail.com <mailto:hugues.peccatte@gmail.com> > <mailto:hugues.peccatte@gmail.com <mailto:hugues.peccatte@gmail.com>>>: > > 2014-10-12 21:06 GMT+02:00 Marc Delisle <marc@infomarc.info <mailto:marc@infomarc.info> > <mailto:marc@infomarc.info <mailto:marc@infomarc.info>>>: > > Le 2014-10-12 12:57, Hugues Peccatte a écrit : > > 2014-10-05 20:36 GMT+02:00 Hugues Peccatte < hugues.peccatte@gmail.com <mailto:hugues.peccatte@gmail.com> <mailto:hugues.peccatte@gmail.com <mailto:hugues.peccatte@gmail.com
> > <mailto:hugues.peccatte@gmail.com <mailto:hugues.peccatte@gmail.com> > <mailto:hugues.peccatte@gmail.com <mailto: hugues.peccatte@gmail.com>>>>: > > > > 2014-10-04 9:01 GMT+02:00 Hugues Peccatte < hugues.peccatte@gmail.com <mailto:hugues.peccatte@gmail.com> <mailto:hugues.peccatte@gmail.com <mailto:hugues.peccatte@gmail.com
> > <mailto:hugues.peccatte@gmail.com <mailto:hugues.peccatte@gmail.com> > <mailto:hugues.peccatte@gmail.com <mailto: hugues.peccatte@gmail.com>>>>: > > > > Le 4 oct. 2014 03:22, "Madhura Jayaratne" < madhura.cj@gmail.com <mailto:madhura.cj@gmail.com> <mailto:madhura.cj@gmail.com <mailto:madhura.cj@gmail.com>> > > <mailto:madhura.cj@gmail.com <mailto:madhura.cj@gmail.com> > <mailto:madhura.cj@gmail.com <mailto:madhura.cj@gmail.com>>>> a écrit : > > > > > > > > > > > > > > > > On Sat, Oct 4, 2014 at 1:24 AM, Hugues Peccatte > > <hugues.peccatte@gmail.com <mailto: hugues.peccatte@gmail.com> > <mailto:hugues.peccatte@gmail.com <mailto: hugues.peccatte@gmail.com>> > <mailto:hugues.peccatte@gmail.com <mailto:hugues.peccatte@gmail.com> > <mailto:hugues.peccatte@gmail.com <mailto: hugues.peccatte@gmail.com>>>> > > wrote: > > >> > > >> 2014-10-03 12:26 GMT+02:00 Marc Delisle < marc@infomarc.info <mailto:marc@infomarc.info> <mailto:marc@infomarc.info <mailto:marc@infomarc.info>> > > <mailto:marc@infomarc.info <mailto:marc@infomarc.info> <mailto:marc@infomarc.info <mailto:marc@infomarc.info>>>>: > > >>> > > >>> Hi Hugues, > > >>> I retested this morning on a laptop, importing a > SQL file > > containing > > >>> 10000 employees from the sample employees > database. This is > > a small file > > >>> (660 KB). > > >>> > > >>> Current master: 3 min 25 sec (and ends with > JSON.parse: > > unexpected > > >>> character) > > >>> > > >>> Current Tithugues/stringFunctions_master: 2 min 10 > sec (same > > js error) > > >>> > > >>> Current QA_4_2: 0 min 5 sec > > >>> > > >>> There has been improvement, but we cannot release > 4.3 with > > this import > > >>> speed. > > >>> > > >>> -- > > >>> Marc Delisle | phpMyAdmin > > >> > > >> > > >> Hi, > > >> > > >> I agree… But I'm afraid this is linked to > multibytes functions… > > >> Maybe we shouldn't use the multibytes functions > everywhere… > > >> > > >> I'll still try to improve performances. > > >> > > >> Hugues. > > >> > > > > > > Indeed, I also think that we should use mb_* > function only > > when necessary and choice to use them should be made > in case by > > case basis. > > > > > > -- > > > Thanks and Regards, > > > > > > Madhura Jayaratne > > > > Hi, > > > > I didn't push my commits, but that's what I've started. I > > replaced the mb_* calls by standard calls on configuration > > variables, reserved words, etc. > > > > Hugues. > > > > > > Hi, > > > > Out of desperation, I try another algorithm. Instead of > buffering > > data until SQL delimiter, I'll try to parse all lines. > > So, I won't parse 1000 times a buffer of 50000 characters, > but less > > than 10 times many buffers of 500 characters. I hope this > will be > > faster. > > > > Hugues. > > > > > > Hi, > > > > The new algorithm is over. There are still some controls to > add, but it > > is usable with the file in this ticket: [1] > > You can find my modifications here: [2] > > > > Marc, is it faster for you ? > > It seems that I won ~33% of time. We're still far from 5 seconds… > > Maybe I'll try to use standard PHP functions to see the > difference. If > > the standard PHP functions are really faster, I'll try to add > an option > > to use mb_* functions or standard PHP functions, as you said. > > > > [1] https://sourceforge.net/p/phpmyadmin/bugs/4536/ > > [2] > > >
https://github.com/Tithugues/phpmyadmin/tree/stringFunctions_useStandardFunc...
> > Hi Hugues, > yes it's faster. With the same testing conditions, the
import
> takes 1 > min 20 sec. > > > -- > Marc Delisle | phpMyAdmin > > > Thanks for your feedback. > I'll try another another improvement to be faster. > > Note for my self: > * read X characters but don't restart the search from 0 each
time
> * search for the escaped quote with a lookbehind expression, > something like `(?!<\\)(\\\\)*'` > > Hugues. > > > Hi, > > As asked by Marc, I added on option to import by reading as a multibytes > string or not. > The default configuration won't read as multibytes string (because it's > too long…). It seems that the DnD to import doesn't use the default > configuration, so what ever you define as default, it won't be use
in
> this process. > Should we create a ticket for this ? I think it's possible to get it in > javascript. > > Hugues. >
With the new option "Read as multibytes" left to its default disabled value, my test file imports in 5 seconds.
When this option is ticked, it takes 60 seconds.
Note that in both cases, I get a Javascript error at the end (the
error
report does not show the same error in Firefox than in Chromium). Can anyone confirm this?
-- Marc Delisle | phpMyAdmin
Marc,
Could we please have both errors? I don't have it on Chrome. I'll try on Firefox and IE.
Thanks, Hugues.
From Iceweasel 24.8.1:
{ "pma_version": "4.3.0-dev", "browser_name": "FIREFOX", "browser_version": "24.0", "user_os": "Linux", "server_software": "Apache/2.2.22 (Debian)", "user_agent_string": "Mozilla/5.0 (X11; Linux x86_64; rv:24.0) Gecko/20140924 Firefox/24.0 Iceweasel/24.8.1", "locale": "en", "configuration_storage": "enabled", "php_version": "5.4.4-14+deb7u14", "exception_type": "js", "exception": { "mode": "stack", "name": "SyntaxError", "message": "JSON.parse: unexpected character", "stack": [ { "func": ".parseJSON", "args": "", "line": 2, "column": "", "context": [ "/*! jQuery v1.8.3 jquery.com | jquery.org/license */\r", "(function(e,t){function _(e){var t=M[e]={};return v.each(e.split(y),functio//...", "", "function sprintf() {", "/*", " * Copyright (c) 2013 Kevin van Zonneveld (http://kvz.io)", " * and Contributors (http://phpjs.org/authors)" ], "filename": "jquery/jquery-1.8.3.min.js" }, { "func": "PMA_console.initialize/<", "args": "", "line": 164, "column": "", "context": [ " $('#pma_console_options input[name=current_query]').prop('c//...", " PMA_console.updateConfig();", " });", "", " $(document).ajaxComplete(function (event, xhr) {", " var data = $.parseJSON(xhr.responseText);", " PMA_console.ajaxCallback(data);", " });", "", " PMA_console.isInitialized = true;", " }" ], "filename": "console.js" }, { "func": "ErrorReport.wrap_function/new_func", "args": "", "line": 262, "column": "", "context": [ " */", " wrap_function: function (func) {", " if (!func.wrapped) {", " var new_func = function () {", " try {", " return func.apply(this, arguments);", " } catch (x) {", " TraceKit.report(x);", " }", " };", " new_func.wrapped = true;" ], "filename": "error_report.js" }, { "func": "v.event.dispatch", "args": "", "line": 2, "column": "", "context": [ "/*! jQuery v1.8.3 jquery.com | jquery.org/license */\r", "(function(e,t){function _(e){var t=M[e]={};return v.each(e.split(y),functio//...", "", "function sprintf() {", "/*", " * Copyright (c) 2013 Kevin van Zonneveld (http://kvz.io)", " * and Contributors (http://phpjs.org/authors)" ], "filename": "jquery/jquery-1.8.3.min.js" }, { "func": "$event.dispatch", "args": "", "line": 372, "column": "", "context": [ "$event.dispatch = function( event ){", "\tif ( $.data( this, \"suppress.\"+ event.type ) - new Date().getTime() > 0 ){", "\t\t$.removeData( this, \"suppress.\"+ event.type );", "\t\treturn;", "\t}", "\treturn $dispatch.apply( this, arguments );", "};", "", "// event fix hooks for touch events...", "var touchHooks = ", "$event.fixHooks.touchstart = " ], "filename": "jquery/jquery.event.drag-2.2.js" }, { "func": "v.event.add/u", "args": "", "line": 2, "column": "", "context": [ "/*! jQuery v1.8.3 jquery.com | jquery.org/license */\r", "(function(e,t){function _(e){var t=M[e]={};return v.each(e.split(y),functio//...", "", "function sprintf() {", "/*", " * Copyright (c) 2013 Kevin van Zonneveld (http://kvz.io)", " * and Contributors (http://phpjs.org/authors)" ], "filename": "jquery/jquery-1.8.3.min.js" }, { "func": "v.event.trigger", "args": "", "line": 2, "column": "", "context": [ "/*! jQuery v1.8.3 jquery.com | jquery.org/license */\r", "(function(e,t){function _(e){var t=M[e]={};return v.each(e.split(y),functio//...", "", "function sprintf() {", "/*", " * Copyright (c) 2013 Kevin van Zonneveld (http://kvz.io)", " * and Contributors (http://phpjs.org/authors)" ], "filename": "jquery/jquery-1.8.3.min.js" }, { "func": "v.event.trigger", "args": "", "line": 2, "column": "", "context": [ "/*! jQuery v1.8.3 jquery.com | jquery.org/license */\r", "(function(e,t){function _(e){var t=M[e]={};return v.each(e.split(y),functio//...", "", "function sprintf() {", "/*", " * Copyright (c) 2013 Kevin van Zonneveld (http://kvz.io)", " * and Contributors (http://phpjs.org/authors)" ], "filename": "jquery/jquery-1.8.3.min.js" }, { "func": "T", "args": "", "line": 2, "column": "", "context": [ "/*! jQuery v1.8.3 jquery.com | jquery.org/license */\r", "(function(e,t){function _(e){var t=M[e]={};return v.each(e.split(y),functio//...", "", "function sprintf() {", "/*", " * Copyright (c) 2013 Kevin van Zonneveld (http://kvz.io)", " * and Contributors (http://phpjs.org/authors)" ], "filename": "jquery/jquery-1.8.3.min.js" }, { "func": ".send/r", "args": "", "line": 2, "column": "", "context": [ "/*! jQuery v1.8.3 jquery.com | jquery.org/license */\r", "(function(e,t){function _(e){var t=M[e]={};return v.each(e.split(y),functio//...", "", "function sprintf() {", "/*", " * Copyright (c) 2013 Kevin van Zonneveld (http://kvz.io)", " * and Contributors (http://phpjs.org/authors)" ], "filename": "jquery/jquery-1.8.3.min.js" } ], "useragent": "Mozilla/5.0 (X11; Linux x86_64; rv:24.0) Gecko/20140924 Firefox/24.0 Iceweasel/24.8.1", "incomplete": "false", "uri": "db_import.php?target=" }, "script_name": "db_import.php", "microhistory": { "pages": [ { "hash":
"index.php?db=&table=&server=1&target=&token=c455fd8a0595a5032d18c07a64e7ab0a" }, { "hash":
"db_structure.php?db=test&table=&server=1&target=&token=c455fd8a0595a5032d18c07a64e7ab0a", "params": { "opendb_url": "db_structure.php", "safari_browser": "0", "collation_connection": "utf8mb4_general_ci", "lang": "en", "server": "1", "text_dir": "ltr", "pma_text_default_tab": "Browse", "pma_text_left_default_tab": "Structure", "LimitChars": "50", "pftext": "", "confirm": "true", "LoginCookieValidity": "144000", "logged_in": "true" } }, { "hash":
"index.php?db=&table=&server=1&target=&token=c455fd8a0595a5032d18c07a64e7ab0a", "params": { "opendb_url": "db_structure.php", "safari_browser": "0", "collation_connection": "utf8mb4_general_ci", "lang": "en", "server": "1", "text_dir": "ltr", "pma_text_default_tab": "Browse", "pma_text_left_default_tab": "Structure", "LimitChars": "50", "pftext": "", "confirm": "true", "LoginCookieValidity": "144000", "logged_in": "true" } }, { "hash":
"db_structure.php?db=test&table=&server=1&target=&token=c455fd8a0595a5032d18c07a64e7ab0a", "params": { "opendb_url": "db_structure.php", "safari_browser": "0", "collation_connection": "utf8mb4_general_ci", "lang": "en", "server": "1", "text_dir": "ltr", "pma_text_default_tab": "Browse", "pma_text_left_default_tab": "Structure", "LimitChars": "50", "pftext": "", "confirm": "true", "LoginCookieValidity": "144000", "logged_in": "true" } }, { "hash":
"db_import.php?db=test&table=&server=1&target=&token=c455fd8a0595a5032d18c07a64e7ab0a", "params": { "opendb_url": "db_structure.php", "safari_browser": "0", "collation_connection": "utf8mb4_general_ci", "lang": "en", "server": "1", "text_dir": "ltr", "pma_text_default_tab": "Browse", "pma_text_left_default_tab": "Structure", "LimitChars": "50", "pftext": "", "confirm": "true", "LoginCookieValidity": "144000", "logged_in": "true" } } ], "current_index": "5" } }
-- Marc Delisle | phpMyAdmin
Hi, It seems that the new algorithm is over, see [1]. The unit tests are now ok (except for the one failing on master). I did some tests with escaped quotes (with quote or backslash), delimiters into strings, modified delimiters. Marc, do you still have your error please? (I didn't work on it, but I hope that your issue is linked to the not-finished version…) Feel free to look at the new algorithm and to tell me if you see any flaw. If you can lead to an error with a correct SQL file, please send it to me. Thanks for your feedback. [1] https://github.com/Tithugues/phpmyadmin/tree/stringFunctions_useStandardFunc... Hugues.
2014-10-28 20:58 GMT+01:00 Hugues Peccatte <hugues.peccatte@gmail.com>:
2014-10-19 21:53 GMT+02:00 Marc Delisle <marc@infomarc.info>:
Le 2014-10-19 15:01, Hugues Peccatte a écrit :
2014-10-19 13:21 GMT+02:00 Marc Delisle <marc@infomarc.info <mailto:marc@infomarc.info>>:
Le 2014-10-18 14:48, Hugues Peccatte a écrit : > 2014-10-13 12:21 GMT+02:00 Hugues Peccatte < hugues.peccatte@gmail.com <mailto:hugues.peccatte@gmail.com> > <mailto:hugues.peccatte@gmail.com <mailto:hugues.peccatte@gmail.com>>>: > > 2014-10-12 21:06 GMT+02:00 Marc Delisle <marc@infomarc.info <mailto:marc@infomarc.info> > <mailto:marc@infomarc.info <mailto:marc@infomarc.info>>>: > > Le 2014-10-12 12:57, Hugues Peccatte a écrit : > > 2014-10-05 20:36 GMT+02:00 Hugues Peccatte < hugues.peccatte@gmail.com <mailto:hugues.peccatte@gmail.com> <mailto:hugues.peccatte@gmail.com <mailto:hugues.peccatte@gmail.com
> > <mailto:hugues.peccatte@gmail.com <mailto:hugues.peccatte@gmail.com> > <mailto:hugues.peccatte@gmail.com <mailto: hugues.peccatte@gmail.com>>>>: > > > > 2014-10-04 9:01 GMT+02:00 Hugues Peccatte < hugues.peccatte@gmail.com <mailto:hugues.peccatte@gmail.com> <mailto:hugues.peccatte@gmail.com <mailto:hugues.peccatte@gmail.com
> > <mailto:hugues.peccatte@gmail.com <mailto:hugues.peccatte@gmail.com> > <mailto:hugues.peccatte@gmail.com <mailto: hugues.peccatte@gmail.com>>>>: > > > > Le 4 oct. 2014 03:22, "Madhura Jayaratne" < madhura.cj@gmail.com <mailto:madhura.cj@gmail.com> <mailto:madhura.cj@gmail.com <mailto:madhura.cj@gmail.com>> > > <mailto:madhura.cj@gmail.com <mailto:madhura.cj@gmail.com> > <mailto:madhura.cj@gmail.com <mailto:madhura.cj@gmail.com>>>> a écrit : > > > > > > > > > > > > > > > > On Sat, Oct 4, 2014 at 1:24 AM, Hugues Peccatte > > <hugues.peccatte@gmail.com <mailto: hugues.peccatte@gmail.com> > <mailto:hugues.peccatte@gmail.com <mailto: hugues.peccatte@gmail.com>> > <mailto:hugues.peccatte@gmail.com <mailto:hugues.peccatte@gmail.com> > <mailto:hugues.peccatte@gmail.com <mailto: hugues.peccatte@gmail.com>>>> > > wrote: > > >> > > >> 2014-10-03 12:26 GMT+02:00 Marc Delisle < marc@infomarc.info <mailto:marc@infomarc.info> <mailto:marc@infomarc.info <mailto:marc@infomarc.info>> > > <mailto:marc@infomarc.info <mailto:marc@infomarc.info> <mailto:marc@infomarc.info <mailto:marc@infomarc.info>>>>: > > >>> > > >>> Hi Hugues, > > >>> I retested this morning on a laptop, importing a > SQL file > > containing > > >>> 10000 employees from the sample employees > database. This is > > a small file > > >>> (660 KB). > > >>> > > >>> Current master: 3 min 25 sec (and ends with > JSON.parse: > > unexpected > > >>> character) > > >>> > > >>> Current Tithugues/stringFunctions_master: 2 min 10 > sec (same > > js error) > > >>> > > >>> Current QA_4_2: 0 min 5 sec > > >>> > > >>> There has been improvement, but we cannot release > 4.3 with > > this import > > >>> speed. > > >>> > > >>> -- > > >>> Marc Delisle | phpMyAdmin > > >> > > >> > > >> Hi, > > >> > > >> I agree… But I'm afraid this is linked to > multibytes functions… > > >> Maybe we shouldn't use the multibytes functions > everywhere… > > >> > > >> I'll still try to improve performances. > > >> > > >> Hugues. > > >> > > > > > > Indeed, I also think that we should use mb_* > function only > > when necessary and choice to use them should be made > in case by > > case basis. > > > > > > -- > > > Thanks and Regards, > > > > > > Madhura Jayaratne > > > > Hi, > > > > I didn't push my commits, but that's what I've started. I > > replaced the mb_* calls by standard calls on configuration > > variables, reserved words, etc. > > > > Hugues. > > > > > > Hi, > > > > Out of desperation, I try another algorithm. Instead of > buffering > > data until SQL delimiter, I'll try to parse all lines. > > So, I won't parse 1000 times a buffer of 50000 characters, > but less > > than 10 times many buffers of 500 characters. I hope this > will be > > faster. > > > > Hugues. > > > > > > Hi, > > > > The new algorithm is over. There are still some controls to > add, but it > > is usable with the file in this ticket: [1] > > You can find my modifications here: [2] > > > > Marc, is it faster for you ? > > It seems that I won ~33% of time. We're still far from 5 seconds… > > Maybe I'll try to use standard PHP functions to see the > difference. If > > the standard PHP functions are really faster, I'll try to add > an option > > to use mb_* functions or standard PHP functions, as you said. > > > > [1] https://sourceforge.net/p/phpmyadmin/bugs/4536/ > > [2] > > >
https://github.com/Tithugues/phpmyadmin/tree/stringFunctions_useStandardFunc...
> > Hi Hugues, > yes it's faster. With the same testing conditions, the
import
> takes 1 > min 20 sec. > > > -- > Marc Delisle | phpMyAdmin > > > Thanks for your feedback. > I'll try another another improvement to be faster. > > Note for my self: > * read X characters but don't restart the search from 0 each
time
> * search for the escaped quote with a lookbehind expression, > something like `(?!<\\)(\\\\)*'` > > Hugues. > > > Hi, > > As asked by Marc, I added on option to import by reading as a multibytes > string or not. > The default configuration won't read as multibytes string (because it's > too long…). It seems that the DnD to import doesn't use the
default
> configuration, so what ever you define as default, it won't be
use in
> this process. > Should we create a ticket for this ? I think it's possible to get it in > javascript. > > Hugues. >
With the new option "Read as multibytes" left to its default
disabled
value, my test file imports in 5 seconds.
When this option is ticked, it takes 60 seconds.
Note that in both cases, I get a Javascript error at the end (the
error
report does not show the same error in Firefox than in Chromium).
Can
anyone confirm this?
-- Marc Delisle | phpMyAdmin
Marc,
Could we please have both errors? I don't have it on Chrome. I'll try on Firefox and IE.
Thanks, Hugues.
From Iceweasel 24.8.1:
{ "pma_version": "4.3.0-dev", "browser_name": "FIREFOX", "browser_version": "24.0", "user_os": "Linux", "server_software": "Apache/2.2.22 (Debian)", "user_agent_string": "Mozilla/5.0 (X11; Linux x86_64; rv:24.0) Gecko/20140924 Firefox/24.0 Iceweasel/24.8.1", "locale": "en", "configuration_storage": "enabled", "php_version": "5.4.4-14+deb7u14", "exception_type": "js", "exception": { "mode": "stack", "name": "SyntaxError", "message": "JSON.parse: unexpected character", "stack": [ { "func": ".parseJSON", "args": "", "line": 2, "column": "", "context": [ "/*! jQuery v1.8.3 jquery.com | jquery.org/license */\r", "(function(e,t){function _(e){var t=M[e]={};return v.each(e.split(y),functio//...", "", "function sprintf() {", "/*", " * Copyright (c) 2013 Kevin van Zonneveld (http://kvz.io)", " * and Contributors (http://phpjs.org/authors)" ], "filename": "jquery/jquery-1.8.3.min.js" }, { "func": "PMA_console.initialize/<", "args": "", "line": 164, "column": "", "context": [ " $('#pma_console_options input[name=current_query]').prop('c//...", " PMA_console.updateConfig();", " });", "", " $(document).ajaxComplete(function (event, xhr) {", " var data = $.parseJSON(xhr.responseText);", " PMA_console.ajaxCallback(data);", " });", "", " PMA_console.isInitialized = true;", " }" ], "filename": "console.js" }, { "func": "ErrorReport.wrap_function/new_func", "args": "", "line": 262, "column": "", "context": [ " */", " wrap_function: function (func) {", " if (!func.wrapped) {", " var new_func = function () {", " try {", " return func.apply(this, arguments);", " } catch (x) {", " TraceKit.report(x);", " }", " };", " new_func.wrapped = true;" ], "filename": "error_report.js" }, { "func": "v.event.dispatch", "args": "", "line": 2, "column": "", "context": [ "/*! jQuery v1.8.3 jquery.com | jquery.org/license */\r", "(function(e,t){function _(e){var t=M[e]={};return v.each(e.split(y),functio//...", "", "function sprintf() {", "/*", " * Copyright (c) 2013 Kevin van Zonneveld (http://kvz.io)", " * and Contributors (http://phpjs.org/authors)" ], "filename": "jquery/jquery-1.8.3.min.js" }, { "func": "$event.dispatch", "args": "", "line": 372, "column": "", "context": [ "$event.dispatch = function( event ){", "\tif ( $.data( this, \"suppress.\"+ event.type ) - new Date().getTime() > 0 ){", "\t\t$.removeData( this, \"suppress.\"+ event.type );", "\t\treturn;", "\t}", "\treturn $dispatch.apply( this, arguments );", "};", "", "// event fix hooks for touch events...", "var touchHooks = ", "$event.fixHooks.touchstart = " ], "filename": "jquery/jquery.event.drag-2.2.js" }, { "func": "v.event.add/u", "args": "", "line": 2, "column": "", "context": [ "/*! jQuery v1.8.3 jquery.com | jquery.org/license */\r", "(function(e,t){function _(e){var t=M[e]={};return v.each(e.split(y),functio//...", "", "function sprintf() {", "/*", " * Copyright (c) 2013 Kevin van Zonneveld (http://kvz.io)", " * and Contributors (http://phpjs.org/authors)" ], "filename": "jquery/jquery-1.8.3.min.js" }, { "func": "v.event.trigger", "args": "", "line": 2, "column": "", "context": [ "/*! jQuery v1.8.3 jquery.com | jquery.org/license */\r", "(function(e,t){function _(e){var t=M[e]={};return v.each(e.split(y),functio//...", "", "function sprintf() {", "/*", " * Copyright (c) 2013 Kevin van Zonneveld (http://kvz.io)", " * and Contributors (http://phpjs.org/authors)" ], "filename": "jquery/jquery-1.8.3.min.js" }, { "func": "v.event.trigger", "args": "", "line": 2, "column": "", "context": [ "/*! jQuery v1.8.3 jquery.com | jquery.org/license */\r", "(function(e,t){function _(e){var t=M[e]={};return v.each(e.split(y),functio//...", "", "function sprintf() {", "/*", " * Copyright (c) 2013 Kevin van Zonneveld (http://kvz.io)", " * and Contributors (http://phpjs.org/authors)" ], "filename": "jquery/jquery-1.8.3.min.js" }, { "func": "T", "args": "", "line": 2, "column": "", "context": [ "/*! jQuery v1.8.3 jquery.com | jquery.org/license */\r", "(function(e,t){function _(e){var t=M[e]={};return v.each(e.split(y),functio//...", "", "function sprintf() {", "/*", " * Copyright (c) 2013 Kevin van Zonneveld (http://kvz.io)", " * and Contributors (http://phpjs.org/authors)" ], "filename": "jquery/jquery-1.8.3.min.js" }, { "func": ".send/r", "args": "", "line": 2, "column": "", "context": [ "/*! jQuery v1.8.3 jquery.com | jquery.org/license */\r", "(function(e,t){function _(e){var t=M[e]={};return v.each(e.split(y),functio//...", "", "function sprintf() {", "/*", " * Copyright (c) 2013 Kevin van Zonneveld (http://kvz.io)", " * and Contributors (http://phpjs.org/authors)" ], "filename": "jquery/jquery-1.8.3.min.js" } ], "useragent": "Mozilla/5.0 (X11; Linux x86_64; rv:24.0) Gecko/20140924 Firefox/24.0 Iceweasel/24.8.1", "incomplete": "false", "uri": "db_import.php?target=" }, "script_name": "db_import.php", "microhistory": { "pages": [ { "hash":
"index.php?db=&table=&server=1&target=&token=c455fd8a0595a5032d18c07a64e7ab0a" }, { "hash":
"db_structure.php?db=test&table=&server=1&target=&token=c455fd8a0595a5032d18c07a64e7ab0a", "params": { "opendb_url": "db_structure.php", "safari_browser": "0", "collation_connection": "utf8mb4_general_ci", "lang": "en", "server": "1", "text_dir": "ltr", "pma_text_default_tab": "Browse", "pma_text_left_default_tab": "Structure", "LimitChars": "50", "pftext": "", "confirm": "true", "LoginCookieValidity": "144000", "logged_in": "true" } }, { "hash":
"index.php?db=&table=&server=1&target=&token=c455fd8a0595a5032d18c07a64e7ab0a", "params": { "opendb_url": "db_structure.php", "safari_browser": "0", "collation_connection": "utf8mb4_general_ci", "lang": "en", "server": "1", "text_dir": "ltr", "pma_text_default_tab": "Browse", "pma_text_left_default_tab": "Structure", "LimitChars": "50", "pftext": "", "confirm": "true", "LoginCookieValidity": "144000", "logged_in": "true" } }, { "hash":
"db_structure.php?db=test&table=&server=1&target=&token=c455fd8a0595a5032d18c07a64e7ab0a", "params": { "opendb_url": "db_structure.php", "safari_browser": "0", "collation_connection": "utf8mb4_general_ci", "lang": "en", "server": "1", "text_dir": "ltr", "pma_text_default_tab": "Browse", "pma_text_left_default_tab": "Structure", "LimitChars": "50", "pftext": "", "confirm": "true", "LoginCookieValidity": "144000", "logged_in": "true" } }, { "hash":
"db_import.php?db=test&table=&server=1&target=&token=c455fd8a0595a5032d18c07a64e7ab0a", "params": { "opendb_url": "db_structure.php", "safari_browser": "0", "collation_connection": "utf8mb4_general_ci", "lang": "en", "server": "1", "text_dir": "ltr", "pma_text_default_tab": "Browse", "pma_text_left_default_tab": "Structure", "LimitChars": "50", "pftext": "", "confirm": "true", "LoginCookieValidity": "144000", "logged_in": "true" } } ], "current_index": "5" } }
-- Marc Delisle | phpMyAdmin
Hi,
It seems that the new algorithm is over, see [1]. The unit tests are now ok (except for the one failing on master). I did some tests with escaped quotes (with quote or backslash), delimiters into strings, modified delimiters. Marc, do you still have your error please? (I didn't work on it, but I hope that your issue is linked to the not-finished version…)
Feel free to look at the new algorithm and to tell me if you see any flaw. If you can lead to an error with a correct SQL file, please send it to me.
Thanks for your feedback.
[1] https://github.com/Tithugues/phpmyadmin/tree/stringFunctions_useStandardFunc...
Hugues.
Hi, Since my previous email, I did some modifications to manage the display of queries in a different way (closer to the current display). But there is something strange with the attachment… I can import the file (without error and all content executed), but on master, if I paste the content of the file into the SQL console, I've an error… Is it the expected behavior please? I don't see any exotic content in this file… On my branch, I don't have an error, but only the first statement (DROP PROCEDURE IF EXISTS) is executed… Thanks, Hugues.
Le 2014-10-28 17:24, Hugues Peccatte a écrit :
2014-10-28 20:58 GMT+01:00 Hugues Peccatte <hugues.peccatte@gmail.com <mailto:hugues.peccatte@gmail.com>>:
2014-10-19 21:53 GMT+02:00 Marc Delisle <marc@infomarc.info <mailto:marc@infomarc.info>>:
Le 2014-10-19 15:01, Hugues Peccatte a écrit : > 2014-10-19 13:21 GMT+02:00 Marc Delisle <marc@infomarc.info <mailto:marc@infomarc.info> > <mailto:marc@infomarc.info <mailto:marc@infomarc.info>>>: > > Le 2014-10-18 14:48, Hugues Peccatte a écrit : > > 2014-10-13 12:21 GMT+02:00 Hugues Peccatte <hugues.peccatte@gmail.com <mailto:hugues.peccatte@gmail.com> <mailto:hugues.peccatte@gmail.com <mailto:hugues.peccatte@gmail.com>> > > <mailto:hugues.peccatte@gmail.com <mailto:hugues.peccatte@gmail.com> > <mailto:hugues.peccatte@gmail.com <mailto:hugues.peccatte@gmail.com>>>>: > > > > 2014-10-12 21:06 GMT+02:00 Marc Delisle <marc@infomarc.info <mailto:marc@infomarc.info> <mailto:marc@infomarc.info <mailto:marc@infomarc.info>> > > <mailto:marc@infomarc.info <mailto:marc@infomarc.info> <mailto:marc@infomarc.info <mailto:marc@infomarc.info>>>>: > > > > Le 2014-10-12 12:57, Hugues Peccatte a écrit : > > > 2014-10-05 20:36 GMT+02:00 Hugues Peccatte <hugues.peccatte@gmail.com <mailto:hugues.peccatte@gmail.com> <mailto:hugues.peccatte@gmail.com <mailto:hugues.peccatte@gmail.com>> > <mailto:hugues.peccatte@gmail.com <mailto:hugues.peccatte@gmail.com> <mailto:hugues.peccatte@gmail.com <mailto:hugues.peccatte@gmail.com>>> > > > <mailto:hugues.peccatte@gmail.com <mailto:hugues.peccatte@gmail.com> > <mailto:hugues.peccatte@gmail.com <mailto:hugues.peccatte@gmail.com>> > > <mailto:hugues.peccatte@gmail.com <mailto:hugues.peccatte@gmail.com> <mailto:hugues.peccatte@gmail.com <mailto:hugues.peccatte@gmail.com>>>>>: > > > > > > 2014-10-04 9:01 GMT+02:00 Hugues Peccatte <hugues.peccatte@gmail.com <mailto:hugues.peccatte@gmail.com> <mailto:hugues.peccatte@gmail.com <mailto:hugues.peccatte@gmail.com>> > <mailto:hugues.peccatte@gmail.com <mailto:hugues.peccatte@gmail.com> <mailto:hugues.peccatte@gmail.com <mailto:hugues.peccatte@gmail.com>>> > > > <mailto:hugues.peccatte@gmail.com <mailto:hugues.peccatte@gmail.com> > <mailto:hugues.peccatte@gmail.com <mailto:hugues.peccatte@gmail.com>> > > <mailto:hugues.peccatte@gmail.com <mailto:hugues.peccatte@gmail.com> <mailto:hugues.peccatte@gmail.com <mailto:hugues.peccatte@gmail.com>>>>>: > > > > > > Le 4 oct. 2014 03:22, "Madhura Jayaratne" <madhura.cj@gmail.com <mailto:madhura.cj@gmail.com> <mailto:madhura.cj@gmail.com <mailto:madhura.cj@gmail.com>> > <mailto:madhura.cj@gmail.com <mailto:madhura.cj@gmail.com> <mailto:madhura.cj@gmail.com <mailto:madhura.cj@gmail.com>>> > > > <mailto:madhura.cj@gmail.com <mailto:madhura.cj@gmail.com> > <mailto:madhura.cj@gmail.com <mailto:madhura.cj@gmail.com>> > > <mailto:madhura.cj@gmail.com <mailto:madhura.cj@gmail.com> <mailto:madhura.cj@gmail.com <mailto:madhura.cj@gmail.com>>>>> a écrit : > > > > > > > > > > > > > > > > > > > > > > On Sat, Oct 4, 2014 at 1:24 AM, Hugues Peccatte > > > <hugues.peccatte@gmail.com <mailto:hugues.peccatte@gmail.com> <mailto:hugues.peccatte@gmail.com <mailto:hugues.peccatte@gmail.com>> > > <mailto:hugues.peccatte@gmail.com <mailto:hugues.peccatte@gmail.com> <mailto:hugues.peccatte@gmail.com <mailto:hugues.peccatte@gmail.com>>> > > <mailto:hugues.peccatte@gmail.com <mailto:hugues.peccatte@gmail.com> > <mailto:hugues.peccatte@gmail.com <mailto:hugues.peccatte@gmail.com>> > > <mailto:hugues.peccatte@gmail.com <mailto:hugues.peccatte@gmail.com> <mailto:hugues.peccatte@gmail.com <mailto:hugues.peccatte@gmail.com>>>>> > > > wrote: > > > >> > > > >> 2014-10-03 12:26 GMT+02:00 Marc Delisle <marc@infomarc.info <mailto:marc@infomarc.info> <mailto:marc@infomarc.info <mailto:marc@infomarc.info>> > <mailto:marc@infomarc.info <mailto:marc@infomarc.info> <mailto:marc@infomarc.info <mailto:marc@infomarc.info>>> > > > <mailto:marc@infomarc.info <mailto:marc@infomarc.info> > <mailto:marc@infomarc.info <mailto:marc@infomarc.info>> <mailto:marc@infomarc.info <mailto:marc@infomarc.info> > <mailto:marc@infomarc.info <mailto:marc@infomarc.info>>>>>: > > > >>> > > > >>> Hi Hugues, > > > >>> I retested this morning on a laptop, importing a > > SQL file > > > containing > > > >>> 10000 employees from the sample employees > > database. This is > > > a small file > > > >>> (660 KB). > > > >>> > > > >>> Current master: 3 min 25 sec (and ends with > > JSON.parse: > > > unexpected > > > >>> character) > > > >>> > > > >>> Current Tithugues/stringFunctions_master: 2 > min 10 > > sec (same > > > js error) > > > >>> > > > >>> Current QA_4_2: 0 min 5 sec > > > >>> > > > >>> There has been improvement, but we cannot > release > > 4.3 with > > > this import > > > >>> speed. > > > >>> > > > >>> -- > > > >>> Marc Delisle | phpMyAdmin > > > >> > > > >> > > > >> Hi, > > > >> > > > >> I agree… But I'm afraid this is linked to > > multibytes functions… > > > >> Maybe we shouldn't use the multibytes functions > > everywhere… > > > >> > > > >> I'll still try to improve performances. > > > >> > > > >> Hugues. > > > >> > > > > > > > > Indeed, I also think that we should use mb_* > > function only > > > when necessary and choice to use them should be made > > in case by > > > case basis. > > > > > > > > -- > > > > Thanks and Regards, > > > > > > > > Madhura Jayaratne > > > > > > Hi, > > > > > > I didn't push my commits, but that's what I've > started. I > > > replaced the mb_* calls by standard calls on > configuration > > > variables, reserved words, etc. > > > > > > Hugues. > > > > > > > > > Hi, > > > > > > Out of desperation, I try another algorithm. Instead of > > buffering > > > data until SQL delimiter, I'll try to parse all lines. > > > So, I won't parse 1000 times a buffer of 50000 > characters, > > but less > > > than 10 times many buffers of 500 characters. I hope > this > > will be > > > faster. > > > > > > Hugues. > > > > > > > > > Hi, > > > > > > The new algorithm is over. There are still some controls to > > add, but it > > > is usable with the file in this ticket: [1] > > > You can find my modifications here: [2] > > > > > > Marc, is it faster for you ? > > > It seems that I won ~33% of time. We're still far from 5 > seconds… > > > Maybe I'll try to use standard PHP functions to see the > > difference. If > > > the standard PHP functions are really faster, I'll try > to add > > an option > > > to use mb_* functions or standard PHP functions, as you > said. > > > > > > [1] https://sourceforge.net/p/phpmyadmin/bugs/4536/ > > > [2] > > > > > > https://github.com/Tithugues/phpmyadmin/tree/stringFunctions_useStandardFunc... > > > > Hi Hugues, > > yes it's faster. With the same testing conditions, the import > > takes 1 > > min 20 sec. > > > > > > -- > > Marc Delisle | phpMyAdmin > > > > > > Thanks for your feedback. > > I'll try another another improvement to be faster. > > > > Note for my self: > > * read X characters but don't restart the search from 0 each time > > * search for the escaped quote with a lookbehind expression, > > something like `(?!<\\)(\\\\)*'` > > > > Hugues. > > > > > > Hi, > > > > As asked by Marc, I added on option to import by reading as a > multibytes > > string or not. > > The default configuration won't read as multibytes string (because > it's > > too long…). It seems that the DnD to import doesn't use the default > > configuration, so what ever you define as default, it won't be use in > > this process. > > Should we create a ticket for this ? I think it's possible to get > it in > > javascript. > > > > Hugues. > > > > With the new option "Read as multibytes" left to its default disabled > value, my test file imports in 5 seconds. > > When this option is ticked, it takes 60 seconds. > > Note that in both cases, I get a Javascript error at the end (the error > report does not show the same error in Firefox than in Chromium). Can > anyone confirm this? > > > -- > Marc Delisle | phpMyAdmin > > > Marc, > > Could we please have both errors? I don't have it on Chrome. I'll try on > Firefox and IE. > > Thanks, > Hugues.
>From Iceweasel 24.8.1:
{ "pma_version": "4.3.0-dev", "browser_name": "FIREFOX", "browser_version": "24.0", "user_os": "Linux", "server_software": "Apache/2.2.22 (Debian)", "user_agent_string": "Mozilla/5.0 (X11; Linux x86_64; rv:24.0) Gecko/20140924 Firefox/24.0 Iceweasel/24.8.1", "locale": "en", "configuration_storage": "enabled", "php_version": "5.4.4-14+deb7u14", "exception_type": "js", "exception": { "mode": "stack", "name": "SyntaxError", "message": "JSON.parse: unexpected character", "stack": [ { "func": ".parseJSON", "args": "", "line": 2, "column": "", "context": [ "/*! jQuery v1.8.3 jquery.com <http://jquery.com> | jquery.org/license <http://jquery.org/license> */\r", "(function(e,t){function _(e){var t=M[e]={};return v.each(e.split(y),functio//...", "", "function sprintf() {", "/*", " * Copyright (c) 2013 Kevin van Zonneveld (http://kvz.io)", " * and Contributors (http://phpjs.org/authors)" ], "filename": "jquery/jquery-1.8.3.min.js" }, { "func": "PMA_console.initialize/<", "args": "", "line": 164, "column": "", "context": [ " $('#pma_console_options input[name=current_query]').prop('c//...", " PMA_console.updateConfig();", " });", "", " $(document).ajaxComplete(function (event, xhr) {", " var data = $.parseJSON(xhr.responseText);", " PMA_console.ajaxCallback(data);", " });", "", " PMA_console.isInitialized = true;", " }" ], "filename": "console.js" }, { "func": "ErrorReport.wrap_function/new_func", "args": "", "line": 262, "column": "", "context": [ " */", " wrap_function: function (func) {", " if (!func.wrapped) {", " var new_func = function () {", " try {", " return func.apply(this, arguments);", " } catch (x) {", " TraceKit.report(x);", " }", " };", " new_func.wrapped = true;" ], "filename": "error_report.js" }, { "func": "v.event.dispatch", "args": "", "line": 2, "column": "", "context": [ "/*! jQuery v1.8.3 jquery.com <http://jquery.com> | jquery.org/license <http://jquery.org/license> */\r", "(function(e,t){function _(e){var t=M[e]={};return v.each(e.split(y),functio//...", "", "function sprintf() {", "/*", " * Copyright (c) 2013 Kevin van Zonneveld (http://kvz.io)", " * and Contributors (http://phpjs.org/authors)" ], "filename": "jquery/jquery-1.8.3.min.js" }, { "func": "$event.dispatch", "args": "", "line": 372, "column": "", "context": [ "$event.dispatch = function( event ){", "\tif ( $.data( this, \"suppress.\"+ event.type ) - new Date().getTime() > 0 ){", "\t\t$.removeData( this, \"suppress.\"+ event.type );", "\t\treturn;", "\t}", "\treturn $dispatch.apply( this, arguments );", "};", "", "// event fix hooks for touch events...", "var touchHooks = ", "$event.fixHooks.touchstart = " ], "filename": "jquery/jquery.event.drag-2.2.js" }, { "func": "v.event.add/u", "args": "", "line": 2, "column": "", "context": [ "/*! jQuery v1.8.3 jquery.com <http://jquery.com> | jquery.org/license <http://jquery.org/license> */\r", "(function(e,t){function _(e){var t=M[e]={};return v.each(e.split(y),functio//...", "", "function sprintf() {", "/*", " * Copyright (c) 2013 Kevin van Zonneveld (http://kvz.io)", " * and Contributors (http://phpjs.org/authors)" ], "filename": "jquery/jquery-1.8.3.min.js" }, { "func": "v.event.trigger", "args": "", "line": 2, "column": "", "context": [ "/*! jQuery v1.8.3 jquery.com <http://jquery.com> | jquery.org/license <http://jquery.org/license> */\r", "(function(e,t){function _(e){var t=M[e]={};return v.each(e.split(y),functio//...", "", "function sprintf() {", "/*", " * Copyright (c) 2013 Kevin van Zonneveld (http://kvz.io)", " * and Contributors (http://phpjs.org/authors)" ], "filename": "jquery/jquery-1.8.3.min.js" }, { "func": "v.event.trigger", "args": "", "line": 2, "column": "", "context": [ "/*! jQuery v1.8.3 jquery.com <http://jquery.com> | jquery.org/license <http://jquery.org/license> */\r", "(function(e,t){function _(e){var t=M[e]={};return v.each(e.split(y),functio//...", "", "function sprintf() {", "/*", " * Copyright (c) 2013 Kevin van Zonneveld (http://kvz.io)", " * and Contributors (http://phpjs.org/authors)" ], "filename": "jquery/jquery-1.8.3.min.js" }, { "func": "T", "args": "", "line": 2, "column": "", "context": [ "/*! jQuery v1.8.3 jquery.com <http://jquery.com> | jquery.org/license <http://jquery.org/license> */\r", "(function(e,t){function _(e){var t=M[e]={};return v.each(e.split(y),functio//...", "", "function sprintf() {", "/*", " * Copyright (c) 2013 Kevin van Zonneveld (http://kvz.io)", " * and Contributors (http://phpjs.org/authors)" ], "filename": "jquery/jquery-1.8.3.min.js" }, { "func": ".send/r", "args": "", "line": 2, "column": "", "context": [ "/*! jQuery v1.8.3 jquery.com <http://jquery.com> | jquery.org/license <http://jquery.org/license> */\r", "(function(e,t){function _(e){var t=M[e]={};return v.each(e.split(y),functio//...", "", "function sprintf() {", "/*", " * Copyright (c) 2013 Kevin van Zonneveld (http://kvz.io)", " * and Contributors (http://phpjs.org/authors)" ], "filename": "jquery/jquery-1.8.3.min.js" } ], "useragent": "Mozilla/5.0 (X11; Linux x86_64; rv:24.0) Gecko/20140924 Firefox/24.0 Iceweasel/24.8.1", "incomplete": "false", "uri": "db_import.php?target=" }, "script_name": "db_import.php", "microhistory": { "pages": [ { "hash": "index.php?db=&table=&server=1&target=&token=c455fd8a0595a5032d18c07a64e7ab0a" }, { "hash": "db_structure.php?db=test&table=&server=1&target=&token=c455fd8a0595a5032d18c07a64e7ab0a", "params": { "opendb_url": "db_structure.php", "safari_browser": "0", "collation_connection": "utf8mb4_general_ci", "lang": "en", "server": "1", "text_dir": "ltr", "pma_text_default_tab": "Browse", "pma_text_left_default_tab": "Structure", "LimitChars": "50", "pftext": "", "confirm": "true", "LoginCookieValidity": "144000", "logged_in": "true" } }, { "hash": "index.php?db=&table=&server=1&target=&token=c455fd8a0595a5032d18c07a64e7ab0a", "params": { "opendb_url": "db_structure.php", "safari_browser": "0", "collation_connection": "utf8mb4_general_ci", "lang": "en", "server": "1", "text_dir": "ltr", "pma_text_default_tab": "Browse", "pma_text_left_default_tab": "Structure", "LimitChars": "50", "pftext": "", "confirm": "true", "LoginCookieValidity": "144000", "logged_in": "true" } }, { "hash": "db_structure.php?db=test&table=&server=1&target=&token=c455fd8a0595a5032d18c07a64e7ab0a", "params": { "opendb_url": "db_structure.php", "safari_browser": "0", "collation_connection": "utf8mb4_general_ci", "lang": "en", "server": "1", "text_dir": "ltr", "pma_text_default_tab": "Browse", "pma_text_left_default_tab": "Structure", "LimitChars": "50", "pftext": "", "confirm": "true", "LoginCookieValidity": "144000", "logged_in": "true" } }, { "hash": "db_import.php?db=test&table=&server=1&target=&token=c455fd8a0595a5032d18c07a64e7ab0a", "params": { "opendb_url": "db_structure.php", "safari_browser": "0", "collation_connection": "utf8mb4_general_ci", "lang": "en", "server": "1", "text_dir": "ltr", "pma_text_default_tab": "Browse", "pma_text_left_default_tab": "Structure", "LimitChars": "50", "pftext": "", "confirm": "true", "LoginCookieValidity": "144000", "logged_in": "true" } } ], "current_index": "5" } }
-- Marc Delisle | phpMyAdmin
Hi,
It seems that the new algorithm is over, see [1]. The unit tests are now ok (except for the one failing on master). I did some tests with escaped quotes (with quote or backslash), delimiters into strings, modified delimiters. Marc, do you still have your error please? (I didn't work on it, but I hope that your issue is linked to the not-finished version…)
On my home machine, sadly I still have the js error after importing. Tested on Iceweasel 31.2. Surely someone else can reproduce this problem on Debian wheezy? Because if it's only me, I don't care too much ;) With 3b3017d7, my test file (10000 rows) imports in 40 seconds in multibyte mode.
Feel free to look at the new algorithm and to tell me if you see any flaw. If you can lead to an error with a correct SQL file, please send it to me.
Thanks for your feedback.
[1] https://github.com/Tithugues/phpmyadmin/tree/stringFunctions_useStandardFunc...
Hugues.
Hi,
Since my previous email, I did some modifications to manage the display of queries in a different way (closer to the current display).
But there is something strange with the attachment… I can import the file (without error and all content executed), but on master, if I paste the content of the file into the SQL console, I've an error… Is it the expected behavior please? I don't see any exotic content in this file… On my branch, I don't have an error, but only the first statement (DROP PROCEDURE IF EXISTS) is executed…
Thanks, Hugues.
-- Marc Delisle | phpMyAdmin
2014-10-29 23:32 GMT+01:00 Marc Delisle <marc@infomarc.info>:
On my home machine, sadly I still have the js error after importing. Tested on Iceweasel 31.2.
Surely someone else can reproduce this problem on Debian wheezy? Because if it's only me, I don't care too much ;)
With 3b3017d7, my test file (10000 rows) imports in 40 seconds in multibyte mode.
-- Marc Delisle | phpMyAdmin
Hi, It bothers me… I'll try to read your trace (quite hard) and maybe I'll find the origin of the error. Nevertheless, are 40 seconds ok to you ? I think it's ok, mostly because there is now an option to use multibytes or not. Hugues.
Hugues Peccatte a écrit :
2014-10-29 23:32 GMT+01:00 Marc Delisle <marc@infomarc.info>:
On my home machine, sadly I still have the js error after importing. Tested on Iceweasel 31.2.
Surely someone else can reproduce this problem on Debian wheezy? Because if it's only me, I don't care too much ;)
With 3b3017d7, my test file (10000 rows) imports in 40 seconds in multibyte mode.
-- Marc Delisle | phpMyAdmin
Hi,
It bothers me… I'll try to read your trace (quite hard) and maybe I'll find the origin of the error. Nevertheless, are 40 seconds ok to you ? I think it's ok, mostly because there is now an option to use multibytes or not.
It's ok to me. Previously, users could not import using multibytes and it seems we cannot do better. I suggest you merge, to get more testers. -- Marc Delisle (phpMyAdmin)
2014-10-30 12:45 GMT+01:00 Marc Delisle <marc@infomarc.info>:
Hugues Peccatte a écrit :
2014-10-29 23:32 GMT+01:00 Marc Delisle <marc@infomarc.info>:
On my home machine, sadly I still have the js error after importing. Tested on Iceweasel 31.2.
Surely someone else can reproduce this problem on Debian wheezy? Because if it's only me, I don't care too much ;)
With 3b3017d7, my test file (10000 rows) imports in 40 seconds in multibyte mode.
-- Marc Delisle | phpMyAdmin
Hi,
It bothers me… I'll try to read your trace (quite hard) and maybe I'll find the origin of the error. Nevertheless, are 40 seconds ok to you ? I think it's ok, mostly because there is now an option to use multibytes or not.
It's ok to me. Previously, users could not import using multibytes and it seems we cannot do better.
I suggest you merge, to get more testers.
-- Marc Delisle (phpMyAdmin)
Hi, It's done. Please tell me if you have any issue about multibytes string or SQL import. Thanks a lot to all testers! Hugues.
Hugues Peccatte a écrit :
2014-10-19 21:53 GMT+02:00 Marc Delisle <marc@infomarc.info>:
Le 2014-10-19 15:01, Hugues Peccatte a écrit :
2014-10-19 13:21 GMT+02:00 Marc Delisle <marc@infomarc.info <mailto:marc@infomarc.info>>:
Le 2014-10-18 14:48, Hugues Peccatte a écrit : > 2014-10-13 12:21 GMT+02:00 Hugues Peccatte < hugues.peccatte@gmail.com <mailto:hugues.peccatte@gmail.com> > <mailto:hugues.peccatte@gmail.com <mailto:hugues.peccatte@gmail.com>>>: > > 2014-10-12 21:06 GMT+02:00 Marc Delisle <marc@infomarc.info <mailto:marc@infomarc.info> > <mailto:marc@infomarc.info <mailto:marc@infomarc.info>>>: > > Le 2014-10-12 12:57, Hugues Peccatte a écrit : > > 2014-10-05 20:36 GMT+02:00 Hugues Peccatte < hugues.peccatte@gmail.com <mailto:hugues.peccatte@gmail.com> <mailto:hugues.peccatte@gmail.com <mailto:hugues.peccatte@gmail.com > > <mailto:hugues.peccatte@gmail.com <mailto:hugues.peccatte@gmail.com> > <mailto:hugues.peccatte@gmail.com <mailto: hugues.peccatte@gmail.com>>>>: > > > > 2014-10-04 9:01 GMT+02:00 Hugues Peccatte < hugues.peccatte@gmail.com <mailto:hugues.peccatte@gmail.com> <mailto:hugues.peccatte@gmail.com <mailto:hugues.peccatte@gmail.com > > <mailto:hugues.peccatte@gmail.com <mailto:hugues.peccatte@gmail.com> > <mailto:hugues.peccatte@gmail.com <mailto: hugues.peccatte@gmail.com>>>>: > > > > Le 4 oct. 2014 03:22, "Madhura Jayaratne" < madhura.cj@gmail.com <mailto:madhura.cj@gmail.com> <mailto:madhura.cj@gmail.com <mailto:madhura.cj@gmail.com>> > > <mailto:madhura.cj@gmail.com <mailto:madhura.cj@gmail.com> > <mailto:madhura.cj@gmail.com <mailto:madhura.cj@gmail.com>>>> a écrit : > > > > > > > > > > > > > > > > On Sat, Oct 4, 2014 at 1:24 AM, Hugues Peccatte > > <hugues.peccatte@gmail.com <mailto: hugues.peccatte@gmail.com> > <mailto:hugues.peccatte@gmail.com <mailto: hugues.peccatte@gmail.com>> > <mailto:hugues.peccatte@gmail.com <mailto:hugues.peccatte@gmail.com> > <mailto:hugues.peccatte@gmail.com <mailto: hugues.peccatte@gmail.com>>>> > > wrote: > > >> > > >> 2014-10-03 12:26 GMT+02:00 Marc Delisle < marc@infomarc.info <mailto:marc@infomarc.info> <mailto:marc@infomarc.info <mailto:marc@infomarc.info>> > > <mailto:marc@infomarc.info <mailto:marc@infomarc.info> <mailto:marc@infomarc.info <mailto:marc@infomarc.info>>>>: > > >>> > > >>> Hi Hugues, > > >>> I retested this morning on a laptop, importing a > SQL file > > containing > > >>> 10000 employees from the sample employees > database. This is > > a small file > > >>> (660 KB). > > >>> > > >>> Current master: 3 min 25 sec (and ends with > JSON.parse: > > unexpected > > >>> character) > > >>> > > >>> Current Tithugues/stringFunctions_master: 2 min 10 > sec (same > > js error) > > >>> > > >>> Current QA_4_2: 0 min 5 sec > > >>> > > >>> There has been improvement, but we cannot release > 4.3 with > > this import > > >>> speed. > > >>> > > >>> -- > > >>> Marc Delisle | phpMyAdmin > > >> > > >> > > >> Hi, > > >> > > >> I agree… But I'm afraid this is linked to > multibytes functions… > > >> Maybe we shouldn't use the multibytes functions > everywhere… > > >> > > >> I'll still try to improve performances. > > >> > > >> Hugues. > > >> > > > > > > Indeed, I also think that we should use mb_* > function only > > when necessary and choice to use them should be made > in case by > > case basis. > > > > > > -- > > > Thanks and Regards, > > > > > > Madhura Jayaratne > > > > Hi, > > > > I didn't push my commits, but that's what I've started. I > > replaced the mb_* calls by standard calls on configuration > > variables, reserved words, etc. > > > > Hugues. > > > > > > Hi, > > > > Out of desperation, I try another algorithm. Instead of > buffering > > data until SQL delimiter, I'll try to parse all lines. > > So, I won't parse 1000 times a buffer of 50000 characters, > but less > > than 10 times many buffers of 500 characters. I hope this > will be > > faster. > > > > Hugues. > > > > > > Hi, > > > > The new algorithm is over. There are still some controls to > add, but it > > is usable with the file in this ticket: [1] > > You can find my modifications here: [2] > > > > Marc, is it faster for you ? > > It seems that I won ~33% of time. We're still far from 5 seconds… > > Maybe I'll try to use standard PHP functions to see the > difference. If > > the standard PHP functions are really faster, I'll try to add > an option > > to use mb_* functions or standard PHP functions, as you said. > > > > [1] https://sourceforge.net/p/phpmyadmin/bugs/4536/ > > [2] > > >
https://github.com/Tithugues/phpmyadmin/tree/stringFunctions_useStandardFunc...
> > Hi Hugues, > yes it's faster. With the same testing conditions, the
import
> takes 1 > min 20 sec. > > > -- > Marc Delisle | phpMyAdmin > > > Thanks for your feedback. > I'll try another another improvement to be faster. > > Note for my self: > * read X characters but don't restart the search from 0 each
time
> * search for the escaped quote with a lookbehind expression, > something like `(?!<\\)(\\\\)*'` > > Hugues. > > > Hi, > > As asked by Marc, I added on option to import by reading as a multibytes > string or not. > The default configuration won't read as multibytes string (because it's > too long…). It seems that the DnD to import doesn't use the default > configuration, so what ever you define as default, it won't be use
in
> this process. > Should we create a ticket for this ? I think it's possible to get it in > javascript. > > Hugues. >
With the new option "Read as multibytes" left to its default disabled value, my test file imports in 5 seconds.
When this option is ticked, it takes 60 seconds.
Note that in both cases, I get a Javascript error at the end (the
error
report does not show the same error in Firefox than in Chromium). Can anyone confirm this?
-- Marc Delisle | phpMyAdmin
Marc,
Could we please have both errors? I don't have it on Chrome. I'll try on Firefox and IE.
Thanks, Hugues. From Iceweasel 24.8.1:
{ "pma_version": "4.3.0-dev", "browser_name": "FIREFOX", "browser_version": "24.0", "user_os": "Linux", "server_software": "Apache/2.2.22 (Debian)", "user_agent_string": "Mozilla/5.0 (X11; Linux x86_64; rv:24.0) Gecko/20140924 Firefox/24.0 Iceweasel/24.8.1", "locale": "en", "configuration_storage": "enabled", "php_version": "5.4.4-14+deb7u14", "exception_type": "js", "exception": { "mode": "stack", "name": "SyntaxError", "message": "JSON.parse: unexpected character", "stack": [ { "func": ".parseJSON", "args": "", "line": 2, "column": "", "context": [ "/*! jQuery v1.8.3 jquery.com | jquery.org/license */\r", "(function(e,t){function _(e){var t=M[e]={};return v.each(e.split(y),functio//...", "", "function sprintf() {", "/*", " * Copyright (c) 2013 Kevin van Zonneveld (http://kvz.io)", " * and Contributors (http://phpjs.org/authors)" ], "filename": "jquery/jquery-1.8.3.min.js" }, { "func": "PMA_console.initialize/<", "args": "", "line": 164, "column": "", "context": [ " $('#pma_console_options input[name=current_query]').prop('c//...", " PMA_console.updateConfig();", " });", "", " $(document).ajaxComplete(function (event, xhr) {", " var data = $.parseJSON(xhr.responseText);", " PMA_console.ajaxCallback(data);", " });", "", " PMA_console.isInitialized = true;", " }" ], "filename": "console.js" }, { "func": "ErrorReport.wrap_function/new_func", "args": "", "line": 262, "column": "", "context": [ " */", " wrap_function: function (func) {", " if (!func.wrapped) {", " var new_func = function () {", " try {", " return func.apply(this, arguments);", " } catch (x) {", " TraceKit.report(x);", " }", " };", " new_func.wrapped = true;" ], "filename": "error_report.js" }, { "func": "v.event.dispatch", "args": "", "line": 2, "column": "", "context": [ "/*! jQuery v1.8.3 jquery.com | jquery.org/license */\r", "(function(e,t){function _(e){var t=M[e]={};return v.each(e.split(y),functio//...", "", "function sprintf() {", "/*", " * Copyright (c) 2013 Kevin van Zonneveld (http://kvz.io)", " * and Contributors (http://phpjs.org/authors)" ], "filename": "jquery/jquery-1.8.3.min.js" }, { "func": "$event.dispatch", "args": "", "line": 372, "column": "", "context": [ "$event.dispatch = function( event ){", "\tif ( $.data( this, \"suppress.\"+ event.type ) - new Date().getTime() > 0 ){", "\t\t$.removeData( this, \"suppress.\"+ event.type );", "\t\treturn;", "\t}", "\treturn $dispatch.apply( this, arguments );", "};", "", "// event fix hooks for touch events...", "var touchHooks = ", "$event.fixHooks.touchstart = " ], "filename": "jquery/jquery.event.drag-2.2.js" }, { "func": "v.event.add/u", "args": "", "line": 2, "column": "", "context": [ "/*! jQuery v1.8.3 jquery.com | jquery.org/license */\r", "(function(e,t){function _(e){var t=M[e]={};return v.each(e.split(y),functio//...", "", "function sprintf() {", "/*", " * Copyright (c) 2013 Kevin van Zonneveld (http://kvz.io)", " * and Contributors (http://phpjs.org/authors)" ], "filename": "jquery/jquery-1.8.3.min.js" }, { "func": "v.event.trigger", "args": "", "line": 2, "column": "", "context": [ "/*! jQuery v1.8.3 jquery.com | jquery.org/license */\r", "(function(e,t){function _(e){var t=M[e]={};return v.each(e.split(y),functio//...", "", "function sprintf() {", "/*", " * Copyright (c) 2013 Kevin van Zonneveld (http://kvz.io)", " * and Contributors (http://phpjs.org/authors)" ], "filename": "jquery/jquery-1.8.3.min.js" }, { "func": "v.event.trigger", "args": "", "line": 2, "column": "", "context": [ "/*! jQuery v1.8.3 jquery.com | jquery.org/license */\r", "(function(e,t){function _(e){var t=M[e]={};return v.each(e.split(y),functio//...", "", "function sprintf() {", "/*", " * Copyright (c) 2013 Kevin van Zonneveld (http://kvz.io)", " * and Contributors (http://phpjs.org/authors)" ], "filename": "jquery/jquery-1.8.3.min.js" }, { "func": "T", "args": "", "line": 2, "column": "", "context": [ "/*! jQuery v1.8.3 jquery.com | jquery.org/license */\r", "(function(e,t){function _(e){var t=M[e]={};return v.each(e.split(y),functio//...", "", "function sprintf() {", "/*", " * Copyright (c) 2013 Kevin van Zonneveld (http://kvz.io)", " * and Contributors (http://phpjs.org/authors)" ], "filename": "jquery/jquery-1.8.3.min.js" }, { "func": ".send/r", "args": "", "line": 2, "column": "", "context": [ "/*! jQuery v1.8.3 jquery.com | jquery.org/license */\r", "(function(e,t){function _(e){var t=M[e]={};return v.each(e.split(y),functio//...", "", "function sprintf() {", "/*", " * Copyright (c) 2013 Kevin van Zonneveld (http://kvz.io)", " * and Contributors (http://phpjs.org/authors)" ], "filename": "jquery/jquery-1.8.3.min.js" } ], "useragent": "Mozilla/5.0 (X11; Linux x86_64; rv:24.0) Gecko/20140924 Firefox/24.0 Iceweasel/24.8.1", "incomplete": "false", "uri": "db_import.php?target=" }, "script_name": "db_import.php", "microhistory": { "pages": [ { "hash":
"index.php?db=&table=&server=1&target=&token=c455fd8a0595a5032d18c07a64e7ab0a" }, { "hash":
"db_structure.php?db=test&table=&server=1&target=&token=c455fd8a0595a5032d18c07a64e7ab0a", "params": { "opendb_url": "db_structure.php", "safari_browser": "0", "collation_connection": "utf8mb4_general_ci", "lang": "en", "server": "1", "text_dir": "ltr", "pma_text_default_tab": "Browse", "pma_text_left_default_tab": "Structure", "LimitChars": "50", "pftext": "", "confirm": "true", "LoginCookieValidity": "144000", "logged_in": "true" } }, { "hash":
"index.php?db=&table=&server=1&target=&token=c455fd8a0595a5032d18c07a64e7ab0a", "params": { "opendb_url": "db_structure.php", "safari_browser": "0", "collation_connection": "utf8mb4_general_ci", "lang": "en", "server": "1", "text_dir": "ltr", "pma_text_default_tab": "Browse", "pma_text_left_default_tab": "Structure", "LimitChars": "50", "pftext": "", "confirm": "true", "LoginCookieValidity": "144000", "logged_in": "true" } }, { "hash":
"db_structure.php?db=test&table=&server=1&target=&token=c455fd8a0595a5032d18c07a64e7ab0a", "params": { "opendb_url": "db_structure.php", "safari_browser": "0", "collation_connection": "utf8mb4_general_ci", "lang": "en", "server": "1", "text_dir": "ltr", "pma_text_default_tab": "Browse", "pma_text_left_default_tab": "Structure", "LimitChars": "50", "pftext": "", "confirm": "true", "LoginCookieValidity": "144000", "logged_in": "true" } }, { "hash":
"db_import.php?db=test&table=&server=1&target=&token=c455fd8a0595a5032d18c07a64e7ab0a", "params": { "opendb_url": "db_structure.php", "safari_browser": "0", "collation_connection": "utf8mb4_general_ci", "lang": "en", "server": "1", "text_dir": "ltr", "pma_text_default_tab": "Browse", "pma_text_left_default_tab": "Structure", "LimitChars": "50", "pftext": "", "confirm": "true", "LoginCookieValidity": "144000", "logged_in": "true" } } ], "current_index": "5" } }
-- Marc Delisle | phpMyAdmin
Hi,
It seems that the new algorithm is over, see [1]. The unit tests are now ok (except for the one failing on master). I did some tests with escaped quotes (with quote or backslash), delimiters into strings, modified delimiters. Marc, do you still have your error please? (I didn't work on it, but I hope that your issue is linked to the not-finished version…)
I don't have the error at work but I have to retest at home.
Feel free to look at the new algorithm and to tell me if you see any flaw. If you can lead to an error with a correct SQL file, please send it to me.
Thanks for your feedback.
[1] https://github.com/Tithugues/phpmyadmin/tree/stringFunctions_useStandardFunc...
Hugues.
------------------------------------------------------------------------
------------------------------------------------------------------------------
------------------------------------------------------------------------
_______________________________________________ Phpmyadmin-devel mailing list Phpmyadmin-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/phpmyadmin-devel
-- Marc Delisle (phpMyAdmin)
Hugues Peccatte a écrit :
2014-09-22 18:44 GMT+02:00 Marc Delisle <marc@infomarc.info>:
Hugues, do you still have the branch where each commit was separate, before squashing? -- Marc Delisle (phpMyAdmin)
With "git reflog", I've been able to get it back:
Interesting command!
git checkout -b memory 972e038
What do you have in mind?
Possibly to find which smaller commit caused the problem. -- Marc Delisle (phpMyAdmin)
participants (4)
-
Chirayu Chiripal -
Hugues Peccatte -
Madhura Jayaratne -
Marc Delisle