[Phpmyadmin-devel] #4536 - master: import problem (PMA_String)

Hugues Peccatte hugues.peccatte at gmail.com
Tue Oct 28 22:24:12 CET 2014


2014-10-28 20:58 GMT+01:00 Hugues Peccatte <hugues.peccatte at gmail.com>:

> 2014-10-19 21:53 GMT+02:00 Marc Delisle <marc at infomarc.info>:
>
>> Le 2014-10-19 15:01, Hugues Peccatte a écrit :
>> > 2014-10-19 13:21 GMT+02:00 Marc Delisle <marc at infomarc.info
>> > <mailto:marc at infomarc.info>>:
>> >
>> >     Le 2014-10-18 14:48, Hugues Peccatte a écrit :
>> >     > 2014-10-13 12:21 GMT+02:00 Hugues Peccatte <
>> hugues.peccatte at gmail.com <mailto:hugues.peccatte at gmail.com>
>> >     > <mailto:hugues.peccatte at gmail.com
>> >     <mailto:hugues.peccatte at gmail.com>>>:
>> >     >
>> >     >     2014-10-12 21:06 GMT+02:00 Marc Delisle <marc at infomarc.info
>> <mailto:marc at infomarc.info>
>> >     >     <mailto:marc at infomarc.info <mailto:marc at infomarc.info>>>:
>> >     >
>> >     >         Le 2014-10-12 12:57, Hugues Peccatte a écrit :
>> >     >         > 2014-10-05 20:36 GMT+02:00 Hugues Peccatte <
>> hugues.peccatte at gmail.com <mailto:hugues.peccatte at gmail.com>
>> >     <mailto:hugues.peccatte at gmail.com <mailto:hugues.peccatte at gmail.com
>> >>
>> >     >         > <mailto:hugues.peccatte at gmail.com
>> >     <mailto:hugues.peccatte at gmail.com>
>> >     >         <mailto:hugues.peccatte at gmail.com <mailto:
>> hugues.peccatte at gmail.com>>>>:
>> >     >         >
>> >     >         >     2014-10-04 9:01 GMT+02:00 Hugues Peccatte <
>> hugues.peccatte at gmail.com <mailto:hugues.peccatte at gmail.com>
>> >     <mailto:hugues.peccatte at gmail.com <mailto:hugues.peccatte at gmail.com
>> >>
>> >     >         >     <mailto:hugues.peccatte at gmail.com
>> >     <mailto:hugues.peccatte at gmail.com>
>> >     >         <mailto:hugues.peccatte at gmail.com <mailto:
>> hugues.peccatte at gmail.com>>>>:
>> >     >         >
>> >     >         >         Le 4 oct. 2014 03:22, "Madhura Jayaratne" <
>> madhura.cj at gmail.com <mailto:madhura.cj at gmail.com>
>> >     <mailto:madhura.cj at gmail.com <mailto:madhura.cj at gmail.com>>
>> >     >         >         <mailto:madhura.cj at gmail.com
>> >     <mailto:madhura.cj at gmail.com>
>> >     >         <mailto:madhura.cj at gmail.com <mailto:madhura.cj at gmail.com>>>>
>> a écrit :
>> >     >         >
>> >     >         >
>> >     >         >         >
>> >     >         >         >
>> >     >         >         >
>> >     >         >         > On Sat, Oct 4, 2014 at 1:24 AM, Hugues
>> Peccatte
>> >     >         >         <hugues.peccatte at gmail.com <mailto:
>> hugues.peccatte at gmail.com>
>> >     >         <mailto:hugues.peccatte at gmail.com <mailto:
>> hugues.peccatte at gmail.com>>
>> >     >         <mailto:hugues.peccatte at gmail.com
>> >     <mailto:hugues.peccatte at gmail.com>
>> >     >         <mailto:hugues.peccatte at gmail.com <mailto:
>> hugues.peccatte at gmail.com>>>>
>> >     >         >         wrote:
>> >     >         >         >>
>> >     >         >         >> 2014-10-03 12:26 GMT+02:00 Marc Delisle <
>> marc at infomarc.info <mailto:marc at infomarc.info>
>> >     <mailto:marc at infomarc.info <mailto:marc at infomarc.info>>
>> >     >         >         <mailto:marc at infomarc.info
>> >     <mailto:marc at infomarc.info> <mailto:marc at infomarc.info
>> >     <mailto:marc at infomarc.info>>>>:
>> >     >         >         >>>
>> >     >         >         >>> Hi Hugues,
>> >     >         >         >>> I retested this morning on a laptop,
>> importing a
>> >     >         SQL file
>> >     >         >         containing
>> >     >         >         >>> 10000 employees from the sample employees
>> >     >         database. This is
>> >     >         >         a small file
>> >     >         >         >>> (660 KB).
>> >     >         >         >>>
>> >     >         >         >>> Current master: 3 min 25 sec (and ends with
>> >     >         JSON.parse:
>> >     >         >         unexpected
>> >     >         >         >>> character)
>> >     >         >         >>>
>> >     >         >         >>> Current Tithugues/stringFunctions_master: 2
>> >     min 10
>> >     >         sec (same
>> >     >         >         js error)
>> >     >         >         >>>
>> >     >         >         >>> Current QA_4_2: 0 min 5 sec
>> >     >         >         >>>
>> >     >         >         >>> There has been improvement, but we cannot
>> >     release
>> >     >         4.3 with
>> >     >         >         this import
>> >     >         >         >>> speed.
>> >     >         >         >>>
>> >     >         >         >>> --
>> >     >         >         >>> Marc Delisle | phpMyAdmin
>> >     >         >         >>
>> >     >         >         >>
>> >     >         >         >> Hi,
>> >     >         >         >>
>> >     >         >         >> I agree… But I'm afraid this is linked to
>> >     >         multibytes functions…
>> >     >         >         >> Maybe we shouldn't use the multibytes
>> functions
>> >     >         everywhere…
>> >     >         >         >>
>> >     >         >         >> I'll still try to improve performances.
>> >     >         >         >>
>> >     >         >         >> Hugues.
>> >     >         >         >>
>> >     >         >         >
>> >     >         >         > Indeed, I also think that we should use mb_*
>> >     >         function only
>> >     >         >         when necessary and choice to use them should be
>> made
>> >     >         in case by
>> >     >         >         case basis.
>> >     >         >         >
>> >     >         >         > --
>> >     >         >         > Thanks and Regards,
>> >     >         >         >
>> >     >         >         > Madhura Jayaratne
>> >     >         >
>> >     >         >         Hi,
>> >     >         >
>> >     >         >         I didn't push my commits, but that's what I've
>> >     started. I
>> >     >         >         replaced the mb_* calls by standard calls on
>> >     configuration
>> >     >         >         variables, reserved words, etc.
>> >     >         >
>> >     >         >         Hugues.
>> >     >         >
>> >     >         >
>> >     >         >     Hi,
>> >     >         >
>> >     >         >     Out of desperation, I try another algorithm.
>> Instead of
>> >     >         buffering
>> >     >         >     data until SQL delimiter, I'll try to parse all
>> lines.
>> >     >         >     So, I won't parse 1000 times a buffer of 50000
>> >     characters,
>> >     >         but less
>> >     >         >     than 10 times many buffers of 500 characters. I hope
>> >     this
>> >     >         will be
>> >     >         >     faster.
>> >     >         >
>> >     >         >     Hugues.
>> >     >         >
>> >     >         >
>> >     >         > Hi,
>> >     >         >
>> >     >         > The new algorithm is over. There are still some
>> controls to
>> >     >         add, but it
>> >     >         > is usable with the file in this ticket: [1]
>> >     >         > You can find my modifications here: [2]
>> >     >         >
>> >     >         > Marc, is it faster for you ?
>> >     >         > It seems that I won ~33% of time. We're still far from 5
>> >     seconds…
>> >     >         > Maybe I'll try to use standard PHP functions to see the
>> >     >         difference. If
>> >     >         > the standard PHP functions are really faster, I'll try
>> >     to add
>> >     >         an option
>> >     >         > to use mb_* functions or standard PHP functions, as you
>> >     said.
>> >     >         >
>> >     >         > [1] https://sourceforge.net/p/phpmyadmin/bugs/4536/
>> >     >         > [2]
>> >     >         >
>> >     >
>> >
>> https://github.com/Tithugues/phpmyadmin/tree/stringFunctions_useStandardFunctions_master
>> >     >
>> >     >         Hi Hugues,
>> >     >         yes it's faster. With the same testing conditions, the
>> import
>> >     >         takes 1
>> >     >         min 20 sec.
>> >     >
>> >     >
>> >     >         --
>> >     >         Marc Delisle | phpMyAdmin
>> >     >
>> >     >
>> >     >     Thanks for your feedback.
>> >     >     I'll try another another improvement to be faster.
>> >     >
>> >     >     Note for my self:
>> >     >     * read X characters but don't restart the search from 0 each
>> time
>> >     >     * search for the escaped quote with a lookbehind expression,
>> >     >     something like `(?!<\\)(\\\\)*'`
>> >     >
>> >     >     Hugues.
>> >     >
>> >     >
>> >     > Hi,
>> >     >
>> >     > As asked by Marc, I added on option to import by reading as a
>> >     multibytes
>> >     > string or not.
>> >     > The default configuration won't read as multibytes string (because
>> >     it's
>> >     > too long…). It seems that the DnD to import doesn't use the
>> default
>> >     > configuration, so what ever you define as default, it won't be
>> use in
>> >     > this process.
>> >     > Should we create a ticket for this ? I think it's possible to get
>> >     it in
>> >     > javascript.
>> >     >
>> >     > Hugues.
>> >     >
>> >
>> >     With the new option "Read as multibytes" left to its default
>> disabled
>> >     value, my test file imports in 5 seconds.
>> >
>> >     When this option is ticked, it takes 60 seconds.
>> >
>> >     Note that in both cases, I get a Javascript error at the end (the
>> error
>> >     report does not show the same error in Firefox than in Chromium).
>> Can
>> >     anyone confirm this?
>> >
>> >
>> >     --
>> >     Marc Delisle | phpMyAdmin
>> >
>> >
>> > Marc,
>> >
>> > Could we please have both errors? I don't have it on Chrome. I'll try on
>> > Firefox and IE.
>> >
>> > Thanks,
>> > Hugues.
>>
>> >From Iceweasel 24.8.1:
>>
>> {
>>     "pma_version": "4.3.0-dev",
>>     "browser_name": "FIREFOX",
>>     "browser_version": "24.0",
>>     "user_os": "Linux",
>>     "server_software": "Apache/2.2.22 (Debian)",
>>     "user_agent_string": "Mozilla/5.0 (X11; Linux x86_64; rv:24.0)
>> Gecko/20140924 Firefox/24.0 Iceweasel/24.8.1",
>>     "locale": "en",
>>     "configuration_storage": "enabled",
>>     "php_version": "5.4.4-14+deb7u14",
>>     "exception_type": "js",
>>     "exception": {
>>         "mode": "stack",
>>         "name": "SyntaxError",
>>         "message": "JSON.parse: unexpected character",
>>         "stack": [
>>             {
>>                 "func": ".parseJSON",
>>                 "args": "",
>>                 "line": 2,
>>                 "column": "",
>>                 "context": [
>>                     "/*! jQuery v1.8.3 jquery.com | jquery.org/license
>> */\r",
>>                     "(function(e,t){function _(e){var t=M[e]={};return
>> v.each(e.split(y),functio//...",
>>                     "",
>>                     "function sprintf() {",
>>                     "/*",
>>                     " * Copyright (c) 2013 Kevin van Zonneveld
>> (http://kvz.io)",
>>                     " * and Contributors (http://phpjs.org/authors)"
>>                 ],
>>                 "filename": "jquery/jquery-1.8.3.min.js"
>>             },
>>             {
>>                 "func": "PMA_console.initialize/<",
>>                 "args": "",
>>                 "line": 164,
>>                 "column": "",
>>                 "context": [
>>                     "                $('#pma_console_options
>> input[name=current_query]').prop('c//...",
>>                     "                PMA_console.updateConfig();",
>>                     "            });",
>>                     "",
>>                     "            $(document).ajaxComplete(function
>> (event, xhr) {",
>>                     "                var data =
>> $.parseJSON(xhr.responseText);",
>>                     "                PMA_console.ajaxCallback(data);",
>>                     "            });",
>>                     "",
>>                     "            PMA_console.isInitialized = true;",
>>                     "        }"
>>                 ],
>>                 "filename": "console.js"
>>             },
>>             {
>>                 "func": "ErrorReport.wrap_function/new_func",
>>                 "args": "",
>>                 "line": 262,
>>                 "column": "",
>>                 "context": [
>>                     "     */",
>>                     "    wrap_function: function (func) {",
>>                     "        if (!func.wrapped) {",
>>                     "            var new_func = function () {",
>>                     "                try {",
>>                     "                    return func.apply(this,
>> arguments);",
>>                     "                } catch (x) {",
>>                     "                    TraceKit.report(x);",
>>                     "                }",
>>                     "            };",
>>                     "            new_func.wrapped = true;"
>>                 ],
>>                 "filename": "error_report.js"
>>             },
>>             {
>>                 "func": "v.event.dispatch",
>>                 "args": "",
>>                 "line": 2,
>>                 "column": "",
>>                 "context": [
>>                     "/*! jQuery v1.8.3 jquery.com | jquery.org/license
>> */\r",
>>                     "(function(e,t){function _(e){var t=M[e]={};return
>> v.each(e.split(y),functio//...",
>>                     "",
>>                     "function sprintf() {",
>>                     "/*",
>>                     " * Copyright (c) 2013 Kevin van Zonneveld
>> (http://kvz.io)",
>>                     " * and Contributors (http://phpjs.org/authors)"
>>                 ],
>>                 "filename": "jquery/jquery-1.8.3.min.js"
>>             },
>>             {
>>                 "func": "$event.dispatch",
>>                 "args": "",
>>                 "line": 372,
>>                 "column": "",
>>                 "context": [
>>                     "$event.dispatch = function( event ){",
>>                     "\tif ( $.data( this, \"suppress.\"+ event.type ) -
>> new Date().getTime() > 0 ){",
>>                     "\t\t$.removeData( this, \"suppress.\"+ event.type
>> );",
>>                     "\t\treturn;",
>>                     "\t}",
>>                     "\treturn $dispatch.apply( this, arguments );",
>>                     "};",
>>                     "",
>>                     "// event fix hooks for touch events...",
>>                     "var touchHooks = ",
>>                     "$event.fixHooks.touchstart = "
>>                 ],
>>                 "filename": "jquery/jquery.event.drag-2.2.js"
>>             },
>>             {
>>                 "func": "v.event.add/u",
>>                 "args": "",
>>                 "line": 2,
>>                 "column": "",
>>                 "context": [
>>                     "/*! jQuery v1.8.3 jquery.com | jquery.org/license
>> */\r",
>>                     "(function(e,t){function _(e){var t=M[e]={};return
>> v.each(e.split(y),functio//...",
>>                     "",
>>                     "function sprintf() {",
>>                     "/*",
>>                     " * Copyright (c) 2013 Kevin van Zonneveld
>> (http://kvz.io)",
>>                     " * and Contributors (http://phpjs.org/authors)"
>>                 ],
>>                 "filename": "jquery/jquery-1.8.3.min.js"
>>             },
>>             {
>>                 "func": "v.event.trigger",
>>                 "args": "",
>>                 "line": 2,
>>                 "column": "",
>>                 "context": [
>>                     "/*! jQuery v1.8.3 jquery.com | jquery.org/license
>> */\r",
>>                     "(function(e,t){function _(e){var t=M[e]={};return
>> v.each(e.split(y),functio//...",
>>                     "",
>>                     "function sprintf() {",
>>                     "/*",
>>                     " * Copyright (c) 2013 Kevin van Zonneveld
>> (http://kvz.io)",
>>                     " * and Contributors (http://phpjs.org/authors)"
>>                 ],
>>                 "filename": "jquery/jquery-1.8.3.min.js"
>>             },
>>             {
>>                 "func": "v.event.trigger",
>>                 "args": "",
>>                 "line": 2,
>>                 "column": "",
>>                 "context": [
>>                     "/*! jQuery v1.8.3 jquery.com | jquery.org/license
>> */\r",
>>                     "(function(e,t){function _(e){var t=M[e]={};return
>> v.each(e.split(y),functio//...",
>>                     "",
>>                     "function sprintf() {",
>>                     "/*",
>>                     " * Copyright (c) 2013 Kevin van Zonneveld
>> (http://kvz.io)",
>>                     " * and Contributors (http://phpjs.org/authors)"
>>                 ],
>>                 "filename": "jquery/jquery-1.8.3.min.js"
>>             },
>>             {
>>                 "func": "T",
>>                 "args": "",
>>                 "line": 2,
>>                 "column": "",
>>                 "context": [
>>                     "/*! jQuery v1.8.3 jquery.com | jquery.org/license
>> */\r",
>>                     "(function(e,t){function _(e){var t=M[e]={};return
>> v.each(e.split(y),functio//...",
>>                     "",
>>                     "function sprintf() {",
>>                     "/*",
>>                     " * Copyright (c) 2013 Kevin van Zonneveld
>> (http://kvz.io)",
>>                     " * and Contributors (http://phpjs.org/authors)"
>>                 ],
>>                 "filename": "jquery/jquery-1.8.3.min.js"
>>             },
>>             {
>>                 "func": ".send/r",
>>                 "args": "",
>>                 "line": 2,
>>                 "column": "",
>>                 "context": [
>>                     "/*! jQuery v1.8.3 jquery.com | jquery.org/license
>> */\r",
>>                     "(function(e,t){function _(e){var t=M[e]={};return
>> v.each(e.split(y),functio//...",
>>                     "",
>>                     "function sprintf() {",
>>                     "/*",
>>                     " * Copyright (c) 2013 Kevin van Zonneveld
>> (http://kvz.io)",
>>                     " * and Contributors (http://phpjs.org/authors)"
>>                 ],
>>                 "filename": "jquery/jquery-1.8.3.min.js"
>>             }
>>         ],
>>         "useragent": "Mozilla/5.0 (X11; Linux x86_64; rv:24.0)
>> Gecko/20140924 Firefox/24.0 Iceweasel/24.8.1",
>>         "incomplete": "false",
>>         "uri": "db_import.php?target="
>>     },
>>     "script_name": "db_import.php",
>>     "microhistory": {
>>         "pages": [
>>             {
>>                 "hash":
>>
>> "index.php?db=&table=&server=1&target=&token=c455fd8a0595a5032d18c07a64e7ab0a"
>>             },
>>             {
>>                 "hash":
>>
>> "db_structure.php?db=test&table=&server=1&target=&token=c455fd8a0595a5032d18c07a64e7ab0a",
>>                 "params": {
>>                     "opendb_url": "db_structure.php",
>>                     "safari_browser": "0",
>>                     "collation_connection": "utf8mb4_general_ci",
>>                     "lang": "en",
>>                     "server": "1",
>>                     "text_dir": "ltr",
>>                     "pma_text_default_tab": "Browse",
>>                     "pma_text_left_default_tab": "Structure",
>>                     "LimitChars": "50",
>>                     "pftext": "",
>>                     "confirm": "true",
>>                     "LoginCookieValidity": "144000",
>>                     "logged_in": "true"
>>                 }
>>             },
>>             {
>>                 "hash":
>>
>> "index.php?db=&table=&server=1&target=&token=c455fd8a0595a5032d18c07a64e7ab0a",
>>                 "params": {
>>                     "opendb_url": "db_structure.php",
>>                     "safari_browser": "0",
>>                     "collation_connection": "utf8mb4_general_ci",
>>                     "lang": "en",
>>                     "server": "1",
>>                     "text_dir": "ltr",
>>                     "pma_text_default_tab": "Browse",
>>                     "pma_text_left_default_tab": "Structure",
>>                     "LimitChars": "50",
>>                     "pftext": "",
>>                     "confirm": "true",
>>                     "LoginCookieValidity": "144000",
>>                     "logged_in": "true"
>>                 }
>>             },
>>             {
>>                 "hash":
>>
>> "db_structure.php?db=test&table=&server=1&target=&token=c455fd8a0595a5032d18c07a64e7ab0a",
>>                 "params": {
>>                     "opendb_url": "db_structure.php",
>>                     "safari_browser": "0",
>>                     "collation_connection": "utf8mb4_general_ci",
>>                     "lang": "en",
>>                     "server": "1",
>>                     "text_dir": "ltr",
>>                     "pma_text_default_tab": "Browse",
>>                     "pma_text_left_default_tab": "Structure",
>>                     "LimitChars": "50",
>>                     "pftext": "",
>>                     "confirm": "true",
>>                     "LoginCookieValidity": "144000",
>>                     "logged_in": "true"
>>                 }
>>             },
>>             {
>>                 "hash":
>>
>> "db_import.php?db=test&table=&server=1&target=&token=c455fd8a0595a5032d18c07a64e7ab0a",
>>                 "params": {
>>                     "opendb_url": "db_structure.php",
>>                     "safari_browser": "0",
>>                     "collation_connection": "utf8mb4_general_ci",
>>                     "lang": "en",
>>                     "server": "1",
>>                     "text_dir": "ltr",
>>                     "pma_text_default_tab": "Browse",
>>                     "pma_text_left_default_tab": "Structure",
>>                     "LimitChars": "50",
>>                     "pftext": "",
>>                     "confirm": "true",
>>                     "LoginCookieValidity": "144000",
>>                     "logged_in": "true"
>>                 }
>>             }
>>         ],
>>         "current_index": "5"
>>     }
>> }
>>
>> --
>> Marc Delisle | phpMyAdmin
>>
>
> Hi,
>
> It seems that the new algorithm is over, see [1]. The unit tests are now
> ok (except for the one failing on master). I did some tests with escaped
> quotes (with quote or backslash), delimiters into strings, modified
> delimiters.
> Marc, do you still have your error please? (I didn't work on it, but I
> hope that your issue is linked to the not-finished version…)
>
> Feel free to look at the new algorithm and to tell me if you see any flaw.
> If you can lead to an error with a correct SQL file, please send it to me.
>
> Thanks for your feedback.
>
> [1]
> https://github.com/Tithugues/phpmyadmin/tree/stringFunctions_useStandardFunctions_master
>
> Hugues.
>

Hi,

Since my previous email, I did some modifications to manage the display of
queries in a different way (closer to the current display).

But there is something strange with the attachment… I can import the file
(without error and all content executed), but on master, if I paste the
content of the file into the SQL console, I've an error… Is it the expected
behavior please? I don't see any exotic content in this file…
On my branch, I don't have an error, but only the first statement (DROP
PROCEDURE IF EXISTS) is executed…

Thanks,
Hugues.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.phpmyadmin.net/pipermail/developers/attachments/20141028/e7ec4e8a/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: procedure.sql
Type: application/octet-stream
Size: 161 bytes
Desc: not available
URL: <http://lists.phpmyadmin.net/pipermail/developers/attachments/20141028/e7ec4e8a/attachment.obj>


More information about the Developers mailing list