[Phpmyadmin-devel] #4536 - master: import problem (PMA_String)

Hugues Peccatte hugues.peccatte at gmail.com
Tue Oct 28 20:58:51 CET 2014


2014-10-19 21:53 GMT+02:00 Marc Delisle <marc at infomarc.info>:

> Le 2014-10-19 15:01, Hugues Peccatte a écrit :
> > 2014-10-19 13:21 GMT+02:00 Marc Delisle <marc at infomarc.info
> > <mailto:marc at infomarc.info>>:
> >
> >     Le 2014-10-18 14:48, Hugues Peccatte a écrit :
> >     > 2014-10-13 12:21 GMT+02:00 Hugues Peccatte <
> hugues.peccatte at gmail.com <mailto:hugues.peccatte at gmail.com>
> >     > <mailto:hugues.peccatte at gmail.com
> >     <mailto:hugues.peccatte at gmail.com>>>:
> >     >
> >     >     2014-10-12 21:06 GMT+02:00 Marc Delisle <marc at infomarc.info
> <mailto:marc at infomarc.info>
> >     >     <mailto:marc at infomarc.info <mailto:marc at infomarc.info>>>:
> >     >
> >     >         Le 2014-10-12 12:57, Hugues Peccatte a écrit :
> >     >         > 2014-10-05 20:36 GMT+02:00 Hugues Peccatte <
> hugues.peccatte at gmail.com <mailto:hugues.peccatte at gmail.com>
> >     <mailto:hugues.peccatte at gmail.com <mailto:hugues.peccatte at gmail.com
> >>
> >     >         > <mailto:hugues.peccatte at gmail.com
> >     <mailto:hugues.peccatte at gmail.com>
> >     >         <mailto:hugues.peccatte at gmail.com <mailto:
> hugues.peccatte at gmail.com>>>>:
> >     >         >
> >     >         >     2014-10-04 9:01 GMT+02:00 Hugues Peccatte <
> hugues.peccatte at gmail.com <mailto:hugues.peccatte at gmail.com>
> >     <mailto:hugues.peccatte at gmail.com <mailto:hugues.peccatte at gmail.com
> >>
> >     >         >     <mailto:hugues.peccatte at gmail.com
> >     <mailto:hugues.peccatte at gmail.com>
> >     >         <mailto:hugues.peccatte at gmail.com <mailto:
> hugues.peccatte at gmail.com>>>>:
> >     >         >
> >     >         >         Le 4 oct. 2014 03:22, "Madhura Jayaratne" <
> madhura.cj at gmail.com <mailto:madhura.cj at gmail.com>
> >     <mailto:madhura.cj at gmail.com <mailto:madhura.cj at gmail.com>>
> >     >         >         <mailto:madhura.cj at gmail.com
> >     <mailto:madhura.cj at gmail.com>
> >     >         <mailto:madhura.cj at gmail.com <mailto:madhura.cj at gmail.com>>>>
> a écrit :
> >     >         >
> >     >         >
> >     >         >         >
> >     >         >         >
> >     >         >         >
> >     >         >         > On Sat, Oct 4, 2014 at 1:24 AM, Hugues Peccatte
> >     >         >         <hugues.peccatte at gmail.com <mailto:
> hugues.peccatte at gmail.com>
> >     >         <mailto:hugues.peccatte at gmail.com <mailto:
> hugues.peccatte at gmail.com>>
> >     >         <mailto:hugues.peccatte at gmail.com
> >     <mailto:hugues.peccatte at gmail.com>
> >     >         <mailto:hugues.peccatte at gmail.com <mailto:
> hugues.peccatte at gmail.com>>>>
> >     >         >         wrote:
> >     >         >         >>
> >     >         >         >> 2014-10-03 12:26 GMT+02:00 Marc Delisle <
> marc at infomarc.info <mailto:marc at infomarc.info>
> >     <mailto:marc at infomarc.info <mailto:marc at infomarc.info>>
> >     >         >         <mailto:marc at infomarc.info
> >     <mailto:marc at infomarc.info> <mailto:marc at infomarc.info
> >     <mailto:marc at infomarc.info>>>>:
> >     >         >         >>>
> >     >         >         >>> Hi Hugues,
> >     >         >         >>> I retested this morning on a laptop,
> importing a
> >     >         SQL file
> >     >         >         containing
> >     >         >         >>> 10000 employees from the sample employees
> >     >         database. This is
> >     >         >         a small file
> >     >         >         >>> (660 KB).
> >     >         >         >>>
> >     >         >         >>> Current master: 3 min 25 sec (and ends with
> >     >         JSON.parse:
> >     >         >         unexpected
> >     >         >         >>> character)
> >     >         >         >>>
> >     >         >         >>> Current Tithugues/stringFunctions_master: 2
> >     min 10
> >     >         sec (same
> >     >         >         js error)
> >     >         >         >>>
> >     >         >         >>> Current QA_4_2: 0 min 5 sec
> >     >         >         >>>
> >     >         >         >>> There has been improvement, but we cannot
> >     release
> >     >         4.3 with
> >     >         >         this import
> >     >         >         >>> speed.
> >     >         >         >>>
> >     >         >         >>> --
> >     >         >         >>> Marc Delisle | phpMyAdmin
> >     >         >         >>
> >     >         >         >>
> >     >         >         >> Hi,
> >     >         >         >>
> >     >         >         >> I agree… But I'm afraid this is linked to
> >     >         multibytes functions…
> >     >         >         >> Maybe we shouldn't use the multibytes
> functions
> >     >         everywhere…
> >     >         >         >>
> >     >         >         >> I'll still try to improve performances.
> >     >         >         >>
> >     >         >         >> Hugues.
> >     >         >         >>
> >     >         >         >
> >     >         >         > Indeed, I also think that we should use mb_*
> >     >         function only
> >     >         >         when necessary and choice to use them should be
> made
> >     >         in case by
> >     >         >         case basis.
> >     >         >         >
> >     >         >         > --
> >     >         >         > Thanks and Regards,
> >     >         >         >
> >     >         >         > Madhura Jayaratne
> >     >         >
> >     >         >         Hi,
> >     >         >
> >     >         >         I didn't push my commits, but that's what I've
> >     started. I
> >     >         >         replaced the mb_* calls by standard calls on
> >     configuration
> >     >         >         variables, reserved words, etc.
> >     >         >
> >     >         >         Hugues.
> >     >         >
> >     >         >
> >     >         >     Hi,
> >     >         >
> >     >         >     Out of desperation, I try another algorithm. Instead
> of
> >     >         buffering
> >     >         >     data until SQL delimiter, I'll try to parse all
> lines.
> >     >         >     So, I won't parse 1000 times a buffer of 50000
> >     characters,
> >     >         but less
> >     >         >     than 10 times many buffers of 500 characters. I hope
> >     this
> >     >         will be
> >     >         >     faster.
> >     >         >
> >     >         >     Hugues.
> >     >         >
> >     >         >
> >     >         > Hi,
> >     >         >
> >     >         > The new algorithm is over. There are still some controls
> to
> >     >         add, but it
> >     >         > is usable with the file in this ticket: [1]
> >     >         > You can find my modifications here: [2]
> >     >         >
> >     >         > Marc, is it faster for you ?
> >     >         > It seems that I won ~33% of time. We're still far from 5
> >     seconds…
> >     >         > Maybe I'll try to use standard PHP functions to see the
> >     >         difference. If
> >     >         > the standard PHP functions are really faster, I'll try
> >     to add
> >     >         an option
> >     >         > to use mb_* functions or standard PHP functions, as you
> >     said.
> >     >         >
> >     >         > [1] https://sourceforge.net/p/phpmyadmin/bugs/4536/
> >     >         > [2]
> >     >         >
> >     >
> >
> https://github.com/Tithugues/phpmyadmin/tree/stringFunctions_useStandardFunctions_master
> >     >
> >     >         Hi Hugues,
> >     >         yes it's faster. With the same testing conditions, the
> import
> >     >         takes 1
> >     >         min 20 sec.
> >     >
> >     >
> >     >         --
> >     >         Marc Delisle | phpMyAdmin
> >     >
> >     >
> >     >     Thanks for your feedback.
> >     >     I'll try another another improvement to be faster.
> >     >
> >     >     Note for my self:
> >     >     * read X characters but don't restart the search from 0 each
> time
> >     >     * search for the escaped quote with a lookbehind expression,
> >     >     something like `(?!<\\)(\\\\)*'`
> >     >
> >     >     Hugues.
> >     >
> >     >
> >     > Hi,
> >     >
> >     > As asked by Marc, I added on option to import by reading as a
> >     multibytes
> >     > string or not.
> >     > The default configuration won't read as multibytes string (because
> >     it's
> >     > too long…). It seems that the DnD to import doesn't use the default
> >     > configuration, so what ever you define as default, it won't be use
> in
> >     > this process.
> >     > Should we create a ticket for this ? I think it's possible to get
> >     it in
> >     > javascript.
> >     >
> >     > Hugues.
> >     >
> >
> >     With the new option "Read as multibytes" left to its default disabled
> >     value, my test file imports in 5 seconds.
> >
> >     When this option is ticked, it takes 60 seconds.
> >
> >     Note that in both cases, I get a Javascript error at the end (the
> error
> >     report does not show the same error in Firefox than in Chromium). Can
> >     anyone confirm this?
> >
> >
> >     --
> >     Marc Delisle | phpMyAdmin
> >
> >
> > Marc,
> >
> > Could we please have both errors? I don't have it on Chrome. I'll try on
> > Firefox and IE.
> >
> > Thanks,
> > Hugues.
>
> >From Iceweasel 24.8.1:
>
> {
>     "pma_version": "4.3.0-dev",
>     "browser_name": "FIREFOX",
>     "browser_version": "24.0",
>     "user_os": "Linux",
>     "server_software": "Apache/2.2.22 (Debian)",
>     "user_agent_string": "Mozilla/5.0 (X11; Linux x86_64; rv:24.0)
> Gecko/20140924 Firefox/24.0 Iceweasel/24.8.1",
>     "locale": "en",
>     "configuration_storage": "enabled",
>     "php_version": "5.4.4-14+deb7u14",
>     "exception_type": "js",
>     "exception": {
>         "mode": "stack",
>         "name": "SyntaxError",
>         "message": "JSON.parse: unexpected character",
>         "stack": [
>             {
>                 "func": ".parseJSON",
>                 "args": "",
>                 "line": 2,
>                 "column": "",
>                 "context": [
>                     "/*! jQuery v1.8.3 jquery.com | jquery.org/license
> */\r",
>                     "(function(e,t){function _(e){var t=M[e]={};return
> v.each(e.split(y),functio//...",
>                     "",
>                     "function sprintf() {",
>                     "/*",
>                     " * Copyright (c) 2013 Kevin van Zonneveld
> (http://kvz.io)",
>                     " * and Contributors (http://phpjs.org/authors)"
>                 ],
>                 "filename": "jquery/jquery-1.8.3.min.js"
>             },
>             {
>                 "func": "PMA_console.initialize/<",
>                 "args": "",
>                 "line": 164,
>                 "column": "",
>                 "context": [
>                     "                $('#pma_console_options
> input[name=current_query]').prop('c//...",
>                     "                PMA_console.updateConfig();",
>                     "            });",
>                     "",
>                     "            $(document).ajaxComplete(function
> (event, xhr) {",
>                     "                var data =
> $.parseJSON(xhr.responseText);",
>                     "                PMA_console.ajaxCallback(data);",
>                     "            });",
>                     "",
>                     "            PMA_console.isInitialized = true;",
>                     "        }"
>                 ],
>                 "filename": "console.js"
>             },
>             {
>                 "func": "ErrorReport.wrap_function/new_func",
>                 "args": "",
>                 "line": 262,
>                 "column": "",
>                 "context": [
>                     "     */",
>                     "    wrap_function: function (func) {",
>                     "        if (!func.wrapped) {",
>                     "            var new_func = function () {",
>                     "                try {",
>                     "                    return func.apply(this,
> arguments);",
>                     "                } catch (x) {",
>                     "                    TraceKit.report(x);",
>                     "                }",
>                     "            };",
>                     "            new_func.wrapped = true;"
>                 ],
>                 "filename": "error_report.js"
>             },
>             {
>                 "func": "v.event.dispatch",
>                 "args": "",
>                 "line": 2,
>                 "column": "",
>                 "context": [
>                     "/*! jQuery v1.8.3 jquery.com | jquery.org/license
> */\r",
>                     "(function(e,t){function _(e){var t=M[e]={};return
> v.each(e.split(y),functio//...",
>                     "",
>                     "function sprintf() {",
>                     "/*",
>                     " * Copyright (c) 2013 Kevin van Zonneveld
> (http://kvz.io)",
>                     " * and Contributors (http://phpjs.org/authors)"
>                 ],
>                 "filename": "jquery/jquery-1.8.3.min.js"
>             },
>             {
>                 "func": "$event.dispatch",
>                 "args": "",
>                 "line": 372,
>                 "column": "",
>                 "context": [
>                     "$event.dispatch = function( event ){",
>                     "\tif ( $.data( this, \"suppress.\"+ event.type ) -
> new Date().getTime() > 0 ){",
>                     "\t\t$.removeData( this, \"suppress.\"+ event.type );",
>                     "\t\treturn;",
>                     "\t}",
>                     "\treturn $dispatch.apply( this, arguments );",
>                     "};",
>                     "",
>                     "// event fix hooks for touch events...",
>                     "var touchHooks = ",
>                     "$event.fixHooks.touchstart = "
>                 ],
>                 "filename": "jquery/jquery.event.drag-2.2.js"
>             },
>             {
>                 "func": "v.event.add/u",
>                 "args": "",
>                 "line": 2,
>                 "column": "",
>                 "context": [
>                     "/*! jQuery v1.8.3 jquery.com | jquery.org/license
> */\r",
>                     "(function(e,t){function _(e){var t=M[e]={};return
> v.each(e.split(y),functio//...",
>                     "",
>                     "function sprintf() {",
>                     "/*",
>                     " * Copyright (c) 2013 Kevin van Zonneveld
> (http://kvz.io)",
>                     " * and Contributors (http://phpjs.org/authors)"
>                 ],
>                 "filename": "jquery/jquery-1.8.3.min.js"
>             },
>             {
>                 "func": "v.event.trigger",
>                 "args": "",
>                 "line": 2,
>                 "column": "",
>                 "context": [
>                     "/*! jQuery v1.8.3 jquery.com | jquery.org/license
> */\r",
>                     "(function(e,t){function _(e){var t=M[e]={};return
> v.each(e.split(y),functio//...",
>                     "",
>                     "function sprintf() {",
>                     "/*",
>                     " * Copyright (c) 2013 Kevin van Zonneveld
> (http://kvz.io)",
>                     " * and Contributors (http://phpjs.org/authors)"
>                 ],
>                 "filename": "jquery/jquery-1.8.3.min.js"
>             },
>             {
>                 "func": "v.event.trigger",
>                 "args": "",
>                 "line": 2,
>                 "column": "",
>                 "context": [
>                     "/*! jQuery v1.8.3 jquery.com | jquery.org/license
> */\r",
>                     "(function(e,t){function _(e){var t=M[e]={};return
> v.each(e.split(y),functio//...",
>                     "",
>                     "function sprintf() {",
>                     "/*",
>                     " * Copyright (c) 2013 Kevin van Zonneveld
> (http://kvz.io)",
>                     " * and Contributors (http://phpjs.org/authors)"
>                 ],
>                 "filename": "jquery/jquery-1.8.3.min.js"
>             },
>             {
>                 "func": "T",
>                 "args": "",
>                 "line": 2,
>                 "column": "",
>                 "context": [
>                     "/*! jQuery v1.8.3 jquery.com | jquery.org/license
> */\r",
>                     "(function(e,t){function _(e){var t=M[e]={};return
> v.each(e.split(y),functio//...",
>                     "",
>                     "function sprintf() {",
>                     "/*",
>                     " * Copyright (c) 2013 Kevin van Zonneveld
> (http://kvz.io)",
>                     " * and Contributors (http://phpjs.org/authors)"
>                 ],
>                 "filename": "jquery/jquery-1.8.3.min.js"
>             },
>             {
>                 "func": ".send/r",
>                 "args": "",
>                 "line": 2,
>                 "column": "",
>                 "context": [
>                     "/*! jQuery v1.8.3 jquery.com | jquery.org/license
> */\r",
>                     "(function(e,t){function _(e){var t=M[e]={};return
> v.each(e.split(y),functio//...",
>                     "",
>                     "function sprintf() {",
>                     "/*",
>                     " * Copyright (c) 2013 Kevin van Zonneveld
> (http://kvz.io)",
>                     " * and Contributors (http://phpjs.org/authors)"
>                 ],
>                 "filename": "jquery/jquery-1.8.3.min.js"
>             }
>         ],
>         "useragent": "Mozilla/5.0 (X11; Linux x86_64; rv:24.0)
> Gecko/20140924 Firefox/24.0 Iceweasel/24.8.1",
>         "incomplete": "false",
>         "uri": "db_import.php?target="
>     },
>     "script_name": "db_import.php",
>     "microhistory": {
>         "pages": [
>             {
>                 "hash":
>
> "index.php?db=&table=&server=1&target=&token=c455fd8a0595a5032d18c07a64e7ab0a"
>             },
>             {
>                 "hash":
>
> "db_structure.php?db=test&table=&server=1&target=&token=c455fd8a0595a5032d18c07a64e7ab0a",
>                 "params": {
>                     "opendb_url": "db_structure.php",
>                     "safari_browser": "0",
>                     "collation_connection": "utf8mb4_general_ci",
>                     "lang": "en",
>                     "server": "1",
>                     "text_dir": "ltr",
>                     "pma_text_default_tab": "Browse",
>                     "pma_text_left_default_tab": "Structure",
>                     "LimitChars": "50",
>                     "pftext": "",
>                     "confirm": "true",
>                     "LoginCookieValidity": "144000",
>                     "logged_in": "true"
>                 }
>             },
>             {
>                 "hash":
>
> "index.php?db=&table=&server=1&target=&token=c455fd8a0595a5032d18c07a64e7ab0a",
>                 "params": {
>                     "opendb_url": "db_structure.php",
>                     "safari_browser": "0",
>                     "collation_connection": "utf8mb4_general_ci",
>                     "lang": "en",
>                     "server": "1",
>                     "text_dir": "ltr",
>                     "pma_text_default_tab": "Browse",
>                     "pma_text_left_default_tab": "Structure",
>                     "LimitChars": "50",
>                     "pftext": "",
>                     "confirm": "true",
>                     "LoginCookieValidity": "144000",
>                     "logged_in": "true"
>                 }
>             },
>             {
>                 "hash":
>
> "db_structure.php?db=test&table=&server=1&target=&token=c455fd8a0595a5032d18c07a64e7ab0a",
>                 "params": {
>                     "opendb_url": "db_structure.php",
>                     "safari_browser": "0",
>                     "collation_connection": "utf8mb4_general_ci",
>                     "lang": "en",
>                     "server": "1",
>                     "text_dir": "ltr",
>                     "pma_text_default_tab": "Browse",
>                     "pma_text_left_default_tab": "Structure",
>                     "LimitChars": "50",
>                     "pftext": "",
>                     "confirm": "true",
>                     "LoginCookieValidity": "144000",
>                     "logged_in": "true"
>                 }
>             },
>             {
>                 "hash":
>
> "db_import.php?db=test&table=&server=1&target=&token=c455fd8a0595a5032d18c07a64e7ab0a",
>                 "params": {
>                     "opendb_url": "db_structure.php",
>                     "safari_browser": "0",
>                     "collation_connection": "utf8mb4_general_ci",
>                     "lang": "en",
>                     "server": "1",
>                     "text_dir": "ltr",
>                     "pma_text_default_tab": "Browse",
>                     "pma_text_left_default_tab": "Structure",
>                     "LimitChars": "50",
>                     "pftext": "",
>                     "confirm": "true",
>                     "LoginCookieValidity": "144000",
>                     "logged_in": "true"
>                 }
>             }
>         ],
>         "current_index": "5"
>     }
> }
>
> --
> Marc Delisle | phpMyAdmin
>

Hi,

It seems that the new algorithm is over, see [1]. The unit tests are now ok
(except for the one failing on master). I did some tests with escaped
quotes (with quote or backslash), delimiters into strings, modified
delimiters.
Marc, do you still have your error please? (I didn't work on it, but I hope
that your issue is linked to the not-finished version…)

Feel free to look at the new algorithm and to tell me if you see any flaw.
If you can lead to an error with a correct SQL file, please send it to me.

Thanks for your feedback.

[1]
https://github.com/Tithugues/phpmyadmin/tree/stringFunctions_useStandardFunctions_master

Hugues.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.phpmyadmin.net/pipermail/developers/attachments/20141028/b0c238a7/attachment.html>


More information about the Developers mailing list