[Phpmyadmin-devel] #4536 - master: import problem (PMA_String)
Marc Delisle
marc at infomarc.info
Wed Oct 29 13:08:18 CET 2014
Hugues Peccatte a écrit :
> 2014-10-19 21:53 GMT+02:00 Marc Delisle <marc at infomarc.info>:
>
>> Le 2014-10-19 15:01, Hugues Peccatte a écrit :
>>> 2014-10-19 13:21 GMT+02:00 Marc Delisle <marc at infomarc.info
>>> <mailto:marc at infomarc.info>>:
>>>
>>> Le 2014-10-18 14:48, Hugues Peccatte a écrit :
>>> > 2014-10-13 12:21 GMT+02:00 Hugues Peccatte <
>> hugues.peccatte at gmail.com <mailto:hugues.peccatte at gmail.com>
>>> > <mailto:hugues.peccatte at gmail.com
>>> <mailto:hugues.peccatte at gmail.com>>>:
>>> >
>>> > 2014-10-12 21:06 GMT+02:00 Marc Delisle <marc at infomarc.info
>> <mailto:marc at infomarc.info>
>>> > <mailto:marc at infomarc.info <mailto:marc at infomarc.info>>>:
>>> >
>>> > Le 2014-10-12 12:57, Hugues Peccatte a écrit :
>>> > > 2014-10-05 20:36 GMT+02:00 Hugues Peccatte <
>> hugues.peccatte at gmail.com <mailto:hugues.peccatte at gmail.com>
>>> <mailto:hugues.peccatte at gmail.com <mailto:hugues.peccatte at gmail.com
>>> > > <mailto:hugues.peccatte at gmail.com
>>> <mailto:hugues.peccatte at gmail.com>
>>> > <mailto:hugues.peccatte at gmail.com <mailto:
>> hugues.peccatte at gmail.com>>>>:
>>> > >
>>> > > 2014-10-04 9:01 GMT+02:00 Hugues Peccatte <
>> hugues.peccatte at gmail.com <mailto:hugues.peccatte at gmail.com>
>>> <mailto:hugues.peccatte at gmail.com <mailto:hugues.peccatte at gmail.com
>>> > > <mailto:hugues.peccatte at gmail.com
>>> <mailto:hugues.peccatte at gmail.com>
>>> > <mailto:hugues.peccatte at gmail.com <mailto:
>> hugues.peccatte at gmail.com>>>>:
>>> > >
>>> > > Le 4 oct. 2014 03:22, "Madhura Jayaratne" <
>> madhura.cj at gmail.com <mailto:madhura.cj at gmail.com>
>>> <mailto:madhura.cj at gmail.com <mailto:madhura.cj at gmail.com>>
>>> > > <mailto:madhura.cj at gmail.com
>>> <mailto:madhura.cj at gmail.com>
>>> > <mailto:madhura.cj at gmail.com <mailto:madhura.cj at gmail.com>>>>
>> a écrit :
>>> > >
>>> > >
>>> > > >
>>> > > >
>>> > > >
>>> > > > On Sat, Oct 4, 2014 at 1:24 AM, Hugues Peccatte
>>> > > <hugues.peccatte at gmail.com <mailto:
>> hugues.peccatte at gmail.com>
>>> > <mailto:hugues.peccatte at gmail.com <mailto:
>> hugues.peccatte at gmail.com>>
>>> > <mailto:hugues.peccatte at gmail.com
>>> <mailto:hugues.peccatte at gmail.com>
>>> > <mailto:hugues.peccatte at gmail.com <mailto:
>> hugues.peccatte at gmail.com>>>>
>>> > > wrote:
>>> > > >>
>>> > > >> 2014-10-03 12:26 GMT+02:00 Marc Delisle <
>> marc at infomarc.info <mailto:marc at infomarc.info>
>>> <mailto:marc at infomarc.info <mailto:marc at infomarc.info>>
>>> > > <mailto:marc at infomarc.info
>>> <mailto:marc at infomarc.info> <mailto:marc at infomarc.info
>>> <mailto:marc at infomarc.info>>>>:
>>> > > >>>
>>> > > >>> Hi Hugues,
>>> > > >>> I retested this morning on a laptop,
>> importing a
>>> > SQL file
>>> > > containing
>>> > > >>> 10000 employees from the sample employees
>>> > database. This is
>>> > > a small file
>>> > > >>> (660 KB).
>>> > > >>>
>>> > > >>> Current master: 3 min 25 sec (and ends with
>>> > JSON.parse:
>>> > > unexpected
>>> > > >>> character)
>>> > > >>>
>>> > > >>> Current Tithugues/stringFunctions_master: 2
>>> min 10
>>> > sec (same
>>> > > js error)
>>> > > >>>
>>> > > >>> Current QA_4_2: 0 min 5 sec
>>> > > >>>
>>> > > >>> There has been improvement, but we cannot
>>> release
>>> > 4.3 with
>>> > > this import
>>> > > >>> speed.
>>> > > >>>
>>> > > >>> --
>>> > > >>> Marc Delisle | phpMyAdmin
>>> > > >>
>>> > > >>
>>> > > >> Hi,
>>> > > >>
>>> > > >> I agree… But I'm afraid this is linked to
>>> > multibytes functions…
>>> > > >> Maybe we shouldn't use the multibytes
>> functions
>>> > everywhere…
>>> > > >>
>>> > > >> I'll still try to improve performances.
>>> > > >>
>>> > > >> Hugues.
>>> > > >>
>>> > > >
>>> > > > Indeed, I also think that we should use mb_*
>>> > function only
>>> > > when necessary and choice to use them should be
>> made
>>> > in case by
>>> > > case basis.
>>> > > >
>>> > > > --
>>> > > > Thanks and Regards,
>>> > > >
>>> > > > Madhura Jayaratne
>>> > >
>>> > > Hi,
>>> > >
>>> > > I didn't push my commits, but that's what I've
>>> started. I
>>> > > replaced the mb_* calls by standard calls on
>>> configuration
>>> > > variables, reserved words, etc.
>>> > >
>>> > > Hugues.
>>> > >
>>> > >
>>> > > Hi,
>>> > >
>>> > > Out of desperation, I try another algorithm. Instead
>> of
>>> > buffering
>>> > > data until SQL delimiter, I'll try to parse all
>> lines.
>>> > > So, I won't parse 1000 times a buffer of 50000
>>> characters,
>>> > but less
>>> > > than 10 times many buffers of 500 characters. I hope
>>> this
>>> > will be
>>> > > faster.
>>> > >
>>> > > Hugues.
>>> > >
>>> > >
>>> > > Hi,
>>> > >
>>> > > The new algorithm is over. There are still some controls
>> to
>>> > add, but it
>>> > > is usable with the file in this ticket: [1]
>>> > > You can find my modifications here: [2]
>>> > >
>>> > > Marc, is it faster for you ?
>>> > > It seems that I won ~33% of time. We're still far from 5
>>> seconds…
>>> > > Maybe I'll try to use standard PHP functions to see the
>>> > difference. If
>>> > > the standard PHP functions are really faster, I'll try
>>> to add
>>> > an option
>>> > > to use mb_* functions or standard PHP functions, as you
>>> said.
>>> > >
>>> > > [1] https://sourceforge.net/p/phpmyadmin/bugs/4536/
>>> > > [2]
>>> > >
>>> >
>>>
>> https://github.com/Tithugues/phpmyadmin/tree/stringFunctions_useStandardFunctions_master
>>> >
>>> > Hi Hugues,
>>> > yes it's faster. With the same testing conditions, the
>> import
>>> > takes 1
>>> > min 20 sec.
>>> >
>>> >
>>> > --
>>> > Marc Delisle | phpMyAdmin
>>> >
>>> >
>>> > Thanks for your feedback.
>>> > I'll try another another improvement to be faster.
>>> >
>>> > Note for my self:
>>> > * read X characters but don't restart the search from 0 each
>> time
>>> > * search for the escaped quote with a lookbehind expression,
>>> > something like `(?!<\\)(\\\\)*'`
>>> >
>>> > Hugues.
>>> >
>>> >
>>> > Hi,
>>> >
>>> > As asked by Marc, I added on option to import by reading as a
>>> multibytes
>>> > string or not.
>>> > The default configuration won't read as multibytes string (because
>>> it's
>>> > too long…). It seems that the DnD to import doesn't use the default
>>> > configuration, so what ever you define as default, it won't be use
>> in
>>> > this process.
>>> > Should we create a ticket for this ? I think it's possible to get
>>> it in
>>> > javascript.
>>> >
>>> > Hugues.
>>> >
>>>
>>> With the new option "Read as multibytes" left to its default disabled
>>> value, my test file imports in 5 seconds.
>>>
>>> When this option is ticked, it takes 60 seconds.
>>>
>>> Note that in both cases, I get a Javascript error at the end (the
>> error
>>> report does not show the same error in Firefox than in Chromium). Can
>>> anyone confirm this?
>>>
>>>
>>> --
>>> Marc Delisle | phpMyAdmin
>>>
>>>
>>> Marc,
>>>
>>> Could we please have both errors? I don't have it on Chrome. I'll try on
>>> Firefox and IE.
>>>
>>> Thanks,
>>> Hugues.
>> >From Iceweasel 24.8.1:
>>
>> {
>> "pma_version": "4.3.0-dev",
>> "browser_name": "FIREFOX",
>> "browser_version": "24.0",
>> "user_os": "Linux",
>> "server_software": "Apache/2.2.22 (Debian)",
>> "user_agent_string": "Mozilla/5.0 (X11; Linux x86_64; rv:24.0)
>> Gecko/20140924 Firefox/24.0 Iceweasel/24.8.1",
>> "locale": "en",
>> "configuration_storage": "enabled",
>> "php_version": "5.4.4-14+deb7u14",
>> "exception_type": "js",
>> "exception": {
>> "mode": "stack",
>> "name": "SyntaxError",
>> "message": "JSON.parse: unexpected character",
>> "stack": [
>> {
>> "func": ".parseJSON",
>> "args": "",
>> "line": 2,
>> "column": "",
>> "context": [
>> "/*! jQuery v1.8.3 jquery.com | jquery.org/license
>> */\r",
>> "(function(e,t){function _(e){var t=M[e]={};return
>> v.each(e.split(y),functio//...",
>> "",
>> "function sprintf() {",
>> "/*",
>> " * Copyright (c) 2013 Kevin van Zonneveld
>> (http://kvz.io)",
>> " * and Contributors (http://phpjs.org/authors)"
>> ],
>> "filename": "jquery/jquery-1.8.3.min.js"
>> },
>> {
>> "func": "PMA_console.initialize/<",
>> "args": "",
>> "line": 164,
>> "column": "",
>> "context": [
>> " $('#pma_console_options
>> input[name=current_query]').prop('c//...",
>> " PMA_console.updateConfig();",
>> " });",
>> "",
>> " $(document).ajaxComplete(function
>> (event, xhr) {",
>> " var data =
>> $.parseJSON(xhr.responseText);",
>> " PMA_console.ajaxCallback(data);",
>> " });",
>> "",
>> " PMA_console.isInitialized = true;",
>> " }"
>> ],
>> "filename": "console.js"
>> },
>> {
>> "func": "ErrorReport.wrap_function/new_func",
>> "args": "",
>> "line": 262,
>> "column": "",
>> "context": [
>> " */",
>> " wrap_function: function (func) {",
>> " if (!func.wrapped) {",
>> " var new_func = function () {",
>> " try {",
>> " return func.apply(this,
>> arguments);",
>> " } catch (x) {",
>> " TraceKit.report(x);",
>> " }",
>> " };",
>> " new_func.wrapped = true;"
>> ],
>> "filename": "error_report.js"
>> },
>> {
>> "func": "v.event.dispatch",
>> "args": "",
>> "line": 2,
>> "column": "",
>> "context": [
>> "/*! jQuery v1.8.3 jquery.com | jquery.org/license
>> */\r",
>> "(function(e,t){function _(e){var t=M[e]={};return
>> v.each(e.split(y),functio//...",
>> "",
>> "function sprintf() {",
>> "/*",
>> " * Copyright (c) 2013 Kevin van Zonneveld
>> (http://kvz.io)",
>> " * and Contributors (http://phpjs.org/authors)"
>> ],
>> "filename": "jquery/jquery-1.8.3.min.js"
>> },
>> {
>> "func": "$event.dispatch",
>> "args": "",
>> "line": 372,
>> "column": "",
>> "context": [
>> "$event.dispatch = function( event ){",
>> "\tif ( $.data( this, \"suppress.\"+ event.type ) -
>> new Date().getTime() > 0 ){",
>> "\t\t$.removeData( this, \"suppress.\"+ event.type );",
>> "\t\treturn;",
>> "\t}",
>> "\treturn $dispatch.apply( this, arguments );",
>> "};",
>> "",
>> "// event fix hooks for touch events...",
>> "var touchHooks = ",
>> "$event.fixHooks.touchstart = "
>> ],
>> "filename": "jquery/jquery.event.drag-2.2.js"
>> },
>> {
>> "func": "v.event.add/u",
>> "args": "",
>> "line": 2,
>> "column": "",
>> "context": [
>> "/*! jQuery v1.8.3 jquery.com | jquery.org/license
>> */\r",
>> "(function(e,t){function _(e){var t=M[e]={};return
>> v.each(e.split(y),functio//...",
>> "",
>> "function sprintf() {",
>> "/*",
>> " * Copyright (c) 2013 Kevin van Zonneveld
>> (http://kvz.io)",
>> " * and Contributors (http://phpjs.org/authors)"
>> ],
>> "filename": "jquery/jquery-1.8.3.min.js"
>> },
>> {
>> "func": "v.event.trigger",
>> "args": "",
>> "line": 2,
>> "column": "",
>> "context": [
>> "/*! jQuery v1.8.3 jquery.com | jquery.org/license
>> */\r",
>> "(function(e,t){function _(e){var t=M[e]={};return
>> v.each(e.split(y),functio//...",
>> "",
>> "function sprintf() {",
>> "/*",
>> " * Copyright (c) 2013 Kevin van Zonneveld
>> (http://kvz.io)",
>> " * and Contributors (http://phpjs.org/authors)"
>> ],
>> "filename": "jquery/jquery-1.8.3.min.js"
>> },
>> {
>> "func": "v.event.trigger",
>> "args": "",
>> "line": 2,
>> "column": "",
>> "context": [
>> "/*! jQuery v1.8.3 jquery.com | jquery.org/license
>> */\r",
>> "(function(e,t){function _(e){var t=M[e]={};return
>> v.each(e.split(y),functio//...",
>> "",
>> "function sprintf() {",
>> "/*",
>> " * Copyright (c) 2013 Kevin van Zonneveld
>> (http://kvz.io)",
>> " * and Contributors (http://phpjs.org/authors)"
>> ],
>> "filename": "jquery/jquery-1.8.3.min.js"
>> },
>> {
>> "func": "T",
>> "args": "",
>> "line": 2,
>> "column": "",
>> "context": [
>> "/*! jQuery v1.8.3 jquery.com | jquery.org/license
>> */\r",
>> "(function(e,t){function _(e){var t=M[e]={};return
>> v.each(e.split(y),functio//...",
>> "",
>> "function sprintf() {",
>> "/*",
>> " * Copyright (c) 2013 Kevin van Zonneveld
>> (http://kvz.io)",
>> " * and Contributors (http://phpjs.org/authors)"
>> ],
>> "filename": "jquery/jquery-1.8.3.min.js"
>> },
>> {
>> "func": ".send/r",
>> "args": "",
>> "line": 2,
>> "column": "",
>> "context": [
>> "/*! jQuery v1.8.3 jquery.com | jquery.org/license
>> */\r",
>> "(function(e,t){function _(e){var t=M[e]={};return
>> v.each(e.split(y),functio//...",
>> "",
>> "function sprintf() {",
>> "/*",
>> " * Copyright (c) 2013 Kevin van Zonneveld
>> (http://kvz.io)",
>> " * and Contributors (http://phpjs.org/authors)"
>> ],
>> "filename": "jquery/jquery-1.8.3.min.js"
>> }
>> ],
>> "useragent": "Mozilla/5.0 (X11; Linux x86_64; rv:24.0)
>> Gecko/20140924 Firefox/24.0 Iceweasel/24.8.1",
>> "incomplete": "false",
>> "uri": "db_import.php?target="
>> },
>> "script_name": "db_import.php",
>> "microhistory": {
>> "pages": [
>> {
>> "hash":
>>
>> "index.php?db=&table=&server=1&target=&token=c455fd8a0595a5032d18c07a64e7ab0a"
>> },
>> {
>> "hash":
>>
>> "db_structure.php?db=test&table=&server=1&target=&token=c455fd8a0595a5032d18c07a64e7ab0a",
>> "params": {
>> "opendb_url": "db_structure.php",
>> "safari_browser": "0",
>> "collation_connection": "utf8mb4_general_ci",
>> "lang": "en",
>> "server": "1",
>> "text_dir": "ltr",
>> "pma_text_default_tab": "Browse",
>> "pma_text_left_default_tab": "Structure",
>> "LimitChars": "50",
>> "pftext": "",
>> "confirm": "true",
>> "LoginCookieValidity": "144000",
>> "logged_in": "true"
>> }
>> },
>> {
>> "hash":
>>
>> "index.php?db=&table=&server=1&target=&token=c455fd8a0595a5032d18c07a64e7ab0a",
>> "params": {
>> "opendb_url": "db_structure.php",
>> "safari_browser": "0",
>> "collation_connection": "utf8mb4_general_ci",
>> "lang": "en",
>> "server": "1",
>> "text_dir": "ltr",
>> "pma_text_default_tab": "Browse",
>> "pma_text_left_default_tab": "Structure",
>> "LimitChars": "50",
>> "pftext": "",
>> "confirm": "true",
>> "LoginCookieValidity": "144000",
>> "logged_in": "true"
>> }
>> },
>> {
>> "hash":
>>
>> "db_structure.php?db=test&table=&server=1&target=&token=c455fd8a0595a5032d18c07a64e7ab0a",
>> "params": {
>> "opendb_url": "db_structure.php",
>> "safari_browser": "0",
>> "collation_connection": "utf8mb4_general_ci",
>> "lang": "en",
>> "server": "1",
>> "text_dir": "ltr",
>> "pma_text_default_tab": "Browse",
>> "pma_text_left_default_tab": "Structure",
>> "LimitChars": "50",
>> "pftext": "",
>> "confirm": "true",
>> "LoginCookieValidity": "144000",
>> "logged_in": "true"
>> }
>> },
>> {
>> "hash":
>>
>> "db_import.php?db=test&table=&server=1&target=&token=c455fd8a0595a5032d18c07a64e7ab0a",
>> "params": {
>> "opendb_url": "db_structure.php",
>> "safari_browser": "0",
>> "collation_connection": "utf8mb4_general_ci",
>> "lang": "en",
>> "server": "1",
>> "text_dir": "ltr",
>> "pma_text_default_tab": "Browse",
>> "pma_text_left_default_tab": "Structure",
>> "LimitChars": "50",
>> "pftext": "",
>> "confirm": "true",
>> "LoginCookieValidity": "144000",
>> "logged_in": "true"
>> }
>> }
>> ],
>> "current_index": "5"
>> }
>> }
>>
>> --
>> Marc Delisle | phpMyAdmin
>>
>
> Hi,
>
> It seems that the new algorithm is over, see [1]. The unit tests are now ok
> (except for the one failing on master). I did some tests with escaped
> quotes (with quote or backslash), delimiters into strings, modified
> delimiters.
> Marc, do you still have your error please? (I didn't work on it, but I hope
> that your issue is linked to the not-finished version…)
I don't have the error at work but I have to retest at home.
>
> Feel free to look at the new algorithm and to tell me if you see any flaw.
> If you can lead to an error with a correct SQL file, please send it to me.
>
> Thanks for your feedback.
>
> [1]
> https://github.com/Tithugues/phpmyadmin/tree/stringFunctions_useStandardFunctions_master
>
> Hugues.
>
>
>
> ------------------------------------------------------------------------
>
> ------------------------------------------------------------------------------
>
>
> ------------------------------------------------------------------------
>
> _______________________________________________
> Phpmyadmin-devel mailing list
> Phpmyadmin-devel at lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/phpmyadmin-devel
--
Marc Delisle (phpMyAdmin)
More information about the Developers
mailing list