[Phpmyadmin-devel] #4536 - master: import problem (PMA_String)
Marc Delisle
marc at infomarc.info
Sun Oct 19 21:53:18 CEST 2014
Le 2014-10-19 15:01, Hugues Peccatte a écrit :
> 2014-10-19 13:21 GMT+02:00 Marc Delisle <marc at infomarc.info
> <mailto:marc at infomarc.info>>:
>
> Le 2014-10-18 14:48, Hugues Peccatte a écrit :
> > 2014-10-13 12:21 GMT+02:00 Hugues Peccatte <hugues.peccatte at gmail.com <mailto:hugues.peccatte at gmail.com>
> > <mailto:hugues.peccatte at gmail.com
> <mailto:hugues.peccatte at gmail.com>>>:
> >
> > 2014-10-12 21:06 GMT+02:00 Marc Delisle <marc at infomarc.info <mailto:marc at infomarc.info>
> > <mailto:marc at infomarc.info <mailto:marc at infomarc.info>>>:
> >
> > Le 2014-10-12 12:57, Hugues Peccatte a écrit :
> > > 2014-10-05 20:36 GMT+02:00 Hugues Peccatte <hugues.peccatte at gmail.com <mailto:hugues.peccatte at gmail.com>
> <mailto:hugues.peccatte at gmail.com <mailto:hugues.peccatte at gmail.com>>
> > > <mailto:hugues.peccatte at gmail.com
> <mailto:hugues.peccatte at gmail.com>
> > <mailto:hugues.peccatte at gmail.com <mailto:hugues.peccatte at gmail.com>>>>:
> > >
> > > 2014-10-04 9:01 GMT+02:00 Hugues Peccatte <hugues.peccatte at gmail.com <mailto:hugues.peccatte at gmail.com>
> <mailto:hugues.peccatte at gmail.com <mailto:hugues.peccatte at gmail.com>>
> > > <mailto:hugues.peccatte at gmail.com
> <mailto:hugues.peccatte at gmail.com>
> > <mailto:hugues.peccatte at gmail.com <mailto:hugues.peccatte at gmail.com>>>>:
> > >
> > > Le 4 oct. 2014 03:22, "Madhura Jayaratne" <madhura.cj at gmail.com <mailto:madhura.cj at gmail.com>
> <mailto:madhura.cj at gmail.com <mailto:madhura.cj at gmail.com>>
> > > <mailto:madhura.cj at gmail.com
> <mailto:madhura.cj at gmail.com>
> > <mailto:madhura.cj at gmail.com <mailto:madhura.cj at gmail.com>>>> a écrit :
> > >
> > >
> > > >
> > > >
> > > >
> > > > On Sat, Oct 4, 2014 at 1:24 AM, Hugues Peccatte
> > > <hugues.peccatte at gmail.com <mailto:hugues.peccatte at gmail.com>
> > <mailto:hugues.peccatte at gmail.com <mailto:hugues.peccatte at gmail.com>>
> > <mailto:hugues.peccatte at gmail.com
> <mailto:hugues.peccatte at gmail.com>
> > <mailto:hugues.peccatte at gmail.com <mailto:hugues.peccatte at gmail.com>>>>
> > > wrote:
> > > >>
> > > >> 2014-10-03 12:26 GMT+02:00 Marc Delisle <marc at infomarc.info <mailto:marc at infomarc.info>
> <mailto:marc at infomarc.info <mailto:marc at infomarc.info>>
> > > <mailto:marc at infomarc.info
> <mailto:marc at infomarc.info> <mailto:marc at infomarc.info
> <mailto:marc at infomarc.info>>>>:
> > > >>>
> > > >>> Hi Hugues,
> > > >>> I retested this morning on a laptop, importing a
> > SQL file
> > > containing
> > > >>> 10000 employees from the sample employees
> > database. This is
> > > a small file
> > > >>> (660 KB).
> > > >>>
> > > >>> Current master: 3 min 25 sec (and ends with
> > JSON.parse:
> > > unexpected
> > > >>> character)
> > > >>>
> > > >>> Current Tithugues/stringFunctions_master: 2
> min 10
> > sec (same
> > > js error)
> > > >>>
> > > >>> Current QA_4_2: 0 min 5 sec
> > > >>>
> > > >>> There has been improvement, but we cannot
> release
> > 4.3 with
> > > this import
> > > >>> speed.
> > > >>>
> > > >>> --
> > > >>> Marc Delisle | phpMyAdmin
> > > >>
> > > >>
> > > >> Hi,
> > > >>
> > > >> I agree… But I'm afraid this is linked to
> > multibytes functions…
> > > >> Maybe we shouldn't use the multibytes functions
> > everywhere…
> > > >>
> > > >> I'll still try to improve performances.
> > > >>
> > > >> Hugues.
> > > >>
> > > >
> > > > Indeed, I also think that we should use mb_*
> > function only
> > > when necessary and choice to use them should be made
> > in case by
> > > case basis.
> > > >
> > > > --
> > > > Thanks and Regards,
> > > >
> > > > Madhura Jayaratne
> > >
> > > Hi,
> > >
> > > I didn't push my commits, but that's what I've
> started. I
> > > replaced the mb_* calls by standard calls on
> configuration
> > > variables, reserved words, etc.
> > >
> > > Hugues.
> > >
> > >
> > > Hi,
> > >
> > > Out of desperation, I try another algorithm. Instead of
> > buffering
> > > data until SQL delimiter, I'll try to parse all lines.
> > > So, I won't parse 1000 times a buffer of 50000
> characters,
> > but less
> > > than 10 times many buffers of 500 characters. I hope
> this
> > will be
> > > faster.
> > >
> > > Hugues.
> > >
> > >
> > > Hi,
> > >
> > > The new algorithm is over. There are still some controls to
> > add, but it
> > > is usable with the file in this ticket: [1]
> > > You can find my modifications here: [2]
> > >
> > > Marc, is it faster for you ?
> > > It seems that I won ~33% of time. We're still far from 5
> seconds…
> > > Maybe I'll try to use standard PHP functions to see the
> > difference. If
> > > the standard PHP functions are really faster, I'll try
> to add
> > an option
> > > to use mb_* functions or standard PHP functions, as you
> said.
> > >
> > > [1] https://sourceforge.net/p/phpmyadmin/bugs/4536/
> > > [2]
> > >
> >
> https://github.com/Tithugues/phpmyadmin/tree/stringFunctions_useStandardFunctions_master
> >
> > Hi Hugues,
> > yes it's faster. With the same testing conditions, the import
> > takes 1
> > min 20 sec.
> >
> >
> > --
> > Marc Delisle | phpMyAdmin
> >
> >
> > Thanks for your feedback.
> > I'll try another another improvement to be faster.
> >
> > Note for my self:
> > * read X characters but don't restart the search from 0 each time
> > * search for the escaped quote with a lookbehind expression,
> > something like `(?!<\\)(\\\\)*'`
> >
> > Hugues.
> >
> >
> > Hi,
> >
> > As asked by Marc, I added on option to import by reading as a
> multibytes
> > string or not.
> > The default configuration won't read as multibytes string (because
> it's
> > too long…). It seems that the DnD to import doesn't use the default
> > configuration, so what ever you define as default, it won't be use in
> > this process.
> > Should we create a ticket for this ? I think it's possible to get
> it in
> > javascript.
> >
> > Hugues.
> >
>
> With the new option "Read as multibytes" left to its default disabled
> value, my test file imports in 5 seconds.
>
> When this option is ticked, it takes 60 seconds.
>
> Note that in both cases, I get a Javascript error at the end (the error
> report does not show the same error in Firefox than in Chromium). Can
> anyone confirm this?
>
>
> --
> Marc Delisle | phpMyAdmin
>
>
> Marc,
>
> Could we please have both errors? I don't have it on Chrome. I'll try on
> Firefox and IE.
>
> Thanks,
> Hugues.
>From Iceweasel 24.8.1:
{
"pma_version": "4.3.0-dev",
"browser_name": "FIREFOX",
"browser_version": "24.0",
"user_os": "Linux",
"server_software": "Apache/2.2.22 (Debian)",
"user_agent_string": "Mozilla/5.0 (X11; Linux x86_64; rv:24.0)
Gecko/20140924 Firefox/24.0 Iceweasel/24.8.1",
"locale": "en",
"configuration_storage": "enabled",
"php_version": "5.4.4-14+deb7u14",
"exception_type": "js",
"exception": {
"mode": "stack",
"name": "SyntaxError",
"message": "JSON.parse: unexpected character",
"stack": [
{
"func": ".parseJSON",
"args": "",
"line": 2,
"column": "",
"context": [
"/*! jQuery v1.8.3 jquery.com | jquery.org/license
*/\r",
"(function(e,t){function _(e){var t=M[e]={};return
v.each(e.split(y),functio//...",
"",
"function sprintf() {",
"/*",
" * Copyright (c) 2013 Kevin van Zonneveld
(http://kvz.io)",
" * and Contributors (http://phpjs.org/authors)"
],
"filename": "jquery/jquery-1.8.3.min.js"
},
{
"func": "PMA_console.initialize/<",
"args": "",
"line": 164,
"column": "",
"context": [
" $('#pma_console_options
input[name=current_query]').prop('c//...",
" PMA_console.updateConfig();",
" });",
"",
" $(document).ajaxComplete(function
(event, xhr) {",
" var data =
$.parseJSON(xhr.responseText);",
" PMA_console.ajaxCallback(data);",
" });",
"",
" PMA_console.isInitialized = true;",
" }"
],
"filename": "console.js"
},
{
"func": "ErrorReport.wrap_function/new_func",
"args": "",
"line": 262,
"column": "",
"context": [
" */",
" wrap_function: function (func) {",
" if (!func.wrapped) {",
" var new_func = function () {",
" try {",
" return func.apply(this,
arguments);",
" } catch (x) {",
" TraceKit.report(x);",
" }",
" };",
" new_func.wrapped = true;"
],
"filename": "error_report.js"
},
{
"func": "v.event.dispatch",
"args": "",
"line": 2,
"column": "",
"context": [
"/*! jQuery v1.8.3 jquery.com | jquery.org/license
*/\r",
"(function(e,t){function _(e){var t=M[e]={};return
v.each(e.split(y),functio//...",
"",
"function sprintf() {",
"/*",
" * Copyright (c) 2013 Kevin van Zonneveld
(http://kvz.io)",
" * and Contributors (http://phpjs.org/authors)"
],
"filename": "jquery/jquery-1.8.3.min.js"
},
{
"func": "$event.dispatch",
"args": "",
"line": 372,
"column": "",
"context": [
"$event.dispatch = function( event ){",
"\tif ( $.data( this, \"suppress.\"+ event.type ) -
new Date().getTime() > 0 ){",
"\t\t$.removeData( this, \"suppress.\"+ event.type );",
"\t\treturn;",
"\t}",
"\treturn $dispatch.apply( this, arguments );",
"};",
"",
"// event fix hooks for touch events...",
"var touchHooks = ",
"$event.fixHooks.touchstart = "
],
"filename": "jquery/jquery.event.drag-2.2.js"
},
{
"func": "v.event.add/u",
"args": "",
"line": 2,
"column": "",
"context": [
"/*! jQuery v1.8.3 jquery.com | jquery.org/license
*/\r",
"(function(e,t){function _(e){var t=M[e]={};return
v.each(e.split(y),functio//...",
"",
"function sprintf() {",
"/*",
" * Copyright (c) 2013 Kevin van Zonneveld
(http://kvz.io)",
" * and Contributors (http://phpjs.org/authors)"
],
"filename": "jquery/jquery-1.8.3.min.js"
},
{
"func": "v.event.trigger",
"args": "",
"line": 2,
"column": "",
"context": [
"/*! jQuery v1.8.3 jquery.com | jquery.org/license
*/\r",
"(function(e,t){function _(e){var t=M[e]={};return
v.each(e.split(y),functio//...",
"",
"function sprintf() {",
"/*",
" * Copyright (c) 2013 Kevin van Zonneveld
(http://kvz.io)",
" * and Contributors (http://phpjs.org/authors)"
],
"filename": "jquery/jquery-1.8.3.min.js"
},
{
"func": "v.event.trigger",
"args": "",
"line": 2,
"column": "",
"context": [
"/*! jQuery v1.8.3 jquery.com | jquery.org/license
*/\r",
"(function(e,t){function _(e){var t=M[e]={};return
v.each(e.split(y),functio//...",
"",
"function sprintf() {",
"/*",
" * Copyright (c) 2013 Kevin van Zonneveld
(http://kvz.io)",
" * and Contributors (http://phpjs.org/authors)"
],
"filename": "jquery/jquery-1.8.3.min.js"
},
{
"func": "T",
"args": "",
"line": 2,
"column": "",
"context": [
"/*! jQuery v1.8.3 jquery.com | jquery.org/license
*/\r",
"(function(e,t){function _(e){var t=M[e]={};return
v.each(e.split(y),functio//...",
"",
"function sprintf() {",
"/*",
" * Copyright (c) 2013 Kevin van Zonneveld
(http://kvz.io)",
" * and Contributors (http://phpjs.org/authors)"
],
"filename": "jquery/jquery-1.8.3.min.js"
},
{
"func": ".send/r",
"args": "",
"line": 2,
"column": "",
"context": [
"/*! jQuery v1.8.3 jquery.com | jquery.org/license
*/\r",
"(function(e,t){function _(e){var t=M[e]={};return
v.each(e.split(y),functio//...",
"",
"function sprintf() {",
"/*",
" * Copyright (c) 2013 Kevin van Zonneveld
(http://kvz.io)",
" * and Contributors (http://phpjs.org/authors)"
],
"filename": "jquery/jquery-1.8.3.min.js"
}
],
"useragent": "Mozilla/5.0 (X11; Linux x86_64; rv:24.0)
Gecko/20140924 Firefox/24.0 Iceweasel/24.8.1",
"incomplete": "false",
"uri": "db_import.php?target="
},
"script_name": "db_import.php",
"microhistory": {
"pages": [
{
"hash":
"index.php?db=&table=&server=1&target=&token=c455fd8a0595a5032d18c07a64e7ab0a"
},
{
"hash":
"db_structure.php?db=test&table=&server=1&target=&token=c455fd8a0595a5032d18c07a64e7ab0a",
"params": {
"opendb_url": "db_structure.php",
"safari_browser": "0",
"collation_connection": "utf8mb4_general_ci",
"lang": "en",
"server": "1",
"text_dir": "ltr",
"pma_text_default_tab": "Browse",
"pma_text_left_default_tab": "Structure",
"LimitChars": "50",
"pftext": "",
"confirm": "true",
"LoginCookieValidity": "144000",
"logged_in": "true"
}
},
{
"hash":
"index.php?db=&table=&server=1&target=&token=c455fd8a0595a5032d18c07a64e7ab0a",
"params": {
"opendb_url": "db_structure.php",
"safari_browser": "0",
"collation_connection": "utf8mb4_general_ci",
"lang": "en",
"server": "1",
"text_dir": "ltr",
"pma_text_default_tab": "Browse",
"pma_text_left_default_tab": "Structure",
"LimitChars": "50",
"pftext": "",
"confirm": "true",
"LoginCookieValidity": "144000",
"logged_in": "true"
}
},
{
"hash":
"db_structure.php?db=test&table=&server=1&target=&token=c455fd8a0595a5032d18c07a64e7ab0a",
"params": {
"opendb_url": "db_structure.php",
"safari_browser": "0",
"collation_connection": "utf8mb4_general_ci",
"lang": "en",
"server": "1",
"text_dir": "ltr",
"pma_text_default_tab": "Browse",
"pma_text_left_default_tab": "Structure",
"LimitChars": "50",
"pftext": "",
"confirm": "true",
"LoginCookieValidity": "144000",
"logged_in": "true"
}
},
{
"hash":
"db_import.php?db=test&table=&server=1&target=&token=c455fd8a0595a5032d18c07a64e7ab0a",
"params": {
"opendb_url": "db_structure.php",
"safari_browser": "0",
"collation_connection": "utf8mb4_general_ci",
"lang": "en",
"server": "1",
"text_dir": "ltr",
"pma_text_default_tab": "Browse",
"pma_text_left_default_tab": "Structure",
"LimitChars": "50",
"pftext": "",
"confirm": "true",
"LoginCookieValidity": "144000",
"logged_in": "true"
}
}
],
"current_index": "5"
}
}
--
Marc Delisle | phpMyAdmin
More information about the Developers
mailing list