[Phpmyadmin-devel] #4536 - master: import problem (PMA_String)
Marc Delisle
marc at infomarc.info
Wed Oct 29 23:32:14 CET 2014
Le 2014-10-28 17:24, Hugues Peccatte a écrit :
> 2014-10-28 20:58 GMT+01:00 Hugues Peccatte <hugues.peccatte at gmail.com
> <mailto:hugues.peccatte at gmail.com>>:
>
> 2014-10-19 21:53 GMT+02:00 Marc Delisle <marc at infomarc.info
> <mailto:marc at infomarc.info>>:
>
> Le 2014-10-19 15:01, Hugues Peccatte a écrit :
> > 2014-10-19 13:21 GMT+02:00 Marc Delisle <marc at infomarc.info <mailto:marc at infomarc.info>
> > <mailto:marc at infomarc.info <mailto:marc at infomarc.info>>>:
> >
> > Le 2014-10-18 14:48, Hugues Peccatte a écrit :
> > > 2014-10-13 12:21 GMT+02:00 Hugues Peccatte <hugues.peccatte at gmail.com <mailto:hugues.peccatte at gmail.com>
> <mailto:hugues.peccatte at gmail.com
> <mailto:hugues.peccatte at gmail.com>>
> > > <mailto:hugues.peccatte at gmail.com
> <mailto:hugues.peccatte at gmail.com>
> > <mailto:hugues.peccatte at gmail.com <mailto:hugues.peccatte at gmail.com>>>>:
> > >
> > > 2014-10-12 21:06 GMT+02:00 Marc Delisle <marc at infomarc.info <mailto:marc at infomarc.info>
> <mailto:marc at infomarc.info <mailto:marc at infomarc.info>>
> > > <mailto:marc at infomarc.info
> <mailto:marc at infomarc.info> <mailto:marc at infomarc.info
> <mailto:marc at infomarc.info>>>>:
> > >
> > > Le 2014-10-12 12:57, Hugues Peccatte a écrit :
> > > > 2014-10-05 20:36 GMT+02:00 Hugues Peccatte <hugues.peccatte at gmail.com <mailto:hugues.peccatte at gmail.com>
> <mailto:hugues.peccatte at gmail.com
> <mailto:hugues.peccatte at gmail.com>>
> > <mailto:hugues.peccatte at gmail.com <mailto:hugues.peccatte at gmail.com>
> <mailto:hugues.peccatte at gmail.com
> <mailto:hugues.peccatte at gmail.com>>>
> > > > <mailto:hugues.peccatte at gmail.com <mailto:hugues.peccatte at gmail.com>
> > <mailto:hugues.peccatte at gmail.com <mailto:hugues.peccatte at gmail.com>>
> > > <mailto:hugues.peccatte at gmail.com
> <mailto:hugues.peccatte at gmail.com>
> <mailto:hugues.peccatte at gmail.com
> <mailto:hugues.peccatte at gmail.com>>>>>:
> > > >
> > > > 2014-10-04 9:01 GMT+02:00 Hugues Peccatte <hugues.peccatte at gmail.com <mailto:hugues.peccatte at gmail.com>
> <mailto:hugues.peccatte at gmail.com
> <mailto:hugues.peccatte at gmail.com>>
> > <mailto:hugues.peccatte at gmail.com <mailto:hugues.peccatte at gmail.com>
> <mailto:hugues.peccatte at gmail.com
> <mailto:hugues.peccatte at gmail.com>>>
> > > > <mailto:hugues.peccatte at gmail.com <mailto:hugues.peccatte at gmail.com>
> > <mailto:hugues.peccatte at gmail.com <mailto:hugues.peccatte at gmail.com>>
> > > <mailto:hugues.peccatte at gmail.com
> <mailto:hugues.peccatte at gmail.com>
> <mailto:hugues.peccatte at gmail.com
> <mailto:hugues.peccatte at gmail.com>>>>>:
> > > >
> > > > Le 4 oct. 2014 03:22, "Madhura Jayaratne" <madhura.cj at gmail.com <mailto:madhura.cj at gmail.com>
> <mailto:madhura.cj at gmail.com <mailto:madhura.cj at gmail.com>>
> > <mailto:madhura.cj at gmail.com <mailto:madhura.cj at gmail.com>
> <mailto:madhura.cj at gmail.com <mailto:madhura.cj at gmail.com>>>
> > > > <mailto:madhura.cj at gmail.com <mailto:madhura.cj at gmail.com>
> > <mailto:madhura.cj at gmail.com <mailto:madhura.cj at gmail.com>>
> > > <mailto:madhura.cj at gmail.com
> <mailto:madhura.cj at gmail.com> <mailto:madhura.cj at gmail.com
> <mailto:madhura.cj at gmail.com>>>>> a écrit :
> > > >
> > > >
> > > > >
> > > > >
> > > > >
> > > > > On Sat, Oct 4, 2014 at 1:24 AM, Hugues Peccatte
> > > > <hugues.peccatte at gmail.com <mailto:hugues.peccatte at gmail.com>
> <mailto:hugues.peccatte at gmail.com
> <mailto:hugues.peccatte at gmail.com>>
> > > <mailto:hugues.peccatte at gmail.com <mailto:hugues.peccatte at gmail.com>
> <mailto:hugues.peccatte at gmail.com
> <mailto:hugues.peccatte at gmail.com>>>
> > > <mailto:hugues.peccatte at gmail.com
> <mailto:hugues.peccatte at gmail.com>
> > <mailto:hugues.peccatte at gmail.com
> <mailto:hugues.peccatte at gmail.com>>
> > > <mailto:hugues.peccatte at gmail.com
> <mailto:hugues.peccatte at gmail.com>
> <mailto:hugues.peccatte at gmail.com
> <mailto:hugues.peccatte at gmail.com>>>>>
> > > > wrote:
> > > > >>
> > > > >> 2014-10-03 12:26 GMT+02:00 Marc
> Delisle <marc at infomarc.info <mailto:marc at infomarc.info>
> <mailto:marc at infomarc.info <mailto:marc at infomarc.info>>
> > <mailto:marc at infomarc.info <mailto:marc at infomarc.info>
> <mailto:marc at infomarc.info <mailto:marc at infomarc.info>>>
> > > > <mailto:marc at infomarc.info
> <mailto:marc at infomarc.info>
> > <mailto:marc at infomarc.info <mailto:marc at infomarc.info>>
> <mailto:marc at infomarc.info <mailto:marc at infomarc.info>
> > <mailto:marc at infomarc.info <mailto:marc at infomarc.info>>>>>:
> > > > >>>
> > > > >>> Hi Hugues,
> > > > >>> I retested this morning on a
> laptop, importing a
> > > SQL file
> > > > containing
> > > > >>> 10000 employees from the sample
> employees
> > > database. This is
> > > > a small file
> > > > >>> (660 KB).
> > > > >>>
> > > > >>> Current master: 3 min 25 sec (and
> ends with
> > > JSON.parse:
> > > > unexpected
> > > > >>> character)
> > > > >>>
> > > > >>> Current
> Tithugues/stringFunctions_master: 2
> > min 10
> > > sec (same
> > > > js error)
> > > > >>>
> > > > >>> Current QA_4_2: 0 min 5 sec
> > > > >>>
> > > > >>> There has been improvement, but we
> cannot
> > release
> > > 4.3 with
> > > > this import
> > > > >>> speed.
> > > > >>>
> > > > >>> --
> > > > >>> Marc Delisle | phpMyAdmin
> > > > >>
> > > > >>
> > > > >> Hi,
> > > > >>
> > > > >> I agree… But I'm afraid this is
> linked to
> > > multibytes functions…
> > > > >> Maybe we shouldn't use the
> multibytes functions
> > > everywhere…
> > > > >>
> > > > >> I'll still try to improve performances.
> > > > >>
> > > > >> Hugues.
> > > > >>
> > > > >
> > > > > Indeed, I also think that we should
> use mb_*
> > > function only
> > > > when necessary and choice to use them
> should be made
> > > in case by
> > > > case basis.
> > > > >
> > > > > --
> > > > > Thanks and Regards,
> > > > >
> > > > > Madhura Jayaratne
> > > >
> > > > Hi,
> > > >
> > > > I didn't push my commits, but that's
> what I've
> > started. I
> > > > replaced the mb_* calls by standard
> calls on
> > configuration
> > > > variables, reserved words, etc.
> > > >
> > > > Hugues.
> > > >
> > > >
> > > > Hi,
> > > >
> > > > Out of desperation, I try another
> algorithm. Instead of
> > > buffering
> > > > data until SQL delimiter, I'll try to
> parse all lines.
> > > > So, I won't parse 1000 times a buffer of 50000
> > characters,
> > > but less
> > > > than 10 times many buffers of 500
> characters. I hope
> > this
> > > will be
> > > > faster.
> > > >
> > > > Hugues.
> > > >
> > > >
> > > > Hi,
> > > >
> > > > The new algorithm is over. There are still
> some controls to
> > > add, but it
> > > > is usable with the file in this ticket: [1]
> > > > You can find my modifications here: [2]
> > > >
> > > > Marc, is it faster for you ?
> > > > It seems that I won ~33% of time. We're still
> far from 5
> > seconds…
> > > > Maybe I'll try to use standard PHP functions
> to see the
> > > difference. If
> > > > the standard PHP functions are really faster,
> I'll try
> > to add
> > > an option
> > > > to use mb_* functions or standard PHP
> functions, as you
> > said.
> > > >
> > > > [1]
> https://sourceforge.net/p/phpmyadmin/bugs/4536/
> > > > [2]
> > > >
> > >
> >
> https://github.com/Tithugues/phpmyadmin/tree/stringFunctions_useStandardFunctions_master
> > >
> > > Hi Hugues,
> > > yes it's faster. With the same testing
> conditions, the import
> > > takes 1
> > > min 20 sec.
> > >
> > >
> > > --
> > > Marc Delisle | phpMyAdmin
> > >
> > >
> > > Thanks for your feedback.
> > > I'll try another another improvement to be faster.
> > >
> > > Note for my self:
> > > * read X characters but don't restart the search
> from 0 each time
> > > * search for the escaped quote with a lookbehind
> expression,
> > > something like `(?!<\\)(\\\\)*'`
> > >
> > > Hugues.
> > >
> > >
> > > Hi,
> > >
> > > As asked by Marc, I added on option to import by reading
> as a
> > multibytes
> > > string or not.
> > > The default configuration won't read as multibytes
> string (because
> > it's
> > > too long…). It seems that the DnD to import doesn't use
> the default
> > > configuration, so what ever you define as default, it
> won't be use in
> > > this process.
> > > Should we create a ticket for this ? I think it's
> possible to get
> > it in
> > > javascript.
> > >
> > > Hugues.
> > >
> >
> > With the new option "Read as multibytes" left to its
> default disabled
> > value, my test file imports in 5 seconds.
> >
> > When this option is ticked, it takes 60 seconds.
> >
> > Note that in both cases, I get a Javascript error at the
> end (the error
> > report does not show the same error in Firefox than in
> Chromium). Can
> > anyone confirm this?
> >
> >
> > --
> > Marc Delisle | phpMyAdmin
> >
> >
> > Marc,
> >
> > Could we please have both errors? I don't have it on Chrome.
> I'll try on
> > Firefox and IE.
> >
> > Thanks,
> > Hugues.
>
> >From Iceweasel 24.8.1:
>
> {
> "pma_version": "4.3.0-dev",
> "browser_name": "FIREFOX",
> "browser_version": "24.0",
> "user_os": "Linux",
> "server_software": "Apache/2.2.22 (Debian)",
> "user_agent_string": "Mozilla/5.0 (X11; Linux x86_64; rv:24.0)
> Gecko/20140924 Firefox/24.0 Iceweasel/24.8.1",
> "locale": "en",
> "configuration_storage": "enabled",
> "php_version": "5.4.4-14+deb7u14",
> "exception_type": "js",
> "exception": {
> "mode": "stack",
> "name": "SyntaxError",
> "message": "JSON.parse: unexpected character",
> "stack": [
> {
> "func": ".parseJSON",
> "args": "",
> "line": 2,
> "column": "",
> "context": [
> "/*! jQuery v1.8.3 jquery.com
> <http://jquery.com> | jquery.org/license <http://jquery.org/license>
> */\r",
> "(function(e,t){function _(e){var
> t=M[e]={};return
> v.each(e.split(y),functio//...",
> "",
> "function sprintf() {",
> "/*",
> " * Copyright (c) 2013 Kevin van Zonneveld
> (http://kvz.io)",
> " * and Contributors (http://phpjs.org/authors)"
> ],
> "filename": "jquery/jquery-1.8.3.min.js"
> },
> {
> "func": "PMA_console.initialize/<",
> "args": "",
> "line": 164,
> "column": "",
> "context": [
> " $('#pma_console_options
> input[name=current_query]').prop('c//...",
> " PMA_console.updateConfig();",
> " });",
> "",
> " $(document).ajaxComplete(function
> (event, xhr) {",
> " var data =
> $.parseJSON(xhr.responseText);",
> "
> PMA_console.ajaxCallback(data);",
> " });",
> "",
> " PMA_console.isInitialized = true;",
> " }"
> ],
> "filename": "console.js"
> },
> {
> "func": "ErrorReport.wrap_function/new_func",
> "args": "",
> "line": 262,
> "column": "",
> "context": [
> " */",
> " wrap_function: function (func) {",
> " if (!func.wrapped) {",
> " var new_func = function () {",
> " try {",
> " return func.apply(this,
> arguments);",
> " } catch (x) {",
> " TraceKit.report(x);",
> " }",
> " };",
> " new_func.wrapped = true;"
> ],
> "filename": "error_report.js"
> },
> {
> "func": "v.event.dispatch",
> "args": "",
> "line": 2,
> "column": "",
> "context": [
> "/*! jQuery v1.8.3 jquery.com
> <http://jquery.com> | jquery.org/license <http://jquery.org/license>
> */\r",
> "(function(e,t){function _(e){var
> t=M[e]={};return
> v.each(e.split(y),functio//...",
> "",
> "function sprintf() {",
> "/*",
> " * Copyright (c) 2013 Kevin van Zonneveld
> (http://kvz.io)",
> " * and Contributors (http://phpjs.org/authors)"
> ],
> "filename": "jquery/jquery-1.8.3.min.js"
> },
> {
> "func": "$event.dispatch",
> "args": "",
> "line": 372,
> "column": "",
> "context": [
> "$event.dispatch = function( event ){",
> "\tif ( $.data( this, \"suppress.\"+
> event.type ) -
> new Date().getTime() > 0 ){",
> "\t\t$.removeData( this, \"suppress.\"+
> event.type );",
> "\t\treturn;",
> "\t}",
> "\treturn $dispatch.apply( this, arguments );",
> "};",
> "",
> "// event fix hooks for touch events...",
> "var touchHooks = ",
> "$event.fixHooks.touchstart = "
> ],
> "filename": "jquery/jquery.event.drag-2.2.js"
> },
> {
> "func": "v.event.add/u",
> "args": "",
> "line": 2,
> "column": "",
> "context": [
> "/*! jQuery v1.8.3 jquery.com
> <http://jquery.com> | jquery.org/license <http://jquery.org/license>
> */\r",
> "(function(e,t){function _(e){var
> t=M[e]={};return
> v.each(e.split(y),functio//...",
> "",
> "function sprintf() {",
> "/*",
> " * Copyright (c) 2013 Kevin van Zonneveld
> (http://kvz.io)",
> " * and Contributors (http://phpjs.org/authors)"
> ],
> "filename": "jquery/jquery-1.8.3.min.js"
> },
> {
> "func": "v.event.trigger",
> "args": "",
> "line": 2,
> "column": "",
> "context": [
> "/*! jQuery v1.8.3 jquery.com
> <http://jquery.com> | jquery.org/license <http://jquery.org/license>
> */\r",
> "(function(e,t){function _(e){var
> t=M[e]={};return
> v.each(e.split(y),functio//...",
> "",
> "function sprintf() {",
> "/*",
> " * Copyright (c) 2013 Kevin van Zonneveld
> (http://kvz.io)",
> " * and Contributors (http://phpjs.org/authors)"
> ],
> "filename": "jquery/jquery-1.8.3.min.js"
> },
> {
> "func": "v.event.trigger",
> "args": "",
> "line": 2,
> "column": "",
> "context": [
> "/*! jQuery v1.8.3 jquery.com
> <http://jquery.com> | jquery.org/license <http://jquery.org/license>
> */\r",
> "(function(e,t){function _(e){var
> t=M[e]={};return
> v.each(e.split(y),functio//...",
> "",
> "function sprintf() {",
> "/*",
> " * Copyright (c) 2013 Kevin van Zonneveld
> (http://kvz.io)",
> " * and Contributors (http://phpjs.org/authors)"
> ],
> "filename": "jquery/jquery-1.8.3.min.js"
> },
> {
> "func": "T",
> "args": "",
> "line": 2,
> "column": "",
> "context": [
> "/*! jQuery v1.8.3 jquery.com
> <http://jquery.com> | jquery.org/license <http://jquery.org/license>
> */\r",
> "(function(e,t){function _(e){var
> t=M[e]={};return
> v.each(e.split(y),functio//...",
> "",
> "function sprintf() {",
> "/*",
> " * Copyright (c) 2013 Kevin van Zonneveld
> (http://kvz.io)",
> " * and Contributors (http://phpjs.org/authors)"
> ],
> "filename": "jquery/jquery-1.8.3.min.js"
> },
> {
> "func": ".send/r",
> "args": "",
> "line": 2,
> "column": "",
> "context": [
> "/*! jQuery v1.8.3 jquery.com
> <http://jquery.com> | jquery.org/license <http://jquery.org/license>
> */\r",
> "(function(e,t){function _(e){var
> t=M[e]={};return
> v.each(e.split(y),functio//...",
> "",
> "function sprintf() {",
> "/*",
> " * Copyright (c) 2013 Kevin van Zonneveld
> (http://kvz.io)",
> " * and Contributors (http://phpjs.org/authors)"
> ],
> "filename": "jquery/jquery-1.8.3.min.js"
> }
> ],
> "useragent": "Mozilla/5.0 (X11; Linux x86_64; rv:24.0)
> Gecko/20140924 Firefox/24.0 Iceweasel/24.8.1",
> "incomplete": "false",
> "uri": "db_import.php?target="
> },
> "script_name": "db_import.php",
> "microhistory": {
> "pages": [
> {
> "hash":
> "index.php?db=&table=&server=1&target=&token=c455fd8a0595a5032d18c07a64e7ab0a"
> },
> {
> "hash":
> "db_structure.php?db=test&table=&server=1&target=&token=c455fd8a0595a5032d18c07a64e7ab0a",
> "params": {
> "opendb_url": "db_structure.php",
> "safari_browser": "0",
> "collation_connection": "utf8mb4_general_ci",
> "lang": "en",
> "server": "1",
> "text_dir": "ltr",
> "pma_text_default_tab": "Browse",
> "pma_text_left_default_tab": "Structure",
> "LimitChars": "50",
> "pftext": "",
> "confirm": "true",
> "LoginCookieValidity": "144000",
> "logged_in": "true"
> }
> },
> {
> "hash":
> "index.php?db=&table=&server=1&target=&token=c455fd8a0595a5032d18c07a64e7ab0a",
> "params": {
> "opendb_url": "db_structure.php",
> "safari_browser": "0",
> "collation_connection": "utf8mb4_general_ci",
> "lang": "en",
> "server": "1",
> "text_dir": "ltr",
> "pma_text_default_tab": "Browse",
> "pma_text_left_default_tab": "Structure",
> "LimitChars": "50",
> "pftext": "",
> "confirm": "true",
> "LoginCookieValidity": "144000",
> "logged_in": "true"
> }
> },
> {
> "hash":
> "db_structure.php?db=test&table=&server=1&target=&token=c455fd8a0595a5032d18c07a64e7ab0a",
> "params": {
> "opendb_url": "db_structure.php",
> "safari_browser": "0",
> "collation_connection": "utf8mb4_general_ci",
> "lang": "en",
> "server": "1",
> "text_dir": "ltr",
> "pma_text_default_tab": "Browse",
> "pma_text_left_default_tab": "Structure",
> "LimitChars": "50",
> "pftext": "",
> "confirm": "true",
> "LoginCookieValidity": "144000",
> "logged_in": "true"
> }
> },
> {
> "hash":
> "db_import.php?db=test&table=&server=1&target=&token=c455fd8a0595a5032d18c07a64e7ab0a",
> "params": {
> "opendb_url": "db_structure.php",
> "safari_browser": "0",
> "collation_connection": "utf8mb4_general_ci",
> "lang": "en",
> "server": "1",
> "text_dir": "ltr",
> "pma_text_default_tab": "Browse",
> "pma_text_left_default_tab": "Structure",
> "LimitChars": "50",
> "pftext": "",
> "confirm": "true",
> "LoginCookieValidity": "144000",
> "logged_in": "true"
> }
> }
> ],
> "current_index": "5"
> }
> }
>
> --
> Marc Delisle | phpMyAdmin
>
>
> Hi,
>
> It seems that the new algorithm is over, see [1]. The unit tests are
> now ok (except for the one failing on master). I did some tests with
> escaped quotes (with quote or backslash), delimiters into strings,
> modified delimiters.
> Marc, do you still have your error please? (I didn't work on it, but
> I hope that your issue is linked to the not-finished version…)
On my home machine, sadly I still have the js error after importing.
Tested on Iceweasel 31.2.
Surely someone else can reproduce this problem on Debian wheezy? Because
if it's only me, I don't care too much ;)
With 3b3017d7, my test file (10000 rows) imports in 40 seconds in
multibyte mode.
>
> Feel free to look at the new algorithm and to tell me if you see any
> flaw.
> If you can lead to an error with a correct SQL file, please send it
> to me.
>
> Thanks for your feedback.
>
> [1] https://github.com/Tithugues/phpmyadmin/tree/stringFunctions_useStandardFunctions_master
>
> Hugues.
>
>
> Hi,
>
> Since my previous email, I did some modifications to manage the display
> of queries in a different way (closer to the current display).
>
> But there is something strange with the attachment… I can import the
> file (without error and all content executed), but on master, if I paste
> the content of the file into the SQL console, I've an error… Is it the
> expected behavior please? I don't see any exotic content in this file…
> On my branch, I don't have an error, but only the first statement (DROP
> PROCEDURE IF EXISTS) is executed…
>
> Thanks,
> Hugues.
>
--
Marc Delisle | phpMyAdmin
More information about the Developers
mailing list