[Phpmyadmin-devel] #4536 - master: import problem (PMA_String)
Hugues Peccatte
hugues.peccatte at gmail.com
Tue Oct 28 22:24:12 CET 2014
2014-10-28 20:58 GMT+01:00 Hugues Peccatte <hugues.peccatte at gmail.com>:
> 2014-10-19 21:53 GMT+02:00 Marc Delisle <marc at infomarc.info>:
>
>> Le 2014-10-19 15:01, Hugues Peccatte a écrit :
>> > 2014-10-19 13:21 GMT+02:00 Marc Delisle <marc at infomarc.info
>> > <mailto:marc at infomarc.info>>:
>> >
>> > Le 2014-10-18 14:48, Hugues Peccatte a écrit :
>> > > 2014-10-13 12:21 GMT+02:00 Hugues Peccatte <
>> hugues.peccatte at gmail.com <mailto:hugues.peccatte at gmail.com>
>> > > <mailto:hugues.peccatte at gmail.com
>> > <mailto:hugues.peccatte at gmail.com>>>:
>> > >
>> > > 2014-10-12 21:06 GMT+02:00 Marc Delisle <marc at infomarc.info
>> <mailto:marc at infomarc.info>
>> > > <mailto:marc at infomarc.info <mailto:marc at infomarc.info>>>:
>> > >
>> > > Le 2014-10-12 12:57, Hugues Peccatte a écrit :
>> > > > 2014-10-05 20:36 GMT+02:00 Hugues Peccatte <
>> hugues.peccatte at gmail.com <mailto:hugues.peccatte at gmail.com>
>> > <mailto:hugues.peccatte at gmail.com <mailto:hugues.peccatte at gmail.com
>> >>
>> > > > <mailto:hugues.peccatte at gmail.com
>> > <mailto:hugues.peccatte at gmail.com>
>> > > <mailto:hugues.peccatte at gmail.com <mailto:
>> hugues.peccatte at gmail.com>>>>:
>> > > >
>> > > > 2014-10-04 9:01 GMT+02:00 Hugues Peccatte <
>> hugues.peccatte at gmail.com <mailto:hugues.peccatte at gmail.com>
>> > <mailto:hugues.peccatte at gmail.com <mailto:hugues.peccatte at gmail.com
>> >>
>> > > > <mailto:hugues.peccatte at gmail.com
>> > <mailto:hugues.peccatte at gmail.com>
>> > > <mailto:hugues.peccatte at gmail.com <mailto:
>> hugues.peccatte at gmail.com>>>>:
>> > > >
>> > > > Le 4 oct. 2014 03:22, "Madhura Jayaratne" <
>> madhura.cj at gmail.com <mailto:madhura.cj at gmail.com>
>> > <mailto:madhura.cj at gmail.com <mailto:madhura.cj at gmail.com>>
>> > > > <mailto:madhura.cj at gmail.com
>> > <mailto:madhura.cj at gmail.com>
>> > > <mailto:madhura.cj at gmail.com <mailto:madhura.cj at gmail.com>>>>
>> a écrit :
>> > > >
>> > > >
>> > > > >
>> > > > >
>> > > > >
>> > > > > On Sat, Oct 4, 2014 at 1:24 AM, Hugues
>> Peccatte
>> > > > <hugues.peccatte at gmail.com <mailto:
>> hugues.peccatte at gmail.com>
>> > > <mailto:hugues.peccatte at gmail.com <mailto:
>> hugues.peccatte at gmail.com>>
>> > > <mailto:hugues.peccatte at gmail.com
>> > <mailto:hugues.peccatte at gmail.com>
>> > > <mailto:hugues.peccatte at gmail.com <mailto:
>> hugues.peccatte at gmail.com>>>>
>> > > > wrote:
>> > > > >>
>> > > > >> 2014-10-03 12:26 GMT+02:00 Marc Delisle <
>> marc at infomarc.info <mailto:marc at infomarc.info>
>> > <mailto:marc at infomarc.info <mailto:marc at infomarc.info>>
>> > > > <mailto:marc at infomarc.info
>> > <mailto:marc at infomarc.info> <mailto:marc at infomarc.info
>> > <mailto:marc at infomarc.info>>>>:
>> > > > >>>
>> > > > >>> Hi Hugues,
>> > > > >>> I retested this morning on a laptop,
>> importing a
>> > > SQL file
>> > > > containing
>> > > > >>> 10000 employees from the sample employees
>> > > database. This is
>> > > > a small file
>> > > > >>> (660 KB).
>> > > > >>>
>> > > > >>> Current master: 3 min 25 sec (and ends with
>> > > JSON.parse:
>> > > > unexpected
>> > > > >>> character)
>> > > > >>>
>> > > > >>> Current Tithugues/stringFunctions_master: 2
>> > min 10
>> > > sec (same
>> > > > js error)
>> > > > >>>
>> > > > >>> Current QA_4_2: 0 min 5 sec
>> > > > >>>
>> > > > >>> There has been improvement, but we cannot
>> > release
>> > > 4.3 with
>> > > > this import
>> > > > >>> speed.
>> > > > >>>
>> > > > >>> --
>> > > > >>> Marc Delisle | phpMyAdmin
>> > > > >>
>> > > > >>
>> > > > >> Hi,
>> > > > >>
>> > > > >> I agree… But I'm afraid this is linked to
>> > > multibytes functions…
>> > > > >> Maybe we shouldn't use the multibytes
>> functions
>> > > everywhere…
>> > > > >>
>> > > > >> I'll still try to improve performances.
>> > > > >>
>> > > > >> Hugues.
>> > > > >>
>> > > > >
>> > > > > Indeed, I also think that we should use mb_*
>> > > function only
>> > > > when necessary and choice to use them should be
>> made
>> > > in case by
>> > > > case basis.
>> > > > >
>> > > > > --
>> > > > > Thanks and Regards,
>> > > > >
>> > > > > Madhura Jayaratne
>> > > >
>> > > > Hi,
>> > > >
>> > > > I didn't push my commits, but that's what I've
>> > started. I
>> > > > replaced the mb_* calls by standard calls on
>> > configuration
>> > > > variables, reserved words, etc.
>> > > >
>> > > > Hugues.
>> > > >
>> > > >
>> > > > Hi,
>> > > >
>> > > > Out of desperation, I try another algorithm.
>> Instead of
>> > > buffering
>> > > > data until SQL delimiter, I'll try to parse all
>> lines.
>> > > > So, I won't parse 1000 times a buffer of 50000
>> > characters,
>> > > but less
>> > > > than 10 times many buffers of 500 characters. I hope
>> > this
>> > > will be
>> > > > faster.
>> > > >
>> > > > Hugues.
>> > > >
>> > > >
>> > > > Hi,
>> > > >
>> > > > The new algorithm is over. There are still some
>> controls to
>> > > add, but it
>> > > > is usable with the file in this ticket: [1]
>> > > > You can find my modifications here: [2]
>> > > >
>> > > > Marc, is it faster for you ?
>> > > > It seems that I won ~33% of time. We're still far from 5
>> > seconds…
>> > > > Maybe I'll try to use standard PHP functions to see the
>> > > difference. If
>> > > > the standard PHP functions are really faster, I'll try
>> > to add
>> > > an option
>> > > > to use mb_* functions or standard PHP functions, as you
>> > said.
>> > > >
>> > > > [1] https://sourceforge.net/p/phpmyadmin/bugs/4536/
>> > > > [2]
>> > > >
>> > >
>> >
>> https://github.com/Tithugues/phpmyadmin/tree/stringFunctions_useStandardFunctions_master
>> > >
>> > > Hi Hugues,
>> > > yes it's faster. With the same testing conditions, the
>> import
>> > > takes 1
>> > > min 20 sec.
>> > >
>> > >
>> > > --
>> > > Marc Delisle | phpMyAdmin
>> > >
>> > >
>> > > Thanks for your feedback.
>> > > I'll try another another improvement to be faster.
>> > >
>> > > Note for my self:
>> > > * read X characters but don't restart the search from 0 each
>> time
>> > > * search for the escaped quote with a lookbehind expression,
>> > > something like `(?!<\\)(\\\\)*'`
>> > >
>> > > Hugues.
>> > >
>> > >
>> > > Hi,
>> > >
>> > > As asked by Marc, I added on option to import by reading as a
>> > multibytes
>> > > string or not.
>> > > The default configuration won't read as multibytes string (because
>> > it's
>> > > too long…). It seems that the DnD to import doesn't use the
>> default
>> > > configuration, so what ever you define as default, it won't be
>> use in
>> > > this process.
>> > > Should we create a ticket for this ? I think it's possible to get
>> > it in
>> > > javascript.
>> > >
>> > > Hugues.
>> > >
>> >
>> > With the new option "Read as multibytes" left to its default
>> disabled
>> > value, my test file imports in 5 seconds.
>> >
>> > When this option is ticked, it takes 60 seconds.
>> >
>> > Note that in both cases, I get a Javascript error at the end (the
>> error
>> > report does not show the same error in Firefox than in Chromium).
>> Can
>> > anyone confirm this?
>> >
>> >
>> > --
>> > Marc Delisle | phpMyAdmin
>> >
>> >
>> > Marc,
>> >
>> > Could we please have both errors? I don't have it on Chrome. I'll try on
>> > Firefox and IE.
>> >
>> > Thanks,
>> > Hugues.
>>
>> >From Iceweasel 24.8.1:
>>
>> {
>> "pma_version": "4.3.0-dev",
>> "browser_name": "FIREFOX",
>> "browser_version": "24.0",
>> "user_os": "Linux",
>> "server_software": "Apache/2.2.22 (Debian)",
>> "user_agent_string": "Mozilla/5.0 (X11; Linux x86_64; rv:24.0)
>> Gecko/20140924 Firefox/24.0 Iceweasel/24.8.1",
>> "locale": "en",
>> "configuration_storage": "enabled",
>> "php_version": "5.4.4-14+deb7u14",
>> "exception_type": "js",
>> "exception": {
>> "mode": "stack",
>> "name": "SyntaxError",
>> "message": "JSON.parse: unexpected character",
>> "stack": [
>> {
>> "func": ".parseJSON",
>> "args": "",
>> "line": 2,
>> "column": "",
>> "context": [
>> "/*! jQuery v1.8.3 jquery.com | jquery.org/license
>> */\r",
>> "(function(e,t){function _(e){var t=M[e]={};return
>> v.each(e.split(y),functio//...",
>> "",
>> "function sprintf() {",
>> "/*",
>> " * Copyright (c) 2013 Kevin van Zonneveld
>> (http://kvz.io)",
>> " * and Contributors (http://phpjs.org/authors)"
>> ],
>> "filename": "jquery/jquery-1.8.3.min.js"
>> },
>> {
>> "func": "PMA_console.initialize/<",
>> "args": "",
>> "line": 164,
>> "column": "",
>> "context": [
>> " $('#pma_console_options
>> input[name=current_query]').prop('c//...",
>> " PMA_console.updateConfig();",
>> " });",
>> "",
>> " $(document).ajaxComplete(function
>> (event, xhr) {",
>> " var data =
>> $.parseJSON(xhr.responseText);",
>> " PMA_console.ajaxCallback(data);",
>> " });",
>> "",
>> " PMA_console.isInitialized = true;",
>> " }"
>> ],
>> "filename": "console.js"
>> },
>> {
>> "func": "ErrorReport.wrap_function/new_func",
>> "args": "",
>> "line": 262,
>> "column": "",
>> "context": [
>> " */",
>> " wrap_function: function (func) {",
>> " if (!func.wrapped) {",
>> " var new_func = function () {",
>> " try {",
>> " return func.apply(this,
>> arguments);",
>> " } catch (x) {",
>> " TraceKit.report(x);",
>> " }",
>> " };",
>> " new_func.wrapped = true;"
>> ],
>> "filename": "error_report.js"
>> },
>> {
>> "func": "v.event.dispatch",
>> "args": "",
>> "line": 2,
>> "column": "",
>> "context": [
>> "/*! jQuery v1.8.3 jquery.com | jquery.org/license
>> */\r",
>> "(function(e,t){function _(e){var t=M[e]={};return
>> v.each(e.split(y),functio//...",
>> "",
>> "function sprintf() {",
>> "/*",
>> " * Copyright (c) 2013 Kevin van Zonneveld
>> (http://kvz.io)",
>> " * and Contributors (http://phpjs.org/authors)"
>> ],
>> "filename": "jquery/jquery-1.8.3.min.js"
>> },
>> {
>> "func": "$event.dispatch",
>> "args": "",
>> "line": 372,
>> "column": "",
>> "context": [
>> "$event.dispatch = function( event ){",
>> "\tif ( $.data( this, \"suppress.\"+ event.type ) -
>> new Date().getTime() > 0 ){",
>> "\t\t$.removeData( this, \"suppress.\"+ event.type
>> );",
>> "\t\treturn;",
>> "\t}",
>> "\treturn $dispatch.apply( this, arguments );",
>> "};",
>> "",
>> "// event fix hooks for touch events...",
>> "var touchHooks = ",
>> "$event.fixHooks.touchstart = "
>> ],
>> "filename": "jquery/jquery.event.drag-2.2.js"
>> },
>> {
>> "func": "v.event.add/u",
>> "args": "",
>> "line": 2,
>> "column": "",
>> "context": [
>> "/*! jQuery v1.8.3 jquery.com | jquery.org/license
>> */\r",
>> "(function(e,t){function _(e){var t=M[e]={};return
>> v.each(e.split(y),functio//...",
>> "",
>> "function sprintf() {",
>> "/*",
>> " * Copyright (c) 2013 Kevin van Zonneveld
>> (http://kvz.io)",
>> " * and Contributors (http://phpjs.org/authors)"
>> ],
>> "filename": "jquery/jquery-1.8.3.min.js"
>> },
>> {
>> "func": "v.event.trigger",
>> "args": "",
>> "line": 2,
>> "column": "",
>> "context": [
>> "/*! jQuery v1.8.3 jquery.com | jquery.org/license
>> */\r",
>> "(function(e,t){function _(e){var t=M[e]={};return
>> v.each(e.split(y),functio//...",
>> "",
>> "function sprintf() {",
>> "/*",
>> " * Copyright (c) 2013 Kevin van Zonneveld
>> (http://kvz.io)",
>> " * and Contributors (http://phpjs.org/authors)"
>> ],
>> "filename": "jquery/jquery-1.8.3.min.js"
>> },
>> {
>> "func": "v.event.trigger",
>> "args": "",
>> "line": 2,
>> "column": "",
>> "context": [
>> "/*! jQuery v1.8.3 jquery.com | jquery.org/license
>> */\r",
>> "(function(e,t){function _(e){var t=M[e]={};return
>> v.each(e.split(y),functio//...",
>> "",
>> "function sprintf() {",
>> "/*",
>> " * Copyright (c) 2013 Kevin van Zonneveld
>> (http://kvz.io)",
>> " * and Contributors (http://phpjs.org/authors)"
>> ],
>> "filename": "jquery/jquery-1.8.3.min.js"
>> },
>> {
>> "func": "T",
>> "args": "",
>> "line": 2,
>> "column": "",
>> "context": [
>> "/*! jQuery v1.8.3 jquery.com | jquery.org/license
>> */\r",
>> "(function(e,t){function _(e){var t=M[e]={};return
>> v.each(e.split(y),functio//...",
>> "",
>> "function sprintf() {",
>> "/*",
>> " * Copyright (c) 2013 Kevin van Zonneveld
>> (http://kvz.io)",
>> " * and Contributors (http://phpjs.org/authors)"
>> ],
>> "filename": "jquery/jquery-1.8.3.min.js"
>> },
>> {
>> "func": ".send/r",
>> "args": "",
>> "line": 2,
>> "column": "",
>> "context": [
>> "/*! jQuery v1.8.3 jquery.com | jquery.org/license
>> */\r",
>> "(function(e,t){function _(e){var t=M[e]={};return
>> v.each(e.split(y),functio//...",
>> "",
>> "function sprintf() {",
>> "/*",
>> " * Copyright (c) 2013 Kevin van Zonneveld
>> (http://kvz.io)",
>> " * and Contributors (http://phpjs.org/authors)"
>> ],
>> "filename": "jquery/jquery-1.8.3.min.js"
>> }
>> ],
>> "useragent": "Mozilla/5.0 (X11; Linux x86_64; rv:24.0)
>> Gecko/20140924 Firefox/24.0 Iceweasel/24.8.1",
>> "incomplete": "false",
>> "uri": "db_import.php?target="
>> },
>> "script_name": "db_import.php",
>> "microhistory": {
>> "pages": [
>> {
>> "hash":
>>
>> "index.php?db=&table=&server=1&target=&token=c455fd8a0595a5032d18c07a64e7ab0a"
>> },
>> {
>> "hash":
>>
>> "db_structure.php?db=test&table=&server=1&target=&token=c455fd8a0595a5032d18c07a64e7ab0a",
>> "params": {
>> "opendb_url": "db_structure.php",
>> "safari_browser": "0",
>> "collation_connection": "utf8mb4_general_ci",
>> "lang": "en",
>> "server": "1",
>> "text_dir": "ltr",
>> "pma_text_default_tab": "Browse",
>> "pma_text_left_default_tab": "Structure",
>> "LimitChars": "50",
>> "pftext": "",
>> "confirm": "true",
>> "LoginCookieValidity": "144000",
>> "logged_in": "true"
>> }
>> },
>> {
>> "hash":
>>
>> "index.php?db=&table=&server=1&target=&token=c455fd8a0595a5032d18c07a64e7ab0a",
>> "params": {
>> "opendb_url": "db_structure.php",
>> "safari_browser": "0",
>> "collation_connection": "utf8mb4_general_ci",
>> "lang": "en",
>> "server": "1",
>> "text_dir": "ltr",
>> "pma_text_default_tab": "Browse",
>> "pma_text_left_default_tab": "Structure",
>> "LimitChars": "50",
>> "pftext": "",
>> "confirm": "true",
>> "LoginCookieValidity": "144000",
>> "logged_in": "true"
>> }
>> },
>> {
>> "hash":
>>
>> "db_structure.php?db=test&table=&server=1&target=&token=c455fd8a0595a5032d18c07a64e7ab0a",
>> "params": {
>> "opendb_url": "db_structure.php",
>> "safari_browser": "0",
>> "collation_connection": "utf8mb4_general_ci",
>> "lang": "en",
>> "server": "1",
>> "text_dir": "ltr",
>> "pma_text_default_tab": "Browse",
>> "pma_text_left_default_tab": "Structure",
>> "LimitChars": "50",
>> "pftext": "",
>> "confirm": "true",
>> "LoginCookieValidity": "144000",
>> "logged_in": "true"
>> }
>> },
>> {
>> "hash":
>>
>> "db_import.php?db=test&table=&server=1&target=&token=c455fd8a0595a5032d18c07a64e7ab0a",
>> "params": {
>> "opendb_url": "db_structure.php",
>> "safari_browser": "0",
>> "collation_connection": "utf8mb4_general_ci",
>> "lang": "en",
>> "server": "1",
>> "text_dir": "ltr",
>> "pma_text_default_tab": "Browse",
>> "pma_text_left_default_tab": "Structure",
>> "LimitChars": "50",
>> "pftext": "",
>> "confirm": "true",
>> "LoginCookieValidity": "144000",
>> "logged_in": "true"
>> }
>> }
>> ],
>> "current_index": "5"
>> }
>> }
>>
>> --
>> Marc Delisle | phpMyAdmin
>>
>
> Hi,
>
> It seems that the new algorithm is over, see [1]. The unit tests are now
> ok (except for the one failing on master). I did some tests with escaped
> quotes (with quote or backslash), delimiters into strings, modified
> delimiters.
> Marc, do you still have your error please? (I didn't work on it, but I
> hope that your issue is linked to the not-finished version…)
>
> Feel free to look at the new algorithm and to tell me if you see any flaw.
> If you can lead to an error with a correct SQL file, please send it to me.
>
> Thanks for your feedback.
>
> [1]
> https://github.com/Tithugues/phpmyadmin/tree/stringFunctions_useStandardFunctions_master
>
> Hugues.
>
Hi,
Since my previous email, I did some modifications to manage the display of
queries in a different way (closer to the current display).
But there is something strange with the attachment… I can import the file
(without error and all content executed), but on master, if I paste the
content of the file into the SQL console, I've an error… Is it the expected
behavior please? I don't see any exotic content in this file…
On my branch, I don't have an error, but only the first statement (DROP
PROCEDURE IF EXISTS) is executed…
Thanks,
Hugues.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.phpmyadmin.net/pipermail/developers/attachments/20141028/e7ec4e8a/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: procedure.sql
Type: application/octet-stream
Size: 161 bytes
Desc: not available
URL: <http://lists.phpmyadmin.net/pipermail/developers/attachments/20141028/e7ec4e8a/attachment.obj>
More information about the Developers
mailing list