[Phpmyadmin-devel] GSoC Removal of Frames : New Navigation
Hi Rouslan, I was just reading your weekly report. I have a question about the 'refreshing of navigation' : You mention that the state of the navigation (which branches are open) has to be remembered, when sending it over to the client/browser. Does this mean that the entire navigation tree is generated server side on every change to the nav tree (f.e. clicking on a branch to open it, or when tables/databases are added/deleted/renamed/...)? Or only in case of a full refresh (or first load)? Would it make sense to keep the nav tree state stored client side and let the script repopulate the tree based on data it gets from the server, after a request initiated by user interaction (navigating the tree) or when the server sends new/changed/deleted tables when something is changed by the user in the main panel (triggering refreshNavigation)? Or would this overcomplicate things? -- Kind regards, Dieter Adriaenssens
On 02/07/12 21:28, Dieter Adriaenssens wrote:
Hi Rouslan,
I was just reading your weekly report. I have a question about the 'refreshing of navigation' : You mention that the state of the navigation (which branches are open) has to be remembered, when sending it over to the client/browser. Does
I think that I just worded it badly. The state of the tree is never, in fact, recorded anywhere. Instead it's calculated with a JS function.
this mean that the entire navigation tree is generated server side on every change to the nav tree (f.e. clicking on a branch to open it, or
When clicking on a branch to open, only that branch, its immediate children and all its parents are loaded into memory. Then only its immediate children are rendered into HTML.
when tables/databases are added/deleted/renamed/...)? Or only in case of a full refresh (or first load)?
On first load, only the list of databases (limited by $cfg[' MaxDbList']) is loaded into memory and rendered. A full refresh is used for when a db/table is added, removed, renamed, copied, etc. During the full refresh, only the nodes that were visible before the refresh are loaded into memory and rendered. The only expection to this are tables, as any request to render a table will force their children to also be loaded and rendered. The idea here seems nice as we can avoid some extra ajax requests after loading a list of tables. However this proved to be slow during full refreshes as all index and column information my have to be loaded for dozens of tables.
Would it make sense to keep the nav tree state stored client side and let the script repopulate the tree based on data it gets from the server, after a request initiated by user interaction (navigating the tree) or when the server sends new/changed/deleted tables when something is changed by the user in the main panel (triggering refreshNavigation)?
Or would this overcomplicate things?
Sorry, I'm not sure what you are trying to say here. That said, I haven't really explained to anyone how the whole thing works, so maybe I should do that. So here is the basic workflow as it is currently in my repo: During the initial page load, the navigation is not sent to the client at all, but just the links at the top of the panel and the logo. As soon as the ready event is fired on the page, the list of databases is fetched via ajax. During this request only these databases are loaded into memory and rendered (but none of their children). Then when a user clicks on a link to expand a particular branch, the path to that node (at this point it's just "root.sakila", for example) is sent to the server. The server will then load the databases into memory and check what kind of items are present in the db (tables, views, procedures, events). If there are more than one type of items, then containers for them (but not the items themselves) are loaded into memory, rendered and sent back to the client. Otherwise, if there are only tables present in a db, then the tables themselves are loaded into memory, rendered and sent back. Let's say that we have tables and events in the sakila db and we opened its branch. At this point the list of containers is received from the server and attached to the sakila node. If we click on "Tables", the following path is sent to the server "root.sakila.tables" and in the reply we will receive the list of tables and for each table a list of columns and a list of indexes, if any. This gets a bit more complicated when nodes are grouped. Whether this happens depends on the names of items and the $cfg['LeftFrameDBSeparator'], $cfg['LeftFrameTableSeparator '] and $cfg['LeftFrameTableLevel'] directives. If we have a hyphen as a db separator and sakila_one and sakila_two dbs, then when we try to expand the tables branch in sakila_one, the following paths will be sent to the server for processing: actual_path -> "root.sakila_two.tables"; virtual_path -> "root.sakila_.two.Tables". As far as refreshing goes, there is a JS function that traverses the rendered tree on the page and finds the deepest nodes in the hierarchy that have a "minus" icon near them. Then the list paths to these node is compiled and sent to the server. The server then knows how deep to load each branch into memory and which parts should be rendered. Let's look at a concrete example. Here's a tree that we have started browsing: + phpmyadmin - sakila - film + columns + film_actor - country + columns - indexes someindex someindex2 - testdb + tables - events someevent someevent2 + mysql + information_schema If we hit refresh. Then the following paths will be sent to the server: root.sakila.tables.country.indexes root.sakila.tables.film root.testdb.events The server will load into memory, all tables, columns and indexes from sakila and render all of them. It will check if there are still tables in testdb, but it won't load any of them. And it will load and render the events from testdb. The resulting HTML code is then sent back to the server and the tree that is on the page is simply swapped for the new one. The improvements that I was talking about on the blog for this system would be partial refreshes and removal of preloading for children of table nodes. * Partial refreshes would be nice when we add/modify a table, for example. In this case, we only need to reload the database, not the whole tree, but we need to know which database to reload, so it would be necessary to pass some parameters to the refreshing function, which at the moment takes none. * Preloading of indexes and columns for tables seemed like a good idea when I was first writing the code, but I didn't foresee that it would considerable slow down the refreshing of the tree, so it need to go. I hope this makes it clearer how things are done in the new navigation. Feel free to ask if there are still things that are unclear, and any suggestion are always welcome :) Bye, Rouslan
2012/7/3 Rouslan Placella <rouslan@placella.com>:
On 02/07/12 21:28, Dieter Adriaenssens wrote:
Hi Rouslan,
I was just reading your weekly report. I have a question about the 'refreshing of navigation' : You mention that the state of the navigation (which branches are open) has to be remembered, when sending it over to the client/browser. Does
I think that I just worded it badly. The state of the tree is never, in fact, recorded anywhere. Instead it's calculated with a JS function.
this mean that the entire navigation tree is generated server side on every change to the nav tree (f.e. clicking on a branch to open it, or
When clicking on a branch to open, only that branch, its immediate children and all its parents are loaded into memory. Then only its immediate children are rendered into HTML.
when tables/databases are added/deleted/renamed/...)? Or only in case of a full refresh (or first load)?
On first load, only the list of databases (limited by $cfg[' MaxDbList']) is loaded into memory and rendered.
A full refresh is used for when a db/table is added, removed, renamed, copied, etc. During the full refresh, only the nodes that were visible before the refresh are loaded into memory and rendered. The only expection to this are tables, as any request to render a table will force their children to also be loaded and rendered. The idea here seems nice as we can avoid some extra ajax requests after loading a list of tables. However this proved to be slow during full refreshes as all index and column information my have to be loaded for dozens of tables.
Would it make sense to keep the nav tree state stored client side and let the script repopulate the tree based on data it gets from the server, after a request initiated by user interaction (navigating the tree) or when the server sends new/changed/deleted tables when something is changed by the user in the main panel (triggering refreshNavigation)?
Or would this overcomplicate things?
Sorry, I'm not sure what you are trying to say here. That said, I haven't really explained to anyone how the whole thing works, so maybe I should do that. So here is the basic workflow as it is currently in my repo:
During the initial page load, the navigation is not sent to the client at all, but just the links at the top of the panel and the logo.
As soon as the ready event is fired on the page, the list of databases is fetched via ajax. During this request only these databases are loaded into memory and rendered (but none of their children).
Then when a user clicks on a link to expand a particular branch, the path to that node (at this point it's just "root.sakila", for example) is sent to the server. The server will then load the databases into memory and check what kind of items are present in the db (tables, views, procedures, events). If there are more than one type of items, then containers for them (but not the items themselves) are loaded into memory, rendered and sent back to the client. Otherwise, if there are only tables present in a db, then the tables themselves are loaded into memory, rendered and sent back.
Let's say that we have tables and events in the sakila db and we opened its branch. At this point the list of containers is received from the server and attached to the sakila node. If we click on "Tables", the following path is sent to the server "root.sakila.tables" and in the reply we will receive the list of tables and for each table a list of columns and a list of indexes, if any.
This gets a bit more complicated when nodes are grouped. Whether this happens depends on the names of items and the $cfg['LeftFrameDBSeparator'], $cfg['LeftFrameTableSeparator '] and $cfg['LeftFrameTableLevel'] directives. If we have a hyphen as a db separator and sakila_one and sakila_two dbs, then when we try to expand the tables branch in sakila_one, the following paths will be sent to the server for processing: actual_path -> "root.sakila_two.tables"; virtual_path -> "root.sakila_.two.Tables".
As far as refreshing goes, there is a JS function that traverses the rendered tree on the page and finds the deepest nodes in the hierarchy that have a "minus" icon near them. Then the list paths to these node is compiled and sent to the server. The server then knows how deep to load each branch into memory and which parts should be rendered. Let's look at a concrete example. Here's a tree that we have started browsing:
+ phpmyadmin - sakila - film + columns + film_actor - country + columns - indexes someindex someindex2 - testdb + tables - events someevent someevent2 + mysql + information_schema
If we hit refresh. Then the following paths will be sent to the server:
root.sakila.tables.country.indexes root.sakila.tables.film root.testdb.events
The server will load into memory, all tables, columns and indexes from sakila and render all of them. It will check if there are still tables in testdb, but it won't load any of them. And it will load and render the events from testdb. The resulting HTML code is then sent back to the server and the tree that is on the page is simply swapped for the new one.
Hi Rouslan, Thanks for explaining how the navigation is generated. It is clearer to me now. So when clicking on a table, an AJAX request is sent to the server to generate HTML only for everything below (columns/indexes) that table, which on it's turn is added by a JS script to the HTML defining the current expanded tree in the navigation? Or is the structure of the parent nodes generated as well?
The improvements that I was talking about on the blog for this system would be partial refreshes and removal of preloading for children of table nodes.
* Partial refreshes would be nice when we add/modify a table, for example. In this case, we only need to reload the database, not the whole tree, but we need to know which database to reload, so it would be necessary to pass some parameters to the refreshing function, which at the moment takes none.
Where would this parameter come from? I guess it would come from the process that renamed/removed/added a table/database/column.
* Preloading of indexes and columns for tables seemed like a good idea when I was first writing the code, but I didn't foresee that it would considerable slow down the refreshing of the tree, so it need to go.
Can't you do a two stage refreshing? First the visible things (tables of a database, when opening a database with only tables (no events, ..)) and then in a second stage (when the tree is populated with tables) the level below that (in this example columns and indexes for every table). This way the users sees the tree populated quite fast, because only the tables are displayed. In the background, what's below the tables is populated, resulting in a quick opening of a node when clicking on a table, because what's below is already loaded. With database with many tables this might be a problem, so there probably should be some kind of limit to the amount of preloading. Kind regards, Dieter
I hope this makes it clearer how things are done in the new navigation. Feel free to ask if there are still things that are unclear, and any suggestion are always welcome :)
Bye, Rouslan
------------------------------------------------------------------------------ Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ _______________________________________________ Phpmyadmin-devel mailing list Phpmyadmin-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/phpmyadmin-devel
-- Kind regards, Dieter Adriaenssens
On 03/07/12 21:07, Dieter Adriaenssens wrote:
2012/7/3 Rouslan Placella<rouslan@placella.com>:
On 02/07/12 21:28, Dieter Adriaenssens wrote:
Hi Rouslan,
I was just reading your weekly report. I have a question about the 'refreshing of navigation' : You mention that the state of the navigation (which branches are open) has to be remembered, when sending it over to the client/browser. Does
I think that I just worded it badly. The state of the tree is never, in fact, recorded anywhere. Instead it's calculated with a JS function.
this mean that the entire navigation tree is generated server side on every change to the nav tree (f.e. clicking on a branch to open it, or
When clicking on a branch to open, only that branch, its immediate children and all its parents are loaded into memory. Then only its immediate children are rendered into HTML.
when tables/databases are added/deleted/renamed/...)? Or only in case of a full refresh (or first load)?
On first load, only the list of databases (limited by $cfg[' MaxDbList']) is loaded into memory and rendered.
A full refresh is used for when a db/table is added, removed, renamed, copied, etc. During the full refresh, only the nodes that were visible before the refresh are loaded into memory and rendered. The only expection to this are tables, as any request to render a table will force their children to also be loaded and rendered. The idea here seems nice as we can avoid some extra ajax requests after loading a list of tables. However this proved to be slow during full refreshes as all index and column information my have to be loaded for dozens of tables.
Would it make sense to keep the nav tree state stored client side and let the script repopulate the tree based on data it gets from the server, after a request initiated by user interaction (navigating the tree) or when the server sends new/changed/deleted tables when something is changed by the user in the main panel (triggering refreshNavigation)?
Or would this overcomplicate things?
Sorry, I'm not sure what you are trying to say here. That said, I haven't really explained to anyone how the whole thing works, so maybe I should do that. So here is the basic workflow as it is currently in my repo:
During the initial page load, the navigation is not sent to the client at all, but just the links at the top of the panel and the logo.
As soon as the ready event is fired on the page, the list of databases is fetched via ajax. During this request only these databases are loaded into memory and rendered (but none of their children).
Then when a user clicks on a link to expand a particular branch, the path to that node (at this point it's just "root.sakila", for example) is sent to the server. The server will then load the databases into memory and check what kind of items are present in the db (tables, views, procedures, events). If there are more than one type of items, then containers for them (but not the items themselves) are loaded into memory, rendered and sent back to the client. Otherwise, if there are only tables present in a db, then the tables themselves are loaded into memory, rendered and sent back.
Let's say that we have tables and events in the sakila db and we opened its branch. At this point the list of containers is received from the server and attached to the sakila node. If we click on "Tables", the following path is sent to the server "root.sakila.tables" and in the reply we will receive the list of tables and for each table a list of columns and a list of indexes, if any.
This gets a bit more complicated when nodes are grouped. Whether this happens depends on the names of items and the $cfg['LeftFrameDBSeparator'], $cfg['LeftFrameTableSeparator '] and $cfg['LeftFrameTableLevel'] directives. If we have a hyphen as a db separator and sakila_one and sakila_two dbs, then when we try to expand the tables branch in sakila_one, the following paths will be sent to the server for processing: actual_path -> "root.sakila_two.tables"; virtual_path -> "root.sakila_.two.Tables".
As far as refreshing goes, there is a JS function that traverses the rendered tree on the page and finds the deepest nodes in the hierarchy that have a "minus" icon near them. Then the list paths to these node is compiled and sent to the server. The server then knows how deep to load each branch into memory and which parts should be rendered. Let's look at a concrete example. Here's a tree that we have started browsing:
+ phpmyadmin - sakila - film + columns + film_actor - country + columns - indexes someindex someindex2 - testdb + tables - events someevent someevent2 + mysql + information_schema
If we hit refresh. Then the following paths will be sent to the server:
root.sakila.tables.country.indexes root.sakila.tables.film root.testdb.events
The server will load into memory, all tables, columns and indexes from sakila and render all of them. It will check if there are still tables in testdb, but it won't load any of them. And it will load and render the events from testdb. The resulting HTML code is then sent back to the server and the tree that is on the page is simply swapped for the new one.
Hi Rouslan,
Thanks for explaining how the navigation is generated. It is clearer to me now.
So when clicking on a table, an AJAX request is sent to the server to generate HTML only for everything below (columns/indexes) that table, which on it's turn is added by a JS script to the HTML defining the current expanded tree in the navigation? Or is the structure of the parent nodes generated as well?
You are correct, the parent nodes need to be loaded into memory in the form of objects that inherit from the Node class. This way we can have a representation of a valid branch of the tree. Then only the active node and it's children are rendered into HTML (which is the most expensive operation).
The improvements that I was talking about on the blog for this system would be partial refreshes and removal of preloading for children of table nodes.
* Partial refreshes would be nice when we add/modify a table, for example. In this case, we only need to reload the database, not the whole tree, but we need to know which database to reload, so it would be necessary to pass some parameters to the refreshing function, which at the moment takes none.
Where would this parameter come from? I guess it would come from the process that renamed/removed/added a table/database/column.
That's right. And it means that it's necessary to go every piece of JS code that interacts with the DBMS in a way that requires a refresh of some part of the navigation tree.
* Preloading of indexes and columns for tables seemed like a good idea when I was first writing the code, but I didn't foresee that it would considerable slow down the refreshing of the tree, so it need to go.
Can't you do a two stage refreshing? First the visible things (tables of a database, when opening a database with only tables (no events, ..)) and then in a second stage (when the tree is populated with tables) the level below that (in this example columns and indexes for every table). This way the users sees the tree populated quite fast, because only the tables are displayed. In the background, what's below the tables is populated, resulting in a quick opening of a node when clicking on a table, because what's below is already loaded.
With database with many tables this might be a problem, so there probably should be some kind of limit to the amount of preloading.
I think that I would like to keep this simple for the moment. I might dedicate it more time at a later stage.
Kind regards,
Dieter
I hope this makes it clearer how things are done in the new navigation. Feel free to ask if there are still things that are unclear, and any suggestion are always welcome :)
Bye, Rouslan
------------------------------------------------------------------------------ Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ _______________________________________________ Phpmyadmin-devel mailing list Phpmyadmin-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/phpmyadmin-devel
Hi Rouslan,
Thanks for explaining how the navigation is generated. It is clearer to me now.
So when clicking on a table, an AJAX request is sent to the server to generate HTML only for everything below (columns/indexes) that table, which on it's turn is added by a JS script to the HTML defining the current expanded tree in the navigation? Or is the structure of the parent nodes generated as well?
You are correct, the parent nodes need to be loaded into memory in the form of objects that inherit from the Node class. This way we can have a representation of a valid branch of the tree. Then only the active node and it's children are rendered into HTML (which is the most expensive operation).
The improvements that I was talking about on the blog for this system would be partial refreshes and removal of preloading for children of table nodes.
* Partial refreshes would be nice when we add/modify a table, for example. In this case, we only need to reload the database, not the whole tree, but we need to know which database to reload, so it would be necessary to pass some parameters to the refreshing function, which at the moment takes none.
Where would this parameter come from? I guess it would come from the process that renamed/removed/added a table/database/column.
That's right. And it means that it's necessary to go every piece of JS code that interacts with the DBMS in a way that requires a refresh of some part of the navigation tree.
True, and you would still miss the changes to the database done outside phpMyAdmin.
* Preloading of indexes and columns for tables seemed like a good idea when I was first writing the code, but I didn't foresee that it would considerable slow down the refreshing of the tree, so it need to go.
Can't you do a two stage refreshing? First the visible things (tables of a database, when opening a database with only tables (no events, ..)) and then in a second stage (when the tree is populated with tables) the level below that (in this example columns and indexes for every table). This way the users sees the tree populated quite fast, because only the tables are displayed. In the background, what's below the tables is populated, resulting in a quick opening of a node when clicking on a table, because what's below is already loaded.
With database with many tables this might be a problem, so there probably should be some kind of limit to the amount of preloading.
I think that I would like to keep this simple for the moment. I might dedicate it more time at a later stage.
Agreed! Spending too much time on this could disrupt your GSoC schedule. This is something that can be improved afterwards. ;) Kind regards, Dieter
participants (2)
-
Dieter Adriaenssens -
Rouslan Placella