[Phpmyadmin-devel] GSoC - Server Stats
Hello everyone, I'm a student from Munich, Germany and stumbled across the GSoC Idea of collecting the data that is shown on the server status page and rendering them as graphs over time. I will apply in the next days, here is what I thought of implementing when I'm accepted: Data collection: In the past, I used rrdtool to collect stats on a server. Its datastore has a limited size - over time old entries get merged together and the data gets thinner, but is never completely deleted. Unfortunately, it is not very portable. Therefor, I would implement a similarily behaving datastore ontop of a mysql table. I think of storing each set of measures as a row and limit the number of total rows, making room for new entries by combining one pair of neighbouring old entries together. Plotting: On the ideas page GD is suggeted, but I would prefer to use processing.js, which is a javascript library that draws on a canvas element with a similar usage. To connect to the datastore I would build a json interface for the collected data. Let me know, what you think of it! Best regards, Philip
Philip Frank a écrit :
Hello everyone,
I'm a student from Munich, Germany and stumbled across the GSoC Idea of collecting the data that is shown on the server status page and rendering them as graphs over time. I will apply in the next days, here is what I thought of implementing when I'm accepted:
Data collection: In the past, I used rrdtool to collect stats on a server. Its datastore has a limited size - over time old entries get merged together and the data gets thinner, but is never completely deleted. Unfortunately, it is not very portable. Therefor, I would implement a similarily behaving datastore ontop of a mysql table. I think of storing each set of measures as a row and limit the number of total rows, making room for new entries by combining one pair of neighbouring old entries together.
Hi Philip, so you plan that an external process will collect data? Please have a look on this list archives, this was discussed a few days ago.
Plotting: On the ideas page GD is suggeted, but I would prefer to use processing.js, which is a javascript library that draws on a canvas element with a similar usage. To connect to the datastore I would build a json interface for the collected data.
Let me know, what you think of it!
Best regards, Philip
-- Marc Delisle http://infomarc.info
2010/4/1 Marc Delisle <marc@infomarc.info>:
Philip Frank a écrit :
Hello everyone,
I'm a student from Munich, Germany and stumbled across the GSoC Idea of collecting the data that is shown on the server status page and rendering them as graphs over time. I will apply in the next days, here is what I thought of implementing when I'm accepted:
Data collection: In the past, I used rrdtool to collect stats on a server. Its datastore has a limited size - over time old entries get merged together and the data gets thinner, but is never completely deleted. Unfortunately, it is not very portable. Therefor, I would implement a similarily behaving datastore ontop of a mysql table. I think of storing each set of measures as a row and limit the number of total rows, making room for new entries by combining one pair of neighbouring old entries together.
Hi Philip, so you plan that an external process will collect data? Please have a look on this list archives, this was discussed a few days ago.
Hi Marc, i thought the process could be triggered by a cron, but if that is not an option, historic data can not be collected, i guess. How about a live monitoring tool? I could create a visualisation of the ratios of the different query types. Constantly updating that with ajax would be nice, and maybe even collecting the data in the browser and plotting graphes. Philip
Plotting: On the ideas page GD is suggeted, but I would prefer to use processing.js, which is a javascript library that draws on a canvas element with a similar usage. To connect to the datastore I would build a json interface for the collected data.
Let me know, what you think of it!
Best regards, Philip
-- Marc Delisle http://infomarc.info
------------------------------------------------------------------------------ Download Intel® Parallel Studio Eval Try the new software tools for yourself. Speed compiling, find bugs proactively, and fine-tune applications for parallel performance. See why Intel Parallel Studio got high marks during beta. http://p.sf.net/sfu/intel-sw-dev _______________________________________________ Phpmyadmin-devel mailing list Phpmyadmin-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/phpmyadmin-devel
Philip Frank a écrit :
2010/4/1 Marc Delisle <marc@infomarc.info>:
Philip Frank a écrit :
Hello everyone,
I'm a student from Munich, Germany and stumbled across the GSoC Idea of collecting the data that is shown on the server status page and rendering them as graphs over time. I will apply in the next days, here is what I thought of implementing when I'm accepted:
Data collection: In the past, I used rrdtool to collect stats on a server. Its datastore has a limited size - over time old entries get merged together and the data gets thinner, but is never completely deleted. Unfortunately, it is not very portable. Therefor, I would implement a similarily behaving datastore ontop of a mysql table. I think of storing each set of measures as a row and limit the number of total rows, making room for new entries by combining one pair of neighbouring old entries together. Hi Philip, so you plan that an external process will collect data? Please have a look on this list archives, this was discussed a few days ago.
Hi Marc, i thought the process could be triggered by a cron, but if that is not an option, historic data can not be collected, i guess. How about a live monitoring tool? I could create a visualisation of the ratios of the different query types. Constantly updating that with ajax would be nice, and maybe even collecting the data in the browser and plotting graphes.
As discussed on the list, asking users to setup cron is out of phpMyAdmin's scope. Live monitoring tool is interesting. -- Marc Delisle http://infomarc.info
2010/4/1 Marc Delisle <marc@infomarc.info>:
Philip Frank a écrit :
2010/4/1 Marc Delisle <marc@infomarc.info>:
Philip Frank a écrit :
Hello everyone,
I'm a student from Munich, Germany and stumbled across the GSoC Idea of collecting the data that is shown on the server status page and rendering them as graphs over time. I will apply in the next days, here is what I thought of implementing when I'm accepted:
Data collection: In the past, I used rrdtool to collect stats on a server. Its datastore has a limited size - over time old entries get merged together and the data gets thinner, but is never completely deleted. Unfortunately, it is not very portable. Therefor, I would implement a similarily behaving datastore ontop of a mysql table. I think of storing each set of measures as a row and limit the number of total rows, making room for new entries by combining one pair of neighbouring old entries together. Hi Philip, so you plan that an external process will collect data? Please have a look on this list archives, this was discussed a few days ago.
Hi Marc, i thought the process could be triggered by a cron, but if that is not an option, historic data can not be collected, i guess. How about a live monitoring tool? I could create a visualisation of the ratios of the different query types. Constantly updating that with ajax would be nice, and maybe even collecting the data in the browser and plotting graphes.
As discussed on the list, asking users to setup cron is out of phpMyAdmin's scope. Live monitoring tool is interesting.
-- Marc Delisle http://infomarc.info
Alright, that live monitoring could be awesome, too. The user could select what values should be measured, they would be constantly updated via ajax and it would be graphed with processing.js. The data could be stored as long as the browser window is open, so it could still be displayed on a time axis. At its best, one could just leave pma running in his browser while testing different configurations. With the html5 client storage, even saving snapshots persistently and comparing them could be done. How does this sound?
------------------------------------------------------------------------------ Download Intel® Parallel Studio Eval Try the new software tools for yourself. Speed compiling, find bugs proactively, and fine-tune applications for parallel performance. See why Intel Parallel Studio got high marks during beta. http://p.sf.net/sfu/intel-sw-dev _______________________________________________ Phpmyadmin-devel mailing list Phpmyadmin-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/phpmyadmin-devel
Philip Frank a écrit :
2010/4/1 Marc Delisle <marc@infomarc.info>:
Philip Frank a écrit :
2010/4/1 Marc Delisle <marc@infomarc.info>:
Philip Frank a écrit :
Hello everyone,
I'm a student from Munich, Germany and stumbled across the GSoC Idea of collecting the data that is shown on the server status page and rendering them as graphs over time. I will apply in the next days, here is what I thought of implementing when I'm accepted:
Data collection: In the past, I used rrdtool to collect stats on a server. Its datastore has a limited size - over time old entries get merged together and the data gets thinner, but is never completely deleted. Unfortunately, it is not very portable. Therefor, I would implement a similarily behaving datastore ontop of a mysql table. I think of storing each set of measures as a row and limit the number of total rows, making room for new entries by combining one pair of neighbouring old entries together. Hi Philip, so you plan that an external process will collect data? Please have a look on this list archives, this was discussed a few days ago.
Hi Marc, i thought the process could be triggered by a cron, but if that is not an option, historic data can not be collected, i guess. How about a live monitoring tool? I could create a visualisation of the ratios of the different query types. Constantly updating that with ajax would be nice, and maybe even collecting the data in the browser and plotting graphes. As discussed on the list, asking users to setup cron is out of phpMyAdmin's scope. Live monitoring tool is interesting.
-- Marc Delisle http://infomarc.info
Alright, that live monitoring could be awesome, too. The user could select what values should be measured, they would be constantly updated via ajax and it would be graphed with processing.js. The data could be stored as long as the browser window is open, so it could still be displayed on a time axis. At its best, one could just leave pma running in his browser while testing different configurations.
With the html5 client storage, even saving snapshots persistently and comparing them could be done.
How does this sound?
Before reading your post about "as long as the browser window is open" I had the same idea. About selecting what values should be measured, I think that this interface should at least propose a list of the most popular things to measure, to guide the admin. I guess that admins will want to use this to find problems on their servers. -- Marc Delisle http://infomarc.info
2010/4/2 Marc Delisle <marc@infomarc.info>:
Philip Frank a écrit :
2010/4/1 Marc Delisle <marc@infomarc.info>:
Philip Frank a écrit :
2010/4/1 Marc Delisle <marc@infomarc.info>:
Philip Frank a écrit :
Hello everyone,
I'm a student from Munich, Germany and stumbled across the GSoC Idea of collecting the data that is shown on the server status page and rendering them as graphs over time. I will apply in the next days, here is what I thought of implementing when I'm accepted:
Data collection: In the past, I used rrdtool to collect stats on a server. Its datastore has a limited size - over time old entries get merged together and the data gets thinner, but is never completely deleted. Unfortunately, it is not very portable. Therefor, I would implement a similarily behaving datastore ontop of a mysql table. I think of storing each set of measures as a row and limit the number of total rows, making room for new entries by combining one pair of neighbouring old entries together. 2010/4/2 Raj Shekhar <rajlist@rajshekhar.net>: In infinite wisdom Philip Frank <ich@philipfrank.de> wrote:
Data collection: In the past, I used rrdtool to collect stats on a server. Its datastore has a limited size - over time old entries get merged together and the data gets thinner, but is never completely deleted. Unfortunately, it is not very portable.
http://infomarc.info 2010/4/2 Raj Shekhar <rajlist@rajshekhar.net>: In infinite wisdom Philip Frank <ich@philipfrank.de> wrote:
Data collection: In the past, I used rrdtool to collect stats on a server. Its datastore has a limited size - over time old entries get merged together and the data gets thinner, but is never completely deleted. Unfortunately, it is not very portable.
The size of the RRD file does not increase over time. THat is a feature, not a bug :-) . Merging of the older data (reducing the granularity) is a side effect of this. In my opinion, the way RRD handles that is quite intelligent. It would be hard to get that functionality right using mysql and rollup tables.
Why do you say that RRDs are not portable? I have seen them work on bsd/linux (and on 32/64 bit).
-- Raj Shekhar
Hi Raj, thanks for your feedback! The only way to write or access rrds is the rrdtool executable. You can only get data out by printing it out as graphs, thats what I mean by not portable.
From what Marc wrote and reading the past discussions on the list I rethough of my project to not require any configurations on the server besides running pma. I won't use any server side datastore so rrdtool is off the table, anyway.
Hi Philip, so you plan that an external process will collect data? Please have a look on this list archives, this was discussed a few days ago.
Hi Marc, i thought the process could be triggered by a cron, but if that is not an option, historic data can not be collected, i guess. How about a live monitoring tool? I could create a visualisation of the ratios of the different query types. Constantly updating that with ajax would be nice, and maybe even collecting the data in the browser and plotting graphes. As discussed on the list, asking users to setup cron is out of phpMyAdmin's scope. Live monitoring tool is interesting.
-- Marc Delisle http://infomarc.info
Alright, that live monitoring could be awesome, too. The user could select what values should be measured, they would be constantly updated via ajax and it would be graphed with processing.js. The data could be stored as long as the browser window is open, so it could still be displayed on a time axis. At its best, one could just leave pma running in his browser while testing different configurations.
With the html5 client storage, even saving snapshots persistently and comparing them could be done.
How does this sound?
Before reading your post about "as long as the browser window is open" I had the same idea.
About selecting what values should be measured, I think that this interface should at least propose a list of the most popular things to measure, to guide the admin. I guess that admins will want to use this to find problems on their servers.
-- Marc Delisle
Hi Marc, There are really a lot of values displayed, and I don't have a clue what most of them say. Ordering them somehow when allowing the admin to pick is a good idea. Some show up in red on my server, I guess these could also be more prominent since they possibly indicate a problem. I found this reference guide for the status variables: http://dev.mysql.com/doc/refman/5.0/en/server-status-variables.html Do you know any other good documents that describe what these values are all about? I wrote most of my application today, this is my list of milestones: - JSON Protocol is defined - PHP middleware running - Client side datastore planned and implemented - Mocked UI & created HTML - Data gets filtered and recorded into datastore - Datastore benchmarked, browsers don't crash - Documentation on datastore written - Study processing.js - JS object hierarchy for plotting planned - Defined interface to feed plotter from datastore - Glue code to datastore is written - Plotters' common features implemented - Bar graph plotter done - Curve plotter for temporal data done - Documentation about plotters written - Bonus: Implement additional plotters I'm pretty confident this can be done in the allocated time.
------------------------------------------------------------------------------ Download Intel® Parallel Studio Eval Try the new software tools for yourself. Speed compiling, find bugs proactively, and fine-tune applications for parallel performance. See why Intel Parallel Studio got high marks during beta. http://p.sf.net/sfu/intel-sw-dev _______________________________________________ Phpmyadmin-devel mailing list Phpmyadmin-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/phpmyadmin-devel
Hi Dne Fri, 2 Apr 2010 14:26:16 +0200 Philip Frank <ich@philipfrank.de> napsal(a):
thanks for your feedback! The only way to write or access rrds is the rrdtool executable. You can only get data out by printing it out as graphs, thats what I mean by not portable.
There are ways to access RRD at least from Python and PHP.
From what Marc wrote and reading the past discussions on the list I rethough of my project to not require any configurations on the server besides running pma. I won't use any server side datastore so rrdtool is off the table, anyway.
The problem is not using server datastore - we already optionally use lot of tables to store our information, but doing some periodic collection. However I think having such thing as optional module is okay.
There are really a lot of values displayed, and I don't have a clue what most of them say. Ordering them somehow when allowing the admin to pick is a good idea. Some show up in red on my server, I guess these could also be more prominent since they possibly indicate a problem.
I found this reference guide for the status variables: http://dev.mysql.com/doc/refman/5.0/en/server-status-variables.html Do you know any other good documents that describe what these values are all about?
You can look at server status page in phpMyAdmin. -- Michal Čihař | http://cihar.com | http://blog.cihar.com
In infinite wisdom Michal Čihař <michal@cihar.com> wrote:
The problem is not using server datastore - we already optionally use lot of tables to store our information, but doing some periodic collection. However I think having such thing as optional module is okay.
mysql 5.1 has an event scheduler. If you want to use mysql tables to store these counters, you can use that. <http://dev.mysql.com/doc/refman/5.1/en/events-overview.html> In my experience, any serious DBA will have a nagios/cacti monitoring and metrics collection on their mysql box. Has there been an interest shown by the users for this monitoring being done by phpmyadmin itself? OTOH, this feature will allow any small site to have a good collection of its database's metrics. (Excuse my ignorance about phpmyadmin's history and how its users use it)
2010/4/2 Raj Shekhar <rajlist@rajshekhar.net>:
In infinite wisdom Michal Čihař <michal@cihar.com> wrote:
The problem is not using server datastore - we already optionally use lot of tables to store our information, but doing some periodic collection. However I think having such thing as optional module is okay.
mysql 5.1 has an event scheduler. If you want to use mysql tables to store these counters, you can use that. <http://dev.mysql.com/doc/refman/5.1/en/events-overview.html>
In my experience, any serious DBA will have a nagios/cacti monitoring and metrics collection on their mysql box. Has there been an interest shown by the users for this monitoring being done by phpmyadmin itself? OTOH, this feature will allow any small site to have a good collection of its database's metrics.
(Excuse my ignorance about phpmyadmin's history and how its users use it)
------------------------------------------------------------------------------ Download Intel® Parallel Studio Eval Try the new software tools for yourself. Speed compiling, find bugs proactively, and fine-tune applications for parallel performance. See why Intel Parallel Studio got high marks during beta. http://p.sf.net/sfu/intel-sw-dev _______________________________________________ Phpmyadmin-devel mailing list Phpmyadmin-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/phpmyadmin-devel
Hi, thanks for pointing me towards the event scheduler, this could probably be used to collect data on the server side. I would like to keep my project focus as it is, but will keep in mind that adding a server side data collector could be desired one day. The goal of my project is not to compete with full-scale monitoring tools like cacti, but simply to build something useful from the data currently displayed on the server status page.
Hi Dne Fri, 02 Apr 2010 12:47:12 -0700 Raj Shekhar <rajlist@rajshekhar.net> napsal(a):
mysql 5.1 has an event scheduler. If you want to use mysql tables to store these counters, you can use that. <http://dev.mysql.com/doc/refman/5.1/en/events-overview.html>
Did not notice this, and we could definitely optionally use this.
In my experience, any serious DBA will have a nagios/cacti monitoring and metrics collection on their mysql box. Has there been an interest shown by the users for this monitoring being done by phpmyadmin itself? OTOH, this feature will allow any small site to have a good collection of its database's metrics.
Well we don't intend to compete with big monitoring systems. However it would be nice to display some simple chart how server was loaded and it has been requested several times. -- Michal Čihař | http://cihar.com | http://blog.cihar.com
In infinite wisdom Philip Frank <ich@philipfrank.de> wrote:
Data collection: In the past, I used rrdtool to collect stats on a server. Its datastore has a limited size - over time old entries get merged together and the data gets thinner, but is never completely deleted. Unfortunately, it is not very portable.
The size of the RRD file does not increase over time. THat is a feature, not a bug :-) . Merging of the older data (reducing the granularity) is a side effect of this. In my opinion, the way RRD handles that is quite intelligent. It would be hard to get that functionality right using mysql and rollup tables. Why do you say that RRDs are not portable? I have seen them work on bsd/linux (and on 32/64 bit). -- Raj Shekhar - If there's anything more important than my ego around, I want it caught and shot now. - Read the latest at my blog: "casio's g-shock watch" <http://rajshekhar.net/blog/archives/360-casios-g-shock-watch.html>
Raj Shekhar wrote:
In infinite wisdom Philip Frank <ich@philipfrank.de> wrote:
Data collection: In the past, I used rrdtool to collect stats on a server. Its datastore has a limited size - over time old entries get merged together and the data gets thinner, but is never completely deleted. Unfortunately, it is not very portable.
The size of the RRD file does not increase over time. THat is a feature, not a bug :-) . Merging of the older data (reducing the granularity) is a side effect of this. In my opinion, the way RRD handles that is quite intelligent. It would be hard to get that functionality right using mysql and rollup tables.
Why do you say that RRDs are not portable? I have seen them work on bsd/linux (and on 32/64 bit).
I believe that their are rrd tools for many platforms available. But this would add new requirements to the environment in which pma is installed. It would also burden the administrator with extra complexity, as a writable filesystem location and the privilege to execute external binaries would be required. One of the sweet things of pma is in my opinion that it works for the most part out-of-the-box. We should be very careful not to lure people with nice features, which these statistics would be, and then jump through a complex installation. I'm very curious if the mysql 5.1 event scheduler combined with pmadb storage would be reliable and easy maintainable. -- Met vriendelijke groet / Regards, Herman van Rink Initfour websolutions
2010/4/6 Herman van Rink <rink@initfour.nl>:
Raj Shekhar wrote:
In infinite wisdom Philip Frank <ich@philipfrank.de> wrote:
Data collection: In the past, I used rrdtool to collect stats on a server. Its datastore has a limited size - over time old entries get merged together and the data gets thinner, but is never completely deleted. Unfortunately, it is not very portable.
The size of the RRD file does not increase over time. THat is a feature, not a bug :-) . Merging of the older data (reducing the granularity) is a side effect of this. In my opinion, the way RRD handles that is quite intelligent. It would be hard to get that functionality right using mysql and rollup tables.
Why do you say that RRDs are not portable? I have seen them work on bsd/linux (and on 32/64 bit).
I believe that their are rrd tools for many platforms available. But this would add new requirements to the environment in which pma is installed. It would also burden the administrator with extra complexity, as a writable filesystem location and the privilege to execute external binaries would be required.
One of the sweet things of pma is in my opinion that it works for the most part out-of-the-box. We should be very careful not to lure people with nice features, which these statistics would be, and then jump through a complex installation.
I'm very curious if the mysql 5.1 event scheduler combined with pmadb storage would be reliable and easy maintainable.
-- Met vriendelijke groet / Regards,
Herman van Rink Initfour websolutions
Snip from my application: ----- Collecting data on the server side was discussed on phpmyadmins mailing list several times and a consensus could not yet be found. The major objections raised were that additional steps would have to be taken by the user to set up a periodical collection of data, and that competition with established monitoring tools like cacti was not desired. To stay flexible regarding this issue, the protocols used in my implementation will also allow for data possibly recorded in the past to be transferred, although server side data collection will not be part of my project. ----- I'll try to get something with scheduled events started as a parallel project.
------------------------------------------------------------------------------ Download Intel® Parallel Studio Eval Try the new software tools for yourself. Speed compiling, find bugs proactively, and fine-tune applications for parallel performance. See why Intel Parallel Studio got high marks during beta. http://p.sf.net/sfu/intel-sw-dev _______________________________________________ Phpmyadmin-devel mailing list Phpmyadmin-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/phpmyadmin-devel
participants (5)
-
Herman van Rink -
Marc Delisle -
Michal Čihař -
Philip Frank -
Raj Shekhar