On Fri, Mar 23, 2012 at 5:40 AM, Isaac Bennetch bennetch@gmail.com wrote:
Hi,
On 3/22/12 8:00 PM, adeel khan wrote:
[[snip]]
I thought about the solution and here what i can think of it right now First as your app generate everything before showing the difference stuff to the user, this is the major cause of bottleneck. In my 20K tuples example i was amazed why it didnt give me any timeout error on the first dialog. In a worst case scenarios generating a diff for the entire db is simply not scalable approach.Even if you have a lot of time/memory the user would have other stuff to do and would not wait for a long time for the sync thing to finish. I think the whole thing should be done in batches.The idea is that the structural diff is mostly less time and memory consuming, so its better to generate it for the entire db, but for data, just generate enough so that the script doesnot time out. (cases where we have huge tables). Now any user would get all the structural diff , plus what data diff we can able to get through in that pass. The time allowed for pass is bounded by script execution time which is 5 minutes as i see.
For what it's worth, the script execution time is configurable and will differ depending on the hosting environment; so we can't count on any specific predefined number.
But as we can set the timeout value to a minimum execution time programmatically we would always be assured that we would able to get that time for a pass.
adeel