Hi Marc,
Here is a clue. In Firebug (network panel), compare the GET command that
is sent when everything is alright, with the other GET command where
there are delays. The parameters of the GET command where delays happen,
are weird (I see an underscore as one of the parameters).
As a result, when all is fine, most GET commands are replied with a "304
not modified" and served by the cache, whereas the weird parameter
causes a not cacheable request (replied by "200 OK").
Try to find where this underscore as a GET parameter comes from.