Google Map takes VERY long to download tiles

I'm having problems with tile downloading in a Google map, with angular-google-maps.

HTTP requests go off as soon as I scroll or zoom, and the download takes milliseconds, but the TIME-TO-FIRST-BYTE delay can go up to several seconds as if the socket was queued for something, somewhere.

• Rest of site functionality is not affected by the problem
• Map has around 250 objects (markers, polygons, etc)
• Map is computationally responsive, interaction is fast
• Removal of these objects seems to eliminate the problem

A specific client of mine sent a video where it took THREE MINUTES for the tiles to draw on the map. Again, the rest of the site works.

I can reproduce the problem on a smaller time-scale. It's somehow related to objects on the map, but I can't figure out how. What should I do to fix or diagnose this better?