Geographic Information Systems

Q How to avoid Google map geocode limit?

I'm creating a custom google map that has 125 markers plotted via a cms. When loading the map I get this message:

Geocode was not successful for the following reason: OVER_QUERY_LIMIT

I'm pretty sure it's the way in which I've geocoded the markers.

How can I avoid these warnings and is there a more efficient way to geocode the results?

UPDATE: This is my attempt at Casey's answer, I'm just getting a blank page at the moment.

<script type="text/javascript"> 
(function() { 

window.onload = function() { 
 var mc;
// Creating an object literal containing the properties we want to pass to the map 
var options = { 
zoom: 10, 
center: new google.maps.LatLng(52.40, -3.61), 
mapTypeId: google.maps.MapTypeId.ROADMAP 
}; 

// Creating the map 
var map = new google.maps.Map(document.getElementById('map'), options); 

// Creating a LatLngBounds object 
var bounds = new google.maps.LatLngBounds(); 

// Creating an array that will contain the addresses 
var places = []; 

// Creating a variable that will hold the InfoWindow object 
var infowindow; 
mc = new MarkerClusterer(map);
<?php
$pages = get_pages(array('child_of' => $post->ID, 'sort_column' => 'menu_order'));
$popup_content = array();
foreach($pages as $post)
    {
    setup_postdata($post);
    $fields = get_fields(); 
    $popup_content[] = '<p>'.$fields->company_name.'</p><img src="'.$fields->company_logo.'" /><br /><br /><a href="'.get_page_link($post->ID).'">View profile</a>';
    $comma = ", ";
    $full_address = "{$fields->address_line_1}{$comma}{$fields->address_line_2}{$comma}{$fields->address_line_3}{$comma}{$fields->post_code}";
    $address[] = $full_address;
    }
wp_reset_query();
echo 'var popup_content = ' . json_encode($popup_content) . ';';
echo 'var address = ' . json_encode($address) . ';';
?>

var geocoder = new google.maps.Geocoder(); 

var markers = [];

// Adding a LatLng object for each city  
for (var i = 0; i < address.length; i++) { 
    (function(i) { 
        geocoder.geocode( {'address': address[i]}, function(results, status) {
            if (status == google.maps.GeocoderStatus.OK) {
                places[i] = results[0].geometry.location;

                // Adding the markers 
                var marker = new google.maps.Marker({position: places[i], map: map});
                markers.push(marker);
                mc.addMarker(marker);

                // Creating the event listener. It now has access to the values of i and marker as they were during its creation
                google.maps.event.addListener(marker, 'click', function() {
                    // Check to see if we already have an InfoWindow
                    if (!infowindow) {
                        infowindow = new google.maps.InfoWindow();
                    }

                    // Setting the content of the InfoWindow
                    infowindow.setContent(popup_content[i]);

                    // Tying the InfoWindow to the marker 
                    infowindow.open(map, marker);
                });

                // Extending the bounds object with each LatLng 
                bounds.extend(places[i]); 

                // Adjusting the map to new bounding box 
                map.fitBounds(bounds) 
            } else { 
            alert("Geocode was not successful for the following reason: " + status); 
            }

        });

    })(i);

} 
var markerCluster = new MarkerClusterer(map, markers); 
} 
})
(); 
</script> 

UPDATE: It doesn't really matter what the solution is as long as the markers load instantly and it's not breaking any terms & conditions. This is all new to me in the first place so why not learn some more new things!!

A

Like everybody else, I could give you an answer with code, but I don't think somebody has explained to you that you are doing something that is fundamentally wrong.

Why are you hitting this error? Because you are calling geocode every time somebody views your page and you are not caching your results anywhere in the db!

The reason that limit exists is to prevent abuse from Google's resources (whether it is willingly or unwillingly) - which is exactly what you are doing :)

Although google's geocode is fast, if everybody used it like this, it would take their servers down. The reason why Google Fusion Tables exist is to do a lot of the heavy server side lifting for you. The geocoding and tile caching is done on their servers. If you do not want to use that, then you should cache them on your server.

If still, 2500 request a day is too little, then you have to look at Google Maps Premier (paid) license that gives you 100,000 geocoding requests per day for something around 10k a year (that is a lot - with server side caching you should not be reaching this limit unless you are some huge site or are doing heavy data processing). Without server side caching and using your current approach, you would only be able to do 800 pageviews a day!

Once you realize that other providers charge per geocode, you'll understand that you should cache the results in the db. With this approach it would cost you about 10 US cents per page view!

Your question is, can you work around the throttle limit that Google gives you? Sure. Just make a request from different ip addresses. Heck, you could proxy the calls through amazon elastic ips and would always have a new fresh 2500 allotted calls. But of course, besides being illegal (you are effectively circumventing the restriction given to you by the Google Maps terms of service), you would be doing a hack to cover the inherent design flaw you have in your system.

So what is the right way for that use-case? Before you call the google geocode api, send it to your server and query if it is in your cache. If it is not, call the geocode, store it in your cache and return the result.

There are other approaches, but this should get you started in the right direction.

Update: From your comments below, it said you are using PHP, so here is a code sample on how to do it correctly (recommendation from the Google team itself) https://developers.google.com/maps/articles/phpsqlsearch_v3

A

I think Sasa is right, on both counts.

In terms of sending all your addresses at once, one option is to send the requests at intervals. In the past when using JavaScript I have opted to delay requests by 0.25 (seems to work!) seconds using the

setTimeout( [FUNCTION CALL] , 2500 )

method.

In .NET i have opted for:

System.Threading.Thread.Sleep(250);

Seems to work.

EDIT: Cant really test it, but this should/might work!

Javascript example. The addressArray holds strings that are addresses...

for (var i = 0; i < addressArray.length; i++0 
{

setTimeout('googleGeocodingFunction(' + addressArray[i] + ')' , 2500);

}

EDIT:

for (var i = 0; i < address.length; i++) {
    function(i) {
        setTimeout(geocoder.geocode({ 'address': address[i] }, function(results, status) {
            if (status == google.maps.GeocoderStatus.OK) {
                places[i] = results[0].geometry.location;

                var marker = new google.maps.Marker({ position: places[i], map: map });
                markers.push(marker);
                mc.addMarker(marker);
                google.maps.event.addListener(marker, 'click', function() {
                    if (!infowindow) {
                        infowindow = new google.maps.InfoWindow();
                    }

                    // Setting the content of the InfoWindow
                    infowindow.setContent(popup_content[i]);

                    // Tying the InfoWindow to the marker 
                    infowindow.open(map, marker);
                });

                // Extending the bounds object with each LatLng 
                bounds.extend(places[i]);

                // Adjusting the map to new bounding box 
                map.fitBounds(bounds)
            } else {
                alert("Geocode was not successful for the following reason: " + status);
            }
        }), 2500);// End of setTimeOut Function - 2500 being a quarter of a second. 
    } 
}
A

It sounds like you are hitting the simultaneous request limit imposed by Google (though I cannot find a reference to what the limit actually is). You will need to space your requests out so that you do not send 125 requests all at once. Note that there is also a 2500 geocode request per day limit.

Consult the Google Geocoding Strategies document for more information.

Update: As an added solution inspired thanks to a post by Mapperz, you could think about creating a new Google Fusion Table, storing your address and any related data in the table, and geocoding the table through their web interface. There is a limit to the number of geocode requests a Fusion Table will make, however the limit is quite large. Once you reach the limit, you will need to re-submit the geocode request, and Fusion Tables will pick up where it left off. Bonus to this approach is a huge speed improvement, as you will only need to geocode ONCE, where with your current solution you will need to geocode on every load, quickly reaching daily limits.

A

Here is how I throttle my geocode requests in javascript, addresses being an array containing each address to geocode:

for (i=0;i<=addresses.length;i++) {
    setTimeout( function () {
            geocoder.geocode( { 'address': addresses[i]}, function(results, status) {

                if (status == google.maps.GeocoderStatus.OK) { 

                    //create and add marker to map based off geocode result
                    var marker = new google.maps.Marker({  
                        map: map,
                        title: results[0].formatted_address,
                        position: results[0].geometry.location
                    });

                 } //EDIT, was missing the closing bracket here
            });

    }, i * 1500);
}

This essentially adds a second and half between each geocoding request.

A

I have tried the page now and it seems to be working right now.

While I think the the timer approach you are currently using is correct and as far as I know the only one that will work w/o requiring some server side support I should suggest the following changes (I took the liberty of rewriting your geocoding functions) to take advantage of localStorage.

LocalStorage is a relatively new technology built into most browsers that allows the web page to store data on the client. It somewhat similar to cookies, but it is much more powerful and it is not always resent to the server (unlike cookies).

My aim is to save geocoded address on the client so that if a user refreshes the page there is no need to invoke Google's geocoder functions again on them. The implementation below is crude (we should add code to check for stale records and probably need a better caching key than the index) but should be enough to get you started. Hope it helps.

(function() { 

window.onload = function() { 
 var mc;
// Creating an object literal containing the properties we want to pass to the map 
var options = { 
zoom: 0, maxZoom: 0, 
center: new google.maps.LatLng(52.40, -3.61), 
mapTypeId: google.maps.MapTypeId.ROADMAP 
}; 

// Creating the map 
var map = new google.maps.Map(document.getElementById('map'), options); 

// Creating a LatLngBounds object 
var bounds = new google.maps.LatLngBounds(); 

// Creating an array that will contain the addresses 
var places = []; 

// Creating a variable that will hold the InfoWindow object 
var infowindow; 
mc = new MarkerClusterer(map);
var popup_content = [..redacted..];

var geocoder = new google.maps.Geocoder(); 

var markers = [];

// Adding a LatLng object for each city 
function geocodeAddress(i) {
     // check if localStorage is available (it is now available on most modern browsers)
     // http://html5tutorial.net/tutorials/working-with-html5-localstorage.html
     // NB: this *must* be improved to handle quota limits, age/freshness, etc
     if(localStorage && localStorage['address_'+i]) {
        places[i]=JSON.parse(localStorage['address_'+i]);
        addPlace(i);
     } else {
      geocoder.geocode( {'address': address[i]}, function(results, status) {
        if (status == google.maps.GeocoderStatus.OK) {
            places[i] = results[0].geometry.location;
            if(localStorage) {
               // cache result locally on the browser, this will help reducing the number of requests
               // to the google geocoder in case the user refreshes the page
               // remember: the more he will refresh, the more he's likely to hit the limit
               // (this is one case where refreshing or closing the browser does not work)
               localStorage['address_'+i]=JSON.stringify(results[0].geometry.location);
            }

            addPlace(i);
        } else { 
            console.log("Geocoding of address "+address[i]+" failed");
        }
    })
}

function addPlace(i) {
    // Adding the markers 
    var marker = new google.maps.Marker({position: places[i], map: map});
    markers.push(marker);
    mc.addMarker(marker);

    // Creating the event listener. It now has access to the values of i and marker as they were during its creation
    google.maps.event.addListener(marker, 'click', function() {
        // Check to see if we already have an InfoWindow
        if (!infowindow) {
            infowindow = new google.maps.InfoWindow();
        }

        // Setting the content of the InfoWindow
        infowindow.setContent(popup_content[i]);

        // Tying the InfoWindow to the marker 
        infowindow.open(map, marker);
    });

    // Extending the bounds object with each LatLng 
    bounds.extend(places[i]); 

    // Adjusting the map to new bounding box 
    map.fitBounds(bounds);
}

function geocode() {
    if (geoIndex < address.length) {
        geocodeAddress(geoIndex);
        ++geoIndex;
    }
    else {
        clearInterval(geoTimer);
    }
}
var geoIndex = 0;
var geoTimer = setInterval(geocode, 600);  // 200 milliseconds (to try out)

var markerCluster = new MarkerClusterer(map, markers); 
} 
})
(); 
A

using geopy geocoders I'm able to geocode a large number of addresses without a hitch (i'm not sure of the posters setup, or if he can include a python dependency to his project)

here is my test:

from geopy import geocoders

g = geocoders.Google()

for x in xrange(1,10000):
    print x
    a = g.geocode("AV JOAO NAVES DE AVILA " + str(x) + " UBERLANDIA MG BRASIL")
    print a

result snippet

7595 (u'Av. Jo\xe3o Naves de \xc1vila, 7595 - Segismundo Pereira, Uberl\xe2ndia - Minas Gerais, 38408-288, Brazil', (-18.9388154, -48.2220562))

7596 (u'Av. Jo\xe3o Naves de \xc1vila, 7596 - Segismundo Pereira, Uberl\xe2ndia - Minas Gerais, 38408-680, Brazil', (-18.938814, -48.2217423))

7597 (u'Av. Jo\xe3o Naves de \xc1vila, 7597 - Segismundo Pereira, Uberl\xe2ndia - Minas Gerais, 38408-288, Brazil', (-18.9388128, -48.2220381))

7598 (u'Av. Jo\xe3o Naves de \xc1vila, 7598 - Segismundo Pereira, Uberl\xe2ndia - Minas Gerais, 38408-680, Brazil', (-18.9388114, -48.2217242))

7599 (u'Av. Jo\xe3o Naves de \xc1vila, 7599 - Segismundo Pereira, Uberl\xe2ndia - Minas Gerais, 38408-288, Brazil', (-18.9388102, -48.2220201))

7600 (u'Av. Jo\xe3o Naves de \xc1vila, 7600 - Segismundo Pereira, Uberl\xe2ndia - Minas Gerais, 38408-680, Brazil', (-18.9388088, -48.2217062))

This is still running and I'm with 7k results.

EDIT: After a while i've hitted the end of the street and google started to give me same points, but looks like it's still making requests and giving me accurate answers.

A

This php function (i dont know which language youre working in, but you can pick it apart im sure) uses google geolocator api. Given an address, it will return the normalised address and latlong. HTH.

// GEOLOCATEBYADDRESS
// @arg (string)address
// @return (array) ((int)flag,(array)address,array(rawresponse))

function geolocatebyaddress($lookupaddress){

    $lookupaddress=trim("$lookupaddress New Zealand");
    $lookupaddress=urlencode($lookupaddress);

    //send off google api lookup request
    $apiurl="http://maps.google.com/maps/api/geocode/json";
    $apiurl.="?address=$lookupaddress";
    $apiurl.="&region=NZ";
    $apiurl.="&sensor=false";

    if (function_exists("curl_init")) {
        $ch = curl_init();
        curl_setopt($ch, CURLOPT_URL, $apiurl);
        curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
        $json = curl_exec($ch);
        curl_close($ch);
    }
    else     $json= file_get_contents($apiurl);

    //process response
    $response= json_decode($json,true);
    $approxflag=0;
    if ($response['status']=='OK' and count($response['results'])) {

        $aa= array();
        foreach($response['results'] as $cc=>$result) {
            //set keys
            foreach(array('street_number','road','suburb','city','postcode','region','lat','long') as $t){
                $aa[$cc][$t]='';
            }
            if ($result['geometry']){
                $aa[$cc]['lat']= $result['geometry']['location']['lat'];
                $aa[$cc]['long']=$result['geometry']['location']['lng'];
         }

            if (count($result['address_components'])){
                foreach ($result['address_components'] as $acs){
                    if ($acs['types'][0]=='street_number')               $aa[$cc]['street_number']= $acs['long_name']; 
                    if ($acs['types'][0]=='route')                       $aa[$cc]['road']=      $acs['long_name']; 
                    if ($acs['types'][0]=='sublocality')                 $aa[$cc]['suburb']=   $acs['long_name']; 
                    if ($acs['types'][0]=='locality')                    $aa[$cc]['city']=    $acs['long_name']; 
                    if ($acs['types'][0]=='postal_code')                 $aa[$cc]['postcode']= $acs['long_name']; 
                    if ($acs['types'][0]=='administrative_area_level_1') $aa[$cc]['region']=   $acs['long_name']; 
                }    
            }
            //successful?
            if ($result['geometry']['location_type']=='APPROXIMATE') $approxflag++;
            if (isset($result['partial_match']))  $approxflag++;
            if (!$aa[$cc]['street_number'])         $approxflag++;
        }
        if ($approxflag) return (array(1,$aa,$response));
        else return (array(2,$aa,$response));
    }
    else return (array(0,array(),$response));
}