The Yahoo Search API has been driving me nuts. I started to get 999 errors on all the calls I was making from a new app I was developing (“999 Rate Limit Exceeded” was returned in the response). What was really driving me nuts was that I was only making a handful of calls per day – I knew I wasn’t exceeding the 5000 rate limit advertised by Yahoo for the Search API.
FYI – skip to the end for the fix…
Since the rate limit is applied per IP address I convinced myself that my problem was due to somebody else’s app running on the same server. Not ideal but it would be a downside of shared hosting. So I immediately signed up for a unique IP. The next day was spent figuring out why this didn’t make any difference. Turns out a unique IP on shared hosting isn’t actually a complete IP – it’s only half an IP. Unique/dedicated IPs are generally provided on shared hosting for SSL. This only requires incoming requests for your site to be directed to a unique IP address. However outgoing requests (eg from a PHP script) do *not* originate from the dedicated IP but from the IP address of the shared Apache instance. Ugh. Understandable in hindsight but it totally didn’t meet my expectations. The upgrade to a private server or virtual private server would get me a “whole” dedicated IP… maybe when this app starts making cash I’ll spring for it.
Anyway, after figuring out the above, spending an hour setting up routes through my various firewalls, installing a proxy (CCProxy worked perfectly) I was able to pass the request through my laptop and hence originate it from a different IP address which I knew for sure wouldn’t be rate limited.
Guess what. Still getting 999 errors. However copying the URL to my browser returns the entire set of results immediately. Same IP same request, no? Ugh! Something crucial had to be different between the request coming from my browser and my script. After experimenting with every cURL option in the known universe I finally figured out the undocumented API requirement.
The Yahoo Search API now requires the HTTP User Agent to be set. Eg:
curl_setopt($session,CURLOPT_USERAGENT,"Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)");
This appears to be a new undocumented requirement but it does match the User Agent requirement listed against the shopping API. I tried various User Agent strings and just about anything seems to be accepted but Yahoo suggests faking a commonly used User Agent string for the shopping API so it’s probably best to stick with a browser UA.
Anyway, worked like a treat. The stupid little things always suck up the most time.