Thanks for sharing this helpful code. I use it daily to check about 10000 urls. On pretty new hardware, this script would take about 10 minutes to execute. I made some ugly hacks to copy the url_hashes_table to an in-memory sqlite DB before querying, and running time is down to about 15 seconds. I'd be happy to share the changes I've made - let me know.
Thanks for sharing this helpful code. I use it daily to check about 10000 urls. On pretty new hardware, this script would take about 10 minutes to execute. I made some ugly hacks to copy the url_hashes_table to an in-memory sqlite DB before querying, and running time is down to about 15 seconds. I'd be happy to share the changes I've made - let me know.
@yolabingo: Yes, I would love to hear from you on the changes. I actually use postgres and it returns results within 3 seconds.
There is already a patch that uses a key-value store(memcached), which gives blazing fast results.
I would like to actually love to benchmark both and decide for myself.
Here's a tarball of the code I'm using. README file has some notes. Hope it's useful for you.
http://www.lincolnlattonsoftware.com/safebrowsing.tar.gz