The 5 Best Free Proxy Lists for Web Scraping
Free proxies are like free advice, both can go south. In this post, we will talk about free proxy list providers and how they can be used. People use free web proxy for anonymous web browsing, web scraping, etc.
Now, what is a proxy server anyway? A proxy server is like an intermediary between yours and a website, allowing you to anonymously browse the Web. When a user makes a request for a certain web page, the proxy responds by pulling that page. The website will see that proxy instead of your internal address. So, your identity will remain safe.
But, is it even secure to use a free proxy list service? Christian Haschek a security researcher wrote a script to test 443 web proxies and the results he got were pretty obvious. His experiment shows that only 21 percent of the tested proxies were not ominous. The rest of the 79 percent of surveyed web proxies forced users to load web pages in the unencrypted (HTTP) form.
The major disadvantage of using a free proxy list is that you don’t know who is operating the address. They can be secret agencies, data-stealing companies, hackers, etc. Hiding under an umbrella from the website you are visiting does not mean that you are completely safe, the proxy server can track your activities.
Are there any better options? We have created a list of the 5 best free proxy list providers that you can use for your daily routine tasks like browsing, scraping, verification, etc. Remember the only advantage they provide you is that they are free until they are not blocked. We will share those 21% proxies and the rest. Choose wisely!
In this post, we will test all these proxies on websites like google, amazon, eBay, and Yellowpages. We will create a web scraper to make 500 requests to each website and then we will judge them on the basis of errors, captcha, success, and response time. We have created a web scraper using python, you can choose any language you like.
This is the list of proxy providers we are going to talk about today:
- Scrapingdog
- Proxyscrape
- Free Proxy lists
- Proxy Nova
- SSL Proxy
Scrapingdog
Scrapingdog is a web scraping API. Using the API you can scrape any web page. You will get raw HTML in response with all the essential data from your target website. You are just a GET request away from your data.
They provide a free trial with 1000 API calls. You can test all the features in the free trial. Scrapingdog offers multiple features like:
- Headless Chrome and Javascript rendering
- API to generate HTML of any web page.
- Linkedin Scraping API
Other than Web Scraping API, they also provide rotating proxies. Using these proxies you can verify ads, scrape websites, browse the internet safely, etc. Our rotating proxies are a mixed batch of residential and datacenter proxies. If you want to use just residential proxies then pass country=random as param after API key. You can even target any specific country while using a residential proxy.
The best part of using Scrapingdog is that if you get a response other than 200 then the request will not be deducted from your account. You can even customize the request headers by using custom_headers=true as an extra parameter. In the free plan, you can make a maximum of 5 concurrent requests and we provide 24*7 support. We don’t differentiate between the free and the paid user. A free user can test all of our proxy networks before upgrading to a premium account.
Scrapingdog has also posted some great tutorials on how to build web scrapers using Nodejs, Python, Scrapy, Java, and even Ruby. Even if you are a beginner in the web scraping world you can read these articles to get some idea on how to build your own web scraper. If you want to build your own web scraper then you can use our rotating proxies to rotate IPs to remain unblocked. Would also recommend reading 10 tips to avoid getting blocked while scraping.
Testing
500 requests were sent for each website.
Proxyscrape
Proxyscrape provides you with a standard list of proxies in a .txt file. You can either filter proxies according to countries or you can opt for a mixed batch of proxies. Not just that you can even filter proxies according to their anonymity levels.
There are three types of anonymity levels:
- Transparent proxy: does not hide your IP Address.
- Anonymous proxy: hides your IP address but does reveal that you are using a proxy server.
- Elite proxy: hides both your IP address and the proxy server.
Plus they also offer a choice between proxies that support SSL and the proxies which do not. It’s a great package.
Another feature they offer is a timeout slider. The timeout slider helps you to decide the threshold time limit of connecting to any website. After certain milliseconds, the connection between your proxy and the target website will break if the proxy is taking too much time to connect. So, it’s a feature-packed proxy.
They offer HTTP, Socks4 and Socks5 proxy list which keeps on updating after every 24 hours. They have a large batch of Socks4 proxies as compare to the other two. Also, the filters we have mentioned earlier are only available for HTTP proxies except for selecting a country.
They have shut down their proxy checker tool due to abuse, but it was a great tool to check the quality of any proxy. Proxyscrape does not any free trial for their premium service which is kind of a negative point. You have to pay to test their services. For commercial usage, you have to upgrade to their premium packs without even knowing whether the proxies will satisfy your purpose or not.
But in the end, these are free web proxy. People who host a proxy can modify the content you see when visiting a website. These modifications can be malicious. You have to be very careful when using any kind of free proxy list.
Testing
500 requests were sent for each website.
Free Proxy List
Just like any other free web proxy provider free proxy list also provides various filters like country selection, port number, anonymity, and protocol. But the problem is you cannot download the proxies. You have to refer to the table for proxies and for more proxies, you have to click the “next” button to get more proxies.
In the case of Proxyscrape you get a timeout slider but in this case, there are two colored boxes on the very right of the table. Using this you can identify whether the proxy can be used for scraping or not. I know this is a bit inconvenient method. You can measure uptime with a percentage.
But the great part is they keep updating the proxy list on a regular basis. For support, they have provided an email.
Once again be careful while using these proxies. You can end up leaking your project modules.
Testing
500 requests were sent for each website.
Proxy Nova
Proxy Nova also provides a list of proxies in a table form. They claim they have the largest database of public proxies. These proxies are tested once every 15 minutes. This increases the reliability of the service. They offer a country-level filter along with that you get a filter for anonymity.
Proxy offered can be used for hiding your real IP address or maybe for unblocking some blocked websites in your country. Their proxy list is updated after every 60 seconds but the best thing is their page will not auto-refresh. This helps in using the good proxies without losing them.
Testing
500 requests were sent for each website.
SSL Proxy
SSL Proxies provides a list of proxies in a table form. The table has eight columns in which two columns are for filtering countries and anonymity just like other proxy providers. There is a third column by the name Google which I suppose means proxy originated from a Google source. Let me know if I am wrong.
They claim that their proxies are tested every 10 minutes but according to their table, the claim falls flat. There are proxies that were tested like 50 minutes ago.
They only offer HTTPS proxies. You have to pay for HTTP and Socks5 proxies. Their proxy plan offers rotating proxies that rotate every minute. For mass data collection this proxy could be banned in no time.
Testing
500 requests were sent for each website.
Analyzing the results
We used a small script to test all these free proxy list providers with 500 requests each on four websites. Now we have to aggregate all the results for a final verdict.
Amazon
eBay
Yellow Pages
Verdict
As you can see most of the free web proxy cannot scrape google and amazon except Scrapingdog. The reason is many of them are already used for scraping google or amazon and now they are permanently banned. While selecting a proxy provider we mainly focus on the validity of the proxy. Getting captchas and errors on every request is really frustrating.
Free web proxy is used by many developers on many different sites. Many websites like search engines, eCommerce websites, social networking websites, etc have already blocked these proxies. Many SEO agencies use these proxies to scrape emails and scrape google search results to generate SEO reports. Search engines use honeypot traps to block proxies. This ends up increasing the error rate. Free web proxy is also blocked by many ISPs, so be careful while subscribing to services like Free proxy lists or SSL proxy.
But you can use Scrapingdog to scrape almost any website. Apart from that, you can use Proxyscrape for web scraping or anonymous browsing. With low-quality proxies, you can end up getting blocked.
Additional Resources
Here are a few additional resources that you may find helpful during your web scraping journey:
