Set Jsoup proxy in Java example shows how to set proxy with Jsoup in Java. The example also shows how to set HTTP and HTTPS proxy with username and password.
How to set Jsoup proxy?
If you are behind the internet proxy, you will get “java.net.SocketTimeoutException: connect timed out” exception while trying to connect to any URL using Jsoup like given below. (time out full example)
|
1 2 3 4 5 6 7 8 9 10 11 12 |
java.net.SocketTimeoutException: connect timed out at java.net.DualStackPlainSocketImpl.waitForConnect(Native Method) at java.net.DualStackPlainSocketImpl.socketConnect(Unknown Source) at java.net.AbstractPlainSocketImpl.doConnect(Unknown Source) at java.net.AbstractPlainSocketImpl.connectToAddress(Unknown Source) at java.net.AbstractPlainSocketImpl.connect(Unknown Source) at java.net.PlainSocketImpl.connect(Unknown Source) at java.net.SocksSocketImpl.connect(Unknown Source) at java.net.Socket.connect(Unknown Source) at sun.net.NetworkClient.doConnect(Unknown Source) at sun.net.www.http.HttpClient.openServer(Unknown Source) … |
You need to set the proxy before you can access any URLs using Jsoup. Under the hood, Jsoup uses Java’s built-in Connection and HTTPURLConnection classes. That means we can set the proxy in the same standard Java way using system properties.
Setting HTTP proxy
Use setProperty method of System class to set proxy host and proxy port properties before you connect to a URL using Jsoup as given below.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 |
package com.javacodeexamples.libraries.jsoup; import java.io.IOException; import org.jsoup.Jsoup; public class JsoupSetProxyExample { public static void main(String[] args) { try{ //set HTTP proxy host to 127.0.0.1 (localhost) System.setProperty("http.proxyHost", "127.0.0.1"); //set HTTP proxy port to 3128 System.setProperty("http.proxyPort", "3128"); String strText = Jsoup .connect("http://www.example.com") .userAgent("Mozilla/5.0") .timeout(10 * 1000) .get() .text(); System.out.println(strText); }catch(IOException ioe){ System.out.println("Exception: " + ioe); } } } |
The above example sets “127.0.0.1” as a proxy host and “3128” as a proxy port. Change these values according to your proxy server details.
Setting HTTPS proxy
If you are trying to parse HTTPS pages using the Jsoup, you also need to set the HTTPS proxy. It can be done by setting below given properties.
|
1 2 |
System.setProperty("https.proxyHost", "127.0.0.1"); System.setProperty("https.proxyPort", "3128"); |
Setting proxy authentication
Does your proxy server need authentication (username and password) to access the internet? If yes, you also need to set a proxy username and password using the below-given properties.
For HTTP,
|
1 2 |
System.setProperty("http.proxyUser", "username"); System.setProperty("http.proxyPassword", "password"); |
For HTTPS,
|
1 2 |
System.setProperty("https.proxyUser", "username"); System.setProperty("https.proxyPassword", "password"); |
UPDATE (Recommended way):
If you are using the Jsoup 1.9.1 version or later, you should use the proxy method of the Connection class.
|
1 |
Connection proxy(String proxyHost, int proxyPort) |
It sets the connection proxy to provided host and port values.
|
1 2 3 4 5 6 7 8 9 |
String strText = Jsoup.connect("http://www.example.com") .userAgent("Mozilla/5.0") .timeout(10 * 1000) .proxy("127.0.0.1", 3128) .get() .text(); System.out.println(strText); |
The Jsoup connection class also now provides an overloaded proxy method that accepts the Proxy object.
|
1 |
Connection proxy(@Nullable Proxy proxy) |
If you pass null as a proxy object, it will remove the previously set proxy.
|
1 2 3 4 5 6 7 8 9 10 11 |
Proxy proxy = new Proxy(Proxy.Type.HTTP, new InetSocketAddress("127.0.0.1", 3128)); String strText = Jsoup.connect("http://www.example.com") .userAgent("Mozilla/5.0") .timeout(10 * 1000) .proxy(proxy) .get() .text(); System.out.println(strText); |
Additionally, it is always a best practice to set user agent and referrer headers in the Jsoup connection.
This example is a part of the Jsoup tutorial with examples.
Please let me know your views in the comments section below.
About the author
- Gson – Serialize HashMap null entries - April 5, 2023
- Gson: Convert String to JsonObject without POJO - April 5, 2023
- Gson – Convert JSON to a Typed ArrayList - April 5, 2023
code is as follows:
import java.io.IOException;
import org.jsoup.Jsoup;
public class JsoupSetProxyExample {
public static void main(String[] args) {
try{
//set HTTP proxy host to 127.0.0.1 (localhost)
System.setProperty(“http.proxyHost”, “10.1.1.18”);
//set HTTP proxy port to 3128
System.setProperty(“http.proxyPort”, “80”);
System.setProperty(“http.proxyUser”, “user_id”);
System.setProperty(“http.proxyPassword”, “password”);
String strText =
Jsoup
.connect(“http://htmlunit.sourceforge.net/gettingStarted.html”)
.userAgent(“Mozilla/5.0”)
.timeout(10 * 1000)
.get()
.text();
System.out.println(strText);
}catch(IOException ioe){
System.out.println(“Exception: ” + ioe);
}
}
}
You need to provide proxy user id and password in order for this to work.
System.setProperty(“http.proxyUser”, “your_user_id”);
System.setProperty(“http.proxyPassword”, “your_password”);
yes i provided the my proxy user_id and my_password , i have not posted in this web page.
in following order
System.setProperty(“http.proxyHost”, “10.1.1.18”);
System.setProperty(“http.proxyPort”, “80”);
System.setProperty(“http.proxyUser”, “my_user_id”);
System.setProperty(“http.proxyPassword”, “my_password”);
but the same exception is coming as below:
Exception: org.jsoup.HttpStatusException: HTTP error fetching URL. Status=407, URL=http://htmlunit.sourceforge.net/gettingStarted.html
Hi !
So you set the proxy for the whole connections from the JVM.
Do you have an idea to set the proxy for this Jsoup.connect() only ?