Java Web Scraping: Complete Guide

Java is a less common starting point for scraping than Python, but plenty of teams already run JVM services and would rather not add a second runtime. The ecosystem is capable: a mature HTTP client in the standard library, Jsoup for parsing, and Selenium for browser work.

This is the path through all three, with proxy configuration at each layer.

Fetching with HttpClient

Since Java 11, java.net.http.HttpClient is in the standard library, so you need no dependency to make requests.

import java.net.URI;
import java.net.http.*;
import java.net.ProxySelector;
import java.time.Duration;

HttpClient client = HttpClient.newBuilder()
    .connectTimeout(Duration.ofSeconds(10))
    .proxy(ProxySelector.of(new InetSocketAddress("proxy.lightningbytes.com", 1080)))
    .build();

HttpRequest request = HttpRequest.newBuilder()
    .uri(URI.create("https://example.com/products"))
    .timeout(Duration.ofSeconds(30))
    .header("User-Agent", "Mozilla/5.0 (compatible; research-bot)")
    .GET()
    .build();

HttpResponse<String> response = client.send(request, HttpResponse.BodyHandlers.ofString());

Two notes. ProxySelector does not carry credentials, so authenticated proxies need an Authenticator:

HttpClient client = HttpClient.newBuilder()
    .authenticator(new Authenticator() {
        @Override
        protected PasswordAuthentication getPasswordAuthentication() {
            return new PasswordAuthentication("lb-USERNAME", "SECRET".toCharArray());
        }
    })
    .proxy(ProxySelector.of(new InetSocketAddress("proxy.lightningbytes.com", 1080)))
    .build();

And prefer HttpClient as a long-lived object. Creating one per request wastes connection pooling, which is the same lesson as reusing a session in Using Python requests with Proxies.

Parsing with Jsoup

Jsoup is the Java equivalent of BeautifulSoup: tolerant of malformed HTML, with CSS selectors.

import org.jsoup.Jsoup;
import org.jsoup.nodes.Document;
import org.jsoup.nodes.Element;

Document doc = Jsoup.parse(response.body());
for (Element card : doc.select("[data-testid=product-card]")) {
    String name = card.selectFirst("[data-testid=product-title]") != null
        ? card.selectFirst("[data-testid=product-title]").text()
        : null;
    String price = card.selectFirst("[data-price]") != null
        ? card.selectFirst("[data-price]").attr("data-price")
        : null;
}

Select on stable attributes such as data-testid rather than generated class names, and null-check every element. The selector discipline is the same in every language, and it is what we cover in Beautiful Soup: Parsing HTML Without the Headache.

Jsoup can also fetch directly, with proxy support:

Document doc = Jsoup.connect("https://example.com")
    .proxy("proxy.lightningbytes.com", 1080)
    .userAgent("Mozilla/5.0 (compatible; research-bot)")
    .timeout(20000)
    .get();

That is concise, but it gives you less control over connection reuse and retries than HttpClient plus Jsoup.

Browser automation with Selenium

For JavaScript-rendered pages, Selenium in Java works much as it does elsewhere.

ChromeOptions options = new ChromeOptions();
options.addArguments("--proxy-server=http://proxy.lightningbytes.com:1080");
options.addArguments("--headless=new");

WebDriver driver = new ChromeDriver(options);
driver.get("https://api.ipify.org");
System.out.println(driver.findElement(By.tagName("body")).getText());
driver.quit();

The authenticated-proxy limitation is identical to the Python case, and the workarounds are the same. See Selenium Proxy Setup and Rotation.

Waiting, the usual flakiness source

Java Selenium has explicit waits, and you should use them rather than Thread.sleep.

WebDriverWait wait = new WebDriverWait(driver, Duration.ofSeconds(20));
wait.until(ExpectedConditions.presenceOfElementLocated(
    By.cssSelector("[data-testid=product-card]")));

Intermittent failures in browser scraping are almost always waiting problems, not proxy problems. Misdiagnosing that sends you tuning the wrong layer.

Concurrency

Java's virtual threads, available since Java 21, make high-concurrency request fan-out straightforward. Two cautions apply regardless of language: keep concurrency per endpoint at one, and cap the total so you do not trip a target's rate limits. The reasoning is in Rate Limiting vs Blocking and the async equivalent in Async Python Scraping Without Breaking Rate Limits.

Validation

Parse leniently, validate strictly. Use records or a validation library to reject records with missing required fields rather than letting them flow downstream. The principle is in Parsing HTML and JSON Reliably.

Operational habits that matter more than the language

• Log the exit IP per request, so failures are attributable.
• Set timeouts at both connect and read level.
• Retry with backoff, and honour Retry-After.
• Verify the proxy is in use rather than assuming. The proxy checker reports exit IP and added latency.

None of that is Java-specific, which is the point: the choice of language changes the code, not the discipline. For the surrounding context, What Is Web Scraping and Data Mining vs Web Scraping cover where this fits.