Collecting Public X/Twitter Data
What remains publicly accessible on X, the API tiers versus collection, and why logged-in scraping carries legal uncertainty that public data does not.
- social-media
- web-scraping
X is one of the more complicated collection targets, not because the technical problem is hard but because the access model changed substantially. The platform now gates most programmatic access behind a paid API, restricts anonymous viewing, and has litigated against scrapers.
Any project here should start with the access model rather than the code.
Start with the access model
X offers an official API with tiered pricing. Tiers differ in request volume, endpoint access and permitted use cases. For many commercial needs, this is the intended and supported route.
The alternative, scraping the web interface, runs against the platform's terms and has been the subject of litigation. That does not make every collection unlawful, but it does mean the risk profile is different from scraping a site that merely dislikes it.
Read the developer terms and the platform's automation rules before building. They are explicit about what is permitted.
What is publicly visible
Without logging in, the platform surfaces a limited view: some individual posts, some profile information, and a restricted search experience. Much of what used to be anonymously accessible is now behind authentication.
That matters because it draws the legal line. Publicly accessible data has been treated differently from data behind a login, as we explain in Is Web Scraping Legal. Collecting what anyone can see without an account is a materially different activity from automating an authenticated session.
In the United States, hiQ Labs v. LinkedIn (938 F.3d 985, affirmed on remand in April 2022) held that scraping publicly accessible data does not violate the Computer Fraud and Abuse Act, while the Supreme Court in Van Buren v. United States, 593 U.S. 374 (2021) narrowed "exceeds authorized access". Notably, hiQ still lost on breach of contract and trespass claims, which is the practical warning: a contract claim under the terms of service is the exposure that survives.
Why logged-in scraping is a different risk
Automating an authenticated account raises issues that anonymous collection does not:
Terms of service. The contract you accept when creating an account governs your use of it. Automated access is usually restricted explicitly.
Access controls. Data behind a login is not public, and circumventing authentication is treated more seriously than reading an open page.
Account consequences. The platform can and does suspend accounts used for automated collection, which is a practical risk before any legal one.
Personal data. Logged-in content frequently includes personal data, which brings data protection law into scope regardless of the access route.
If your requirement can only be met by automating an authenticated session, that is a decision to take deliberately with legal input, not a technical detail to solve with infrastructure.
What is technically involved
Should you proceed with publicly accessible collection:
Rendering. The web client is a client-side application, so a plain HTTP client gets little. A browser engine is usually required, with the trade-offs in Puppeteer Web Scraping.
Client fingerprinting. The platform checks client characteristics, so a realistic browser matters more than usual. The general mechanics are in How Anti-Bot Systems Detect Scrapers.
IP reputation. Datacenter ranges are challenged quickly. Residential or mobile addresses are the practical requirement, as described in Choosing a Proxy for Web Scraping.
Rate limiting. Aggressive request patterns produce empty responses or challenges rather than clear errors, so validation is essential. The detection approach is in Parsing HTML and JSON Reliably.
Session consistency. Hold one identity per unit of work rather than rotating per request, per Rotating vs Sticky Proxies.
A more sensible framing
Rather than asking how to scrape X, ask which of these you actually need:
- Monitoring your own brand's mentions. The API is the supported route, and where it does not cover your need, a manual or semi-manual process may be adequate.
- Sentiment or trend analysis. Aggregated and licensed datasets exist commercially, which shifts the cost from engineering to procurement, often a good trade.
- Academic research. Research access programmes exist for some platforms and are worth investigating before building anything.
- Competitive analysis. Check whether the API tier you need is affordable before assuming collection is cheaper. The full comparison is in Web Scraping vs APIs.
A large share of projects that start as scraping projects end as API or licensed-data projects once the maintenance cost is priced in, and rightly so.
If you do collect
Keep it narrow and defensible:
- Public data only. No authenticated sessions.
- Low volume, paced like a human, with explicit handling of empty responses.
- No personal data beyond what is necessary, and a documented purpose. The framework is in Data Collection Ethics for Engineering Teams.
- Residential or mobile IPs, one session per logical unit.
- Validation on every record, because a challenge page looks like a successful response with no content.
Confirm endpoints with the proxy checker, and coverage is on the residential and mobile pages.