Concerns about AI scraper problem evidence and blocking measures
Dispute over evidence for AI scraping as cause of Sourceware load issues and the resulting access blocks.
Xe Iaso, creator of the Anubis project, defends the characterization of an AI scraper problem impacting Sourceware, citing multiple organizations reaching similar conclusions. This follows concerns raised by Kalamatee regarding the sufficiency of public evidence linking traffic to AI scraping and resource exhaustion. The discussion touches on the nature of the traffic, whether it’s illegitimate or merely automated, and the potential for expensive operations to be abused by unauthenticated requests.
- proposer
Asserts that the AI scraper problem is real and corroborated by multiple organizations, defending the necessity of blocking measures.
“As a rule of thumb: if multiple organizations are all coming to the same conclusions without colluding, it's probably because a thing is actually happening.”
- other
Questions the evidence presented for the AI scraper problem, arguing that traffic is not definitively proven to be illegitimate or the sole cause of resource exhaustion.
“I don't doubt that Sourceware has seen unusual or excessive traffic, and I don't think it is unreasonable to investigate whether automated scraping is contributing to load. What I am less convinced about is the extent to which the public evidence actually supports the conclusions being drawn from that traffic.”
- other
Agrees that the underlying issue is expensive operations being triggered by unauthenticated requests, suggesting that securing these operations is more important than identifying the requester as AI.
“If the expensive operations are made difficult to abuse, it matters much less whether the requester is an AI scraper, CI, curl or a human.”
- other
States that `git blame` has been disabled in cgit and gitweb, implying a mitigation measure has already been taken.
“git blame has been disabled in cgit (and gitweb).”
Technical Tradeoffs
- Blocking potentially malicious AI scrapers versus inconveniencing legitimate users.
- Resource exhaustion due to inefficient operations versus proactive security measures.
In Details
The discussion revolves around unusual traffic patterns observed on Sourceware, suspected to be from AI scrapers. The core of the debate is whether the evidence for this being malicious AI scraping is sufficient to justify blocking measures, which are causing access issues for legitimate users. The impact on git blame queries and other resource-intensive operations is a key concern.
- AI scraper
- Automated software that browses websites to collect data, often for training artificial intelligence models. In this context, it refers to traffic identified as potentially malicious or excessive.
- Sourceware
- A web-based infrastructure for open-source projects, providing services like Git repositories, bug tracking, and mailing lists.
- RFC 9309
- A Request for Comments document that specifies internet standards. The specific content of RFC 9309 is not detailed here but is mentioned as being disrespected.
- git blame
- A Git command that shows what revision and author last modified each line of a file, often used for understanding code history.
- nginx
- A web server that can also act as a reverse proxy, load balancer, and HTTP cache. In this context, it is serving the GCC Git web interface and returning 429 errors.
- 429 Too Many Requests
- An HTTP status code indicating that the user has sent too many requests in a given amount of time ('rate limiting').