What this really says that they opened a hole which got them into a questionable legal area by removing them, and they regret not just remaining neutral.
It comes down to content moderation. If a company begins to moderate content that travels through its network, it can be held liable and forced to remove things that any entity or governmental body could disagree with.
For example that means if the Republican Party was hosting on Cloudflare and the Democrat Party disagreed with the Republican Party, they could force Cloudflare to remove the Republican Party's websites from transiting their network.
Another example is if the Washington Post and Fox news both went through Cloudflare, and Fox disagreed with the Washington Post they could force Cloudflare to remove the Washington Post.
The moment you set the precedence it becomes an expectation -- by Cloudflare remaining neutral they can't be forced into that position.
> It comes down to content moderation. If a company begins to moderate content that travels through its network, it can be held liable and forced to remove things that any entity or governmental body could disagree with.
Cloudflare already does this, especially to services involving sex workers.
I’m glad Cloudflare is taking a neutral stance to this. I’ve long said if a service provider, in any capacity, starts moderating content without a court order then it’s a slippery slope. I’m not saying the sites are good, but, why does Cloudflare need to be the internet police here? They can’t and shouldn’t be.
Neutrality is key, how would you feel if your content was suddenly pulled because someone on the internet disagreed?
We're a young, energetic company hiring for multiple roles on contract. We help companies build everything from custom-designed software to entire networks to power hundreds of gigabits a second of traffic.
We're hiring for the following positions:
- Jr Systems Administrator (EST timezone preferred)
- Jr Network Engineer (EST timezone preferred)
- Jr Systems Administrator (GMT to GMT+2 timezone preferred)
A brief:
- Our choice of Linux OS on managed systems and machines is Ubuntu (mixture of 20.04 LTS, 21.04 currently), with some various Alma/CentOS thrown in the mix
- We work with various industries (ecommerce, networking/security, IoT, etc) and take security very seriously
- Our clients span multiple countries and timezones, from LA to AU
Here's what we're looking for:
- You know Linux pretty good
- You know Ansible, you've used it (we do it a lot!)
- You have a good understanding of Python (we use Python to automate a lot of system monitoring alongside ansible for automating deployments/updates/tasks)
- You're comfortable with some PHP (Symfony/Laravel/Wordpress)
- You're comfortable with some Go (we're just trying some new things here, so more the better!)
- You understand security in environments (eg, PCI compliance, SOC2, etc)
- You're good at communicating and are able to self-prioritize tasks
- You maintain a good work life balance
If this sounds like fun to you, drop your resume to: team[@]packetrabbitinc[.]com along with a brief cover letter and we'll be in touch!
> There were many times where I saw a big traffic jump and I realized the traffic was coming from the same ASN, and likely from the same company. I tried reaching out to these companies when I saw it but they rarely ever replied. Some even became extremely hostile to my emails.
A hostile reply from a netblock operator seems like a perfectly valid reason to block their traffic.
The problem is that you don't know what the source of the traffic is. It could be an incompetent network operator/sysadmin, but it could just as well be something like an IP camera that people bought in good faith. If you block the CGNAT system of an operator that has a hundred million subscribers because it all seems to come from a single IP range you know nothing about, you could be hurting innocent users with the block.
That being said, a service like this doesn't come with any guarantees and if it'd disappear from the net tomorrow, I wouldn't blame the author. Blocking is a perfectly valid solution to this problem, but assuming malice isn't always the right answer.
Were I in this situation, I'd rate limit networks per /24 (maybe even /16?) as much as I could, and work together with antivirus companies to help identify infections of malware known to use the service to discourage criminals from abusing the system. I wouldn't even bother hosting the site on IPv6 since those addresses are supposed to be public anyway. The author clearly has more patience than I do.
In some sense, it might not matter. If an ASN/company admin responds to emails in a hostile fashion, does it matter if they bought their devices in good faith? They're still assholes.
Hostility can often come from a place of ignorance or misunderstanding. I can't say much for the former, but the latter can easily go wrong with the cultural and linguistic barrier between operators.
The guy operating the NOC may be a dick, but is taking down the IoT networks for all of their customers unknowingly relying on your services really the right way?
Personally, I'd say yes, it'd help. However, there's an argument to be made that the hostile ASN operator doesn't represent the people behind the network in the slightest. I can understand that someone may give such an asshat the benefit of doubt and drop it despite their abuse.
This isn’t “one person bought a bad camera”, it was certain ASNs accounting for a huge portion of the traffic. If the operators are unresponsive to the abuse request (making them incompetent network operators), then you absolutely block them. At that point the fallout is the fault of the network operators for operating an abuse friendly network.
This is how cloudflare handles it for normal web services. If you’re coming from trash IPs there is no chance a curl request is going to make it through to a backend without an onerous captcha.
I wouldn't expect one person with one camera to cause such a load, but popular, cheap internet cameras pull this crap all the time. I remember reading a story here about one company that hardcoded a particular IP address for their NTP bootstrapping in their firmware, with thousands of devices all across the world and no way to easily update them. Such a thing can easily happen with consumer routers and other networking equipment, generating a publicly accessible link for their user's convenience.
If I saw the Time Warner ASN send too many requests, my first thought wouldn't be to just block a huge ISP. Who knows what mihjt be causing these issues and what you could be breaking by interrupting service.
The Time Warner NOC wouldn't be able to completely fix the problem if the source of the issue is the firmware of a certain shitty IoT device. If someone emailed their NOC about some weird IP cams installed by their customers causing load on their servers, they could feel like that's a problem between icanhazip and the camera manufacturer, not something they can fix.
The author is quite tolerant of the obviously malicious behaviour others are attacking his servers with. I'd have taken more aggressive measures instead of scaling up capacities myself. Because the problem is volume and not necessarily anything complex, I'd wager that even a simple block could be quite expensive because that traffic and the associated retries will be going somewhere. Directing the traffic towards the last router in their ASN through DNS would be something I'd consider, making it the problem of the network operators.
Looking at the icanhazip.com site, I wonder how much any kind of rate-limiting per address/block would even help.
At the HTTP level it's probably cheaper to just return the HTTP 200 response. I suppose if you're doing TLS handshakes then a packet-level rate-limit would help significantly, but at the same time I'd be wary of triggering any kind of retry-behavior.
Worst-case scenario for a service like this would be having an error response/timeout trigger some kind of unlimited retry flood.
The block route I'd go with is blackholing the entire range into nothing through BGP or similar so the servers wouldn't have to deal with the traffic, similar to how anti DDOS tools often work. Might even redirect the DNS for that subnet to the IP of the people running the network, let them deal with the abuse. That'd be a very offensive approach, though.
I probably wouldn't bother with TLS either, just a plain HTTP 0.1 response with minimum information should be enough.
This raises something I've wondered for a while: is there a service or database that can give an indication of how many humans are behind a particular IP address? e.g. with CGNAT, there might be many thousands of people sharing a single IP. For some residential services, it might be 1-2 people.
It feels like this sort of data (even if only providing order of magnitude estimates) would help greatly with deciding on appropriate rate limits for small operators who don't have the time to research all the traffic they're receiving.
This is not an unimportant or victimless problem, however said problem is the network operator's entire job. Making them deal with this is not uncalled-for.
As a network operator, you would be surprised how many AS operators are hostile or simply don’t respond. It’s unfortunately very common, even Tier-1’s are hostile.
Out of the last month, I sent out 191 abuse reports, of which 10 got replied to, 2 were resolved 6 were “no f** off” style, and 2 were told “can’t fix / won’t fix / don’t know how to fix”.
I’m not just referring to Chinese ASNs either, some US Telco’s, German, Australia even.
We're entering an era where the content providers and the major clouds are >50% of demand on new subsea capacity. Feels like this blurs 'Tier 1' and the role of backbones because major eyeball networks now interconnect directly with e.g. Facebook or Google.
Anyway, on abuse@ response rates, my probably unpopular but realistic take based on looking at tens of thousands of such complaints over the years and having worked for ASNs which have received millions, I'd hazard everyone has an SNR and ROI problem with handling these. There's just too many of them and most aren't actionable.
Some examples, "I saw a failed SSH login attempt from 1.2.3.4 and OMG that's a huge issue, you have been compromised, and you must solve this immediately!". OK, well, the subscriber might have: a) Typoed your IP address, b) Been running nmap/zmap over a wide range of IPs for research purposes; c) You're on an IP with a provider who recycled it to you, subscriber has outdated DNS records.
What do you expect a 'Tier 1' to do with your report?
Many ASNs are now just looking at the pattern of reports per IP address or subscriber, are automating scanning for e.g. open mail relays when whatever processes abuse@ determines the person is complaining about spam, or automating looking for anomalies in flows for DDoS complaints, a human may not even see the ticket unless the automation was able to confirm a problem may exist, and the human will probably only engage the subscriber and won't respond to the 1-1000 things received to abuse@ related to the issue.
In Major's case with icanhazip.com it looks like pretty bad behavior from the Chinese ASNs mentioned, but could just be IOT configured to fetch its IP every minute instead of every 60 minutes of 24 hours because someone misunderstood cron. Unfortunate that nobody responded but 30B a day is ~350kRPS (which isn't a lot, in the grand scheme of the internet). I'm sure 30B requests per day is nothing at Cloudflare's scale and they have options to cure these ASNs behavior should they choose, including stuff like IP-based or ASN-based ratelimiting, or even IP/ASN restrictions.
I'm sure Cloudflare will learn some interesting things about both the accidental contributors (e.g. cron) and intentional contributors (e.g. botnets) from analyzing the sources generating the requests, and I'm ultimately glad it is them picking this up, their other initiatives like 1.1.1.1 have had been positive for the internet (IMHO).
If I were playing the game at that level, I’d do it too. The best I can do to evade my taxes is to get customers to pay me via cryptocurrency which I then sell to people locally in person in cash for a 10% discount.
> Uber probably are doing something dodgy, with all these shell companies, but this opening remark just shouts "Don't trust this source - they don't know what they are talking about".
Uber, like all massive companies, are always doing dodgy tax avoidance things. But, even if they're doing tax avoidance, it's still legal as long as it's not blatant fraud.
Like all media companies, Business Insider has manipulated the content to cause outrage with their target markets for readers.
Though, on the topic of wording data, it's always interesting to see how media organizations & companies as a whole swing things, for example:
"Uber claimed $4.5 billion in global operating losses ... in reality, it brought in $5.8 billion in operating revenue" could've been written as:
"Uber brought in $5.8 billion in operating revenue, however, declared a loss of $4.5 billion"
or
"In a 2019 filing, Uber declared a $4.5 billion loss, despite earning $5.8 billion in operating revenue."
Frankly, it's been long suggested by the government to do that. I would not be surprised if they did as well.
In Canada, our freedom of expression is limited by whatever the Government deems appropriate:
> Freedom of expression in Canada is protected as a "fundamental freedom" by Section 2 of the Canadian Charter of Rights and Freedoms. The Charter also permits the government to enforce "reasonable" limits.
Unfortunately, there is not much guidelines these days on what "reasonable" is considered. I like to hope it would not be abused...
> In Canada, our freedom of expression is limited by whatever the Government deems appropriate:
The same could be said for the United States or really just about any other country, as there are restrictions on free speech here (think "fighting words," etc.)
In Canada, what the government deems appropriate is under a version of the Sword of Damocles known as the electorate. A government can lose confidence of the house at any time in the way we know it. We can start to see this with a polarized cabinet in Alberta right now.
If we continue to teach good civics and encourage people promise no safe votes to politicians, they won't make reasonable limits unreasonable.
I'm not going to rely on Cloudflare Pages personally... It's concerning to see they let free traffic in regions be broken for over 8 hours _without_ redirecting traffic to another working PoP as well[0].
I was personally affected by it, it appears they kept the announcement of the prefixes live in those datacenters but it was similar to being blackholed in those POPs. It caused 8 hour of outages for me and many people I know who go through these POPs.
All the free CF sites I've got were down for those 8 hours via those POPs, and, I had many customers calling to ask why their site was doing and had to explain "sorry, Cloudflare broke just these two POPs..."
