Musk’s Perplexing Battle With Nitter and the Problem With Locking the Internet Behind Login Screens

It starts with a login; it ends with an internet entirely under government control. In this article I describe how online trends risk further enabling censorious governments, and how this relates to the dispute between Elon Musk’s X and the open-source Nitter project.

Several sinister patterns are becoming widespread online.

Firstly: sites are now so determined to obstruct AI crawlers from scraping material for training that human users are caught in the crossfire. Increasingly, webpage failures come not from developer mistakes, but deliberate actions by CDNs (Content Delivery Networks), such as Cloudflare and Akamai. CDNs serve as middlemen to route traffic but block webpages – or parts thereof – on the slightest suspicion of your browser being a bot. Even if you ‘lower shields’ by relaxing browser security settings, they often block you simply on the grounds of your IP address – provided randomly and temporarily to you by your ISP – having a poor reputation. A VPN which could bypass an age-lock on taxi-tracking imposed by an ISP or government, or bypass geoblocks run by a site itself, still won’t get you through to the contact form on a utility company’s website behind an overzealous CDN dragnet.

Secondly: actions which were formerly available to all passing visitors, particularly read-only actions like viewing content, are now being placed behind the wall of creating an account. Pinterest, Quora, LinkedIn, and Instagram were early to try this, deciding that images, answers to questions and resumés – all intended to be shared with any member of the public by the people posting them – would henceforth only be available to logged-in users. This trend has spread like a panic amid a pandemic since July 2023 when Twitter, re-branding to X shortly afterwards, locked profile pages – where the chronological records of an author’s tweets are displayed – behind a login wall.

Thirdly, and finally, the burden involved in creating accounts has grossly increased. Many sites now force convoluted verification via SMS message, phone numbers being dangerously close to de facto Digital IDs in many countries. Even this level of burden can be highly dangerous, such as when its use on mainstream Western websites obstructs Iranian dissidents’ ability to organise online over cobbled-together connections. Some sites are even more intrusive, demanding app installations, iris scans or credit card numbers – even for free services. Such accumulations of identity data become an attractive target for cyber-criminals, who find that even a phishing email from a supposedly official source can be enough to make companies spill user details. These deanonymising demands also prevent people who are receiving online abuse from simply abandoning old accounts and creating fresh ones which would not be targeted. In the era just before social media, one was always taught that the key to online safety was never to give websites your real details. Anonymity is safety, particularly when the most serious danger to internet users today comes with a knock at the door. Remember too that sacrificing anonymity makes people vulnerable not only to today’s Government, but also any future government which engages in ‘offence archaeology’ as a prelude to persecution.

Summed together, these are equivalent to a high-street shop demanding that pedestrians pass a background check before they can view items in the window. Apologists will say that treating every potential customer as a hostile trespasser in this manner is necessary to appease advertising partners, who insist on proof that their ads are placed before real human eyeballs; but it sacrifices the spontaneity upon which the web rose to prominence. What is the point in advertising to an audience which resents the biometric strip-search they endured to enter the showroom? A free market cannot be saved by enveloping it in the infrastructure of a digital prison.

Despite all this, crawlers continue unimpeded. Scraping scripts can forge the appearance of browsers more ‘realistic’ than your real browser. Residential proxies can be hired by AI companies to forward their traffic through an IP address which doesn’t yet have a bad reputation – but will after they’re finished with it and it gets assigned to an innocent user. And where website operators fear the sheer traffic volumes associated with crawling, they have only themselves to blame when they decide to bundle a firehose of autoplaying background videos alongside the plain text and images the user actually wanted to see, and to require an average of over 70 loading requests per page – over 20 typically for JavaScript alone.

Static HTML pages can reduce the burden on servers; when combined with proof-of-work challenges – or simple rate limiting – they can stem the tides of bots without requiring a digital panopticon. Yet this supposed online crisis, like every other, prompts calls from the usual corporate and government voices for centralised regulations and control, rather than encouraging the development of open tools to give individuals an edge over spammers, scammers and bots.

Enclosing the internet within walled gardens, an activity which Big Tech performs all by itself without the need for governmental pressure, plays right into government hands. Now a citizen can be tracked and punished simply for viewing something critical of establishment viewpoints with no need for messy tracing of IP addresses and metadata; if he’s had to log in with an identifiable account, then the privacy advantages usually conferred by a VPN or Tor have been reduced to little more than cosplaying as a covert operative. Walled gardens also benefit those seeking to memory-hole inconvenient records: when requiring accounts to view pages prevents crawling by independent archives, who is to say that a page ever existed?

This confluence of interests between Big Tech and Big Government is highlighted by recent court proceedings which, though reported as if punishing Meta (Facebook’s parent company), actually appear a godsend to it. For a mere $17 billion, Meta gets a mandate to tighten its monopoly, gather even more private data under the pretence of age verification, and keep at bay any startup competitors who cannot clear the onerous hurdles which Big Tech legal departments pass at full gallop. The payout pales into insignificance compared with the future profits this situation secures.

It is amid this internet-wide infestation of walled gardens that the decision by X to send a cease-and-desist letter to Zedeus, pseudonymous lead developer of the Nitter project, is so perplexing.

Nitter is open-source software which runs on a web server; anyone can host an individual instance. When a visitor to an instance requests a page, the instance uses throwaway X accounts to fetch data on-demand from X. These data are then presented on a lightweight page resembling the classic Twitter interface. Where the official X interface has begun forcing visitors to log-in even to unmute videos, Nitter shows text, images, profile pages and videos in full. Any URL for a page on X transforms to a Nitter URL by substituting just the x.com part with the domain name of a working instance.

The view provided by Nitter is unaffected by the age-verification checkpoints which ‘protect the children’ from historic oil paintings, video journalism and Parliamentary speeches. Nitter’s bypass method works when a VPN cannot, such as if the exit-node country becomes as heavily censored as the visitor’s own, or if an account opening via SMS verification is forced – thereby alerting X to the visitor’s true location via phone number prefixes. If Elon Musk feels he cannot deactivate censorship within X, then helping – rather than allowing his legal department to hound – a service which provides a censorship-bypass is surely a good fallback option?

Many commentators suggest switching to alternative microblogging platforms, but Bluesky is as willing as X to comply with diktats, and Mastodon is badly fragmented due to Left-wing servers enforcing cascading bans on any neutral server which allows users from Right-wing servers to post – even on apolitical topics. Nostr is censorship-proof at the architectural level, allowing any user to follow posts from any combination of others with any combination of viewpoints – anyone wanting to see all sides of an argument truly can – but suffers from the self-reinforcing problem of a lack of users. Anyway, such suggestions are only applicable to whole communities of people wishing to move. Whilst I am loath to imply that government content is of unique importance, the fact that official bodies around the world use X to share information – often sidelining their own websites – shows what a de facto public square X has become. Aside from officialdom, many commentators with unique insights on newsworthy current affairs, local events, politics and countless apolitical topics continue to use X as their only distribution channel. The dedication of Nitter’s developers to preserving a non-login route for viewing this shows they understand the limitations of an exodus.

Copyright-focused criticisms also implode under moral scrutiny, falsely equating Nitter to a service which might bypass paywalls at the New York Times, Netflix or a Substack blog – paywalled by author’s choice. Tweets were written by their authors, not by X’s employees, and – especially for the vast archives of material predating July 2023 – were posted in the hope of being freely read by a global audience, not left to wither in a walled garden only account-holders could enter. Musk’s “How dare they steal the stuff Anthropic stole from human coders” observation on AI crawling applies here; when all of Big Tech are busily scraping one another, one cannot complain when the tables are turned. As a self-declared free-speech absolutist Musk would – I hope – understand that time-invariant moral constants should take precedence over mere legislation or terms of service, both being mutable on a middle manager or minister’s whim. Tolerance for all speech, except where calculated to incite imminent lawless violence, is a reliable compass to which one can pledge loyalty. By contrast, obeying the laws of the day risks inconsistency when activist judges and authoritarian politicians pervert them against common-sense as if shipping a shifting cargo of magnets right beside the binnacle. This principle-based philosophy is becoming worryingly rare among Big Tech. But platforms which adhere to free-speech principles, rather than treating every – usually widely resented – regulation as sacred, still enjoy surging popularity.

Though today’s Big Tech has often betrayed these ideals, services with a backbone can still decline to assist with overbearing enforcement and avoid complicity in forcing a population’s opposition to overreach to operate entirely within the very strictures the people wish to abolish. If you run a site with Winnie the Pooh art you should not block downloads from Chinese IP addresses; if you run a blog about the war in Ukraine you should not geoblock Russian IP addresses, thereby helping Putin to hide inconvenient news away from the Russian people. If you run a privacy or messaging tool designed to work amid extreme levels of censorship you have no excuse not to make it available to dissidents in Tehran – Tor, Briar and Signal ensure their systems are optimised for the sort of user who may yet Make Iran Persia Again.

Whilst Zedeus’s own Nitter instances are presently inactive and replaced with a donation request for legal funding, X’s legal department cannot coerce the open-source project as a whole. There are now more Nitter instances active than before the cease-and-desist letter. A visit to xxcancel.com shows the variety of these instances, each being independent of xxcancel.com and nitter.net, hence, able to run without either. This is how the good old internet, the one Under Secretary Rogers says she hopes to preserve, works. It is a version of the Streisand effect, and a reality – quite immune to legal technicalities – which Musk must come to accept. His best move at this point – should he wish to demonstrate that his free-speech absolutist statements were not an empty gesture – would be to sharply call his legal department to heel, withdraw the cease-and-desist, and actively donate some pocket-change to Nitter’s server costs.

And for websites beyond X and Nitter, the conclusion is clear: put the user first, don’t let a few Big Tech giants herd you into walled gardens where visitors’ access to your content is conditional on their approval. The danger of a two-tier internet, accessible only with ‘Approved’ devices, still looms over us; the undermining of ownership through ‘Smart’ devices dependent on cloud accounts forms the other jaw of a vice; and walled gardens online serve to normalise this extreme level of dependency. Tolerating some crawlers is the price of a connection to your visitors which your competitors cannot disrupt: Tear down this login wall!

In Other News: The Home Office’s attempts to backdoor Apple’s encryption, as discussed previously, continue. Again the main lesson, particularly for opposition politicians whom Labour might be targeting, is to trust in open-source code which – for all of Big Tech’s supposed immunity to legal interference – remains the gold standard for government-proofing.

Dr R P completed a robotics PhD during the global over-reaction to Covid. He spends his time with one eye on an oscilloscope, one hand on a soldering iron and one ear waiting for the latest bad news.