NewsGroups

ShscWiki :: LogIn :: PageIndex :: RecentChanges

A Short Guide to Liberating Data from Usenet

by inpheaux

Usenet is the primordial soup which all modern internet discussion forums evolved from. Back in the dark ages it was used for actual discussion, and in some small secluded corners you may actually still find it being used that way, but that isn't why you're here.

Today, Usenet is primarily used for the distribution of 'binaries', which can be everything from real honest-to-goodness Linux ISOs to every kind of illicit digital commodity under the sun. Furthermore, Usenet tends to be one of the first places said goods arrive, shortly after topsites and IRC, and generally before BitTorrent. Usenet is popular in this respect because it was the first service to provide such a wide-scale distribution channel, and it has had staying power due to being decentralized and thus quite hard to shut down.

 Table of content 


Boring History Lesson

Usenet is made up of a hierarchical system of newsgroups, sorted roughly by what goes on in said newsgroup. After The Great Renaming, we ended up with the broad top-level categories like comp.*, sci.* and rec.*; and then you had alt.*, the catchall for Everything Else. Binaries, so-called because they're not text, generally live somewhere within the great tangled mass that is the alt.binaries.* hierarchy. To start cruising through Usenet, there's a few things you'll need:

Newsreaders

To get files out of Usenet, you first need a program called a Newsreader. There are three kinds of newsreaders: Text-Oriented, Binary-Oriented and Pure NZB Processors. We'll be covering the latter two.

Binary-Oriented Newsreaders

These programs are designed for the old way of downloading binaries from Usenet. They generally require you to download large quantities of headers, read through all the headers, and manually pick and choose what stuff you want to download. All of these newsreaders have gotten with the times and can process NZB's, but they're still geared towards the Old Way.

Pure NZB Processors

NZB is an XML format developed by Newzbin to side-step the process of downloading headers. Instead of subscribing groups, downloading and then combing through hundreds of thousands of headers, you search Newzbin and (assuming you have premium credit) generate an .nzb. Simply hand that .nzb off to one of the following programs and you'll skip downloading headers for hours, and directly start leeching just what you want.

Other Software

If you do things manually, you're going to end up with a ton of rars and pars. So you'll need a few more bits of software. It also never hurts to have this stuff installed.

Indexing Services

Usenet is a Big Place - like, terabytes-posted-per-day big - and sometimes people don't post things where you expect. These services let you skip all that.

Usenet Servers

Many ISPs provide you with free Usenet access, either through their own server or through a third party. Determine this FIRST so you don't shell out for a service you don't need. As of late, more ISPs have gone to outsourcing their news servers. . . but this is a good thing, because they've been outsourced to companies that know how to run a news server. Comcast outsources their stuff to Giganews, and gives you a 2gb/mo account there, which may translate to a discount on a full Giganews account. Time Warner has recently switched their news service to an unlimited Newshosting account. Your ISP's Usenet access might not suck, so do some research regarding what they offer before shelling out for a dedicated Usenet provider. Newzbin provides a rather extensive list of what assorted ISPs worldwide offer with regard to Usenet.

If your ISP doesn't offer Usenet access, or if their access sucks, you'll want a dedicated Usenet Provider. When it comes to determining what makes a news server suck, there are four things to be concerned with:

  1. Completion - What percentage of posts actually make it to the server? Because Usenet is decentralized, you can only download what files have shown up on your server. Sometimes this just causes lag, where it can take a few hours for a post to actually get there. Most times it results in the file just not showing up at all, which sucks because then you've got an incomplete binary and will have to use Pars.
  2. Retention - How long do posts actually stay on the server? Again, because Usenet is decentralized, and the files you see are just what's on your specific news server, your news server's admin will generally have things on a rolling cycle where posts get nuked after [x] days. Obviously, higher is better, unless you don't mind checking Usenet daily for your fix.
  3. Monthly Quota - How much can you download per month? Bandwidth isn't free, so servers generally give you a monthly quota. To me, this is the most important aspect, because who cares if you have access to 80 bazillion terabytes of posts and can download them at the fastest speed your hardware can handle if you have to essentially pay per gig? Luckily, pay-per-gig services are few and far between these days.
  4. Price - DURRRRRRRRRR

With that out of the way, here are some commercial news servers:

An In-Depth Walkthrough

We used to have a lengthy walkthrough here explaining in-depth how to set up and use SABnzbd, but that was only really needed because there was no official SABnzbd documentation. We don't have to do that anymore, because the SABnzbdPlus team is staffed by competent people who like writing documentation. So here's the only three links you'll need to get everything up and running once you've secured a news host, some Newzbin premium credit, and have downloaded the latest build of SABnzbdPlus.

  1. SABnzbdPlus Setup Guide
  2. SABnzbdPlus Usage Guide (I, again, strongly suggest using nzbdStatus and Newzbin)
  3. SABnzbdPlus Wiki Manual

Once you get it all configured, it's just a matter of searching for something at Newzbin and clicking the pretty orange arrows. SABnzbdPlus will gleefully download everything automatically, verify it, repair it, extract it, and clean up the remaining files for you. Anything else is insane and needlessly complicated.

This article is ©2008 by the respective authors. Reproduction is prohibited without express permission from all contributors.