As of late, the SH/SC Thread of the Week has been "HEY GUYS WHAT FORUM SHOULD I USE?!". In the past, the answer has begrudgingly been "Just use phpBB, because it's free and not too broken", but phpBB has taken a significant nosedive in quality, and several quite suitable replacements have cropped up in its place. Apart from the freebies there's also the industry standards that have never gone away. In this article I'll be outlining the systems worth talking about, some essential mods for them, and weighing all their assorted pros and cons.
What You Need to Know
Forums are pretty complex things, and before diving in there are a few things you need to take into consideration to figure out what system is best for you.
Security - This should be your #1 concern. What kind of security track record does a given system have? How frequently are exploits released? How quickly do the developers respond? How easy is the upgrade process?
Customizability - Mods and Themes are often important, as it's rare that you'll find a forum with everything you could ever want out-of-the-box, so an active modding and theming community plus good documentation for both will be important.
Price - Do you really want to pay for the right to use a given forum system on a yearly basis? Is paid support really important to you?
Spam Protection - Without some kind of CAPTCHA system, most forum systems are open to Spambots auto-registering accounts and plastering your forums with all manner of spam. Some forums have built-in CAPTCHAs, while others require mods.
Experience - How much experience do you have with PHP? How about some form of SQL? It will help immensely to have experience with both, especially if you're going to be looking to do any level of customization. Some systems like SMF are a bit friendlier to people with less coding experience, but a good background is going to help you across the board.
Security - Yes I know it's here twice, because it needs to be. You need to be reminded that developers fixing code is just half of the security process. You have to know that if you're going to run a forum, or any kind of publicly accessable software application on the internet, you have to stay up to date on it. Get on their release mailing list, bookmark their Secunia product page, check out milw0rm every once in awhile. BE PARANOID. Just remember that the upgrade you decide to skip or postpone because "Aw man, it's Friday afternoon, I don't feel like upgrading my forum installation" will be the one that would have patched an exploit that some script kiddy's worm will be using that weekend to turn your server into molten slag. I wish I was making this up.
Highly Recommended Free Software
These are your best bets, really. Free, Modern, Secure, and Actively developed.
PunBB
PunBB is a free lightweight opensource forum system, geared towards being simple and straightforward while still including most of the features expected from something like phpBB.
Security Record: Very Good. Exploits are rare, generally low in severity, and get patched in a very timely fashion.
Spam Protection: Project Founder Rickard Andersson is strictly against CAPTCHAs in all forms, and refuses to add one by default in PunBB on the grounds that it ruins accessibility. Luckily, this mod with these further tweaks and a readable font like DejaVu can result in a rather strong defense against spam.
Modding: PunBB mods are generally installed the Hard Way by editing some source files and running an installer PHP script to make any necessary database changes. There's a wide variety of mods available at PunRes, but not all of them will work immediately since installer scripts have a hard-coded max version and people seem to rarely update that. Luckily, any mod made for the 1.2.x line should work for any 1.2.x version of PunBB, you just have to edit the max version in the install_mod.php to match what you have.
Theming: PunBB Themes are purely CSS based, not template based like phpBB or SMF. This can be somewhat daunting if you have minimal CSS experience, but if you do have CSS experience it's amazing. PunRes has a decent amount of themes available in their database, but if you just want to tweak the colors there's a site called Spinkbb that lets you interactively muck around with the forum colors and ends up spitting out a custom .css file for you.
Support: The Official PunBB forums are quite helpful in troubleshooting, and it's not uncommon for Rickard to show up in a thread and tell you exactly how to do whatever you want to do. Then again, it's also not uncommon to come across some bug that you might be stuck troubleshooting yourself. Luckily, PunBB is very logically laid out, which makes troubleshooting quite easy if you're relatively competent when it comes to PHP/MySQL.
Upgrading: PunBB upgrades like a lot of complex PHP scripts. Download changed-files-only package, upload changed files, run an included update script to mess with the database if necessary. Only pot-holes to watch for are that any files you modified due to mods either aren't changed or are re-patched once you upgrade.
Migration: If you're jumping ship from from some other forum system to PunBB, a nice converter script is available that will migrate your old database to a new shiny PunBB one. It can convert from pretty much all major boards, including everything in this article save for Vanilla (too new) and UBB.threads, with varying levels of success. phpBB, for instance, will migrate users, threads, posts and postcounts, but things like Ranks, administrators, user groups, etc are all up to you. Migrating the things it misses isn't very difficult if you're good with MySQL, though.
Simple Machines
SMF is a more advanced forum package compared to things like PunBB. Rather than attempting to be a streamlined package, SMF is geared more towards being a complete 1:1 replacement for systems like phpBB, but done right. SMF has its roots in YaBB, by way of the short-lived "YaBB SE" which was a port of YaBB to PHP. The devs decided to scrap all that YaBB stuff and start from scratch as Simple Machines.
Security Record: Pretty clean. This shouldn't be a sign that their software is bulletproof, though, just that no one has tried to seriously attack it yet.
Spam Protection: None built in, however mods are available like this one.
Modding: Mods for SMF are package based. This means no editing of source files and no desync when you upgrade. Neat.
Theming: Themes are installed via the same package based system. If you want to make your own, you can either use the web-based system in the admin panel or make your own offline. Either way, it uses a full templating system, similar to YaBB.
Support: SMF has a rather extensive support section, featuring a detailed FAQ, a Wiki-based Online Manual, and then of course there's the forums you can search.
Upgrading: Upgrading works just about like all other boards. Upload stuff, run an upgrade script. But since mods and themes are package based, you don't have to worry about file desync. Plus there are a bunch of nice extras in the upgrade script, like integrated database backup and the ability to putt the forum into maintenance mode during the upgrade.
Migration: SMF has a couple converters available, covering most common forum systems. Not sure how good they are, though. You should probably be ready with some manual SQL action.
Vanilla
Vanilla is a very new forum system [initially released 1 JUL 2006] that attempts to completely reinvision the forum system. So much so that it probably isn't what you're looking for. Don't expect a lot of structure and don't expect to use it for a high-volume forum.
Security Record: Technically impeccable. However, this is just due to the fact that it's so new. phpBB didn't have any exploits in the wild after a month, either.
Spam Protection: Vanilla's idea of "Spam Protection" is "force the admin to manually approve all registrations". Brilliant, guys. Brilliant.
Modding: Vanilla really plays up the whole Modding thing. Vanilla mods (called "extensions") are all self contained like SMF, but . . uh . . they all seem to do stuff I can't imagine anyone ever caring about. Like this amazing To-Do List. A forum with a To-Do list. Just what I need.
Theming: Since it's new and all, there are apparently only four themes. One's the default, one's a very minor mod, and one's a placeholder, leaving one actual theme. Super.
Upgrading / Migration: As far as I can tell, this stuff is nonexistant. I assume upgrading instructions might show up when there's a version to upgrade to, but Migration stuff might take awhile.
Other Stuff: Vanilla only supports PHP/MySQL. Sorry Postgresql users! Sorry SQLite crew! You get to sit this Web2.0 shitfest out.
Good, but Distinctly Not Free
If you've got money to blow and want paid support, these are ones to watch.
vBulletin
vBulletin is the big dog in the non-free forum market. Though I'd never pay for it myself, I figure if you're going to shell out big bucks for a forum, you'd might as well go with the most popular. Nearly all of the top-traffic forums out there use a flavor of VBulletin if they don't have something custom. Newer versions (3.5+) have a totally rebuilt product/plugin system that makes adding modifications to your installation easy and fast. All templates can be edited on the fly within the Administration Control Panel, no need to manually edit files on the server. There are ton of other features, check out their official website for more information. But in summary, this is the cock of the walk for commercial forum software.
Security Record: Not horrible. They've had a couple somewhat high-profile exploits roll through, but since they're getting paid to fix bugs, bugs get fixed nice and quick.
Pricing: $160 to own it, with 1 year of upgrades, which can be expanded for $30/yr. $85 to "lease" it for 1 year at a time, which includes upgrades.
Spam Protection: They've got a built in CAPTCHA, but it's pretty weak according to PWNtcha. I am uncertain of the status of vB spambots, but if the CAPTCHA is weak, they probably exist.
Modding: Huge community based around modding on VB.org (see above).
Theming: That same community, VB.org (see above) provides hundreds of free themes and template modifications.
Support: Docs, Forum-based support and email support are included in your license fee. Toll-free phone support is anywhere from $60-25/mo, depending on how long of a support contract you buy.
Upgrading: Upgrading is very easy, with an idiot-proof upgrading script. Basically, you upload modified files, run the upgrade script, press next a few times, and you're done! It really is that easy.
Migration: Impressive support using VB's own Impex script. It has support for pretty much every forum system out there.
Invision Power Board
IPB has always sorta been an odd-man-out. It was started when the original developer of IkonBoard left that project, and was initially free for non-commercial installations. Then they pulled a bait & switch, by continuing to promise that it would be free until the last minute when in 2004 they made it cost for everyone. Now you get to pay no matter what.
Security Record: Currently good, however, since IPB has gone through Licensing Hell, a lot of people are still out there using the old free version which has 5 unpatched vulnerabilities, ranging from XSS to Arbitrary Remote Code Execution.
Pricing: $185 to own it and get upgrades for life, but only 1 year of technical support, which can be extended for $30/yr. $69.95 for a 1-year license that only shuts off upgrades after your year runs out if you don't renew it. The board will still work, you just won't be able to patch it.
Spam Protection: They've got a CAPTCHA system in place, but it looks pretty weak. Fixed glyph positions, very little glyph obfuscation, no rotation or manipulation, etc.
Modding: Mods are available and are the standard messy manually-edit-a-bunch-of-files type. Yuck.
Theming: IPB themes seem to be template based, and interestingly modern ones seem to be applicable to all other IPS products like their Wiki, Gallery and Blog systems.
Support: Again, standard stuff, you've got docs, forums, and since you pay for it, toll-free phone support. One nice note is that their bug reporting forum shows status of the bug right on the topic listing, so you can easily see if something is resolved or not.
Upgrading: Invision claims that not only is their upgrade system easy, but they've got techs that will do it for you. Well great. I'd believe it when I see it, though.
Migration: Similar to their upgrading claims, they'll also do migration for you. Their conversion team can supposedly convert from everything listed here except PunBB and Vanilla, plus there's a use-it-yourself converter tool that can do most major forum systems and a couple little ones. They certainly do seem agressive on these fronts, so much so that if you've got a forum they can't convert, they'd still be willing to try.
UBB.threads
UBB was one of the original examples of a full-featured web-based forum, credited with developing the BBCode markup system now used just about everywhere. Over the past decade the original UBB has come and gone, but along the way WWWThreads (a PHP-based threaded forum system) was bought by InfoPop and rebranded into UBB.threads.
Security Record: Who doesn't care about security? UBB doesn't care about security. Both UBB.Threads 5.x and 6.x have unpatched vulnerabilities according to Secunia. SQL injection, XSS, System Access, you name it. To be fair, though, the only places I've actually seen UBB.Threads used in modern implementations are places like ArsTechnica, which use Groupee's (formerly InfoPop) hosted forum solutions. One would assume that these exploits would get fixed for their hosted clients, and just don't trickle down to pesky little things like "released versions".
Pricing: $229 for a one-year license with no support whatsoever, $499 for a 5-year license with an amazing three hours of support, $725 for a lifetime license and an unbelievable five hours worth of support.
Spam Protection: Spam Protection? They can't get security down, and you expect Spam Protection? Luckily there are a couple uglymods to apply some form of CAPTCHA. I have yet to be able to find one of these actually in use, so I have no clue how easily-breakable it is.
Modding: Mods exist, but are unimaginably ugly. I don't know what it is about them, but they just seem like such dirty hacks compared to what else is out there. UBB.threads in general feels like a big dirty hack in general, really.
Theming: UBB.threads themes seem to largely be based around just mucking around with the CSS. Good luck finding documentation, though.
Support: I have no idea what support is like, because I lack a license. Considering their track record with bugfixes, I wouldn't expect much unless you're paying the big bucks for a hosted solution.
Upgrading: Details are sparse, but it doesn't seem too complex to upgrade forums, seems just like most other upgrade systems.
Migration: I can't find anything about this at all. Again, since it seems they're trying to focus on hosted solutions, they'd probably jump through hoops for you to migrate whatever you have to them, but who knows.
Wow, the guys at Groupee/InfoPop really seem to have fallen on hard times since the dark ages when UBB reigned supreme. Their name change, constant shuffling around of software, constant rebranding of software, and their distinct focus on hosted solutions rather than shipping good software is really sort of depressing. I really can't recommend even considering using them.
Not Recommended Whatsoever
(But still included so people didn't think we forgot about them)
phpBB
Oh, phpBB. Back in 2000 you seemed perfect. Fresh and active development team. Free and Open Source. Highly extensible and customizable. And hell, at the time the main competition (UBB) was still using Perl and flat files, so phpBB looked amazing by comparison. Unfortunately, as time has gone on, phpBB has turned out to be highly exploitable. XSS exploits are common, same with SQL injection. Though uncommon, phpBB has in the past been subject to a massive worm attack, where board after board was hijacked, the hosting server was trashed, and further attacks were launched from the then-compromised server. Their built-in CAPTCHA (which was added incredibly late in the game) is a joke.
The good news is that phpBB's failings have contributed significantly to other projects listed in this article, as it leaves a distinct opening for learning from phpBB's glaring mistakes and writing your own board from scratch and doing it the right way.
Security Record: Abysmal. While everything gets patched, and generally in a timely fashion, the vulnerabilities are frequently critical. Furthermore, even phpBB mods get targeted by vulnerabilities, and not little pidly ones, I'm talking highly-critical systemaccessexploits. Some of which remain unpatched. The whole thing is just a giant security minefield.
Spam Protection: They've included a CAPTCHA system since part-way through the 2.x series, but it's absolutely pathetic. Easily defeated by PWNtcha or a similar system.
Modding: Mods to do just about everything you could ever want are available, but you have to install them the Hard Way by modifying source files, and as mentioned above, security is frequently just as awful as phpBB itself.
Theming: The one undeniably good thing about phpBB is the theme system. Not only are there tons of great themes to work with, but developing themes is dead simple, and even easier now than back in the dark ages since there's actually documentation. Back in my day we had to guess what all these arcane template variables meant. Now you kids have convenient lists to work with.
Support: Pretty standard stuff. Docs, Forums, and as a bonus, they've got a Mailing list so you can be kept extra up to date on when there's a super critical bug you have to wake up at 4am to patch so you don't get owned by breakfast.
Upgrading: Upgrading phpBB sucks. You have to track files you've modded, re-mod changed files, and possibly have to tweak your theme too because sometimes they just LOVE hiding new cosmetic changes
Migration: phpBB has a pretty wide variety of migrators available. Just about the most of the other systems I've reviewed, including systems I've never even heard of. Ironic that the system most people want to switch away from has the most ways to switch to it.
Bonus Warning: Please please please please please don't use phpBB. Please. I really mean it.
YaBB
Yet Another (see what I did there?) forum system that grew out of UBB's shortcomings around 2000, like vB and phpBB. Unfortunately, it didn't grow enough. Even to this day, YaBB still uses Perl, still uses flat file storage, and is still not worth talking about whatsoever. But at least it's free, right?!
YaBB is so ancient and decrepit that I'm not even going to go through the whole list of details for it. Not only is it old and outmoded, but the developers have moved on not once but twice, from YaBB to YaBB SE (PHP-based YaBB) and now to SMF. If you for some sick reason really want to use YaBB, please punch yourself in the face and use Simple Machines instead. Even the YaBB developers want you to use SMF instead of YaBB. If you won't listen to me, listen to them.
These days, there's very little reason to pay for forum software. Modern systems like PunBB and SMF are just as good as ones you'd have to pay a hundred bucks a year for. So lets recap:
If you don't mind getting your hands dirty, and feel sufficiently experienced with PHP/SQL, look towards PunBB.
If you consider yourself less experienced, or just want something more full-featured out of the box and an easier modding/upgrade procedure, check out Simple Machines
If you absolutely feel the need to throw money at someone, consider vBulletin, or alternatively, a paid hosted solution of PunBB or SMF.
But remember: No matter what system you choose, stay up to date!