Password Cracking Rig Setup
A Quick Note to N00bs
This document is primarly for people who have already fallen in love with password cracking and are ready to take their relationship to the next level.
If you are brand-new to hash cracking, please do not be discouraged by this document! While it is true that you really should not be cracking passwords
on a laptop or a low/mid-range desktop, or even an HEDT with inadequate power/cooling, that's likely all you will have available to you when you are
first starting out and that's okay! As a beginner, these systems are perfectly fine to learn on so long as you keep the workload light (
hashcat -w 1 or hashcat -w 2) and keep your runs fairly short (30 minutes - 1 hour at most.) As you start to learn the ropes you will
want to quench your thirst for more speed, and that's when this document will be more valuable to you. Until then, use what you have and just have fun
learning!
The Wrong Tools for the Job
Password cracking is hands-down the most brutal workload you can throw at a computer — absolutely nothing taxes computer hardware like password cracking does. So while it is technically possible to crack passwords on just about anything with a processor, there are many devices on which you probably should not attempt to crack passwords, such as your mobile phone, your Tesla's infotainment system, or the casino ATMs. Attempting to crack passwords on systems that are not designed to handle sustained compute workloads for long durations will definitively ensure their untimely demise. So if you manage to get Hashcat running on your Nintendo Switch, be fully prepared to buy a new Nintendo Switch tomorrow.
It also is generally not advisable to crack passwords on your laptop. Most laptops do not have a discrete GPU, but rather have an integrated GPU or
an APU. And most laptops which do have a discrete GPU do not have adquate cooling for compute workloads, as typically the GPU and CPU share a single
common heatpipe and heatsink. If you happen to have an ultra high-end "desktop replacement laptop" or "mobile workstation" with a desktop-class GPU,
then you may actually have discrete thermal zones for your GPU and CPU and thus may actually be able to get away with cracking on your laptop. For
example, I have a Clevo P650HP with a GTX 1060 and a Dell Precision 7530 with a Quadro P4200, both of which can actually handle full compute workloads
(hashcat -w 3) for moderately long durations (but not without burning my fingers when I touch the keyboard). However, do not expect
to get amazing speeds out of even the highest-end mobile workstations, as these are still relatively low-power devices (< 200W) with performance that
reflects such.
It also may not be advisable to crack passwords on your current desktop. Low/Mid-range desktops typically have an integrated GPU, an APU, or a low- power budget GPU that simply cannot cope with compute workloads. Even a high-end gaming desktop may not have a GPU or chassis with adequate cooling for password cracking workloads. If you find your temperatures soaring past 90°, it's time to crack open your wallet.
Finally, while GPU mining rigs are technically designed for dedicated compute workloads (but only by the narrowest definition), they tend
to break nearly every rule when it comes to building a dedicated cracking rig. These rigs are designed to perform brute force attacks against a single
target hash with the absolute minimum amount of hardware necessary to do so, in an effort to keep operating overhead as low as possible and profits
as high as possible. This is the polar opposite of what you want in a password cracking rig. While you can get away with using a GPU mining rig for
some password cracking workloads and you might post some exciting numbers with hashcat -b, you will ultimately run into problems
when attempting to do anything fun or worthwhile. If you already have the hardware, you might as well use it until you start running into walls. Otherwise,
please don't rush out to buy a milkcrate miner off eBay.
The Right Tool for the Job
Building a proper password cracking rig is nowhere near as simple as shoving a shitload of GPUs into a computer and calling it a day. Imagine running the most wicked stress-test ever made on your system, and then continuting to run it for days / weeks / months on end — well, that's password cracking. Power consumption and heat dissipation are the chief demons you must battle, but indeed all components you select for your rig must be balanced and carefully considered. Seriously. There are few if any corners you can cut if you want to do this right.
Power Delivery
Power delivery is a critical aspect of building an operating a password cracking rig. You must understand some basic electrical concepts (W =
V × A × PF) and be capable of doing some basic mathematics, because if you get this part wrong... best-case scenario, you pop a
circuit breaker; worst-case scenario, you cause a fire resulting in property damage, injury, or potentially even death.
Ensure that the electrical outlet(s) you are plugging your rig into are capable of supporting the amount of power your rig will draw. Most modern residential circuits are only 15 or 20 amps (110V), and you should never place more than 80% load on a circuit (e.g., 12A for a 15A circuit, or 16A for a 20A circuit.) Depending on the number of GPUs you have, your rig might draw significantly more amperage than this. And your rig may not be the only thing drawing power on the circuit (you'll be competing for electrons with the lighting, the microwave oven, the Hitachi, etc.)
Also, be mindful of your power supply's power factor. Your power supply's job is to convert AC current from the wall into usable DC current, and it cannot do this with perfect efficiency. Power supplies are most efficient at 50% load (typically 85% - 95%), and are typically only 80% - 92% efficient at 100% load. Suppose you have a rig with 4 x RTX 2080 whose total component draw is 1320W, but your power supply is only 85% efficient at 100% load:
1320W active power ÷ 0.85 power factor ≈ 1553W apparent power
While your total component draw may only be 1320W (active power), due to your power supply's power factor, you are actually drawing more than 1550W from the outlet (apparent power.) This may not seem like that big of a deal until you do the math and realize that, while a 15A 110V circuit can support 1320W, it cannot safely support 1550W. On a similar note, it's important to understand that your power supply's wattage rating is the number of watts it's capable of delivering to your components — it's an output rating, not an input rating. If you have a 1600W power supply that has a power factor of 87% on full load, your power supply is actually capable of drawing 1839W from the outlet. You can learn more about your power supply's power factor and efficiency at various loads on the 80 Plus Certification website.
On the topic of power supplies, ensure you use quality power supplies that can deliver the amount of power your system will demand. You may need more than one power supply depending on the number of GPUs and other components in your rig, especially if you have a power-hungry CPU like AMD Threadripper. Your power supplies should be rated 80 Plus Gold at an absolute minimum, and ideally should be capable of supplying 120% - 200% of your total system power consumption under load. Remember, power supplies are most efficient (have the highest power factor) when they are in the 20% - 50% load range.
Additionally — and this is extremely important — DO NOT BUY OR USE CHEAP OR UNDERSIZED POWER SUPPLIES. I will say it again, because this is absolutely critical: DO NOT USE CHEAP OR UNDERSIZED POWER SUPPLIES IN YOUR PASSWORD CRACKING RIG. YOU WILL START A FIRE. A LITERAL FUCKING FIRE, CRAIG. A FIRE WITH FLAMES. REAL FLAMES. THE BURNING KIND. There are some areas where you may be able to cut some corners to save cost in your password cracking rig, but power supplies absolutely must not be an area where you cheap out.
Finally, ensure the power cables you use are of sufficient gauge. Most all power supplies — even power supplies capable of delivering >1000 watts — come with flimsy 18 AWG C13 power cables which can quickly overheat and melt. You will likely need to use 16 AWG power cables at a minimum, and if you have a monster 1500W+ power supply, 14 AWG power cables will be required.
Chassis & Heat Dissipation
The chassis, also known as the case, is the foundation of your build. The chassis you select for your rig must be large enough to accomodate the hardware you plan to install, while also supporting an appropriate number of power supplies and providing an unobstructed straight-line path for strong positive air flow. Most mid-tower ATX chassis can only support two or three GPUs, while full-tower and super-tower EATX / SSI-EEB chassis can support four or five GPUs. Specialty server chassis can accomodate multiple power supplies, as well as support eight, ten, sixteen, and even twenty (single-width) GPUs.
The stock fans that come with your chassis will almost surely be woefully insufficient for the amount of heat your rig will generate (there are a few exceptions to this, but pretend there aren't.) You will need to replace the stock fans with high CFM, high static pressure fans. Delta and San Ace are good options, but be aware some of these fans can draw up to 75W each. Also, these fans can be inordinately loud, which may be bothersome to some. Water cooling may be an appropriate alternative in some scenarios.
If you're looking for an easy win when it comes to chassis selection, both Tyan and Supermicro offer server barebones that are specifically designed for GPGPU computing (caveat emptor: many of these chassis are designed for low-power, passively-cooled server GPUs like NVIDIA Tesla and cannot accomodate desktop GPUs.)
Graphics Processing Units (GPUs)
If you're a n00b, you may be wondering why we keep talking about graphics cards when we're not talking about gaming or graphics. Or maybe you already knew that GPUs are used for password cracking, but don't really understand why. Well, password cracking is what's referred to as an embarrassingly parallel problem (little effort is needed to split the problem into a number of parallel tasks), and as such, password cracking workloads benefit greatly from parallelism. This means the more processors we have working on a problem, the less time we have to wait for a task to complete. Your CPU likely has somewhere between 4 - 16 cores, and with simultaneous multithreading (SMT, also known as hyperthreading) may be able to support 8 - 32 execution threads. And with Single Instruction Multiple Data (SIMD) instructions, you may be able to perform anywhere from 16 - 256 simultaneous calculations. This may seem like a lot, and you're right — it is! Or at least it would be, if not for the fact that GPUs have a literal fuckton of cores. Modern high-end GPUs can have anywhere from 2560 - 5120 cores, and while they are slower and have limited capabilities compared to the cores in your CPU, there are enough of them to make GPUs 100+ times faster than your CPU for password cracking workloads. And performance scales linearly with each GPU you add to your rig, too! So if you have 10× RTX 2080 Ti in your rig, you would have an orgasmic 43,520 GPU cores at your disposal that will never push a single pixel.
To better understand this concept, here's an illustrative example:
Let's say you own a 19th century pudding factory (pre industrial revolution.) Many tonnes of bananas are required to make your delicious banana pudding,
but all of those bananas have to be peeled first (an embarrassingly parallel problem). You and your five business partners (a multi-core CPU) are really
smart businessmen and are more than capable of peeling bananas; in fact, you know of the most optimal way to peel a banana and can make complex banana-
related decisions. But the six of you can't peel very many bananas in one day. Even if you peeled one banana in each hand (SMT) or peeled four bananas
in each hand (SIMD) it would still take many weeks to peel all of the bananas. There's just too many damn bananas! So you decide to hire and train 10,000
monkeys (GPUs) to peel the bananas for you. The monkeys aren't very fast, nor are they very intelligent; why, they know nothing of the complexities of
making banana pudding and operating a factory! But with 10,000 of them working simultaneously, they sure can peel the fuck out of some bananas — in
fact, they can peel them all in one morning! If you give the monkeys cocaine* (overclocking) they can work even faster. But exercise caution, as sometimes
the monkeys overheat, pass out from exhaustion (fell off the bus), or stop listening to instructions altogether (ASIC hang.)
* Legal disclaimer: Never give cocaine to 10,000 monkeys without providing adequate cooling.
From this example, it is clear to see why we use GPUs for password cracking. If not, it should at least provide sufficient justification for staffing your Victorian pudding factory with coked-up monkeys.
So which GPU should you use in your rig? Well, once upon a time ATi / AMD reigned supreme in the password cracking world. With more cores than NVIDIA plus special hardware instructions (BIT_ALIGN and BFI_INT) that enabled them to reduce the number of instructions needed to calculate a hash value, ATi / AMD GPUs were easily 4× faster than their rivals. But NVIDIA stole the crown from AMD back in 2014 with the release of their Maxwell architecture (GTX 900 series) and AMD have yet to win it back (and quite likely never will.) So you unequivocally will want to use modern NVIDIA GPUs in your rig.
But which NVIDIA GPUs, you ask? Great question! You want to use desktop GPUs (GTX / RTX), not workstation (Quadro) or server (Tesla) GPUs, even if you are using a server chassis. Why, you ask? You sure do ask a lot of questions. You know I'm not getting paid for this, right? They literally have me chained to a desk in a musty basement, forcing me to write all this for you on an Eee PC 700. I haven't eaten in a month and I defecate through a hole in the chair. Please send help. Oh shit, they're coming; be cool, you know nothing. *Types out loud* ... H A P P Y T O ... A N S W E R ... T H A T Q U E S T I O N ... F O R ... Y O U... Right, so, first reason is that desktop GPUs are an order of magnitude or two less expensive, as desktop GPUs are subsidized by gamers and you have an economy of scale working in your favor. Workstation and server GPUs also have features that you'll never use for password cracking (such as FP64 performance), so you're paying extra for basically nothing. If you're really into giving away assloads of money for absolutely nothing in return, I will happily give you my ACH info. Second, workstation and server GPUs are traditionally slightly slower than desktop GPUs for password cracking workloads. There have been some recent exceptions to this — chiefly the Tesla P100 and V100 — but even though they're faster they cost more than a human liver. We typically measure a GPU's value in terms of performance per dollar and performance per watt. Pick your most-used algorithm (if you're in a corp environment this will likely be NTLM) and divide the hashrate of the GPU by how much it costs. You will see the perf:dollar ratio for Titan, Quadro, and Tesla GPUs are abysmally low, while flagship desktop GPUs (model numbers ending in "80") have the best perf:dollar ratios. Similarly, steer clear of mid-range and low-end GPUs. While the price tag may look attractive, their perf:dollar ratios reveal they're a waste of money.
So you now know to buy high-end NVIDIA desktop GPUs with the highest perf:dollar ratios, but there's still one critical piece of information I haven't yet told you, so DELETE THAT GPU FROM YOUR NEWEGG CART, CRAIG. FUCK. You need to buy GPUs with a blower-style fan and a heatsink with horizontal fins. Do NOT buy GPUs with axial fans and a heatsink with vertical fins. Just because a GPU has two or three fans does NOT mean it will have better cooling than a GPU with a single blower fan — quite the opposite is true. Those flashy multiple-fan GPU coolers actually vent hot air into your chassis rather than out of it, and are designed to handle "bursty" workloads (like gaming) and absolutely cannot cope with sustained compute workloads. Simply put:
Central Processing Units (CPU)
First and foremost, your CPU selection will be primarily driven by your motherboard selection. So pick your motherboard first. There's not even much to say about that, and that's why motherboard doesn't have its own section. Get what supports you need, it's as simple as that.
Many people mistakenly believe that it doesn't matter what CPU you install in your cracking rig if you plan to only ever do GPU cracking. This mentality largely stems from the mining community, where you want to buy the cheapest and shittiest CPU possible to keep infrastructure costs low. But password cracking is not mining. IT JUST FUCKING ISN'T, CRAIG. I DON'T CARE WHAT YOU SAW ON REDDIT. While the CPU you select will be largely driven by your choice of motherboard, there are still some rules that need to be followed.
Generally speaking, your CPU needs to have N + 2 threads available for GPU cracking, where N is the number of GPUs in your rig. For example, if you have an 8-GPU rig, then your CPU needs to be able to support 10 threads. A quadcore CPU with simultaneous multithreading (SMT, such as Intel HyperThreading) only supports 8 threads, which would leave you a bit starved for resources; a hexacore CPU with SMT would be far more appropriate.
If you are using a dual-socket motherboard, pay attention to whether your board is Single Root Complex or Dual Root Complex. Single Root Complex means all GPUs are routed through a single CPU, while Dual Root Complex means half of the GPUs are routed through one CPU, and half are routed through the second CPU. If your board is Single Root Complex, this typically means you can get away with only installing one CPU provided you have enough threads to support the number of GPUs.
Also, keep in mind that while you think you may only ever do GPU cracking, in practice you surely won't. Some algorithms are not amenable to GPU acceleration and you will need to use your CPU. Even if an algorithm is amenable to GPU acceleration, an attack still may be more efficient on CPU (for example, a straight dictionary attack against a fast hash algorithm, or when working against hash lists with hundreds of millions of hashes.) You will also need to do things like wordlist manipulation, ruleset generation, potfile parsing, etc., and you don't want a dog-ass slow CPU for these tasks.
Finally, let's talk brand. Again, this will primarily be driven by your choice of motherboard. But if you have a choice, go with AMD. Intel was king for a very long time, but AMD is finally competitive again for the first time since 2006 and they're absolutely crushing it with EPYC, Threadripper, and Ryzen. If you need help selecting a CPU, check out the High End CPU and CPU Price Performance charts from Passmark. I personally would not consider not buying anything with a CPU Mark score of less than 15,000.
RAM
This will also be primarily driven by your choice of motherboard, but there two hard and fast rules to follow:
Rule #0: You need at least twice as much host memory as you do video memory.
Host RAM ≥ 2 × VRAM
So if you have 8 × RTX 2080 SUPER with 8GB VRAM each, then you will need at least 64GB of host memory. You may be able to get away with less for a little while, but when you start doing more advanced stuff you'll receive CL_OUT_OF_HOST_MEMORY errors.
Rule #1: Always buy reputable, name-brand RAM. Do not buy cheap RAM. Do not buy off-brand RAM. Do not buy refurbished RAM. Do not buy a Ram truck. Do not buy a literal ram.
Storage
Repeat after me:
I do not need multiple terabytes of storage for rainbow tables.
I do not need multiple terabytes of storage for rainbow tables.
I do not need multiple terabytes of storage for rainbow tables.
I do not need multiple terabytes of storage for rainbow tables.
I do not need multiple terabytes of storage for rainbow tables.
It is 2020 and I still get people asking me about rainbow tables at least once a quarter. Rainbow tables are an ancient relic of the past that simply have no place in modern password cracking. Modern password cracking is highly dynamic and requires agility, flexibility, and scalability. Rainbow tables are static, rigid, and not at all scalable - they are the antithesis of modern password cracking. When you look at what the world's most successful password crackers — Team Hashcat, Team CynoSure Prime, Team Radeon 9800 — you'll see that none of them have touched rainbow tables in nearly a decade. So no, you don't need multiple terabytes of rainbow table storage. You don't even need 1 TB of storage for your password cracking rig. Seriously, even 250 GB is probably way more disk than you'll ever use unless you're doing some crazy passphrase research or something.
Many people are under the impression that disk read/write speed and IOPS do not matter for a cracking rig, but I assure you they most certainly do. When possible, NVMe storage is by far your best bet. If not possible, a SATA SSD will work just fine as well. Like RAM, try to stick to the higher-end models from reputable brands.
Operating System (OS)
There's an old computer proverb that says "don't choose your apps based on the OS you use; choose your OS based on the apps you use." I actually don't know if that's a real proverb or if it's just some bullshit I thought of one day and have been repeating for the past decade; either way, it's absolutely true. Keeping this in mind, the clear choice is Ubuntu Linux. Ubuntu is the only Linux distribution that has had consistently excellent OpenCL and proprietary GPU driver support, and Hashcat is developed on Ubuntu as well, which pretty much guarantees that you will have the overall best experience with Ubuntu. Other Linux distributions have varying degrees of OpenCL and proprietary GPU driver support, with bleeding-edge distros like Fedora and Arch being the most painful to use. Kali is also known to have consistently poor OpenCL support.
If you're not comfortable with Linux, Windows is also a vaiable option. But if you do go with Windows you'll need to supplement it with Cygwin, WSL, or similar, as
successful password crackers make heavy use of tools like awk, grep, and sed, as well as quick 'n dirty shell scripts, Perl
scripts, and Python scripts. All of these things are organic to Linux, but you'll have to go out of your way to make them available to you on Windows.
If the thought of Windows is absolutely nauseating to you and you also happen to be a "never Linux" graybeard, FreeBSD and Solaris may be viable options depending on which GPU you use. macOS is unfortunately no longer a viable option, as Apple has deprecated OpenCL support in favor of Metal and they perpetually lag supporting newer GPUs.
If at all possible, your cracking rig should be headless (no monitor.) Cracking passwords and using a graphical desktop at the same time is rather difficult, as your GPU will be too busy crunching numbers to update your display. If you're on Linux and you insist on using a graphical desktop, try running a bare window manager (such as blackbox, openbox, i3, or xmonad) without a desktop manager (such as Gnome, KDE, XFCE, or Mate) to make the desktop load as light as possible.