For a Better, More Open Future...

Overview

For the past few months now, our public threat intelligence feed has been using an unsuitable hashing algorithm that enabled attackers/malicious actors to reverse the hash. Today, we are changing that.

TL;DR:

Instead of using the old SHA256 hash matching code, please migrate to Argon2id. Here is a sample of the code change required (in Python):

Before:

import hashlib

def check_match(old_hash, input_data):
    input_hash = hashlib.sha256(input_data.encode()).hexdigest()
    return input_hash == old_hash

After:

import argon2
from argon2 import PasswordHasher

ph = PasswordHasher()

def check_match(new_hash, input_data):
    try:
        ph.verify(new_hash, input_data)
        return True

    except argon2.exceptions.VerifyMismatchError:
        return False

Please refer to the Argon2id documentation for more details on installation and usage. This example is provided with the argon2-cffi Python package.

So, why?

SHA256 has long been considered a secure hashing algorithm. However, with advancements in computational power and techniques, it has become feasible for attackers to reverse-engineer SHA256 hashes, especially when combined with techniques like rainbow tables. This poses a significant risk to the privacy and confidentiality of our threat intelligence data.

Discord invites, unfortunately, have very low entropy, making them particularly vulnerable to such attacks. Brute-force attacks can easily guess the original invite from its SHA256 hash in a matter of seconds to minutes.

To clarify, SHA256 is not broken. It is still a mathematically secure hashing algorithm. However, it is not suitable for our specific use case due to the low entropy of the data being hashed.

Entropy Explained

Entropy is a measure of randomness or unpredictability in data. High-entropy strings are complex and difficult to guess, while low-entropy strings are simpler and more predictable.

# Example of a high-entropy string
"XyZ!9@#aBcDeFgHiJkLmNoPqRsTuVwXyZ1234567890"

# Example of a low-entropy string
"abcde" <- Discord invite codes are similar in entropy to this

Traditionally, Discord has used a 8-character alphanumeric code for invites, which results in a limited number of possible combinations (aside from vanity URLs). This limited keyspace makes it feasible for attackers to generate and test all possible combinations against the SHA256 hashes.

So, although SHA256 is cryptographically secure, our specific use case does not provide enough complexity to prevent reversal attacks.

The solution

We are switching to Argon2id, a modern hashing algorithm specifically designed to be resistant to brute-force attacks. Argon2id incorporates memory-hard functions, making it significantly more challenging for attackers to reverse-engineer hashes, even with substantial computational resources.

With this change, it would make brute-forcing computationally infeasible within a reasonable timeframe for such a piece of information. No realistic computer/cluster today is able to reverse the hash into its original invite in any feasible timeframe.

Why was SHA256 used initially?

When we first launched our public threat intelligence feed, we had considered computing resource constraints on our side. SHA256 was extremely fast to compute, which made it possible to scale the dataset to an enormous size without incurring significant computational costs. However, as our infrastructure has improved and the number of entries were not as high as our initial anticipations, we decided to prioritize security over speed.

Why Argon2id now?

Argon2id uses an independent approach for the first pass (to secure against side-channel attacks) and a data-dependent approach for the second pass (to secure against GPU cracking attacks). This hybrid approach provides a balanced defense against various attack vectors, making it a robust choice for hashing sensitive data like Discord invites. This algorithm is also the winner of the Password Hashing Competition (PHC) and is widely used today as a password hashing algorithm.

The core process

• Memory-Hardness: Unlike traditional hashing algorithms, Argon2id fills the memory with a matrix of pseudo-random data. An attacker would need to allocate a certain amount of physical memory for each guess. This is what makes large-scale reversal attempts impractical.

• Time Cost: It performs multiple passes over that memory, with computing time increasing with each pass.

• Parallelism: Argon2id can be configured to use multiple threads, making it efficient on modern multi-core processors while still being resistant to brute-force attacks.
  In a nutshell, Argon2id's memory requirements renders the technologies of today (GPUs, ASICs, etc.) useless at cracking these hashes due to high resource requirements.

What changed on our end

"So you've just said it's computationally expensive for your systems, what changed?"

Before this change, we had a mechanism to create a hash every single time the dataset was updated. This meant that everyday, we would re-hash all invites using SHA256 and update the dataset. This would be fine, if we were to continue using SHA256. With Argon, this would cause a significant load on our backend servers daily during the update, potentially slowing down our processing of messages.

To mitigate this, we are now creating a hash during data ingestion. This means that when we first see a new invite, we create an Argon2id hash of it and store it in our database. During dataset updates, we simply fetch this hash from our database instead of re-computing it. This effectively spreads out the workload over time, rather than concentrating it during updates.

Before:

# This runs during a dataset update
async def update_dataset():
    invites = await db.fetch_all_invites()
    for invite in invites:
        invite_hash = hashlib.sha256(invite.encode()).hexdigest()
        store_in_dataset(invite_hash)

After

# This runs whenever we see a new invite
async def ingest_invite(invite):
    invite_hash = ph.hash(invite)
    await db.store_invite_hash(invite, invite_hash)

# This runs during a dataset update
async def update_dataset():
    invites = await db.fetch_all_invites()
    for invite in invites:
        invite_hash = await db.get_invite_hash(invite)
        store_in_dataset(invite_hash)

These code snippets do not accurately represent our internal systems, but are simplified versions to illustrate the change.

Conclusion

We believe this change is crucial to maintaining our goal of providing an accessible and confidential threat intelligence feed. By adopting Argon2id, we are significantly enhancing the security of our dataset and protecting the privacy of our users.

We will continue to make improvements to our systems and processes. Thank you for your understanding and support as we roll out this important update over the next few days.

Best regards,

The Cockatoo Team