I Shipped a Race Condition That Oversold Stock (And Other War Stories)

These are bugs I shipped between 2024 and 2026. Most were in Polaris, the retail system I built for shops that bill all day, and one was in an AI pipeline at Entropy Labs.

Two cashiers, one item

Two cashiers at different counters scan the same product. The shop has one left. Both press "Complete sale" within a fraction of a second of each other.

One sale should go through and the other cashier should be told the item is gone. Instead both went through and stock went to -1. The shop had sold something it didn't have, and someone had to explain that to a customer.

The code wrapped the sale in transaction.atomic(), which I had been treating as "safe" for no good reason. Both transactions read quantity = 1, both passed the quantity > 0 check, and both committed. Atomic means all-or-nothing, not one-at-a-time.

The textbook fix is a row lock:

product = Product.objects.select_for_update().get(id=product_id)

That's correct, and at rush hour it meant cashiers staring at a spinner while one sale waited for another. So the lock became select_for_update(nowait=True). The second transaction fails immediately, the app retries it with a short backoff, and the cashier sees a short blip.

A separate version check catches a slower cousin of the same bug: someone opens a product, goes to lunch, comes back and saves over three edits that happened in the meantime.

updated = Product.objects.filter(pk=product.pk, version=product.version).update(
    stock=new_stock,
    version=F("version") + 1,
)
if not updated:
    raise StaleDataError("Someone else changed this product. Reload and try again.")

I keep relearning that transaction.atomic() gives you consistency and does nothing to keep the other cashier out.

A chain that returned nothing, politely

At Entropy Labs I had a LangChain retrieval chain that worked in tests and in staging, and in production returned an empty string with no exception and nothing in the logs.

I checked the model config, the keys and the rate limits, then added logging to every step, and every step ran. The cause was further up: under certain inputs the prompt template rendered an empty message list. The model received nothing, replied with nothing, and the chain passed that along as a successful result.

I put a guard in front of the call:

messages = prompt.format_messages(**inputs)
if not messages or all(not m.content.strip() for m in messages):
    raise ValueError(f"Prompt rendered empty for inputs: {sorted(inputs)}")

A direct API call would have failed loudly on an empty prompt. The framework's helpfulness turned a five-minute bug into a long afternoon. I still use LangChain, but I validate what goes into it and what comes out, and I don't assume silence means success. More on that in LangChain in production.

A lock that only worked on one process

Customer balances in Polaris come from a ledger: every sale, payment and return is an entry, and the balance is derived from them. Two operations on the same customer at once can each read the entries, compute a balance and write a new one, and you end up with a balance that matches neither.

Row locks don't help much when the operation reads many rows, so the ledger code took a PostgreSQL advisory lock per customer, a named lock that isn't tied to any row and is held while the balance work runs. Using an advisory lock was right. The key I built for it was wrong:

lock_id = abs(hash(f"customer_{customer_id}")) % 2147483647
cursor.execute("SELECT pg_advisory_lock(%s)", [lock_id])

Python salts hash() for strings differently in every process, and Gunicorn runs several. So two requests for the same customer, landing on different workers, computed different lock IDs and happily ran side by side. The lock only serialized requests that happened to share a process.

I found it in May 2026 while hardening the refund paths, which is an embarrassingly long time for that line to have lived. The replacement uses PostgreSQL's two-integer form with a fixed namespace, and takes the transaction-scoped variant so the lock can't outlive the transaction:

class AdvisoryLockNamespace(IntEnum):
    LEDGER_CUSTOMER = 10_001
    LEDGER_SUPPLIER = 10_002
 
 
cursor.execute(
    "SELECT pg_advisory_xact_lock(%s, %s)",
    [AdvisoryLockNamespace.LEDGER_CUSTOMER, customer_id],
)

It came with a test that computes the key under different PYTHONHASHSEED values and fails if they disagree. That test looks silly until you remember it would have caught this on day one.

What the three have in common

Each bug lived in the gap between one user on my laptop and many users in a shop. On my laptop there was one process, one cashier and clean data. A shop has several processes and several cashiers, all at the same moment, and the data is messier.

The fixes were all small: fail fast when the row is locked, check the prompt before sending it, and use a lock key that means the same thing in every process. The earlier and smaller mistakes, like refunds that queried once per line item, are in the Polaris performance log.