why my basis trading bot hasn't blown up my account yet

Nine months. Five hundred tests. One 3 AM panic attack. Here's what I actually learned building a delta-neutral crypto bot — and most of it has nothing to do with trading.

Let me skip the part where I pretend I'm some kind of quant genius. I'm not. I'm a software engineer who got tired of reading "I built a crypto bot in a weekend" posts that conveniently end at the part where the work actually begins.

You know the ones. The backtest chart goes up and to the right, the author deploys to a $5 VPS, posts the Medium article, and collects the Twitter followers. What nobody writes about is what happens three weeks later when the VPS reboots mid-trade, or when Binance returns -1111 Precision is over the maximum defined for this asset for reasons that make no sense, or when one leg of a two-leg trade fills and the other one silently doesn't, and you're suddenly long ETH with no hedge while the price is moving against you at 2 AM.

This isn't a tutorial. It's a design post-mortem on a system I've been building for the better part of a year. 520 tests. Eight stages shipped, one left before mainnet. The strategy is delta-neutral basis trading on Binance — long ETH spot, short ETH perpetual, collect funding payments. Nothing novel about it. People have been doing cash-and-carry forever.

What is worth writing about is every decision I made along the way, and why. Because the interesting parts of a trading system aren't the strategy. They're the plumbing.

No code. Code's easy to skim and hard to evaluate. What I want to show you is the reasoning — the specific failure modes each design choice is there to prevent. The things I only understood after getting burned.

The strategy is boring on purpose

Here's the unglamorous truth: I picked basis trading because it makes the problem easier, not harder.

Directional strategies ask "where's ETH going?" The honest answer is nobody knows. You can build all the indicators and ML models you want, you're still fundamentally guessing. Which means the single most important variable in your system — whether the premise is even correct — lives outside your control.

Basis trading sidesteps this. The premise isn't a prediction. It's a mechanism: perpetual futures need to track the underlying asset, and the way exchanges force them to is by making one side periodically pay the other. Every 8 hours on Binance. Usually the longs pay the shorts (contango, happens most of the time in crypto bull markets). You go long spot, short perp, you're market-neutral, and you collect funding.

Target's 10–30% APR from funding alone. Not life-changing. But also not dependent on being right about anything.

Okay, so where does the money actually come from

Concrete example. $10,000 capital per leg on ETH at $3,000:

LONG  SPOT  : +3.333 ETH bought  at $3,000   (notional: $10,000)
SHORT PERP  : -3.333 ETH shorted at $3,005   (notional: $10,015)
─────────────────────────────────────────────────────────────
Net delta   : +3.333 − 3.333 = 0    (market-neutral)

ETH rips to $3,300. Am I rich?

Spot P&L    : +3.333 × ($3,300 − $3,000)  =  +$1,000
Perp P&L    : -3.333 × ($3,300 − $3,005)  =    -$983
──────────────────────────────────────────────────────
Price P&L   :  ≈ $17

Not even close. The spot gain almost perfectly offsets the perp loss. That ~$17 residual is just the basis narrowing slightly. The entire position is numb to price direction. That's the whole point.

What I'm actually exposed to is funding:

Funding_Payment = Notional × Funding_Rate
Funding_APR     = Funding_Rate × 3 × 365 × 100      (3 fundings/day on Binance)

Example: funding rate = 0.01% per 8h
  APR              = 0.0001 × 3 × 365 × 100  = 10.95% annualized
  Per-period       = $10,015 × 0.0001        = $1.00 every 8 hours
  Daily            = $3.00 on $10K per leg

When funding's positive, shorts collect. I'm the short. Money lands in the account.

Break-even is where it gets engineering-y:

Funding_Collected  >  4 × Fee_Rate × Notional  +  Slippage

Four legs of fees (enter spot + enter perp + exit spot + exit perp), plus whatever the market ate while I was trying to execute. That's the whole edge. There's no alpha to discover. The variables that move P&L are execution quality, fee tier, and how tight I can keep the spread. All engineering variables.

So when the hard problem is engineering instead of prediction, the engineering becomes the alpha. Every design decision I make either collects more of that funding reliably, or it leaks it back out through some failure mode I didn't anticipate.

Why I didn't roll my own framework (and why you probably shouldn't either)

Every engineer who's tried to build a trading system from raw WebSocket handlers eventually builds, accidentally and half-assedly, a framework. The state management layer grows organically in response to production bugs. The order reconciliation logic gets bolted on after the first partial fill. The event routing gets refactored three times as requirements become clear. It's a framework. It's just a bad framework that only one person understands.

I used NautilusTrader. Python interface, Rust core, LGPL license. Is it perfect? No. The documentation is thin in places — there are corners where the only way to figure out what the framework expects is to read the source. The Rust layer gives you tracebacks that sometimes point at the wrong thing entirely. The learning curve is real, and the first two weeks I was actively frustrated. Several times I thought, I could build this piece myself in an afternoon.

I was wrong every time. What looked like overhead was actually the framework enforcing constraints I didn't know I needed.

Here's the thing about a good framework in a high-stakes domain: its opinions are the product. NautilusTrader has opinions about how state is managed, how you're allowed to access Redis, how numbers are represented, how events flow. Most of these opinions inconvenienced me at first. Every single one prevented a bug I later saw someone else hit in a forum post.

When a mature framework tells you "no, you can't do that," it's because someone already did that, and it caused a production incident. You're not paying for the code. You're paying to inherit the scars.

The alternative is discovering those failure modes yourself. In a trading system, that involves real money.

Events are just what the market is

Trading systems aren't request-response. You don't ask the market for a price. Prices show up. Orders fill when the book says they fill. Funding events happen on the 8-hour clock whether you're paying attention or not. Mark prices tick continuously in the background.

An event-driven system just matches what's actually happening. A polling system, on the other hand, is pretending. You ask "what's the price?" every 100ms, and between those polls anything can happen. Missed events. Stale state. A liquidation cascade that moves ETH 3% while you're still sitting on the last quote.

I had this mental model wrong for a while. I kept wanting to write code that said "get the current state and decide what to do." That's not how any of this works. The code is "here's what just happened, update my understanding." Different mode of thinking.

The concrete payoff is that time-ordering stops being ambiguous. Every event has a timestamp. They process in order. When something goes sideways later and I have to figure out what the system knew and when, there's a real answer. In a polling architecture that question doesn't have a clean answer, and I've spent enough hours in Slack threads trying to reconstruct "what did prod think was true at 14:32" to never want to do it again.

Cache is the truth. Don't touch Redis directly.

If you want to understand how trading systems fail in ways that don't look like bugs, look for places where the same fact is stored in two different places and those two places can disagree.

I'll give you the war story so this lands. Early on, I needed to peek at a position. The "correct" way was through the NautilusTrader Cache API. But I was debugging, and I just wanted to redis-cli GET the key and move on. Fine. Then, a few days later — this is the part I'm still embarrassed about — I wrote a helper that instantiated its own redis.asyncio.Redis client because the Cache API was returning something I didn't understand and I was impatient.

Three days after that I was staring at a bug I could not reproduce. The system believed a position didn't exist. It did exist. It existed in Redis. It existed on the exchange. My own helper had written to a slightly different key pattern than the Cache was reading from. Two sources of truth. Both internally consistent. Both wrong in different ways.

I ripped the helper out. Changed my rule: never instantiate a Redis client. Ever. If the Cache API doesn't do what I want, I fix the Cache API, or I fix my understanding of what it's doing. The Cache is the plumbing. You don't bypass the plumbing because you're annoyed.

This is the single most important architectural rule in the whole system. It's also the rule I most wanted to violate. Both facts are related.

Broader lesson: when incorrect state has real-money consequences, the architecture needs to make bad state hard to create, not just discouraged. Conventions break under deadline pressure. Enforced constraints hold.

Trading logic doesn't care about your analytics

When an order fills, two things need to happen. The strategy needs to update its view of position state. And the fill needs to end up somewhere durable for later — reporting, reconciliation, performance analysis, end-of-month tax pain.

These are not the same job.

I split them. Strategies hold trading logic — enter, exit, size, manage risk. Actors handle durable writes to PostgreSQL for analytics. If you squint it's CQRS, but I wouldn't lead with that framing because it invites people to over-architect.

The practical reason is simple: the path that touches the market can't depend on anything that isn't essential. PostgreSQL can be slow. It can be down. It can be undergoing a long-running query that locks a table. If any of that propagates into the strategy's event loop, the critical path just got a dependency it shouldn't have.

So now, if the analytics writer throws an exception, the strategy doesn't know and doesn't care. Fills still happen. Positions still close. The analytics backlog gets caught up later, or some rows get dropped, and that's fine because I can reconstruct analytics from the exchange. I can't reconstruct a missed exit.

Testing gets easier too. Trading logic tested without a database. Database writes tested without a running strategy. The boundary's explicit, which means the coupling's explicit, which means I can reason about it.

The general version of this rule: if you've got a critical path and a non-critical path, the boundary needs to be architectural, not a comment that says "careful, this part matters."

Two databases, because they answer different questions

The system runs Redis and PostgreSQL. People sometimes look at that and say "over-engineered." It's not.

Redis answers what's true right now. What positions exist. What orders are open. What the last mark price was. It needs to be fast, simple, and authoritative. On a crash, Redis is what I read to figure out what I was doing before I stopped. It's the hot state. Redis does one job and does it well.

PostgreSQL answers what did this system do over time. Every fill. Every position. Every funding payment received. Queries across weeks. Joins between positions and their fills. Reconciling against the month-end statement from Binance. That's a relational question and needs a relational tool.

Using Redis for analytics would mean writing query engines on top of a key-value store. I've seen that movie. It ends badly.

Using PostgreSQL for hot state would mean my critical path now has a database dependency with all the fun that implies — connection pool exhaustion, vacuum operations blocking writes, the replica being a few seconds behind. Not for hot state. No thanks.

And here's where the split pays for itself: if PostgreSQL goes down, trading continues. The analytics actor logs errors, retries, or drops writes. I'll fix it in the morning. If Redis goes down, trading stops. Immediately. Because trading without the source of truth for position state is how you end up holding a position you didn't know about.

The criticality hierarchy lives in which failures I tolerate. PostgreSQL I can lose. Redis I cannot.

Decimal everywhere. This isn't negotiable.

Okay, story time.

Early in the project, I was running the strategy on testnet. Signal fires. The system calculates a quantity. Submits the spot buy. Rejected: -1111 Precision is over the maximum defined for this asset.

I thought it was a Binance issue. I checked the instrument's tick size — it was 0.00001. My quantity was 0.15. How is that over precision? I spent — and I'm not exaggerating — about 40 minutes debugging this. Reading Binance's filter docs. Checking my symbol lookup. Logging more. Reading more.

It was me.

The quantity wasn't 0.15. It was 0.15000000000000002. I'd computed it from a float, and that's what floats do. Binance's validator is exact. My number had extra trailing garbage after the 5th decimal. Rejected.

That bug cost me a trade. On testnet it cost me dignity. On mainnet it would've cost me the basis window I'd been waiting for, or worse — because if the spot leg had somehow squeaked through and the perp one got rejected for the same reason, I'd have been long ETH with no hedge.

I now use Python's Decimal everywhere. Strings at serialization boundaries. No float anywhere near a price, a quantity, or a rate. It's verbose. Decimal("3500.50") is ugly compared to 3500.50. I don't care anymore. Verbosity is not a cost when the alternative is silent numeric corruption.

And the really insidious part: floating-point bugs don't throw exceptions. A position comparison that returns the wrong boolean because two numbers differ in the 6th decimal — nothing happens. No error. No log. The system silently does the wrong thing. You find out during reconciliation, or you find out when your margin hits a threshold it shouldn't have hit.

Decimal everywhere is just paranoia encoded into types. I'll take it.

The atomic execution problem (aka, the thing that keeps me up at night)

Here's the part I spent the most design time on, and the part I still worry about most.

Entering a basis position means two orders: spot buy, perp sell. They need to both fill. If only one fills, the position isn't delta-neutral. It's directional. In the least forgiving direction — because the signal to enter was chosen assuming the other leg would hedge, so the remaining naked leg is now sitting there in a regime I didn't design for.

Exchanges don't give you transactions. You can't say "fill both or neither." You submit them and you watch. Partial states are real, they happen, and if you don't handle them explicitly you end up with the worst possible version of a position you didn't want.

The failure mode I'm most afraid of isn't a 20% market crash. It's a 30-second window where Binance's API is flaky, spot fills instantly, perp gets a -1003 Too many requests rate limit error, and by the time the retry goes through, ETH has moved 2%. My hedge is gone and I'm long.

So the coordinator tracks every leg's state. Explicitly. If spot fills and perp doesn't, there's a protocol: either retry the perp for a bounded window, or close the orphan spot position and walk away. No third option. No "let's see what happens." I pick one of the two, up front, based on how long it's been and what the market's doing.

There is no retry logic that makes this problem disappear. While the retry is pending, I'm directional. The design has to acknowledge the window and decide what's acceptable inside it. Pretending the window doesn't exist is the bug.

Every box in that diagram is reachable in production. The happy path — "BothFilled" — is one of five outcomes. The other four are where production systems die if you haven't thought about them. Demos only hit BothFilled. That's why demos look easy.

The 5x margin buffer is just paranoia with a number

You can do a calculation for how much margin you technically need to avoid liquidation:

Liquidation_Price  ≈  Entry × (1 + 1/Leverage − Maintenance_Margin_Rate)
Margin_Ratio       =  Equity / Required_Margin

I run at a margin ratio of 5. Not 1.5. Not 2. Five.

Is that overkill? Probably, by the math. Here's why I don't care. The math uses historical volatility. It uses the funding rates that have existed. It uses the worst case you've seen. None of that is the worst case that's possible. LUNA collapsed in a week. FTX unwound in three days. During those moments, I don't want my safety margin to be calibrated against the prior month's volatility.

The 5x isn't derived from a formula. It's derived from the admission that I don't know what's coming, and I'd rather have the bot sitting there doing nothing during an event than have it force-liquidated because I wanted a little extra capital efficiency.

Same principle runs through the defaults everywhere. Starts at $100. 1x leverage. High signal thresholds. Signal-only mode by default — paper trading until you explicitly turn on live execution. These aren't conservative because I'm timid. They're conservative because the first production failure of a financial system is really not the moment to find out that your risk assumptions had an optimism bias.

Designing for the restart, not the uptime

Somewhere along the way I got religion about this. Amateur system design tries to prevent crashes. Real system design assumes crashes will happen and makes the recovery clean.

When the system restarts — for any reason, a bad deploy, a VPS reboot, an OOM kill — it's got Redis state saying "here's what you were doing." But Redis might be stale by a few hundred milliseconds. An event might've been in flight when the process died. A write might've been half-committed.

So on startup, the system reads Redis, then calls Binance and asks "what do you think I have open?" Both answers get reconciled. Three cases:

The exchange is the truth. My local state is an approximation that drifts whenever the process isn't running. After any interruption I need to reconcile or I'm operating on a stale cache and making decisions against imaginary state.

This reconciliation code is annoying. It's slow. It touches the exchange on every startup, which adds latency to every deploy. It's also the thing I'd keep if you took everything else away. Because on the day something weird happens — and there will be a day — the reconciliation is what prevents a bad restart from turning into a catastrophic restart.

Related 3 AM war story: the first margin warning I ever got showed up at 03:47. I was asleep. My phone buzzed with the notification. I had never gotten out of bed and opened a laptop faster. Nothing was actually wrong — funding had just moved a lot and the margin ratio dipped from comfortable to slightly-less-comfortable. But the shot of adrenaline at 3:47 AM is, I think, the most effective motivation for defense-in-depth I've ever experienced. You design differently after you've had that feeling once.

Order of development matters more than I thought

The system was built in nine stages. In order. Data models → market data → entry signals → execution → risk management → exit logic → monitoring → production hardening → mainnet.

Risk management is stage 5 of 9. Not stage 1. That was deliberate and took me a while to internalize.

You can't do risk management against a system that doesn't exist yet. When I tried — early on, feature-driven, building "the whole thing" at once — I ended up designing risk logic against an imaginary execution layer. Then I built the execution layer. It behaved differently from the imaginary one. The risk code had to be rewritten. That rewrite surfaced bugs in the execution layer that only showed up because risk was now looking at it differently. Cascading rework.

The stage order fixes this. Each stage produces something tested, useful in isolation, and a stable foundation for the next. Stage 4 (execution) ships with its own tests before stage 5 (risk) goes anywhere near it. Risk is built against a real thing, not an imagined thing.

The 520 tests aren't a vanity metric. They're what happens when each layer gets tested before the next one is written. If I'd built it all at once, I'd have maybe 100 tests, they'd all test the end-to-end happy path, and every production bug would be a surprise.

Mocks will lie to your face

I don't mock the database in integration tests. Real Redis. Real PostgreSQL. Docker compose spins them up, tests run against them, teardown cleans up.

Mocks are fine when the thing you're mocking is simple and well-understood. The Binance REST client? Sure, mock that. The filesystem? Sure.

The NautilusTrader Cache sitting on top of Redis? Absolutely not. Because the bugs I'm afraid of are at that interface. A mock that implements what I think the Cache does won't catch the case where what I think is wrong. A real Redis will.

Integration tests are slower. They need setup. They break if Docker's having a weird day. All true. They're also the tests that catch the bugs that cost real money in production.

"Prove it before you build it" applies to tests too. A test against a mock isn't proof — it's a rehearsal of your own assumptions. Proof needs the real environment. Or at least the closest approximation you can get.

The lessons that aren't about trading

I've been writing about a crypto bot but almost none of this is specific to crypto.

Design for the worst case you can think of, then add margin for the worst case you can't. Works for trading systems. Works for queue consumers that process irreversible actions. Works for anything where incorrect state has real consequences. Conservative defaults aren't timidity — they're the recognition that production is the wrong place to discover your assumptions were optimistic.

Trust the framework's constraints. When a mature library tells you "no," start by assuming it knows something you don't. Route around it only after you understand what it's protecting you from. Most of the time, "inconvenient" means "you're about to make the mistake this was designed to prevent."

Separate concerns at the criticality boundary. If trading depends on analytics, a slow query takes down trading. If you've got a critical path and a non-critical path, the architecture needs to enforce that, not your discipline.

Build in dependency order. Slow at first. Fast later. Reverse that — go fast first, feature by feature — and you get fast at first and then slow forever, because every new thing has to navigate accumulated mess.

The last one's the one I keep coming back to: the interesting problem is almost never the one that looks interesting. For a trading bot, the interesting-looking problem is finding the edge. The actually hard problem is managing state correctly when things go wrong. The first one's visible and exciting. The second one's invisible until it fails. Real engineering is mostly about taking the invisible problems as seriously as the visible ones.

The bot works. 520 tests green. Stage 8 shipped. I'm about to put real money on it — starting at $100, because of course I'm starting at $100. I have no illusions that I've seen every failure mode yet. Mainnet will surface things testnet can't. That's fine. Every new failure is another piece of the design that needs to exist.

The reason I'm not terrified is that I've spent most of the last year on the parts that don't look like trading. The plumbing. The state management. The reconciliation code nobody ever sees. The tests that feel paranoid until the day they catch something.

Which is, I think, what "prove it before you build it" actually means when you apply it to system design. It doesn't mean testing after the fact. It means every decision in the system can be defended with a specific thing it prevents. If you can't say what a piece of architecture protects you from, it probably shouldn't be there — and the pieces you left out because "it'll probably be fine" are the ones that'll get you at 3:47 AM.