NAV Back-Office Balance Pipeline
NAV Back-Office Balance Pipeline
System Architecture Walkthrough
A job-scheduling and balance-ingestion platform that pulls account balances from 500+ crypto exchanges, blockchains, and equity funds on fixed daily cut-offs — engineered around the fact that most clients want the same cut-off: UTC 00:00.
BASIC + IN-DEPTH · KAFKA · gRPC · REDIS · S3 / RABBITMQ · .NET · GO · PYTHON
| 500+ exchanges, chains & funds covered | 15,000 clients | 66,000 accounts under management | 20,000+ accounts requesting the UTC 00:00 cut-off |
Contents
Part I — Basic Architecture
Part II — In-Depth Architecture
- 2. Detailed Architecture
- 2.1 Job Store
- 2.2 Task Status API
- 2.3 Scheduler
- 2.4 Kafka & Event-Driven Communication
- 2.5 Consumer
- 2.6 Account API
- 2.7 Datafetch Service
- 2.8 Caching Strategy
- 2.9 Data Processor & Reporting DB
- 3. End-to-End Job Lifecycle
- 4. Database Design & Data Flow
- 5. Integration Patterns
- 6. Reliability & Failure Handling
- 7. Performance & Scale Reference
- 8. Confirmed Decisions
Answering the Brief
Every aspect asked for in the original brief, answered in one line with a link to the full treatment:
| Requested aspect | Short answer |
|---|---|
| Overall system architecture diagram | Two levels — basic flow (Fig. 02) for a CTO read, full component map (Fig. 03) for an architect read. |
| Microservices and their interactions | Nine services, each broken down in §2.1–2.9, wired together in Fig. 03. |
| Kafka (message broker) and event-driven communication | Partitions, ordering, and how it differs from the RabbitMQ leg — §2.4. |
| Database design and data flow | Job store schema — §2.1; job store vs. reporting DB and how data moves — §4. |
| Caching layer and caching strategy | Two separate caches — credentials and market data — §2.8 (Fig. 04). |
| Integration between services and supporting components | Every hop, sync vs. async, protocol and why — §5. |
| Architecture diagram illustrating the complete solution | End to end — component map (Fig. 03) + single-job sequence (Fig. 05, §3). |
0. The Core Problem
Exchanges don’t expose historical balance. If we don’t capture a balance at the moment a client’s accounting period closes, that number is gone forever — so balance capture is not a “nice to have” poll, it’s the only ledger entry that will ever exist for that slot. Clients pick their own cut-off (once or twice daily), but when 20,000+ of 66,000 accounts independently pick the same natural boundary — midnight UTC — the system inherits one enormous synchronized burst instead of a smooth trickle.
The architecture below isn’t designed to average that burst away. It’s designed to absorb it: every control-plane hop is sub-second, and the one stage that talks to the outside world (Datafetch) is scaled horizontally wide enough that 20,000 concurrent jobs still land inside a single-digit-second window.
Fig. 01 — where the 00:00 UTC burst spends its time (cumulative time since the schedule tick, for the full 20,000-job burst)
| Stage | Cumulative time |
|---|---|
| Scheduler sweep → Kafka publish | 150–200 ms |
| Consumer dispatch (Kafka → gRPC) | 600–700 ms |
| Datafetch execution complete | 7–8 s |
Data processor time isn’t shown — it drains the RabbitMQ queue asynchronously and isn’t part of the client-facing SLA.
Why this works Everything before Datafetch (job store → scheduler → Kafka → consumer) is internal bookkeeping and finishes in under a second even at 20k concurrency. The actual network I/O — 20,000 simultaneous round-trips to 500+ third-party exchanges — is pushed onto 280 Datafetch pods running concurrently, which is what keeps the external burst inside a 7–8 second envelope instead of a multi-minute queue drain.
…