Brokermint Multiple API Keys Scaling Proposal

Problem Statement

A new business requirement asks us to support up to 40 API keys per account under a single BrokerMint integration.

Running all 40 keys sequentially within a single DAG run is not feasible because:

• BrokerMint enforces rate limits on their public API.

• A single API key can return a large number of transactions, and each transaction triggers multiple downstream API calls (details + participants).

• Running all 40 keys back-to-back in one execution window risks:

  • Hitting rate limits and getting throttled / 429 responses.

  • DAG runs exceeding the schedule interval (long-running task with max_active_runs=1 blocking the next run).

  • Increased failure blast radius — one failure can affect the whole batch.

We need a way to distribute the load of 40 API keys across the day while staying within BrokerMint’s rate limits and working cleanly with Airflow’s scheduling model.

Currently, all API keys for the account are processed in one sequential loop within a single DAG execution.

Proposed Solutions

We have evaluated two approaches. Both rely on introducing a new field — run_status — in the ipro_source_settings table to identify which subset of API keys (or which connection) should be processed in a given run.

Option A — 4 Connections × 10 API Keys, Time-of-Day Based (AM / PM Split)

Split the 40 API keys across 4 BrokerMint connections for the same account (10 keys per connection), then process them in two time windows based on whether it is AM or PM.

How it works

• The DAG runs twice a day (one AM trigger, one PM trigger).

• At runtime, the DAG checks the current time (AM vs PM) and selects the connections whose run_status belongs to that window.

• Each selected connection processes its own 10 API keys.

Pros

• Fewer DAG triggers per day (2 instead of 4).

• Clear logical grouping — connections, not the DAG schedule, define the partition.

• Simple operational model: “AM batch” and “PM batch”.

Cons

• More schema and configuration changes — we need to model 4 logical connections per account and ensure each holds its own 10 keys.

• Heavier per-run workload — each run still processes ~20 keys back-to-back (2 connections × 10 keys), keeping rate-limit risk relatively high.

• Less even distribution of API load across the day (two large clusters).

• Higher migration effort: existing single-connection accounts must be split into 4 connections.

Option B — Single Connection, DAG Runs Every 6 Hours, run_status Driven

Keep the existing single-connection structure, but partition the 40 keys into 4 logical groups via run_status (0, 1, 2, 3). Trigger the DAG every 6 hours, and at runtime process only the keys whose run_status matches the current time bucket.

Time-bucket mapping

• DAG schedule_interval = "0 */6 * * *" (every 6 hours).

• At the start of each run, compute the current bucket and load only the API keys (or sub-config) tagged with that run_status.

• Each run handles ~10 API keys, leaving a 6‑hour buffer before the next batch.

Pros

• Evenly spreads load across 24 hours — maximizes safety with respect to BrokerMint rate limits.

• Smaller per-run workload (~10 keys), reducing risk of timeouts and throttling.

• Failures are isolated to a single bucket; the next retry window is only 6 hours away.

• No need to restructure connections — run_status is a lightweight column on ipro_source_settings.

• Highly extensible — we can add more buckets later (e.g., 8 buckets every 3 hours) without changing the connection model.

Cons

• 4 DAG triggers per day — slightly more scheduler activity.

• The DAG must contain time-bucket logic (and be aware of clock skew / DST, if applicable).

• Monitoring and alerting must account for the fact that each run covers only a subset of keys (a single failed run does not imply the entire account failed).

Comparison Summary

Recommendation

We recommend Option B — DAG every 6 hours, partitioned by run_status, because it:

1. Best respects BrokerMint rate limits by spreading 40 API keys evenly over 24 hours.

2. Minimizes schema and migration churn — only a new run_status column on ipro_source_settings is required; no connection restructuring.

3. Reduces blast radius — at most ~10 keys are affected by a single failed run.

4. Scales cleanly — we can introduce more buckets later (e.g., 8 buckets every 3 hours) without re-architecting the integration.