Back to blog

MuleSoft idempotency with Object Store v2 — without the multi-worker race

2026-09-17

Idempotency in integration means: the same request (same business key) processed twice must not create two side effects — a second payment, a second order, a second write to the target system. On Mule 4 / CloudHub many teams reach for Object Store v2 as “I’ve already seen this ID.” The catch: a contains → then store pattern is not atomic under concurrency, and on CloudHub / CloudHub 2.0 distributed locking is not available for OSv2.

What this piece is not: a Studio click-path for Object Store, a substitute for a message queue, or a promise that OSv2 replaces Anypoint MQ between apps. It is a race-safe pattern and limits guide for developers / architects protecting hotspots (payments, orders, webhooks) on multi-replica CloudHub.

Below: why contains+store fails under load, pitfall cards (symptom → cause → docs → pattern), TPS/size limits, when to choose MQ, a checklist, and AEO-ready FAQ.

What OSv2 does well (and what it does not promise)

Object Store v2 holds state inside a single application (CloudHub workers / replicas of that app). Enabled by default for Mule 4 on CloudHub; values up to 10 MB; TTL max 30 days; rolling vs static TTL depending on entryTtl (OSv2 FAQ, Using OSv2, OSv2 Overview).

Docs are explicit: OSv2 is not designed for app-to-app communication. To share data between two Mule 4 apps, use an Anypoint MQ queue (OSv2 FAQ — Can an app access another app’s store?).

Pitfall #1: contains → store under concurrency

Symptom: duplicate payments / orders despite an “idempotency key” in Object Store; often only on ≥2 workers / replicas.

Cause: between contains (false) and store, another thread / replica does the same. Classic check-then-act — not atomic.

Docs: Store with failIfPresent=true throws OS:KEY_ALREADY_EXISTS when the key already exists; default failIfPresent=false overwrites (Object Store Connector Reference — Store; Store and Retrieve example). Troubleshooting: OS:KEY_ALREADY_EXISTS = The Mule app tries to store an object, but the object store already has a value for that key (Troubleshooting Object Store Connector).

Pattern (race-safer):

  1. Define a stable business key (e.g. paymentId, Idempotency-Key header).
  2. os:store with failIfPresent=true (first-writer-wins).
  3. On OS:KEY_ALREADY_EXISTS → treat as already processed (On Error Continue / dedicated handler): return prior result or 200/409 per API contract — without repeating the side effect.
  4. Do not treat “contains then store” as an idempotency guarantee.

Honesty note: failIfPresent=true alone does not invent a distributed lock on CloudHub (pitfall #2). It shrinks the race window versus check-then-act and gives a clear “key taken” signal.

Pitfall #2: Multi-worker CloudHub without distributed lock

Symptom: Object already exists for the key / inconsistent values with ≥2 workers; failures in connectors that cache state in OSv2 (e.g. Salesforce Replay Listener, Confluent Schema Registry — Help articles).

Cause: Using Object Store v2 with multi-worker CloudHub applications might result in data discrepancies or key clashes — Distributed Locking is not available on CloudHub and CloudHub 2.0 with OSv2 (OSv2 Overview; Using OSv2 — Synchronize Access).

Connector docs say Store is synchronized at key level and, in cluster mode, across nodes (Store reference). That does not cancel the CloudHub OSv2 limitation — the overview is the source of truth for CH/CH2.

Docs suggest: use a distributed key-value store as a lock to synchronize access to OSv2; they point to Distributed Locking (LockFactory / custom extension / scripting) — useful on on-prem clusters where the Mule lock factory works cross-node. On CloudHub with OSv2, design as if there is no platform distributed lock: single-writer path, external lock (Redis, etc.), or first-writer-wins + idempotent downstream.

Pattern:

  • Idempotency hot path: failIfPresent=true + KEY_ALREADY_EXISTS handling.
  • Connectors with persistent OSv2 cache: one worker or disable persistent cache (per that connector’s Help).
  • Do not assume worker A’s contains sees worker B’s store in the same millisecond without a race.

Pitfall #3: TPS and size limits (quiet 429s)

Symptom: intermittent peak errors; HTTP 429; “Object Store is slow”; values over the limit.

Docs (OSv2 FAQ):

Limit Value
Value size 10 MB (no limit on key count / total store size)
TPS base 10 TPS per app
TPS premium add-on 100 TPS per app
Key size max 1024 bytes UTF-8
TTL max 2592000 s (30 days)
CloudHub keys `

Every API call (connector or REST) counts toward TPS. Exceeding base without the SKU can pause requests with 429.

Pattern: do not use OSv2 as a per-request cache on a hot path without a TPS budget; batch / local TTL cache where consistency allows; watch Usage Reports; premium when you truly need 100 TPS.

Pitfall #4: OSv2 as a “bus” between applications

Symptom: app A writes, app B reads via REST “because it works”; coupling, TTL surprises, no queue semantics (ack, DLQ, competing consumers).

Docs: you can use the Object Store REST API to store/retrieve from another app, but OSv2 is not for app-to-app — to share data between Mule 4 apps use Anypoint MQ (OSv2 FAQ).

Pattern: OSv2 = idempotency / watermark / state inside an app. MQ / broker = cross-app messaging and durable queues (especially after CH2 without persistent VM queues — see the CH2 migration article).

Pitfall #5: TTL and the “vanishing” idempotency key

Symptom: after ~30 days (or sooner with static TTL) the same business ID is accepted again as “new”.

Docs: max TTL 30 days; rolling TTL (omit entryTtl, Mule ≥4.2.1) vs static TTL (set entryTtl); entryTtl="0" is not rolling — it switches to static (Connector reference — TTL; Configure custom TTL).

Pattern: for idempotency keys set an intentional static TTL matching the business window (e.g. 7–30 days); document that reprocessing is possible after TTL; long-term audit → database / data lake, not OSv2.

Mini decision tree

Need Choice
Dedup / “I already processed this ID” in one app OSv2 + failIfPresent=true + KEY_ALREADY_EXISTS handler
Multi-replica + strong write serialization Assume no CH distributed lock; external lock or single-writer + idempotent target
Cross-app messaging / competing consumers Anypoint MQ (not OSv2)
High-RPS response cache Not OSv2 base 10 TPS — Cache scope / external cache
Audit > 30 days Persist outside OSv2

Checklist (order)

  1. Define the business key (stable, ≤1024 B, no | on CloudHub).
  2. os:store with failIfPresent=true; handle OS:KEY_ALREADY_EXISTS as the dedupe success path.
  3. Remove contains→store as a “guarantee”.
  4. Budget TPS (base 10 / premium 100) and value size (≤10 MB).
  5. Set TTL deliberately (static vs rolling).
  6. Multi-worker: plan for no distributed lock on CH/CH2; review connectors with OSv2 cache.
  7. App-to-app → MQ, not OSv2.
  8. Test: parallel requests with the same key on ≥2 replicas.

FAQ

1. Is contains then store enough for idempotency?

Not under concurrency — that is check-then-act. Prefer store with failIfPresent=true and handle OS:KEY_ALREADY_EXISTS.

2. What does OS:KEY_ALREADY_EXISTS mean?

Store with failIfPresent=true when the key already exists. Use it as “already processed”, not as an unexpected crash.

3. Does Object Store v2 have distributed lock on CloudHub?

Overview docs: distributed locking is not available on CloudHub / CloudHub 2.0 with OSv2 — multi-worker discrepancies / key clashes are possible.

4. What are the TPS and size limits?

Value ≤10 MB; base 10 TPS/app; premium add-on 100 TPS/app; key ≤1024 B; TTL ≤30 days (OSv2 FAQ).

5. When should I choose Anypoint MQ instead of OSv2?

When you need cross-app communication, queue semantics, competing consumers — OSv2 is in-app state storage, not a bus.

6. Is OSv2 suitable as a high-RPS cache?

At base 10 TPS — usually no. Over limit → 429. Prefer Cache scope / external cache.

7. What about TTL for idempotency keys?

Max 30 days. After expiry the same ID can enter again. Match static TTL to the business window; keep long audit outside OSv2.

8. Can the OSv2 REST API read another app’s store?

Technically yes, but docs discourage app-to-app via OSv2 — use Anypoint MQ.

9. How do I test the race?

Send N parallel requests with the same idempotency key to an app with ≥2 replicas; expect one side effect and a controlled KEY_ALREADY_EXISTS path.

Soft CTA

Designing payment / order idempotency on CloudHub (multi-replica) and want a review of OSv2 vs MQ and TPS limits before production catches a race? Solita is a Nordic MuleSoft partner with delivery from Poland (EU-shoring) — we help shape first-writer-wins patterns and OSv2 boundaries. No “#1” claims and no marketing checklists.

Sources

Documentation

Help (multi-worker adjacency)