Table of Contents

Outbox Claim and Lease Design Record

This design record captures the selected direction for multi-worker outbox claim and lease support.

Issue: #407, implemented baseline for #464

Status: Accepted design direction; provider-neutral claim contracts and claim-capable drain/store support implemented. Claim leasing is enabled by default by GovernanceOutboxOptions.UseClaimLeases. Provider-specific stronger atomic claim patterns remain host/provider-owned.

The claim contracts remain explicit and additive, but the drain no longer requires a host to opt into claim leasing. Since #697, a newly constructed GovernanceOutboxOptions uses claim leases by default. Hosts use the legacy non-claiming candidate path only when they explicitly set UseClaimLeases = false.

Context

The provider-neutral outbox drain can read pending or retry-ready entries, emit each envelope to a provider, and then save the resulting delivered, deferred, failed, retryable-failure, or dead-lettered state.

The original non-claiming candidate behavior remains appropriate for a single active worker or a host that partitions workers so each worker reads a disjoint durable outbox slice. It is not sufficient by itself for multiple workers reading the same durable outbox rows. That behavior is no longer the drain default; it is an explicit compatibility opt-out.

The risk is duplicate provider emission:

worker A selects row 123
worker B selects row 123
worker A emits to provider
worker B emits to provider
worker A saves delivered
worker B may fail on concurrency or stale-state checks

Optimistic concurrency can protect the persisted row from some conflicting final-state writes, but it cannot undo a provider call that already occurred. A claim-before-emit strategy is therefore a separate reliability concern from final-state concurrency protection.

Decision

Claim and lease support is treated as an explicit durable outbox capability. The claim APIs remain separate from the selection-only FindPendingAsync and FindRetryReadyAsync APIs, while the default drain uses the claim-capable path because GovernanceOutboxOptions.UseClaimLeases defaults to true.

The implemented baseline direction is:

  1. Keep current single-worker selection APIs supported.
  2. Keep Core claim/lease contracts explicit and provider-neutral.
  3. Use the claim-capable drain path by default; UseClaimLeases = false explicitly selects the non-claiming compatibility path.
  4. Implement baseline claim/lease behavior in the in-memory and EF Core stores.
  5. Continue requiring provider-side idempotency guidance because claim/lease reduces duplicate selection risk but does not create universal exactly-once delivery.

The claim contracts remain additive, but #697 intentionally changed the drain default so cooperating workers claim before provider emission. Hosts that opt out restore the earlier duplicate-selection risk and must provide an appropriate single-worker, partitioning, or idempotency strategy.

Implemented contract shape

The claim-capable contract is additive. It does not replace IGovernanceOutboxStore for hosts that only need local tests, samples, or one active worker.

public interface IGovernanceOutboxClaimStore : IGovernanceOutboxStore
{
    ValueTask<IReadOnlyList<GovernanceOutboxClaim>> ClaimPendingAsync(
        GovernanceOutboxClaimRequest request,
        CancellationToken cancellationToken = default);

    ValueTask<IReadOnlyList<GovernanceOutboxClaim>> ClaimRetryReadyAsync(
        GovernanceOutboxClaimRequest request,
        CancellationToken cancellationToken = default);

    ValueTask<GovernanceOutboxEntry> MarkClaimDeliveredAsync(
        GovernanceOutboxClaim claim,
        GovernanceEmissionResult result,
        CancellationToken cancellationToken = default);

    ValueTask<GovernanceOutboxEntry> MarkClaimFailedAsync(
        GovernanceOutboxClaim claim,
        GovernanceEmissionError governanceEmissionError,
        DateTimeOffset? nextRetryUtc = null,
        CancellationToken cancellationToken = default);

    ValueTask<GovernanceOutboxEntry> MarkClaimDeadLetteredAsync(
        GovernanceOutboxClaim claim,
        GovernanceEmissionError governanceEmissionError,
        string? deadLetterReason = null,
        CancellationToken cancellationToken = default);

    ValueTask<GovernanceOutboxEntry> SaveClaimAsync(
        GovernanceOutboxClaim claim,
        GovernanceOutboxEntry entry,
        CancellationToken cancellationToken = default);

    ValueTask<GovernanceOutboxEntry?> ReleaseClaimAsync(
        GovernanceOutboxClaim claim,
        string? reason = null,
        CancellationToken cancellationToken = default);
}

The important design boundary remains that claim operations are explicit. The drain invokes them by default, while hosts that deliberately disable claim leasing should understand that they are selecting the non-claiming compatibility path.

Claim model fields

Claim support uses separate claim fields rather than a new provider-neutral emission status:

Field Purpose
ClaimOwner Stable worker, process, node, or partition owner identifier.
ClaimedUtc UTC timestamp when the claim was acquired.
ClaimExpiresUtc UTC timestamp when the lease expires and another worker may reclaim.
ClaimToken Opaque value used to verify the same claim owner during final state transition.
ClaimAttemptCount Operational counter for repeated claims/reclaims.

ClaimToken is preferred over relying only on ClaimOwner because worker names can be reused across process restarts. A token allows finalization to prove that the completing worker still holds the same lease instance.

Status model decision

Do not add a provider-neutral Claimed or InProgress status as the first implementation step.

The current GovernanceEmissionStatus values describe provider-neutral emission state: pending, delivered, deferred, failed, retryable failure, and dead-lettered. A claim is a lease over a row, not necessarily an emission state. Keeping claim as separate metadata avoids broadening status semantics and reduces status migration/API churn.

Claim lifecycle

The expected claim lifecycle is:

select eligible pending/retry-ready row
  -> atomically write claim owner, claim token, claimed time, and expiration
  -> emit to provider only after claim succeeds
  -> complete final transition only when claim token still matches
  -> clear claim fields on delivered/deferred/failed/retryable/dead-letter transition

A worker that cannot obtain a claim must not emit that entry.

If a worker crashes after claiming but before completing, the row becomes eligible again after ClaimExpiresUtc. The reclaiming worker must still treat downstream provider emission as at-least-once unless the provider accepts and enforces an idempotency key.

EF Core implementation direction

The EF Core store implements a baseline provider-neutral claim path using optimistic concurrency:

  1. Query eligible unclaimed or expired rows.
  2. Attempt to update claim fields with the current tracked row and concurrency token.
  3. Return only rows whose claim update succeeded.
  4. Complete final state transitions only when the claim owner/token still matches.

This reduces duplicate provider calls when workers respect the claim result and emit only after claim success. It is intentionally not advertised as a universal exactly-once guarantee.

Provider-specific implementations may be better for high-throughput drains:

Provider family Possible host/provider-specific pattern
SQL Server UPDATE TOP (...) ... OUTPUT with UPDLOCK, READPAST, and appropriate transaction isolation.
PostgreSQL SELECT ... FOR UPDATE SKIP LOCKED or UPDATE ... RETURNING patterns.
Cloud queues Visibility timeout or native lease semantics.

These stronger patterns are intentionally provider-specific. They should live in provider-specific packages or host-owned adapters unless a portable EF Core implementation is proven sufficient for the host's workload.

Completion and ownership verification

Claim completion verifies the claim owner/token before writing delivered, failed, retryable, deferred, or dead-lettered state.

A final transition should fail or no-op when:

  • the claim token does not match;
  • the claim has expired and another worker has reclaimed the row;
  • the row is already terminal;
  • the row is not in a status that can be completed by the claim path.

This prevents an old worker from overwriting a newer worker's outcome after a lease expiration/reclaim race.

Delivery guarantees vocabulary

Documentation and API naming should keep the following terms distinct:

Term Meaning
At-least-once delivery The same provider emission may be attempted more than once. Hosts and providers must tolerate duplicates.
Idempotent delivery Duplicate attempts use stable identifiers so the provider or host can collapse duplicates into one effective outcome.
Claim/lease Workers coordinate selection so only a claim holder should emit a given row during a lease window.
Exactly-once delivery A full end-to-end guarantee that a provider observes one and only one effective delivery. AsiBackbone should not claim this generically.

Claim/lease support can reduce duplicate selection risk. It does not by itself prove exactly-once delivery because provider behavior, network failure, transaction scope, and replay/recovery policy remain outside the package boundary.

Compatibility and migration boundaries

Claim support was introduced as an additive API feature where possible. The later #697 default change intentionally made claim leasing the safe default for the drain:

  • Existing selection-only store APIs remain available.
  • Claim-capable APIs remain explicit and additive.
  • GovernanceOutboxOptions.UseClaimLeases defaults to true, so the drain takes the claim-capable path unless the host explicitly opts out.
  • EF Core claim fields require an explicit host migration.
  • Hosts that cannot add claim columns must explicitly set UseClaimLeases = false and accept the duplicate-emission risk of the non-claiming path.
  • The package still does not apply or mutate a host-owned database schema automatically.

The package does not silently change an existing host's deployed database without a host-owned migration.

Documentation outcome

With the initial claim/lease support available, the recommended production guidance is:

  • use one active worker per durable outbox partition when simple operations are preferred;
  • use the default claim-capable drain path when the configured store implements IGovernanceOutboxClaimStore and the host has applied any required schema migration;
  • if claim leases are explicitly disabled, use one active worker, disjoint partitions, or another coordination strategy to address duplicate selection;
  • use provider-side idempotency keys wherever the downstream provider supports them.

This design record records the implemented baseline without overstating runtime guarantees.

Edit this page