Fetch, Coverage, and Sync: agreed design

Status: approved direction from the session Q&A; implementation pending unless explicitly marked complete in the implementation plan. The architecture document describes the current implementation.

1. Decisions

Compare first; publish eligible confirmed-missing copies only on explicit Sync.
Prefer NIP-77 reconciliation; use ordinary Nostr timestamp paging as fallback.
After initial backfill, Fetch gets newer events by default with a ten-second overlap. Recheck history is a separate action. Resume continues unfinished work.
Preserve per-fetch/per-relay range history rather than overwriting one global relay status. Coverage primarily displays ranges, counts, and observation times.
Range Refetch/Resume actions belong in a compact ⋮ menu.
Fetch automatically updates Coverage and the per-relay Sync tree. This never automatically publishes events.
Sync defaults to all eligible confirmed-missing archived events on configured write relays; users may narrow by relay, kind, or date.
Publication acknowledgement is Accepted; successful read-back is Verified. Timeout/inconclusive verification is Unverified, distinct from Rejected.
Ordinary fetch defaults to 100 events per page, adjustable in fetch settings. The effective limit is saved with the run and reused on Resume.
Keep query bounds, request/completion timestamps, and EOSE/NOTICE/rejection/ timeout/disconnect outcomes. Age alone does not erase presence observations.
Specialized-relay kind-filter fallbacks and hostname-specific behavior are postponed. There is no planned special case for purplepag.es.

2. Shared flow and relay roles

Account
   |
   +-- configured NIP-65 read/write relays
   +-- extra fetch sources
                  |
       write relays + extras
                  |
           Fetch / Back up
                  |
       deduplicated local archive
                  |
      per-fetch range evidence/history
                  |
       Coverage + Sync candidates
                  |
          explicit Sync action
                  |
     configured write relays only
                  |
       Accepted -> read-back -> Verified

The local read/write list may differ from its published NIP-65 snapshot during a migration. Existing Reset to published relays and Publish NIP-65 remain separate local-replacement and signing/publication actions. Read-only relays are excluded from authored-history fetching/sync unless separately added as an extra source. An extra source never becomes a sync destination merely because it was queried.

3. Vocabulary and evidence

Found: distinct event IDs established on a relay in this observation's scope.
Fetched: found events whose original signed bodies are validated and archived.
Not fetched: identified remote event IDs still awaiting a validated body. It does not mean events missing from the relay.
Newly archived: bodies first inserted into the shared archive during the run.
Received deliveries: raw repeat deliveries across pages/retries, shown only in details; these are not unique Found or newly archived counts.
Missing copy: an eligible local event absent from a completed, applicable relay comparison. Used by the Sync tree, not as the headline Coverage vocabulary.
Undetermined: insufficient applicable evidence to establish relay absence.
Invalid observation: malformed/future observation metadata, reported separately. This is distinct from an event's created_at and from an incomplete network query.

Signature validation remains mandatory for archive bodies and read-back evidence. Reconciliation can identify remote IDs before their bodies are downloaded; display Found separately from Fetched while that transfer is pending.

Historical facts retain their timestamp and provenance. A failed query does not erase prior successful observations. Present is not a promise of permanent relay retention. A completed observation can update earlier evidence for the same scope; an incremental query does not alter conclusions about older, unqueried ranges.

4. Fetch modes and checkpoints

State is independent for each account/source relay, not one global account cursor. Snapshot source URLs, filters, chosen page limit, and run boundary when starting. Adding a source gives it its own initial-backfill state; edits do not retroactively change an existing run's query scope.

Initial history and fallback paging

Use a fixed run-start timestamp T as the first upper bound. Initial history has no lower bound: Beginning represents an open historical bound, not an invented timestamp for the author's first event.

REQ { authors: [account], until: T, limit: 100 }
                    |
         archive verified returned events
                    |
             literal EOSE arrived?
             /                  \
           no                   yes
           |                     |
     partial/error       page shorter than effective limit?
     keep checkpoint           /                  \
                             yes                  no
                              |                    |
                  history queried to end      sort by created_at
                  under relay semantics      earliest timestamp E
                                                   |
                               next REQ until=E (inclusive)
                               deduplicate repeated boundary IDs

Use the inclusive earliest timestamp to avoid silently skipping events sharing that second. Persist boundary IDs/progress sufficient to detect repeated pages. If paging stalls at a saturated timestamp, leave the scope partial and report the reason; do not advance to E - 1 and pretend the skipped bucket is complete. The normal strategy is backward until paging, replacing the current default recursive window splitting.

A short page with EOSE is a completion signal under the relay's effective query limits, not proof of an immutable/global earliest event. Record the requested limit and completion basis. Relay caps, filters, and retention can limit recovery; reject inconclusive evidence rather than using it to establish Missing.

Incremental fetching

After initial history completes:

since = last successful forward boundary - 10 seconds
until = current run's start timestamp
limit = run's saved page size

Page backward within this fixed interval if needed. Advance the successful forward boundary only after the entire interval finishes. Preserve since, fixed until, current paging cursor, and page progress on failure/interruption. Resume uses those saved values, not a new run boundary or changed settings. Newer activity can be fetched in a subsequent run after the resumed snapshot completes.

Store history_complete separately from the last successful forward boundary. Completed historical work does not mean future Fetch work is unnecessary.

Refetch and Recheck history

Old signed events can be added to a relay later with their old created_at intact. An incremental timestamp query cannot discover every such import. Recheck history revisits older scopes; a range's Refetch revisits that exact scope. Both create new observation attempts, retaining the earlier results. Resume appends query attempts to the same unfinished logical range/run.

5. NIP-77 preferred strategy

NIP-77 compares timestamp/ID sets for a standard filter:

local matching set <-> NEG-OPEN / NEG-MSG <-> relay matching set
                           |
              remote-only IDs -> REQ -> validate/archive
              local-only IDs  -> candidate missing copies
              common IDs      -> relay presence evidence

NIP-77 discovers differences; it does not transfer event bodies or publish them. Fetching from an extra source consumes remote-only events without uploading local- only events. Sync candidates are computed only for configured write destinations.

Use the same logical ranges, history records, and UI for both strategies. Preserve method and outcome per attempt. Advertised NIP-11 support may guide strategy choice; runtime negotiation is authoritative. Unsupported negotiation uses the paging fallback; timeout/NEG-ERR remains recorded and cannot establish absence. Large reconciliation scopes may require bounded subscopes; their completion must be explicit, not inferred from one successful fragment.

Full-range rechecking can discover newly imported old events without redownloading common bodies. Recent default fetching remains incremental as agreed. Reconciliation of a recent interval must not update older history outside that filter.

6. Coverage UI

Coverage includes fetch-source relays (write relays plus extras), grouped by relay and logical range. Sync separately restricts destinations to write relays.

COVERAGE
v Relay A
  > Beginning -> DATEX   Found (n)   Checked T1   Complete   [⋮]
  > DATEX -> DATEY       Found (m)   Checked T2   Complete   [⋮]
  > DATEY -> DATEZ       Found (k)   Checked T3   Partial    [⋮]

Expanded range
  Fetched n / Found n
  > kind 1 (count)
  > kind 7 (count)
  Strategy: NIP-77 / ordinary REQ
  Query bounds / cursor / chosen limit
  Observation attempts and query outcome details

Menu: Refetch for an existing range; Resume for unfinished ranges. Disable actions while that range is active. Failed/partial scopes remain visible, not represented as complete zero-result ranges. Complete means the comparison/query scope finished; show any remaining Not fetched bodies separately.

Display overlapping timestamp endpoints accurately. Inclusive boundary deliveries are deduplicated by ID within an observation; summing overlapping range counts must not be presented as a unique account-event total. The archive count is unique IDs across all runs and relays.

7. Comparison and Sync

Automatically recompute candidates after relevant fetch/comparison evidence is committed. Do not require a second Compare button. A scan on one relay says nothing about an unqueried relay. Newly archived events may be compared against applicable completed inventories from other relay/range observations; keep the evidence's observation time visible. Coverage is historical evidence, not a simultaneous snapshot of the whole network.

Only absence established by a completed applicable comparison becomes a candidate. Not returned by a limited/partial page is insufficient. Absence is neither created by age nor by connection failures. A limited IDs query likewise cannot establish absence if the response may have been truncated.

SYNC                                  total missing copies
v Relay A (count)
  > kind X (count)
  > kind Y (count)
v Relay B (count)
  > kind X (count)

[Sync missing events]

One event missing on two relays is two deliveries. Default scope is all eligible confirmed-missing copies; optional relay/kind/date narrowing includes collapsed branches. Preserve the exact selected candidate/scope snapshot with the sync run.

Send original signed events without re-signing or changing IDs/timestamps. Keep delivery outcomes distinct: Pending, Accepted/already stored, Rejected, Verified, and Unverified with reason. Read back IDs from the corresponding destination, validate responses, and append positive evidence only for bodies actually found. An acknowledgement alone is not Verified. Resume retries unfinished delivery/ verification work; it does not blindly resend verified copies.

Archive and replay eligibility differ: preserve history locally but avoid replaying expired/deleted events, ephemeral/private events, or superseded replaceable events. Honor deletion requests and latest replaceable/addressable versions before forming candidates. This does not implement new deletion-request creation workflows.

8. Persistence direction

Reuse the existing events archive and operations run identities. Add normalized range/query/evidence records where provenance and scoped comparison need indexes; retain JSON for checkpoint/settings payloads rather than a large opaque history blob. Exact migrations/types are implementation-phase work, but these responsibilities are required:

Record	Responsibility
Per-account/relay fetch state	History completion and last successful incremental boundary
Fetch range/attempt	Run, relay, filter/bounds, strategy, settings snapshot, status, counts, timestamps, checkpoint; prior attempt link for Refetch
Query log	Range, exact REQ/NEG scope, sent/completed timestamps, limit, EOSE/NEG completion, NOTICE/error/timeout/disconnect, received/unique counts
Range-event evidence	Range/attempt/query provenance and found/common/local-only IDs; body availability through the shared archive
Current comparison projection	Latest applicable positive/negative evidence by event/relay, with provenance; rebuildable cache, no age TTL
Sync deliveries	Run/event/relay, attempts, acknowledgement/rejection, verification result/time and error

Reconciliation may identify an ID before an archive body exists; an evidence ID must therefore not require a body row prematurely. Failed attempts append outcomes without overwriting earlier evidence. Use query completion times/scopes rather than job-finalization time to order concurrent evidence accurately.

9. Migration and current behavior

The current implementation still uses 500-event split windows, operation-level pendingWindows JSON, one-hour presence expiry, and a global Coverage table. It does not yet implement NIP-77, incremental cursors, per-range Coverage, or Sync.

Keep existing accounts, signed events, publication versions, and operation history. Legacy positive observations can be retained with legacy provenance. Do not invent per-query bounds/completion timestamps absent from old data, or reinterpret prior short-page completion as a fully verified NIP-77 inventory. Preserve old partial checkpoints explicitly; provide a supported legacy-resume path or clearly marked new scan without losing archived events. Never silently reinterpret a 500-event checkpoint as a 100-event/NIP-77 checkpoint.

10. Deferred work

Specialized-relay kind-filter fallback and related capability policy.
Individual selection, Select all/Clear selection, and JSONL export/import beyond the narrowed sync scope required above; discussed earlier, not settled by this Q&A.
New deletion-request workflows, media-file backup, and unattended scheduling.
Relay authentication and persisted preferred-signer references remain existing follow-ups; do not confuse authentication failure with a completed empty scan.

Implementation milestones and exit criteria live in the implementation plan. This document records the agreed direction; its existence does not imply the runtime already implements it.