Repository navigation
Tags
Tags
Filter no-op notifications for whitelisted dot-files and excluded tra… …cked files; bump to 1.2.7 (#222) * Filter no-op notifications for whitelisted dot-files and excluded tracked files The managed watcher drops a file notification when the published state shows that the file's trusted metadata and visibility are unchanged. Two gaps let such notifications invalidate the view and clear `ready` (#221): - Dot-files that an ignore-file whitelist admits. The filter compared the raw hidden status with the recorded one, which the whitelist had already adjusted. The walk now reports the whitelisted entries, and the filter compares raw status with raw status. - Tracked files that ignore rules exclude from the walk. They had no evidence. Each pass now records their trusted version and raw hidden status, and the filter drops a notification only when both are unchanged. Ignore and attributes files are never recorded, so their notifications invalidate exactly as before. A missing directory skips its tracked descendants without a stat each. Notifications for paths without evidence or a record still invalidate. That includes new files and untracked ignored files. Diagnostics are additive: - `watch_events.unchanged_excluded_filtered`; - `watch_events.unfiltered`, which counts why notifications were kept; - `last_reconcile.excluded_tracked`; - `last_reconcile.timings.excluded_records_us`. Nothing persisted changes. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Bump version to 1.2.7 Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Charge excluded-record scratch before allocating it excluded_records charged the listed-path set and the candidate list to the caller's charge, which held both until publication resized it or the pass ended. It charged the candidates only after allocating them, and by length. The directory stack and the per-directory name lists were not charged at all. A sparse checkout with many absent tracked paths could therefore exceed the private-memory reservation while underreporting peak_private_bytes. Scratch now has its own charge, taken before each allocation and released when the buffers drop: - The listed set is charged with an upper bound on its table, and released as soon as the candidates are known. - Candidates are counted first and allocated exactly. - The directory stack grows through a helper that charges the replacement storage while the old storage is still charged. - The per-directory name lists are gone: each directory entry's name is looked up by binary search in the directory's sorted candidate range. The caller's charge now grows only by the records. Records, exact-spelling matching and skipped absent subtrees are unchanged, and every entry visit still checks for cancellation, including the counting pass. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Require ignore rules to prove exclusion before recording excluded_records recorded every present tracked file that the walk had not listed, as if ignore rules excluded it. A tracked file restored after the walk is unlisted too, so it could get a record. If its create notification was missed, a later Modify that left it unchanged was dropped as unchanged, although the published state did not contain it. A candidate now gets a record only when the ignore rules the walk applied exclude it or a parent directory: - The metadata walk returns the point-query matcher it already builds for visibility, and a controlled walk keeps its input control alive with it. - When the walk built no matcher, one is built from the walk's rule files, only once a candidate is found on disk. - Under a permit, as managed views always walk, both matchers parse the rule-file contents the walk read through its input control, so nothing is read again and a rule changed since the walk proves nothing. A rebuilt matcher leaves out the case-insensitive tracked-file layer, which only adds exclusions. The input control's charge for those snapshots and patterns is now held until the records are built, then released. Tracked files that the walk skipped only for other reasons get no record, so their notifications keep invalidating. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Prune whitelisted hidden paths on unchanged verification This PR keeps the raw hidden status of whitelisted files beside the published visibility, so that a no-op notification for a whitelisted dot-file is dropped. A verification that found the publication current refreshed the evidence but kept that set as published. On Windows, a file can lose both its hidden attribute and the whitelist that admitted it without a change to its visibility or its file version, which has no change time. If neither change was notified, verification found nothing to publish and kept the stale entry. The notification for setting the attribute again then looked unchanged and was dropped, so the file stayed visible to queries without --hidden until a later pass or verification. v1.2.6 invalidated in that case. Elsewhere the raw hidden status depends only on the name, so changing it is a rename, which verification sees as a membership change. Unchanged verification now keeps only the entries that its own walk still reports as hidden and whitelisted. It prunes the set rather than replacing it, as it does the excluded records, because retained view memory is charged for the published set. A whitelisted file hidden since the last publication therefore has no entry and keeps invalidating until a pass publishes. Tests: - unchanged_verification_prunes_whitelisted_entries_its_walk_no_longer_reports (Windows) removes a file's hidden attribute and its info/exclude whitelist without a notification, verifies, sets the attribute again and expects HiddenChanged. - unchanged_file_drops_whitelisted_hidden_files now also checks that an unchanged verification keeps the entries its walk still reports. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Prove excluded-file records only with rules the walk applies An excluded-file record lets the watcher drop an unchanged notification for a tracked file the walk did not list. A record is made only when the pass's matcher proves the file excluded, so the matcher must not exclude a file the walk would list. It could in three cases. Below a nested repository, the walk stops applying the outer repository's `.gitignore` and `info/exclude` rules, while the matcher still applies them. A tracked file there that was absent during the walk and restored before the records were built could get a record although the next walk lists it. If its create notification was missed, a later notification that left it unchanged was dropped, although the published state did not contain the file. Records now skip every directory below the root that holds an entry named `.git` or `.jj`, of any type and in any ASCII case, or whose listing fails partway. The walk treats a directory as another repository only when a lookup of `.git` or `.jj` there finds an entry, and never with `--no-require-git`. Unless the file system equates another name with those beyond ASCII case, this skips at least every subtree the walk treats as one, and skipping only removes records. A listing can return the marker after other entries, so the records and queued subdirectories already taken from that directory are then released. Neither exclude file proves exclusion any more. The metadata walk never sets the walker's working directory, so it matches global patterns relative to the process's working directory, while the matcher roots them at the view root. An anchored global pattern could therefore exclude a file in the matcher that the walk lists. Behind a `.git` file, the walk applies the repository's `info/exclude` only when Git is required and the pointer, resolved against the process's working directory, leads to a Git directory with a `commondir` file. The matcher resolves a relative pointer against the directory holding `.git`, and takes a Git directory without `commondir`, such as a submodule's or a `--separate-git-dir` checkout's, as its own common directory, so it applied `info/exclude` where the walk did not. Both files have the lowest precedence of the standard rules, and the p4ignore and case filters don't read the standard decision, so leaving them out of the proof only removes exclusions and stays safe however the walk resolves them. When Git ignores case, the case filter still matches `info/exclude`'s rules, but it never hides a tracked file, and the walk applies the same filter. Tracked files that only an exclude file excludes get no record and keep invalidating, as in 1.2.6. Outside nested repositories, records proven by `.gitignore`, `.ignore` or the p4ignore filter are unchanged, as are the walk, visibility and whitelists. The tests remove tracked files before the walk and restore them before the records are built, with the walk's matcher and with a rebuilt one, an injected matcher whose only proof is global, a daemon whose global excludes file holds anchored and unanchored patterns, and `info/exclude` behind a `.git` directory, a separate Git directory and a linked worktree's relative pointer. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Keep admitted work running when stop-if-idle finds the namespace busy… …; bump to 1.2.6 (#219) * Keep admitted work running when stop-if-idle finds the namespace busy A stop-if-idle probe closed catalog admission and tested activity.lock before it counted protected work. Jobs of operations accepted before the probe then failed with busy/namespace-admission-closed even when the probe decided not to stop. Count catalog work and close admission in one transaction, and close only when no work is visible. Acceptance checks admission in its own write transaction, so the count includes every unfinished operation accepted before the closure. A probe that sees unreleased leases or owners, reservations, unfinished operations, receipt readers or external work now refuses the stop without closing admission or testing activity.lock, discards only expired verification, and records the decision through the same receipt path. The closed path is unchanged: the final all-category count and activity.lock test follow the closure, and only a committed stop ends in-flight work. namespace_readers_or_work is now observed only after admission closes and is false when the probe refuses before closing. Fixes #218 Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Bump version to 1.2.6 Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Prioritize managed work, bound WAL checkpoints, cut scoped query cost… …; bump to 1.2.5 (#217) * Prioritize managed work, bound WAL checkpoints, cut scoped query cost Addresses #212, #214 and #216. #212: scheduling and starvation - Replace the managed daemon's FIFO job dispatch with a bounded, aged job queue with foreground, maintenance (collection) and background (serving verification, automatic advancement) classes. A free worker runs the job whose queue time plus class delay (0 s, 5 s, 10 min) is earliest, so catch-up reconciliation no longer waits behind background work and no class starves. When a view loses readiness, its queued work moves to the foreground class and keeps its queue time. - Back off exponentially when recovery or collection continuations make no progress, instead of counting verification-only activity as productive. - Report queued and running jobs, collection activity and each view's queued work in status, distinguishing accepted, queued and running work. - Run each parallel query window on a dedicated rayon pool (tgrep-search-N), so long reconciliation jobs on the global pool no longer stall searches. #214: WAL growth - Trigger the pre-mutation catalog checkpoint at min(metadata_bytes / 4, 64 MiB). Only the trigger changes; the checkpoint itself, synchronization and crash-credit ordering are unchanged. The trigger does not cap WAL size while readers pin frames. - Document resumable collection verification and how to tune collection budgets safely. #216: scoped queries, verification cost and readiness churn - Restrict indexed candidates to the queried scope before processing paths, check overlay precedence once per selected file, and walk only the matching prefix of the ordered membership set for shared --files listings. - Filter --files listings by glob and type in the daemon. - Let managed v2 clients derive their attachment from the gitfile, commondir and markers without running Git. The derived identity must match the registration bound to the owner-only credential; any mismatch falls back to Git discovery. Discover once per worktree per invocation and revalidate further paths with lookup. The managed accept loop blocks in accept() instead of polling every 10 ms; a waker connection unblocks it at shutdown. - Open sibling candidates through the group's pinned handles, and size sibling groups from the sibling count and query threads, capped at 64 files, so small directories spread across query threads. Containment is still verified when each batch finishes and against the query's final snapshot. - Carry trusted unchanged metadata through reconciliation without content opens or parent verification batches, including for untracked overlay files. Use tree-only invalidation where possible, keep the original hints when a pass fails, coarsen hints that overflow their bound to ancestor directories, and keep the evidence of an unchanged verification that a newer event superseded. An explicit refresh still rereads every file. - Pace native verification in two tiers: metadata checks after at least one poll interval and content rereads after at least 15, each also waiting 19 times its previous admitted cost. Status reports the tier, the delays and metadata reuse, plus tree and Git event counters. - Ignore modifications of the Git metadata root and remote-tracking or prefetch ref updates. Index, HEAD, local branch and directory changes keep tree checks instead of forcing full rereads. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Publish eventual verification differences found alongside newer hints A serving verification that found a difference while a newer invalidation arrived returned ChangedDuringReconcile, and its stale guard then forced a full reread. On an eventual view, an edit made during a content pass therefore cost a second full pass: on a 184,065-file dotnet checkout the operation read 5.6 GB in 802.6 s, and the edit stayed unsearchable for 797 s after the write. An eventual (stale_reads) view now publishes the verified state at a fresh epoch without restoring readiness and returns Verification::PublishedSuperseded. A verification starts only with an empty queue, so everything still queued is newer than the verified state; the same operation's ordinary hinted refresh then processes only those hints. The receipt reports verification "superseded" with changed: true. Strict (v1) views still return ChangedDuringReconcile, and a failed publication still fails closed. Tests: - eventual_verification_publishes_a_difference_and_leaves_only_newer_hints (core): the difference is published and searchable while the view stays unready but serving with one pending path; the follow-up reads one file without a full pass and restores readiness. - superseded_native_content_verification_publishes_edits_without_a_second_full_pass (shared_daemon): the completed receipt reports superseded and changed, full=false, files_read=1, metadata_reused=2; the view serves throughout and the content timer advances. On the same checkout, a forced content pass with an edit 3 s in is now one operation with outcome superseded: one content pass plus a 1-read hinted pass, 2.84 GB of job reads (was 5.64 GB) and 301 CPU-s (was 739). The edit was searchable at the first publish, before the hinted pass; serving held for all 122 polls, and the next content delay is the 1,800 s floor. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Bump version to 1.2.5 Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Apply cost backoff to failed content verification attempts Track terminal content attempts separately from successful rereads so failures and cancellations restart retry backoff without resetting content freshness. Preserve metadata checks and the successful-pass cadence, expose attempt age and retry delay, and cover the behavior with scheduler-boundary and daemon fault-injection regressions. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Make worktree verification tests honor complete file versions Account for Unix ctime changes after size-preserving writes that restore mtime. Assert metadata repair when the complete version changes, content-tier repair otherwise, and the corresponding short-circuit hint-lookup count. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Wait for the requested verification tier in managed tests Do not mistake an earlier metadata result for a content retry while the new operation is queued or just starting. Wait for an empty work queue and the expected tier as well as the terminal outcome. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Bound managed queue aging by operation lifetime Cap class offsets at one quarter of remaining operation lifetime, sharing the durable acceptance clock with work admission. Consider shorter-lived jobs within each class and preserve queue age during promotion. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Allow transient inherited guard locks in lifecycle tests Keep the live-reader protection assertions immediate, but bound release checks while concurrent subprocesses may still hold inherited descriptors before exec. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Back off failed polling reconciliations by terminal attempt cost Record polling failures and cancellations, gate retained work by its retry delay, and allow newer input epochs to request prompt repair. Keep running receipts pending until retry accounting completes and cover both configured polling and native fallback. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Retain coarsened hints that fit the configured capacity Treat half capacity as a compaction target rather than a hard limit. Cover mixed root and nested hints at even and odd capacities while retaining genuine overflow fallback. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Wait for attach reservations before injecting catalog contention A ready publication can precede permit teardown. Drain its reserved slots before taking the test's writer lock so the fixture does not strand a reservation during cleanup. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Review fixes, Chromium worktree benchmark, and managed attach perform… …ance (#210) (#211) * Fix review findings and add Chromium worktree benchmark - query: replace dropped lookarounds with an empty group so neighbours cannot fuse into a different repetition during index relaxation - cli: -M 0 means no column limit, matching ripgrep - cli: explicit file arguments bypass -g/-t/-T filters, matching ripgrep - shared daemons: tolerate transient accept() failures with bounded retry - builder: fsync index.bin/lookup.bin before publication - scripts: benchmark_worktrees.py measures ordinary index, cold managed base build, linked worktree attach, and base vs worktree query times - ci: manual Chromium multi-worktree benchmark workflow Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Address issue #210: faster managed attach, eventual-consistency serving, scheduler fixes - Fix scheduler stall and pace idle catalog ticks; re-enqueued work counts as progress - Serve ready views while reconciliation catches up (eventual consistency) - Filter unchanged watcher events and expose watch counters - Remove query cliffs; page large results; JSON stats fixes - Parallel base generation build; larger proof batches; batched catalog commits - Retry backoff, housekeeping precheck, Windows parent-handle gate 8192 - Keep binary-extension files from splitting verified-read batches - Benchmark harness records per-query timeouts; document Chromium results Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Benchmark: document base vs worktree query latency; skip hidden dirty files Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Serve provisional base on first eventual managed attach An eventual (stale_reads) view's first attachment now publishes the pinned generation after the metadata walk, tombstoning paths whose content is not from the base. The view reports serving:true, ready:false and the scheduler runs full content verification before it becomes ready. FileVersion evidence is recorded only when validated around an actual read. The Chromium harness records time-to-serving separately from time-to-ready and reports verification counters from the reconcile that made the view ready. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Keep legacy attach ready; tests wait for verified managed views Legacy v1 attachments keep their ready-on-attach contract. Managed integration tests that search immediately after a first attach now wait for the provisional view to finish content verification. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Benchmark: document provisional first-attach serving on Chromium Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Fix hook-gated managed tests for provisional first attach Rename the readiness helper to verified_attach so it no longer collides with local 'attached' bindings in managed-test-hooks modules, and make the remaining hook-gated tests wait for (or assert) provisional first-attach verification. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Reclaim catalog WAL before refusing storage admission Storage admission charges catalog WAL bytes, which are only checkpointed once the WAL reaches a quarter of the metadata budget. A retryable namespace-storage-allocation refusal could therefore persist until unrelated writes crossed that threshold. Truncate the WAL once and re-evaluate before refusing, and include the usage breakdown in the collection test's admission panic. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Derive collection admission test limit from checkpointed footprint Admission now reclaims catalog WAL frames before refusing, so the macOS layout's expected refusal can no longer rely on WAL bytes. Size the storage limit from the truncated catalog footprint so retained stages alone close admission and their reclamation alone reopens it on every platform. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Settle view work before applying the test reconcile timer A concurrent scheduler pass can accept a reconciliation while the testing.reconcile-timer hook is between invalidating and recording its own pending operation. The duplicate then stays queued or deferred after the view becomes ready, so the next timer call returned early without invalidating. The hook now waits for the view's running and queued work to settle before evaluating the injected elapsed time. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Address review: page deadline per file, discard eventual group rows on replaced parent - GroupSearch now checks an optional page deadline at every file boundary, so a slow window ends the page with the searched prefix and a next_page cursor instead of overrunning the operation deadline. The single-file progress retry stays deadline-free so every page makes progress. - When an eventual sibling batch fails final parent verification, the group's rows are dropped, their response budget is refunded, and the paths stay requeued, matching how vanished candidates are skipped. - Extract a managed query test fixture and cover both behaviours. - Limit the Chromium benchmark workflow token to contents: read. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Address review: sample catalog usage under the writer lock; render partial worktree reports Storage admission sampled catalog file sizes before catalog_io::begin, which can checkpoint and truncate a WAL past metadata_bytes / 4, so a stale sample refused reservations that fit and the empty-WAL reclaim could not retry. Sample inside the admission transaction instead. The benchmark Markdown report no longer raises KeyError for a worktree whose attachment failed or timed out; it renders an incomplete row and the run error. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Bump version to 1.2.4 Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Read tracked files from split Git indexes Honor the index 'link' extension: load the shared index it names, apply the replace and delete bitmaps, and fall back to a filesystem walk when the shared index is missing or inconsistent. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Keep path names exact in index keys Outside Windows a backslash is an ordinary file-name character, so stop rewriting it to '/'. Files whose names an index cannot represent (not UTF-8, or containing '\' outside Windows) are left out of the index with a warning and remain searchable with --no-index. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Harden the search server and fix CLI output edge cases - fsync staged index files and serve.json, and their directories, when publishing them - record a random server instance in serve.json and verify it on every response, so a stale file pointing at a reused port is not trusted - remove serve.json on SIGTERM as well as Ctrl-C - bound concurrent connections, idle/write timeouts, response size and regex compilation limits on the server - give 'status' a read timeout and fall back to local state - repair a watcher overflow within 10s even when events never pause - exit 2 when a file cannot be read during a search - apply -0 to every printed path and make --trim strip only leading whitespace, as ripgrep does - use path arguments verbatim instead of trimming quotes and spaces Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Restore file permissions when a Unix test read lock drops Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Fix Unix path display tests and agent restart race Verbatim-prefix stripping is Windows-only: on Unix a name beginning with \\?\ is ordinary and must be shown as is. The Windows display tests are gated accordingly, with a Unix test for backslash names. A server now removes serve.json on SIGTERM before exiting, while it still holds the inherited start.lock. The agent runtime waited for neither and fell back to a scan; it now waits briefly for the lock, or for a server to publish. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Gate verbatim display-path tests to Windows Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Exclude .git from hidden parity check; record PR #211 Chromium validation On Windows the shared daemon holds a byte-range lock on .git/tgrep-daemon-v1.lock, so a --no-index --hidden scan reports it unreadable and exits 2 like ripgrep. Indexed queries never include .git, so the parity case excludes it explicitly. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Give collection admission test an operation budget for loaded hosts The warm-admission operation spans the whole collection loop, which can exceed the fixture's 30 s operation_timeout_ms under a parallel test run and fail reserve with operation-deadline. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Retry owner-hold refusal probes on catalog contention A daemon catalog write can outlast the 250 ms SQLite writer wait, so a refusal probe may legally see retryable, uncommitted busy/catalog-io first. Rerun the exited helper with the same claim, as the client contract and retry_catalog_busy already do. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Stabilize managed tests around provisional verification and catalog contention retained_checkpoint now waits for the eventual first attachment's verification and collects the superseded provisional checkpoint before the refresh, so the refresh retires the only collectible checkpoint. Previously a collection pass could reach member-before-io on the provisional checkpoint before skipping the reader-guarded target (reproduced deterministically by delaying the refresh). The missing-evicted exact-commit test retries its final attach on documented retryable, not-committed catalog-io contention with a fresh token. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Prepare v1.2.3 with native watcher, reclamation, and query fixes (#207) * Authenticate Windows producer reobservations before publication Reverify closed managed members against their original bounded block proofs under native write/delete exclusion. Keep the original seal, identity and ownership checks, and charge generation verification to its existing work permit. Reject timestamp-restored rewrites and unverified reobservations; add native metadata-only and protection regressions. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: cff18193-91c7-4186-9e2d-7dfa8433e29d * Keep admitted deadlines during member authentication Reuse the existing WorkPermit deadline and checks for admitted Windows verification. Bound only unreserved observations with current policy, and cover a policy timeout reduction after admission. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: cff18193-91c7-4186-9e2d-7dfa8433e29d * Account for CRLF overlay allocations by capacity Release extraction headroom and consumed input tables instead of retaining cumulative per-posting estimates. Admit hash-table growth with old and replacement capacity both charged, preserve content and mask semantics, and cover CRLF checkpoint/refresh under the existing budget. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: cff18193-91c7-4186-9e2d-7dfa8433e29d * Fix managed native watches for large worktrees Use bounded recursive-root subscriptions on Windows and macOS while retaining per-directory accounting elsewhere. Preserve native coverage after kernel-initiated watch retirement, expose subscription counts, and cover default-budget multi-worktree events, metadata, fallback, and cleanup. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: d2a0130e-af45-49cc-ac06-2c1da73d9d13 * Bump workspace version to 1.2.2 Update both first-party lockfiles and release instructions; preserve historical v1.2.1 deprecation and candidate evidence. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: cff18193-91c7-4186-9e2d-7dfa8433e29d Cherry-picked-from: 2f9d21f Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: d2a0130e-af45-49cc-ac06-2c1da73d9d13 * Preserve overlay allocation charges across replacements and deletions Size replacement growth after removing old entries from the unpublished clone. Track backing-capacity watermarks for retained maps, posting sets, and bookkeeping instead of releasing tombstone storage. Add deterministic red/green replacement and deletion regressions, rehash/pressure coverage, and changed-CRLF checkpoint parity. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: cff18193-91c7-4186-9e2d-7dfa8433e29d Cherry-picked-from: 6627cef Copilot-Session: d2a0130e-af45-49cc-ac06-2c1da73d9d13 * Fix bounded managed qualification handshakes Resume collection only from an evidenced bounded continuation, preserving exact reclamation accounting and native sharing assertions. Discard epoch-proven CLI fallback scans and wait within the existing managed busy budget before comparing indexed results. Add deterministic deadline and real native Git-exclude controls, and clarify the lifecycle contracts. Production code and the sealed 4e2a2a8 candidate are unchanged. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: d2a0130e-af45-49cc-ac06-2c1da73d9d13 * Preserve managed collection progress across busy idle probes Discard cached member authentication only after other shutdown blockers are clear. Preserve natural expiry, active verifier protection, and existing resource limits. Cover bounded stage reclamation, unchanged-cap readmission, restart persistence, and generated-store cleanup. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 23051c7b-9d3c-4f54-9d4e-72897ee5a82a * Reuse bounded sibling handles for indexed query reads Amortize rooted ancestor opens across consecutive sibling candidates without reordering results. Reuse reconciliation readers while preserving snapshot validation, generation and root pins, final ancestor checks, query deadlines, and response limits. Keep singleton candidates on the direct rooted path. Add local, legacy, and managed scan-parity coverage plus containment, failure-classification, and pin-lifetime regressions. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 417710f4-d898-45a9-ade1-dd8f33592373 * Project count queries into bounded per-file results Send the effective line-count mode to servers and omit unused matching-line payloads for both count modes. Preserve original match statistics, binary visibility, multiline line-count semantics, legacy ordinary-server rendering, and existing response limits. Validate projected rows before scope filtering or output, and cover malformed replies, exact byte budgets, dense counts, and local/ordinary/shared route parity. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 417710f4-d898-45a9-ade1-dd8f33592373 * Test compact counts separately from oversized content fallback Preserve full-content response-cap rejection and compare fallback output byte-for-byte with a scan. Assert compact count replies stay on the shared daemon and preserve filename/count parity. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: d2a0130e-af45-49cc-ac06-2c1da73d9d13 * Keep macOS reclamation fixture within unchanged pass budgets macOS deliberately discards verified prefixes after each truncation. Use smaller temporary stages and adjust pinned generations only on macOS, preserving pressure/recovery assertions, the 128-pass bound, all per-pass/resource limits, and the original Windows/Linux fixture inputs. A conservative-prefix model reproduces the old failure and the corrected fixture still fails against the pre-fix idle behavior. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: d2a0130e-af45-49cc-ac06-2c1da73d9d13 * test: reproduce Linux foreign-reader lease failures deterministically An external on-access reader that opens a verified member breaks or refuses the collection's exclusive F_WRLCK lease. Reproduce both variants in-process on Linux, isolated in a child test process, and assert the observed fail-closed signatures followed by recovery once the reader is gone. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: d2a0130e-af45-49cc-ac06-2c1da73d9d13 * Frame typed owner-hold failures before the handshake A typed OwnerGuard::claim failure in `tgrep shared owner-hold` was flattened to exit 2 plus stderr text, so a supervising client could not distinguish retryable not-committed catalog contention from an unknown failure. Before `holding:true`, write exactly one newline-terminated `{"ok":false,"error":...}` stdout line carrying the managed failure as returned, then keep the existing exit status and stderr diagnostic. Claim-file read/size/parse errors stay unclassified and unframed, and no failure after the handshake is framed. The success handshake, guard lifetime and SQLite writer/busy limits are unchanged. Add CLI-boundary regressions: a held catalog writer yields a busy/catalog-io retryable not-committed frame, and the same claim then holds and exits on stdin EOF; a live holder yields owner-guard-already-held; an ended lifetime yields a non-retryable stale-identity frame; malformed claims emit no frame. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: d2a0130e-af45-49cc-ac06-2c1da73d9d13 * Verify healthy native managed views while serving (SHARE-7) Healthy native-watched managed views had no periodic full verification, so a silently missed watcher notification could leave results stale indefinitely. Once the existing --poll-interval has elapsed since a view's last completed full pass, the scheduler now queues one ordinary reconcile operation with verify: true, without invalidating the view. The verification runs under the unchanged operation deadline and CPU/input/handle budgets, reads and hashes every admitted file, and compares membership and visibility with the published snapshot while queries keep using it. If nothing differs it publishes nothing. The first detected difference marks the view stale, closing readiness, and the pass publishes a full repair. Deadline or cancellation leaves the view ready but unverified until the next interval; other errors, watcher uncertainty, explicit invalidation, concurrent events and stale policy keep their fail-closed semantics. Status adds last_verification and native verification policy fields, and hello advertises serving-native-periodic-verification. Docs are updated. Core tests cover unchanged availability, missed-change repair, concurrent invalidation, read failures, cancellation and policy races. CLI tests cover the interval boundary, mid-verification serving and missed nested-directory repair with scan parity. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: d2a0130e-af45-49cc-ac06-2c1da73d9d13 * Update poll-interval help assertion for native verification Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: d2a0130e-af45-49cc-ac06-2c1da73d9d13 * Cover serving-verification budget failure after a detected difference Cancellation, deadline and resource pressure keep a serving verification ready only when it observed no difference. Add a managed regression for both interleavings: a membership difference observed by discovery before cancellation at a paused batch, and a content difference observed by the full read before cancellation at publication. Both must leave the view not ready with a full refresh required. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: d2a0130e-af45-49cc-ac06-2c1da73d9d13 * Add explicit content-only eligibility for managed views Managed v2 indexes can opt out of the binary-extension filename heuristic, so text such as an ASCII RTF .doc is indexed and searched like ripgrep. NUL classification, ignore, hidden, size and filename rules are unchanged. - IndexingProfile gains an optional eligibility ("content-only"); the default is omitted, so default keys, storage names and hello.profile are unchanged. - Worktree verification applies the extension rule only for the default. - Daemons advertise content-eligibility-profile; v1 leases reject it. - `--content-eligibility content` searches use only a matching managed view. The client and server refuse mismatches in either direction, and the client scans with the requested rule. Ordinary servers and local indexes are bypassed. index/serve reject the flag. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: d2a0130e-af45-49cc-ac06-2c1da73d9d13 * Stabilize content-only refresh and retired-owner refusal assertions Invalidation schedules automatic reconciliation even with --no-watch, so the explicit refresh may legitimately run a full pass after the hints are consumed; assert the preserved content-only profile instead (hint-only processing stays covered in tgrep-core). Housekeeping can remove a released owner record before re-hold, which reports owner-retired; both reason codes remain stale-identity, non-retryable and not-committed. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: d2a0130e-af45-49cc-ac06-2c1da73d9d13 * test: retain managed attach and shutdown failure diagnostics Capture the original failed operation and one best-effort view status within its existing wait budget. Report stop-request, observed acceptance, and post-accept exit timing without changing the shared shutdown deadline. Add deterministic diagnostic-output and loopback controls; no production changes or whole-test retries. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: d2a0130e-af45-49cc-ac06-2c1da73d9d13 * Exclude JSON end events from summary bytes_printed ripgrep records each file's bytes_printed before writing its end message, and the summary sums those per-file stats. tgrep added the end line to the running file total after serializing it, so summary.bytes_printed exceeded the sum of end records by every end line on scan, local-index, server and managed v2 routes. Stop counting the end line and keep bytes_printed in the JSON parity comparisons; only elapsed timing remains volatile. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: d2a0130e-af45-49cc-ac06-2c1da73d9d13 * Report actual searched-file work in remote JSON stats Carry per-candidate work receipts from the server matching boundary, including candidates that emit no rows. Populate JSON summary searches/bytes_searched and each file's bytes without extra reads or scan-equivalent guesses. Validate receipts before output and charge them to existing shared response budgets. Keep elapsed conventions unchanged and document backend-work versus result counters. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: d2a0130e-af45-49cc-ac06-2c1da73d9d13 * test: resolve documented preparation invalidation in native watcher attach The native watcher test assumed every initial views.attach succeeds on its first operation. The managed contract documents a terminal failed result with both commitment fields not-committed, category busy, reason view-preparation-invalidated and retryable=true; the sealed 272c059 run hit it. Factor the managed_faults exact-state recovery helper into managed_attach.rs with an explicit per-call-site allowance. Catalog callers keep the unchanged catalog-io allowance and messages. The watcher call site alone permits the exact preparation-invalidation outcome: it inspects the old operation, replays the old token unchanged, recovers the same root/profile/pending identity, uses a fresh token only after the old terminal result is resolved, keeps the original 30-second deadline shared across attempts and the existing 8-attempt ceiling, and requires one lease bound to the first operation. Any other outcome still fails with view diagnostics. No product code changes. Scripted controls cover recovery, rewritten replays, extra leases, exhaustion, deadline, rejected outcomes/identity/responses and the unchanged classifier. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: d2a0130e-af45-49cc-ac06-2c1da73d9d13 * test: inject attach preparation invalidation through real daemon path Add a managed-test-hooks-only fault at the post-checkpoint preparation boundary. Its bounded invalidation action advances and invalidates the real view slot for each of three preparation attempts, producing the actual retryable, not-committed view-preparation-invalidated operation receipt. Production builds do not compile the action or its preparation handling. Exercise the watcher recovery helper against a real managed daemon: verify the failed receipt and three fault hits, then require token 2 to complete against the same pending view and original lease, with native readiness, one lease, indexed search, scan parity, and normal cleanup. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: d2a0130e-af45-49cc-ac06-2c1da73d9d13 * fix: address Copilot review findings Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: d2a0130e-af45-49cc-ac06-2c1da73d9d13 * test: accept procfs ESRCH after descendant exit Linux procfs can report ESRCH when a task exits after its stat file is opened. Treat that observation as task disappearance in the test-only assert_dead helper while preserving live-state deadlines and unrelated errors. Cover the real open/exit/read boundary and scripted live/error observations. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: d2a0130e-af45-49cc-ac06-2c1da73d9d13 * test: accept legal catalog Busy from guarded generation opens open_generation takes the SQLite catalog writer lock with the fixed 250 ms wait, so a concurrent writer can legally return retryable, uncommitted busy/catalog-io (SHARED_WORKTREE_INDEXES.md). The legacy-reader test and the guarded reader subprocess assumed first-attempt success with a bare unwrap. Route both through the existing exact-class owner-claim retry helper, made type-generic with its unchanged 10 s fixture bound and 20 ms spacing. Add a real-boundary control that holds an owned BEGIN IMMEDIATE writer: it fails with the old bare unwrap (same payload as CI run 39065427720), passes with the helper, keeps an expired bound at one attempt, and stops on a non-Busy error. No production changes. The historical lock holder in the CI sample remains unknown. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: d2a0130e-af45-49cc-ac06-2c1da73d9d13 * test(cli): witness streaming sibling opens after a replaced parent Drive the real on-disk indexed CLI route with pipe backpressure so the search is held inside the first sibling while the test replaces their parent. The former batch loop printed the next sibling through the replaced parent before failing ancestor verification; the streaming route resolves it again and prints only the replacement. Unix-only: Windows pins an open parent against replacement. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: d2a0130e-af45-49cc-ac06-2c1da73d9d13 * Test operations.cancel on an in-flight managed collection Add a hooks-gated daemon integration test that pauses a real collections.start operation at object-withdrawn, object-pending-deletion, member-intent-saved, member-before-io and collection-before-complete, services operations.cancel while the worker is parked, and then releases it. The cancelled receipt must be terminal with no delete credit after the cancel point, unchanged checkpoint members for pre-IO points, a discarded verification context, namespace retained memory back at its pre-collection baseline, and an idempotent replay. Fresh operations then resume from the saved cursor and finish collection with scan parity. Removing the cancellation recheck after MemberBeforeIo makes the test fail with 1475 bytes credited after cancellation at member-intent-saved. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: d2a0130e-af45-49cc-ac06-2c1da73d9d13 * test: continue collection barriers past legitimate short passes A bounded collection pass may complete before an armed member/object barrier: an escaped reader of the object skips it without error, or the pass exhausts its elapsed budget and returns a resumable cursor. Windows CI run 39092617708 hit this in the member-before-io iteration of collection_crash_boundaries_preserve_live_views_and_resumable_progress (hook armed with 0 hits, one completed pass, 0 errors, 0 verification pages). The log cannot distinguish the two causes. Fault tests now start continuation passes with fresh tokens only after a classifier accepts the short pass (completed, uncancelled, errorless, no omitted details, target detail limited to object-readers-active, or an exceeded budget with a cursor). Pass completion is observed through the in-memory maintenance counters so the poller never reads the catalog while a pass may hold it at the credit barrier. Recovery assertions keep their committed expectations, accepting an interrupted, uncommitted continuation only at removal barriers after earlier passes deleted every member. Adds a classifier unit test and a witness that drives both legitimate paths deterministically. No production or budget changes. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: d2a0130e-af45-49cc-ac06-2c1da73d9d13 --------- Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: cff18193-91c7-4186-9e2d-7dfa8433e29d Copilot-Session: d2a0130e-af45-49cc-ac06-2c1da73d9d13 Copilot-Session: 23051c7b-9d3c-4f54-9d4e-72897ee5a82a Copilot-Session: 417710f4-d898-45a9-ade1-dd8f33592373
Fix Windows managed publication and CRLF overlay memory accounting (#205 ) * Authenticate Windows producer reobservations before publication Reverify closed managed members against their original bounded block proofs under native write/delete exclusion. Keep the original seal, identity and ownership checks, and charge generation verification to its existing work permit. Reject timestamp-restored rewrites and unverified reobservations; add native metadata-only and protection regressions. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: cff18193-91c7-4186-9e2d-7dfa8433e29d * Keep admitted deadlines during member authentication Reuse the existing WorkPermit deadline and checks for admitted Windows verification. Bound only unreserved observations with current policy, and cover a policy timeout reduction after admission. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: cff18193-91c7-4186-9e2d-7dfa8433e29d * Account for CRLF overlay allocations by capacity Release extraction headroom and consumed input tables instead of retaining cumulative per-posting estimates. Admit hash-table growth with old and replacement capacity both charged, preserve content and mask semantics, and cover CRLF checkpoint/refresh under the existing budget. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: cff18193-91c7-4186-9e2d-7dfa8433e29d * Bump workspace version to 1.2.2 Update both first-party lockfiles and release instructions; preserve historical v1.2.1 deprecation and candidate evidence. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: cff18193-91c7-4186-9e2d-7dfa8433e29d * Preserve overlay allocation charges across replacements and deletions Size replacement growth after removing old entries from the unpublished clone. Track backing-capacity watermarks for retained maps, posting sets, and bookkeeping instead of releasing tombstone storage. Add deterministic red/green replacement and deletion regressions, rehash/pressure coverage, and changed-CRLF checkpoint parity. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: cff18193-91c7-4186-9e2d-7dfa8433e29d --------- Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: cff18193-91c7-4186-9e2d-7dfa8433e29d
Bound and parallelize managed worktree reconciliation (#204) * Bound and instrument managed worktree reconciliation Cache versioned work controls with bounded authorization refresh; verify files in shared CPU, input and parent-handle budgets while preserving rooted reads, epoch checks and full repair fallback. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 9776b17f-6a2d-4c47-9e89-0f8113d14b95 * Document the measured RC2 reconciliation speedup Record the sanitized same-binary Windows/ReFS comparison, phase accounting, bounded resource peaks, correctness gates, and exact provenance. Keep the unchanged periodic full-repair limitation explicit. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 9776b17f-6a2d-4c47-9e89-0f8113d14b95 * Clarify reconciliation read and memory accounting Describe full-reconciliation read operations without claiming EOF consumption. Distinguish the 64 MiB searchable cap from the extra probe byte and describe the conservative read/decode charge rather than physical buffers. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 9776b17f-6a2d-4c47-9e89-0f8113d14b95 * Classify blob-boundary file growth as unstable Check growth past the opened length before the managed blob limit. Add deterministic regressions for one-byte growth at the boundary and stable initially oversized files. Keep benchmark timings attributed to the immutable measured source rather than this follow-up. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 9776b17f-6a2d-4c47-9e89-0f8113d14b95 * Remove periodic safety repair for native managed views Honor the requested event-driven staleness tradeoff for healthy native managed views. Preserve chosen and fallback polling, strict initial/restore verification, explicit refresh and error recovery. Report configured and effective update modes separately, cover the former timer boundary deterministically, and retain the original benchmark attribution. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 9776b17f-6a2d-4c47-9e89-0f8113d14b95 * Make managed timer tests independent of clock epoch Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 9776b17f-6a2d-4c47-9e89-0f8113d14b95 * Clarify bounded ancestor observations and publication checks Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 9776b17f-6a2d-4c47-9e89-0f8113d14b95 --------- Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 9776b17f-6a2d-4c47-9e89-0f8113d14b95
Restore Rust 1.88 Windows release compatibility (#193) Advance the rusqlite Git pin to the compatible development-line revision while retaining native Windows WAL fixes and upgrading bundled SQLite to 3.53.4. Declare and exercise the workspace MSRV, use compatible atomic updates, and preserve OneBranch release hardening and publication behavior. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Fix unnecessary server reindexing and Git-internal indexing (#165) * Fix server indexing noise and exclude Git internals Preserve indexed content fingerprints during deferred-event replay, exclude Git metadata directories from the default indexed corpus, and clarify watcher and save diagnostics. Add regression coverage and document the behavior. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Watch ignore-source parents outside the indexed corpus Keep native per-directory subscriptions for ordinary ignore sources such as .git/info/exclude, while retaining Git-internal indexing filters and in-root containment. Cover regular, missing, and symlinked sources. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Keep Git exclude watches across deletion and recreation Resolve the candidate ignore-file location separately from existing sources, retain its contained parent watch, and recognize newly created exclude files during events and recovery. Cover absent startup and delete-refresh-recreate on native watchers. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Isolate membership refresh test from Git template timestamps Seed a historical exclude file so the exact corrective-pass assertion is not affected by the independent ignore-source recovery window. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * chore: bump version to 1.0.11 Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
PreviousNext