client: add onchain reconciler to daemon with enable/disable CLI

[snormore]

Summary of Changes

• Add an onchain reconciler to doublezerod that polls the DZ Ledger and automatically provisions/removes tunnels when users are activated or deactivated, replacing the CLI-driven /provision flow
• Remove the JSON state file (doublezerod.json) crash recovery mechanism — the daemon now re-provisions from onchain state on restart
• Add doublezero enable / doublezero disable CLI commands to control the reconciler, with connect implicitly enabling it
• Move client IP discovery from the CLI to the daemon (auto-discovered via default route or --client-ip flag), deprecating --client-ip on the CLI
• Simplify the CLI by dropping ~400 lines of provisioning logic (device fetching, tunnel parameter computation, multicast resolution) — the CLI now just creates the onchain user and waits for the daemon's reconciler to provision the tunnel
• Add drift detection: reconciler detects when onchain state changes (e.g., multicast group subscriptions) and re-provisions running services to match
• Add new e2e test (TestE2E_IBRL_EnableDisable) covering the enable/disable lifecycle

RFC: rfcs/rfc17-client-onchain-reconciler.md

Diff Breakdown

Category	Files	Lines (+/-)	Net
Core logic	23	+2,659 / -1,548	+1,111
Scaffolding	6	+70 / -17	+53
Tests	16	+2,436 / -930	+1,506
Fixtures	23	+39 / -208	-169
Docs	7	+221 / -18	+203
Total	75	+5,425 / -2,721	+2,704

~41% of the net additions are tests; core logic is roughly balanced between new reconciler code and removed CLI/state-file code.

Note: This PR exceeds the recommended ~500 lines of new non-test code. The reconciler, state management, client IP discovery, CLI refactor, and service interface changes are tightly coupled and difficult to split further without shipping a half-working intermediate state. The RFC covers the full design.

Key files (click to expand)

• client/doublezerod/internal/manager/reconciler_test.go — comprehensive tests for reconciler provisioning, removal, drift detection, enable/disable lifecycle
• client/doublezero/src/command/status.rs — rewritten to use daemon v2 status endpoint, display reconciler state
• client/doublezerod/internal/manager/manager.go — reconciler loop, onchain data diffing, provision request building, state file removal
• client/doublezero/src/command/connect.rs — stripped provisioning logic; now enables reconciler and polls daemon for tunnel status
• client/doublezerod/internal/manager/state.go — persistent reconciler enabled/disabled state with migration from old state file
• client/doublezero/src/command/disconnect.rs — no longer sends /remove to daemon; reconciler handles teardown after onchain user deletion
• client/doublezerod/internal/runtime/clientip.go — client IP auto-discovery via kernel default route with external fallback
• client/doublezerod/internal/onchain/fetcher.go — caching fetcher wrapping the serviceability SDK for RPC calls

Testing Verification

• 1,427-line reconciler test suite covering: provision on user activation, removal on deactivation, drift detection and re-provisioning, multiple user type handling, enable/disable lifecycle, error recovery, tunnel source resolution
• State persistence tests (fresh install, migration from old state file, read/write cycles)
• Client IP discovery tests (default route resolution, validation, martian filtering)
• Onchain fetcher tests (caching behavior, TTL expiry, error propagation)
• New e2e test TestE2E_IBRL_EnableDisable validating the full enable → connect → disable → re-enable flow against a live devnet
• Existing e2e tests updated to work with the reconciler-driven flow

[Copilot]

Pull request overview

This PR moves tunnel provisioning from the doublezero CLI into an onchain-driven reconciler loop inside the doublezerod daemon, adds runtime enable/disable controls, and updates status/reporting + tests/fixtures accordingly.

Changes:

• Add an onchain reconciler (polling + provisioning/removal) to doublezerod, with persisted enable/disable state and new /enable, /disable, /v2/status endpoints.
• Update CLI flows (connect, status, plus new enable/disable) to interact with the reconciler rather than directly provisioning.
• Add caching onchain fetcher + adjust tests/e2e fixtures to account for reconciler state and updated output.

Reviewed changes

Copilot reviewed 66 out of 66 changed files in this pull request and generated 10 comments.

Show a summary per file

File	Description
rfcs/rfc17-client-onchain-reconciler.md	RFC documenting reconciler design, state, endpoints, and rollout plan
CHANGELOG.md	Changelog entry describing reconciler + CLI enable/disable
e2e/user_ban_test.go	Treat missing tunnel interface as successful route withdrawal
e2e/multicast_test.go	Update status fixture usage to templated reconciler-aware output
e2e/internal/fixtures/diff.go	Make CLI-table diff parsing robust to non-table preamble lines
e2e/ibrl_with_allocated_ip_test.go	Update status fixture usage to templated reconciler-aware output
e2e/ibrl_test.go	Update status fixture usage to templated reconciler-aware output
e2e/ibrl_enable_disable_test.go	New E2E test for connect → disable → enable → restart persistence lifecycle
e2e/fixtures/multicast/doublezero_status_disconnected.txt	Remove old disconnected status fixture
e2e/fixtures/multicast/doublezero_status_disconnected.tmpl	New reconciler-aware disconnected status fixture template
e2e/fixtures/multicast/doublezero_status_connected_subscriber.tmpl	Add “Reconciler” column to connected multicast subscriber fixture
e2e/fixtures/multicast/doublezero_status_connected_publisher.tmpl	Add “Reconciler” column to connected multicast publisher fixture
e2e/fixtures/ibrl_with_allocated_addr/doublezero_status_disconnected.txt	Remove old disconnected status fixture
e2e/fixtures/ibrl_with_allocated_addr/doublezero_status_disconnected.tmpl	New reconciler-aware disconnected status fixture template
e2e/fixtures/ibrl_with_allocated_addr/doublezero_status_connected.tmpl	Add “Reconciler” column to connected allocated-IP fixture
e2e/fixtures/ibrl/doublezero_status_disconnected.txt	Remove old disconnected status fixture
e2e/fixtures/ibrl/doublezero_status_disconnected.tmpl	New reconciler-aware disconnected status fixture template
e2e/fixtures/ibrl/doublezero_status_connected.tmpl	Add “Reconciler” column to connected IBRL fixture
client/doublezerod/cmd/doublezerod/main.go	Add flags for client IP, reconciler poll interval, and state dir; pass into runtime
client/doublezerod/internal/runtime/run.go	Wire reconciler + state migration + caching fetcher; update routes handler wiring; latency now uses fetcher
client/doublezerod/internal/runtime/run_test.go	Update runtime tests to new Run() signature; remove statefile recovery assertions
client/doublezerod/internal/runtime/clientip.go	New client IP auto-discovery (explicit → interfaces → ifconfig.me)
client/doublezerod/internal/runtime/clientip_test.go	Unit tests for public IP classification and explicit IP behavior
client/doublezerod/internal/reconciler/reconciler.go	New reconciler loop + enable/disable endpoints + /v2/status
client/doublezerod/internal/reconciler/state.go	Persisted reconciler enable/disable state + migration from old doublezerod.json
client/doublezerod/internal/reconciler/state_test.go	Unit tests for state load/write + migration behavior
client/doublezerod/internal/reconciler/metrics.go	Prometheus metrics for reconciler polls/provisions/removals/matched users
client/doublezerod/internal/onchain/fetcher.go	New TTL-based caching fetcher shared by reconciler and latency subsystem
client/doublezerod/internal/onchain/fetcher_test.go	Unit tests for caching behavior (TTL, stale-on-error, concurrency)
client/doublezerod/internal/manager/manager.go	Remove DB/statefile dependency; add ResolveTunnelSrc + GetProvisionedServices
client/doublezerod/internal/manager/http_test.go	Update manager HTTP tests for removed DB + new GetProvisionedServices
client/doublezerod/internal/manager/db.go	Remove old on-disk state DB implementation
client/doublezerod/internal/manager/db_test.go	Remove tests for deleted DB/statefile system
client/doublezerod/internal/manager/fixtures/doublezerod.*.json	Remove fixtures used exclusively for deleted DB/statefile behavior
client/doublezerod/internal/services/base.go	Remove DBReaderWriter interface (services now keep ProvisionRequest in memory)
client/doublezerod/internal/services/services_test.go	Update service creation tests after DB removal
client/doublezerod/internal/services/ibrl.go	Store ProvisionRequest in memory; drop DB usage; expose ProvisionRequest()
client/doublezerod/internal/services/edgefiltering.go	Store ProvisionRequest in memory; drop DB usage; expose ProvisionRequest()
client/doublezerod/internal/services/multicast.go	Store ProvisionRequest in memory; drop DB usage; expose ProvisionRequest()
client/doublezerod/internal/latency/smartcontract.go	Remove old direct smartcontract fetcher module (replaced by fetcher integration)
client/doublezerod/internal/latency/manager.go	Support injected Fetcher; adjust SmartContractFunc signature and fetch path
client/doublezerod/internal/latency/manager_test.go	Update tests for new smartcontract func signature and removed program ID options
client/doublezerod/internal/api/routes.go	Replace DBReader with ServiceStateReader (provisioned services now in manager)
client/doublezerod/internal/api/routes_test.go	Update routes tests to use ServiceStateReader mock
client/doublezero/src/servicecontroller.rs	Add v2 status + enable/disable calls; remove CLI provisioning/remove/resolve-route APIs
client/doublezero/src/routes.rs	Remove resolve_route command surface; keep routes retrieval
client/doublezero/src/main.rs	Allow enable/disable commands without version warning gate
client/doublezero/src/cli/command.rs	Add doublezero enable / doublezero disable commands
client/doublezero/src/command/mod.rs	Register enable/disable subcommands
client/doublezero/src/command/enable.rs	New enable command implementation + unit tests
client/doublezero/src/command/disable.rs	New disable command implementation + unit tests
client/doublezero/src/command/status.rs	Use /v2/status; surface reconciler state in output/table; synthesize disconnected row when empty
client/doublezero/src/command/connect.rs	Switch connect flow to best-effort enable reconciler + poll daemon status for provisioning
client/doublezero/src/command/disconnect.rs	Stop calling daemon /remove; poll daemon for deprovision completion after onchain user deletion

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.