GLM-5.3 · Balanced

dual-storage-architecture-grafana-codex

Grafana · Go · L · 300–799 lines · 4 files

The normal review, with one generalist agent. Judge: Claude Haiku 5.5.

Gold bugs found
7 of 9
Comments published
8
Comments matching a bug
7
Comments not matching
1
Cost of this PR
$1.245

Published comments (8)

highbugpkg/apiserver/rest/dualwriter_mode3.go:50-57

Async legacy writes use request context, canceled when handler returns — Legacy Create/Delete/Update/DeleteCollection writes are aborted mid-flight whenever they outlast the response, silently drifting the two stores and risking data loss if reads ever fall back to legacy. Derive the goroutine contexts from `context.WithTimeoutCause(context.WithoutCancel(ctx), 10*time.Second, ...)` so only the deadline applies.

highbugpkg/apiserver/rest/dualwriter_mode3.go:134-141

Legacy Update recomputes patch on stale object instead of unified result — PATCH/Apply requests write different content to legacy than to unified storage, because raw `objInfo` makes legacy `Update` recompute the patch on its own possibly stale copy (see `legacyStorage.Update` in `pkg/registry/apis/playlist/legacy_storage.go`). Wrap the unified result in `updateWrapper`, as `dualwriter_mode2.go` still does.

highbugpkg/apiserver/rest/dualwriter_mode3.go:100-105

Delete no longer tolerates NotFound from unified storage, orphaning legacy-only rows — Objects existing only in legacy (pre-migration rows) now return 404 and stay orphaned forever, blocking mode 3-to-4 migration, since every storage error returns immediately. Restore the `apierrors.IsNotFound` fall-through so legacy cleanup still runs.

mediumbugpkg/apiserver/rest/dualwriter_mode3.go:127-131

Mode 3 records unified-storage errors as legacy duration metrics — `grafana_dual_writer_legacy_duration_seconds{is_error="true"}` gets inflated by unified-storage failures recorded through `d.recordLegacyDuration(true, ...)`, skewing the legacy health metrics operators use for mode migration. Replace both error-path calls with `recordStorageDuration`, as `dualwriter_mode2.go` does.

mediumbugpkg/apiserver/rest/dualwriter_mode3.go:161-167

Async legacy delete-collection recorded under storage metric instead of legacy metric — Every async legacy delete-collection is hidden: `d.recordStorageDuration(err != nil, ...)` attributes it to the storage metric, so `grafana_dual_writer_legacy_duration_seconds` shows zero delete-collection traffic in mode 3. Call `recordLegacyDuration` instead, matching `dualwriter_mode2.go`'s counterpart.

mediumbugpkg/apiserver/rest/dualwriter_mode3.go:106

Delete success path uses object name as metric kind label causing unbounded cardinality — Each distinct deleted object name creates its own `grafana_dual_writer_storage_duration_seconds` series via the `name` label — user-chosen names mean unbounded Prometheus cardinality. Pass `options.Kind` like every sibling call.

mediumbugpkg/apiserver/rest/dualwriter_mode3_test.go:56-76

Mode-3 tests never assert the async legacy write actually happened — Mode 3's defining async legacy write can be deleted or broken without failing any test — no `AssertCalled` verifies the legacy mock after the goroutine. Synchronize on the goroutine (mock channels or a WaitGroup) and assert `LegacyStorage.Create/Update/Delete/DeleteCollection` ran with the unified-written object.

mediumbugpkg/tests/apis/playlist/playlist_test.go:132-144

Mode-3 playlist tests assert legacy state without waiting for async dual-write, causing flakiness — The mode-3 subtests only pass when the async legacy write beats the next HTTP request — `doPlaylistTests` asserts legacy state immediately with no synchronization against the goroutine. Poll for legacy convergence with a timeout instead.

Gold bugs found (7)

mediumThe legacy DeleteCollection goroutine records its duration with recordStorageDuration instead of recordLegacyDuration, misattributing legacy latency and errors to the storage metric.
mediumDelete's success path records the storage metric with the object name in the kind label instead of options.Kind, creating per-object series and inconsistent labels.
mediumOn storage Create/Update failure the duration is recorded with recordLegacyDuration, attributing unified-store errors to the legacy metric.
highMode 3 legacy writes now run in background goroutines derived from the request context, which is cancelled when the request completes, so legacy writes can be cancelled and legacy storage falls out of sync.
highLegacy.Update now receives the original objInfo instead of a wrapper carrying the object stored in unified storage, so legacy recomputes the update and can diverge.
mediumThe Mode 3 tests never wait for or assert the asynchronous legacy writes, and legacy and storage share one mock, so removing the legacy calls would not fail them.
highDualWriterMode3.Delete now returns on any unified-storage error, including NotFound, before deleting from legacy, so objects that exist only in legacy can no longer be deleted.

Gold bugs missed (2)

highIndependent per-request goroutines remove ordering between legacy writes to the same resource, so legacy can end up with an older version or a resurrected object.
mediumMode 3 legacy-storage write errors (create/update/delete/delete-collection) are no longer logged after moving to async goroutines; only a duration metric with an error flag is recorded.