v3.8.49: GET /v1/models pegs the event loop — getModelsDevPricing() re-reads and re-parses the whole pricing set once per model

[turbsmcgurk]

v3.8.49: GET /v1/models pegs the event loop — getModelsDevPricing() re-reads and re-parses the whole pricing set once per model

Summary

Since v3.8.49, building the /v1/models catalog does a full SQL scan + JSON.parse of the entire models_dev_pricing key_value namespace once per model entry. On an instance with ~11,600 models and 186 pricing rows (~443 KB of JSON), that is ~11,600 SQL scans and ~5 GB of JSON parsed per catalog build.

The build takes minutes, but CATALOG_CACHE_TTL_MS_DEFAULT is 60 s, so the cache is always expired by the time a build finishes and the next request immediately starts another one. The result is a permanently blocked Node event loop: the process sits at 100% CPU on the main thread and every HTTP endpoint — including /api/system/version and the web UI — times out.

Call path

src/app/api/v1/models/catalog.ts:200      buildUnifiedModelsResponseCore
 └ src/app/api/v1/models/catalogResponse.ts:125   finalModels.map(... enrichCatalogModelEntry)
    └ src/lib/modelMetadataRegistry.ts:464         resolveCatalogPricing(provider, model)
       └ src/lib/modelMetadataRegistry.ts:285      getModelsDevPricing()
          └ src/lib/modelsDevSync.ts:199           SELECT ... + JSON.parse per row  ← uncached

getModelsDevPricing() (modelsDevSync.ts:199) has no memoization:

export function getModelsDevPricing(): PricingByProvider {
  const db = getDbInstance();
  const rows = db
    .prepare("SELECT key, value FROM key_value WHERE namespace = 'models_dev_pricing'")
    .all();
  const synced: PricingByProvider = {};
  for (const row of rows) {
    ...
    synced[key] = JSON.parse(rawValue) as PricingModels;   // full re-parse, every call
  }

and resolveCatalogPricing calls it per model.

Evidence

V8 CPU profile (15 s, sampled live via the inspector, source-map resolved), main thread:

41.1%  modelsDevSync.ts:199        getModelsDevPricing
15.9%  catalog.ts:200              buildUnifiedModelsResponseCore
10.4%  catalogResponse.ts:113/121  (enrich map)
 3.7%  better-sqlite3 prepare      via modelCapabilities.ts:340 getSyncedCapabilityForResolved

perf on the same process, native symbols:

 8.34%  v8::internal::JSDataObjectBuilder::BuildFromIterator<JsonParser...>
 7.26%  v8::internal::StringTable::LookupKey<SeqSubStringKey...>
 5.22%  v8::internal::JsonParser<unsigned char>::ScanJsonString
 4.96%  v8::internal::BinarySearch<..., TransitionArray>
 4.69%  v8::internal::JsonParser<unsigned char>::ParseJsonValue
 3.84%  v8::internal::JsonParser<unsigned char>::ParseJsonNumber

top -H shows the main JS thread at ~99.5% with the GC and libuv threads idle, i.e. a JS busy loop rather than GC pressure.

Introduced by

fix(models): attach models.dev pricing to GET /v1/models entries (#8018). git show v3.8.48:src/lib/modelMetadataRegistry.ts | grep -c getModelsDevPricing → 0; same on v3.8.49 → 2. v3.8.48 is unaffected.

Environment

• OmniRoute v3.8.49, source install, Node v22.22.3, Ubuntu 24.04 ARM64 (Oracle Cloud, 24 GB)
• ~40 connected accounts, ~11,600 models, 186 rows in models_dev_pricing
• Symptom persisted continuously for 3 days after the upgrade (CPU: 2d 17h over 3 days of uptime)

Suggested fix

Memoize getModelsDevPricing() behind the same invalidation the writers already use — clearModelsDevPricing() and the sync write path both know when the namespace changes, so a module-level cache invalidated there would be sufficient. Hoisting the lookup out of the per-model loop in catalogResponse.ts (resolve once per catalog build, pass down) would also work and is arguably the more robust shape.

getSyncedCapabilityForResolved (modelCapabilities.ts:340) has a smaller version of the same problem — a better-sqlite3 .prepare() per model — worth batching in the same pass.

Workaround

Setting modelsDevSyncEnabled = false and clearing the models_dev_pricing namespace makes the per-model call return an empty set, which restores responsiveness at the cost of pricing metadata on /v1/models.

[Crazyphil]

This looks a bit like my problem, where OmniRoute freezes for half a minute when I visit some pages on the UI, all requests (including background pings) stalling. After a while, during which the container has 100% CPU load, it magically works again. Seems like my system is fast enough to not deteriorate into an endless loop, but half a minute of no response from the server is still unacceptable.