Benchmarks
How well openmemory's recall finds the right conversation on the LongMemEval benchmark, what was and wasn't measured, and how to reproduce it.
Measured on 2 November 2026 with LongMemEval, a public benchmark of 500 questions, each with about 50 past conversations to search through. Unanswerable questions are left out, leaving 470.
What it measures
Each question gets a fresh space with one memory per conversation turn, about 490 in all, and then runs the real recall. A question counts as a hit when a turn from a conversation holding the answer is in the top results.
Results
| Search | Questions | Top 1 | Top 5 | Top 10 |
|---|---|---|---|---|
| Words only | 470 | 85.5% | 94.0% | 96.2% |
| Words only, without word forms | 470 | 84.9% | 93.2% | 96.2% |
| Words only | 20, evenly spaced | 90% | 100% | 100% |
| Words and meaning | the same 20 | 100% | 100% | 100% |
| Meaning only | the same 20 | 95% | 95% | 95% |
| Words and meaning | 12 of the 28 questions words alone missed | 33.3% | 50% | 75% |
Meaning search used @cf/baai/bge-small-en-v1.5 with a similarity cut-off of 0.52.
How to read this
- There is no full score for words and meaning. Embedding all 470 question sets would pass the service's free daily allowance many times over, so that mode ran on samples.
- This is retrieval, not answer accuracy. Nothing here answers the question. Benchmarks that have an LLM answer and a second LLM judge the result measure something harder, so don't compare these numbers with theirs.
- Questions keyword search missed are the hard ones. The 50% is what meaning search adds on those; it isn't a score for all questions.
- It isn't how openmemory is meant to be used. Agents store short facts, and this stores whole conversation turns. The numbers measure search, not an agent's memory end to end.
- Weakest area: preference questions, with 19 of 30 in the top 5 on words alone. They often share few words with the conversation.
Run it yourself
The script is apps/api/scripts/bench-longmemeval.ts. Download longmemeval_s from the dataset, then from apps/api:
pnpm bench longmemeval_s.json --mode keywordFor meaning search, it calls Workers AI, so set CLOUDFLARE_ACCOUNT_ID and CLOUDFLARE_API_TOKEN. Embeddings are cached on disk, and --max-neurons stops the run before it passes a budget:
pnpm bench longmemeval_s.json --mode hybrid --sample 20 --cache ./embed-cache --max-neurons 4500Build your own agent loop
The pattern for a custom agent - get_context at the start of a run, one atomic batch at the end, approvals through the action ledger, and nightly dreaming on your own model.
Self-hosting
Run openmemory yourself on Node.js with a local SQLite file, or on your own Cloudflare account with Workers, D1, and Vectorize. Same API, same MCP server.