---
title: Benchmarks
description: How well openmemory's recall finds the right conversation on the LongMemEval benchmark, what was and wasn't measured, and how to reproduce it.
---

Measured on 2 November 2026 with [LongMemEval](https://huggingface.co/datasets/xiaowu0162/longmemeval), a public benchmark of 500 questions, each with about 50 past conversations to search through. Unanswerable questions are left out, leaving 470.

## What it measures

Each question gets a fresh space with one memory per conversation turn, about 490 in all, and then runs the real `recall`. A question counts as a hit when a turn from a conversation holding the answer is in the top results.

## Results

| Search | Questions | Top 1 | Top 5 | Top 10 |
| --- | --- | --- | --- | --- |
| Words only | 470 | 85.5% | **94.0%** | 96.2% |
| Words only, without word forms | 470 | 84.9% | 93.2% | 96.2% |
| Words only | 20, evenly spaced | 90% | 100% | 100% |
| Words and meaning | the same 20 | 100% | 100% | 100% |
| Meaning only | the same 20 | 95% | 95% | 95% |
| Words and meaning | 12 of the 28 questions words alone missed | 33.3% | **50%** | 75% |

Meaning search used `@cf/baai/bge-small-en-v1.5` with a similarity cut-off of 0.52.

## How to read this

- **There is no full score for words and meaning.** Embedding all 470 question sets would pass the service's free daily allowance many times over, so that mode ran on samples.
- **This is retrieval, not answer accuracy.** Nothing here answers the question. Benchmarks that have an LLM answer and a second LLM judge the result measure something harder, so don't compare these numbers with theirs.
- **Questions keyword search missed are the hard ones.** The 50% is what meaning search adds on those; it isn't a score for all questions.
- **It isn't how openmemory is meant to be used.** Agents store short facts, and this stores whole conversation turns. The numbers measure search, not an agent's memory end to end.
- **Weakest area:** preference questions, with 19 of 30 in the top 5 on words alone. They often share few words with the conversation.

## Run it yourself

The script is `apps/api/scripts/bench-longmemeval.ts`. Download `longmemeval_s` from the dataset, then from `apps/api`:

```bash
pnpm bench longmemeval_s.json --mode keyword
```

For meaning search, it calls Workers AI, so set `CLOUDFLARE_ACCOUNT_ID` and `CLOUDFLARE_API_TOKEN`. Embeddings are cached on disk, and `--max-neurons` stops the run before it passes a budget:

```bash
pnpm bench longmemeval_s.json --mode hybrid --sample 20 --cache ./embed-cache --max-neurons 4500
```