ST-MemRESEARCH arXiv

NeurIPS 2026 Accepted paper

Linguistic Trajectory Encodingfor Efficient Long-Horizon Spatial Memory
in Embodied Agents

Object-centric spatiotemporal memory
for long-horizon embodied agents.

Tianyidan Xie1Shenyi Wang1Qiang Tang2Mingjie Wang3Zhicheng Qiu4 Xuanfu Li4Zhan Xu4Jian Yang1Lanjun Wang5,*Zili Yi1,*
1Nanjing University2University of British Columbia3Zhejiang Sci-Tech University4Huawei Technologies Co., Ltd.5Tianjin University

* Corresponding authors: Zili Yi and Lanjun Wang

Read the paper Watch the film SMB benchmark BibTeX

THE RESEARCH FILM

LTE in motion.

See how object histories connect language, spatial anchors, and visual evidence across real egocentric video.

METHOD · REAL EGOLIFE EXAMPLES · RESULTS Open video ↗
The film uses the paper framework and real EgoLife examples to show trajectory retrieval and linked spatial and visual evidence. SMB and Ego4D numbers are paper-reported results.

Abstract

From arXiv v1 ↗

Embodied agents performing long-horizon tasks require a memory representation in which the state transitions of dynamic objects remain queryable in natural language across hours-to-days observation horizons. Existing systems either drop fine-grained motion (clip-level video-language embeddings), keep it only as raw coordinates (geometric SLAM), or organise it around immediate task context (agent working memories). None of them gives the agent a per-object timeline whose state transitions are themselves queryable in language. Our key contribution is Linguistic Trajectory Encoding (LTE), which compresses dynamic object motion histories via a hybrid representation combining natural language descriptions, sparse spatial anchors, and visual anchors. LTE adapts compression to motion complexity by anchoring periods without reliable observations to the last seen location, while representing motion with geometric waypoints and linguistic descriptions to preserve accuracy. To evaluate these capabilities across extended time horizons, we construct the Spatial Memory Benchmark (SMB) from EgoLife multi-day recordings, targeting capabilities absent in existing benchmarks: semantic trajectory retrieval and long-horizon object retrieval. On SMB, the LTE-based system achieves 45.3% success in semantic trajectory retrieval and 48.7% in long-horizon object retrieval, outperforming structured-memory and VLM baselines (best prior: 31.9% and 34.4%). LTE achieves trajectory compression by factors of 8.7× to 26.1× with sub-second query latency on 24 h video. On Ego4D natural-language queries, the system reaches 28.75% / 55.10% R@1/R@5, +15.80 / +31.30 pts over EgoVLPv2.

01 / THE IDEA AT A GLANCEFigure 1 from the paper
The paper teaser contrasts scanning every video clip with an LTE memory. Language descriptions, spatial anchors and visual evidence connect an apple, a book and keys to their histories in a reconstructed home.
From hours of observations to an object-centric memory: retrieve what happened, where, and when through linked evidence. Numbers in this original figure are paper-reported results, not measurements of the core code release.

THE PROBLEM

Remembering a place is not enough.
You need to remember what changed.

“Where did I last put the washed apple?” asks for an object, a state change, and a location at a particular moment. LTE gives each object a compact, language-queryable history, linking descriptions of its motion to sparse 3D positions and visual evidence. The memory preserves the connections between these three forms of evidence across long observation horizons.

02 / THE METHOD

Three kinds of evidence.
One object history.

LTE combines complementary channels in a single per-object record.

C / LANGUAGE

What changed?

Interval captions describe an object’s motion and state in language, making its history semantically searchable.

“Apple moved from sink to counter.”
Aspatial / GEOMETRY

Where did it happen?

Sparse 3D waypoints preserve the shape of a trajectory. Complex motion keeps more anchors. Simpler motion needs fewer.

World position + observation time
Avisual / IDENTITY

Which object was it?

Crops and bounding boxes ground the record in what was actually seen, supporting visual matching and identity checks.

Object crop + frame reference

Missing observations are not new evidence. Tracking gaps retain the last-seen location. The representation does not pretend to observe an object while it is out of view.

FROM PERCEPTION TO RETRIEVALFigure 2 from the paper
Framework diagram: perception produces structured observations. The memory combines an octree with linguistic trajectory encoding. Object, Scene, Text, Event and Image views support NLQ, VQ2D, STR and LOR queries.
The paper’s full architecture organizes shared object records into five complementary views. The public code includes the memory core and optional video perception integrations. Audio, navigation, and the complete evaluation pipeline remain outside its scope.

03 / ASK THE MEMORY

Different questions.
Connected evidence.

Four retrieval tasks probe time, visual identity, motion, and the last observed location.

EGO4D / NATURAL LANGUAGE QUERIES

“When did I wash the apple?”

Locate relevant moments in an observation history using a natural-language description.

EVIDENCE TO RETRIEVE
  • Language-matched intervals
  • Temporal bounds
  • Linked object context

EGO4D / VISUAL QUERIES 2D

Find this object in the past.

A reference object crop is the query. Visual matching retrieves earlier evidence of that object.

EVIDENCE TO RETRIEVE
  • Matching visual anchors
  • Image-space bounding boxes
  • Frame and time references

SMB / SEMANTIC TRAJECTORY RETRIEVAL

“Show the apple’s path from the sink to the counter.”

Retrieve an object’s trajectory using semantic state, spatial, and temporal constraints.

EVIDENCE TO RETRIEVE
  • Language-described motion phases
  • Sparse 3D trajectory anchors
  • Linked visual evidence

SMB / LONG-HORIZON OBJECT RETRIEVAL

“Where did I last leave my keys?”

Search a long lookback window for an object’s most recent observed location.

EVIDENCE TO RETRIEVE
  • Last observed position
  • Time of the observation
  • Object identity evidence

Illustrative task examples, not live inference or recorded query results. The core code exposes explicit filters. This page does not run models.

04 / RESULTS IN THE PAPER

Compact memory.
Long-horizon retrieval.

The Spatial Memory Benchmark (SMB) contains 600 queries across EgoLife multi-day recordings: 300 trajectory queries and 300 long-horizon object queries.

SMB task definitions, protocol, and availability ↗

45.3%SMB trajectory retrieval+13.4 points over Q3VL-235B + GD
48.7%SMB last-occurrence retrieval+14.3 points over Q3VL-235B + GD
8.7–26.1×Trajectory compressionLTE-only storage vs. dense trajectories
0.43sPer-query latency at 24 hSingle A800 · after memory construction

A memory representation,
not a larger video model.

The strongest VLM baseline scans clips. LTE instead retains object-level histories, with linked language, geometry and visual anchors.

On Ego4D NLQ at IoU = 0.3, the paper reports 28.75% R@1 and 55.10% R@5.

Read the experimental settings and limitations
Spatial Memory Benchmark · success rate (%)
Method STR ↑ LOR ↑
Q3VL-8B + GD 21.5 25.1
Q3VL-235B + GD 31.9 34.4
KFMem 19.8 33.8
VideoAgent 24.7 30.5
LTE-based system (ours) 45.3 48.7

Source: arXiv:2609.04802v1, §4. These are reported research results, not new measurements of the scoped core release. Compression refers to LTE trajectory storage, not the total system footprint. Query latency excludes preprocessing and memory construction.

05 / THE DEMO

Run ST-Mem locally.

Build, save, reload, and query an LTE memory. Start with the model-free CPU example, or process your own video with the optional GPU perception pipeline.

AVAILABLE

The CPU demo uses synthetic observations. The video path uses local model checkpoints. Neither is a live browser demo or benchmark reproduction.

CPU quickstart Real-video guide
CORE + VIDEO INTEGRATIONS

Included

LTE · five linked views
Portable memory · four query APIs
SAM3 tracking · ViPE geometry
DINOv2 features · Qwen3-VL descriptions

Outside this demo

Model weights / source footage
Benchmark evaluation / training
Audio and navigation

The video path requires CUDA and separate model environments. Paper-reported benchmark results are not reproduced by these demos.

The SMB description is public. Benchmark annotations and evaluation scripts are not distributed here.

06 / CITATION

Build on this work.

Accepted at NeurIPS 2026. Please cite the arXiv version below until proceedings metadata is available.

arXiv:2609.04802

DOI: 10.48550/arXiv.2609.04802

BIBTEX
Download .bib 
@misc{xie2026linguistictrajectory,
  title = {Linguistic Trajectory Encoding for Efficient Long-Horizon Spatial Memory in Embodied Agents},
  author = {Tianyidan Xie and Shenyi Wang and Qiang Tang and Mingjie Wang and Zhicheng Qiu and Xuanfu Li and Zhan Xu and Jian Yang and Lanjun Wang and Zili Yi},
  year = {2026},
  note = {Accepted at NeurIPS 2026},
  eprint = {2609.04802},
  archivePrefix = {arXiv},
  primaryClass = {cs.CV},
  doi = {10.48550/arXiv.2609.04802},
  url = {https://arxiv.org/abs/2609.04802}
}

Paper figure

Original resolution · scroll to explore on small screens.