ST-MemRESEARCH arXiv

Embodied intelligence / arXiv preprint · 2026

Linguistic Trajectory Encodingfor Efficient Long-Horizon Spatial Memory
in Embodied Agents

Object-centric spatiotemporal memory
for long-horizon embodied agents.

Linguistic Trajectory Encoding (LTE) connects natural-language descriptions, sparse spatial anchors, and visual evidence into queryable object state histories. ST-Mem is the LTE-based memory system; SMB is the paper’s long-horizon retrieval benchmark.

Tianyidan Xie1,*Shenyi Wang1Qiang Tang2Mingjie Wang3Zhicheng Qiu4 Xuanfu Li4Zhan Xu4Jian Yang1Lanjun Wang5Zili Yi1,*
1Nanjing University2University of British Columbia3Zhejiang Sci-Tech University4Huawei Technologies Co., Ltd.5Tianjin University

* Corresponding authors: Tianyidan Xie and Zili Yi

Read the paper Code coming soon SMB benchmark BibTeX

The paper is public. Code and installation instructions are coming soon.

Abstract

From arXiv v1 ↗

Embodied agents performing long-horizon tasks require a memory representation in which the state transitions of dynamic objects remain queryable in natural language across hours-to-days observation horizons. Existing systems either drop fine-grained motion (clip-level video-language embeddings), keep it only as raw coordinates (geometric SLAM), or organise it around immediate task context (agent working memories). None of them gives the agent a per-object timeline whose state transitions are themselves queryable in language. Our key contribution is Linguistic Trajectory Encoding (LTE), which compresses dynamic object motion histories via a hybrid representation combining natural language descriptions, sparse spatial anchors, and visual anchors. LTE adapts compression to motion complexity by anchoring periods without reliable observations to the last seen location, while representing motion with geometric waypoints and linguistic descriptions to preserve accuracy. To evaluate these capabilities across extended time horizons, we construct the Spatial Memory Benchmark (SMB) from EgoLife multi-day recordings, targeting capabilities absent in existing benchmarks: semantic trajectory retrieval and long-horizon object retrieval. On SMB, the LTE-based system achieves 45.3% success in semantic trajectory retrieval and 48.7% in long-horizon object retrieval, outperforming structured-memory and VLM baselines (best prior: 31.9% and 34.4%). LTE achieves trajectory compression by factors of 8.7× to 26.1× with sub-second query latency on 24 h video. On Ego4D natural-language queries, the system reaches 28.75% / 55.10% R@1/R@5, +15.80 / +31.30 pts over EgoVLPv2.

01 / THE IDEA AT A GLANCEFigure 1 from the paper
The paper teaser contrasts scanning every video clip with an LTE memory. Language descriptions, spatial anchors and visual evidence connect an apple, a book and keys to their histories in a reconstructed home.
From hours of observations to an object-centric memory: retrieve what happened, where, and when through linked evidence. Numbers in this original figure are paper-reported results, not measurements of the core code release.

THE PROBLEM

Remembering a place is not enough.
You need to remember what changed.

“Where did I last put the washed apple?” asks for an object, a state change, and a location at a particular moment. LTE gives each object a compact, language-queryable history, linking descriptions of its motion to sparse 3D positions and visual evidence. The memory preserves the connections between these three forms of evidence across long observation horizons.

02 / THE METHOD

Three kinds of evidence.
One object history.

LTE combines complementary channels in a single per-object record.

C / LANGUAGE

What changed?

Interval captions describe an object’s motion and state in language, making its history semantically searchable.

“Apple moved from sink to counter.”
Aspatial / GEOMETRY

Where did it happen?

Sparse 3D waypoints preserve the shape of a trajectory. Complex motion keeps more anchors; simpler motion needs fewer.

World position + observation time
Avisual / IDENTITY

Which object was it?

Crops and bounding boxes ground the record in what was actually seen, supporting visual matching and identity checks.

Object crop + frame reference

Missing observations are not new evidence. Tracking gaps retain the last-seen location; the representation does not pretend to observe an object while it is out of view.

FROM PERCEPTION TO RETRIEVALFigure 2 from the paper
Framework diagram: perception produces structured observations; the memory combines an octree with linguistic trajectory encoding; Object, Scene, Text, Event and Image views support NLQ, VQ2D, STR and LOR queries.
The paper’s full architecture organizes shared object records into five complementary views. The current code release implements a scoped offline memory core; audio and the complete evaluation pipeline are not included.

03 / ASK THE MEMORY

Different questions.
Connected evidence.

Four retrieval tasks probe time, visual identity, motion, and the last observed location.

EGO4D / NATURAL LANGUAGE QUERIES

“When did I wash the apple?”

Locate relevant moments in an observation history using a natural-language description.

EVIDENCE TO RETRIEVE
  • Language-matched intervals
  • Temporal bounds
  • Linked object context

EGO4D / VISUAL QUERIES 2D

Find this object in the past.

A reference object crop is the query. Visual matching retrieves earlier evidence of that object.

EVIDENCE TO RETRIEVE
  • Matching visual anchors
  • Image-space bounding boxes
  • Frame and time references

SMB / SEMANTIC TRAJECTORY RETRIEVAL

“Show the apple’s path from the sink to the counter.”

Retrieve an object’s trajectory using semantic state, spatial, and temporal constraints.

EVIDENCE TO RETRIEVE
  • Language-described motion phases
  • Sparse 3D trajectory anchors
  • Linked visual evidence

SMB / LONG-HORIZON OBJECT RETRIEVAL

“Where did I last leave my keys?”

Search a long lookback window for an object’s most recent observed location.

EVIDENCE TO RETRIEVE
  • Last observed position
  • Time of the observation
  • Object identity evidence

Illustrative task examples, not live inference or recorded query results. The core code exposes explicit filters; this page does not run models.

04 / RESULTS IN THE PAPER

Compact memory.
Long-horizon retrieval.

The Spatial Memory Benchmark (SMB) contains 600 queries across EgoLife multi-day recordings: 300 trajectory queries and 300 long-horizon object queries.

SMB task definitions, protocol, and availability ↗

45.3%SMB trajectory retrieval+13.4 points over Q3VL-235B + GD
48.7%SMB last-occurrence retrieval+14.3 points over Q3VL-235B + GD
8.7–26.1×Trajectory compressionLTE-only storage vs. dense trajectories
0.43sPer-query latency at 24 hSingle A800 · after memory construction

A memory representation,
not a larger video model.

The strongest VLM baseline scans clips. LTE instead retains object-level histories, with linked language, geometry and visual anchors.

On Ego4D NLQ at IoU = 0.3, the paper reports 28.75% R@1 and 55.10% R@5.

Read the experimental settings and limitations
Spatial Memory Benchmark · success rate (%)
Method STR ↑ LOR ↑
Q3VL-8B + GD 21.5 25.1
Q3VL-235B + GD 31.9 34.4
KFMem 19.8 33.8
VideoAgent 24.7 30.5
LTE-based system (ours) 45.3 48.7

Source: arXiv:2609.04802v1, §4. These are reported research results, not new measurements of the scoped core release. Compression refers to LTE trajectory storage, not the total system footprint; query latency excludes preprocessing and memory construction.

05 / THE CODE

Code coming soon.

We are preparing a focused release of the ST-Mem memory core for long-term embodied agent memory and natural-language spatial retrieval.

COMING SOON

The implementation is not publicly available yet. Code, installation instructions, and functional validation notes will be linked here when the release is ready.

PLANNED CORE RELEASE

Planned scope

LTE · five linked views
Portable memory · four query APIs
Offline perception adapters

Outside the planned release

Full benchmark reproduction
Evaluation / training pipelines
Audio and navigation

The planned release focuses on core functionality, not full reproduction of the paper’s benchmark results.

The SMB description is public; benchmark annotations and evaluation scripts are not distributed here.

06 / CITATION

Build on this work.

If this research is useful to your work, please cite the paper.

arXiv:2609.04802

DOI: 10.48550/arXiv.2609.04802

BIBTEX
Download .bib 
@misc{xie2026linguistictrajectory,
  title = {Linguistic Trajectory Encoding for Efficient Long-Horizon Spatial Memory in Embodied Agents},
  author = {Tianyidan Xie and Shenyi Wang and Qiang Tang and Mingjie Wang and Zhicheng Qiu and Xuanfu Li and Zhan Xu and Jian Yang and Lanjun Wang and Zili Yi},
  year = {2026},
  eprint = {2609.04802},
  archivePrefix = {arXiv},
  primaryClass = {cs.CV},
  doi = {10.48550/arXiv.2609.04802},
  url = {https://arxiv.org/abs/2609.04802}
}

Paper figure

Original resolution · scroll to explore on small screens.