MotionLM: Multi-Agent Motion Forecasting as Language Modeling
AriSeff BrianCera DianChen* MasonNg AurickZhou NigamaaNayakanti
KhaledS.Refaat RamiAl-Rfou BenjaminSapp
Waymo
Vocabulary
Abstract
Reliableforecastingofthefuturebehaviorofroadagents
isacriticalcomponenttosafeplanninginautonomousve-
hicles. Here, we represent continuous trajectories as se-
quencesofdiscretemotiontokensandcastmulti-agentmo-
tion prediction as a language modeling task over this do-
main. Ourmodel,MotionLM,providesseveraladvantages:
First,itdoesnotrequireanchorsorexplicitlatentvariable
optimizationtolearnmultimodaldistributions. Instead,we
leverage a single standard language modeling objective,
maximizing the average log probability over sequence to-
kens. Second, our approach bypasses post-hoc interaction
heuristics where individual agent trajectory generation is
conductedpriortointeractivescoring. Instead,MotionLM
producesjointdistributionsoverinteractiveagentfuturesin
a single autoregressive decoding process. In addition, the
model’ssequentialfactorizationenablestemporallycausal
conditional rollouts. The proposed approach establishes
new state-of-the-art performance for multi-agent motion
predictionontheWaymoOpenMotionDataset,ranking1st
ontheinteractivechallengeleaderboard.
1.Introduction
Modernsequencemodelsoftenemployanext-tokenpre-
dictionobjectivethatincorporatesminimaldomain-specific
assumptions. For example, autoregressive language mod-
els [3, 10] are pre-trained to maximize the probability of
thenextobservedsubwordconditionedontheprevioustext;
there is no predefined notion of parsing or syntax built in.
Thisapproachhasfoundsuccessincontinuousdomainsas
well,suchasaudio[2]andimagegeneration[49].Leverag-
ing the flexibility of arbitrary categorical distributions, the
aboveworksrepresentcontinuousdatawithasetofdiscrete
*WorkdoneduringaninternshipatWaymo.
Contact:{aseff, bensapp}@waymo.com
…
…
Motion token sequence: ...
t=1 t=2 t=T
Figure1. Ourmodelautoregressivelygeneratessequencesofdis-
cretemotiontokensforasetofagentstoproduceconsistentinter-
activetrajectoryforecasts.
tokens,reminiscentoflanguagemodelvocabularies.
In driving scenarios, road users may be likened to par-
ticipantsinaconstantdialogue,continuouslyexchanginga
dynamicseriesofactionsandreactionsmirroringthefluid-
ityofcommunication. Navigatingthisrichwebofinterac-
tionsrequirestheabilitytoanticipatethelikelymaneuvers
and responses of the involved actors. Just as today’s lan-
guage models can capture sophisticated distributions over
conversations,canweleveragesimilarsequencemodelsto
forecastthebehaviorofroadagents?
A common simplification to modeling the full future
worldstatehasbeentodecomposethejointdistributionof
agent behavior into independent per-agent marginal distri-
butions. Although there has been much progress on this
task [8, 47, 12, 25, 31, 5, 6, 21], marginal predictions are
insufficientasinputstoaplanningsystem;theydonotrep-
resentthefuturedependenciesbetweentheactionsofdiffer-
entagents,leadingtoinconsistentscene-levelforecasting.
Oftheexistingjointpredictionapproaches, someapply
3202
peS
82
]VC.sc[
1v43561.9032:viXraaseparationbetweenmarginaltrajectorygenerationandin- predictions, whicharelargelyunsupported bycurrent
teractivescoring[40,42,29]. Forexample,Luoetal.[29] jointforecastingmodels.
initiallyproduceasmallsetofmarginaltrajectoriesforeach
agentindependently,beforeassigningalearnedpotentialto 2.Relatedwork
each inter-agent trajectory pair through a belief propaga-
Marginaltrajectoryprediction. Behaviorpredictorsare
tion algorithm. Sun et al. [42] use a manual heuristic to
often evaluated on their predictions for individual agents,
tag agents as either influencers or reactors, and then pairs
e.g., in recent motion forecasting benchmarks [14, 9, 4,
marginal and conditional predictions to form joint predic-
51, 37]. Previous methods process the rasterized scene
tions.
with CNNs [8, 5, 12, 17]; the more recent works repre-
We also note that because these approaches do not ex-
sent scenes with points and polygraphs and process them
plicitly model temporal dependencies within trajectories,
withGNNs[6,25,47,22]ortransformers[31,40,20]. To
theirconditionalforecastsmaybemoresusceptibletospu-
handlethemultimodalityoffuturetrajectories,somemod-
rious correlations, leading to less realistic reaction predic-
elsmanuallyenforcediversityviapredefinedanchors[8,5]
tions. For example, these models can capture the cor-
or intention points [40, 52, 28]. Other works learn diverse
relation between a lead agent decelerating and a trail-
modeswithlatentvariablemodeling,e.g., [24].
ing agent decelerating, but may fail to infer which one is
While these works produce multimodal future trajecto-
likely causing the other to slow down. In contrast, previ-
riesofindividualagents,theyonlycapturethemarginaldis-
ousjointmodelsemployinganautoregressivefactorization,
tributionsofthepossibleagentfuturesanddonotmodelthe
e.g.,[36,43,39],dorespectfuturetemporaldependencies.
interactionsamongagents.
These models have generally relied on explicit latent vari-
ables for diversity, optimized via either an evidence lower
boundornormalizingflow. Interactive trajectory prediction. Interactive behavior
In this work, we combine trajectory generation and in- predictors model the joint distribution of agents’ futures.
teraction modeling in a single, temporally causal, decod- This task has been far less studied than marginal motion
ing process over discrete motion tokens (Fig. 1), leverag- prediction. Forexample,theWaymoOpenMotionDataset
ing a simple training objective inspired by autoregressive (WOMD)[14]challengeleaderboardcurrentlyhas71pub-
language models. Our model, MotionLM, is trained to lishedentriesformarginalpredictioncomparedtoonly14
directly maximize the log probability of these token se- forinteractionprediction.
quencesamonginteractingagents. Atinferencetime,joint Ngiametal.[32]modelsthedistributionoffuturetrajec-
trajectories are produced step-by-step, where interacting tories with a transformer-based mixture model outputting
agentssampletokenssimultaneously,attendtooneanother, jointmodes. Toavoidtheexponentialblow-upfromafull
andrepeat. Incontrasttopreviousapproacheswhichman- joint model, Luo et al. [29] models pairwise joint distri-
ually enforce trajectory multimodality during training, our butions. Tolstaya et al. [44], Song et al. [41], Sun et al.
modelisentirelylatentvariableandanchor-free,withmul- [42]considerconditionalpredictionsbyexposingthefuture
timodalityemergingsolelyasacharacteristicofsampling. trajectory of one agent when predicting for another agent.
MotionLMmaybeappliedtoseveraldownstreambehavior Shi et al. [40] derives joint probabilities by simply mul-
predictiontasks, includingmarginal, joint, andconditional tiplying marginal trajectory probabilities, essentially treat-
predictions. ing agents as independent, which may limit accuracy. Cui
Thisworkmakesthefollowingcontributions: etal.[11],Casasetal.[7],Girgisetal.[15]reducethefull-
fledgedjointdistributionusinggloballatentvariables. Un-
1. Wecastmulti-agentmotionforecastingasalanguage
likeourautoregressivefactorization,theabovemodelstypi-
modeling task, introducing a temporally causal de-
callyfollow“one-shot”(parallelacrosstime)factorizations
coderoverdiscretemotiontokenstrainedwithacausal
anddonotexplicitlymodeltemporallycausalinteractions.
languagemodelingloss.
2. We pair sampling from our model with a simple roll- Autoregressive trajectory prediction. Autoregressive
outaggregationschemethatfacilitatesweightedmode behaviorpredictorsgeneratetrajectoriesatintervalstopro-
identification for joint trajectories, establishing new duce scene-consistent multi-agent trajectories. Rhinehart
state-of-the-artperformanceontheWaymoOpenMo- et al. [36], Tang and Salakhutdinov [43], Amirloo et al.
tion Dataset interaction prediction challenge (6% im- [1], Salzmann et al. [39], Yuan et al. [50] predict multi-
provementintherankingjointmAPmetric). agent future trajectories using latent variable models. Lu
et al. [27] explores autoregressively outputting keyframes
3. We perform extensive ablations of our approach as via mixtures of Gaussians prior to filling in the remaining
well as analysis of its temporally causal conditional states. In [18], an adversarial objective is combined withScene Decoded motion tokens Ensemble & Rollout aggregation
embeddings
Scene … xE
Encoder
[R,N,ᐧ,H] …
Sample Sample Sample
… NMS
Projection
Autoregressive Transformer
Self Attention [R,N,T,2] [K,N,T,2]
Decoder
K-means
Embed Embed
Cross Attention …
… … A1 A1 AT-1 AT-1 AT AT
Agent 1 Agent 2
Start
token
Multimodal Time
Scene Inputs
t=0 t=1 t=T-1 t=T
Figure2.MotionLMarchitecture.Wefirstencodeheterogeneousscenefeaturesrelativetoeachmodeledagent(left)assceneembeddings
ofshapeR,N,·,H. Here,Rreferstothenumberofrollouts,N referstothenumberof(jointlymodeled)agents,andH isthedimen-
sionalityofeachembedding. WerepeattheembeddingsRtimesinthebatchdimensionforparallelsamplingduringinference. Next,a
trajectorydecoderautoregressivelyrollsoutT discretemotiontokensformultipleagentsinatemporallycausalmanner(center). Finally,
representativemodesoftherolloutsmayberecoveredviaasimpleaggregationutilizingk-meansclusteringinitializedwithnon-maximum
suppression(right).
parallel beam search to learn multi-agent rollouts. Unlike frameworkcapableofcapturingthesubstantialmultimodal-
mostautoregressivetrajectorypredictors, ourmethoddoes ityindrivingscenarios. Inaddition,wetakeconsideration
not rely on latent variables or beam search and generates here to preserve temporal dependencies; i.e., inference in
multimodal joint trajectories by directly sampling from a ourmodelfollowsadirectedacyclicgraphwiththeparents
learneddistributionofdiscretemotiontokensequences. ofeverynoderesidingearlierintimeandchildrenresiding
later (Section 3.3, Fig. 4). This enables conditional fore-
Discrete sequence modeling in continuous domains. casts that more closely resemble causal interventions [34]
Whengeneratingsequencesincontinuousdomains,oneef- byeliminatingcertainspuriouscorrelationsthatcanother-
fectiveapproachistodiscretizetheoutputspaceandpredict wise result from disobeying temporal causality2. We ob-
categoricaldistributionsateachstep. serve that joint models that do not preserve temporal de-
For example, in image generation, van den Oord et al. pendencies may have a limited ability to predict realistic
[45]sequentiallypredicttheuniformlydiscretizedpixelval- agentreactions–akeyuseinplanning(Section4.6).Tothis
ues for each channel and found this to perform better than end, we leverage an autoregressive factorization of our fu-
outputting continuous values directly. Multiple works on turedecoder,whereagents’motiontokensareconditionally
generating images from text such as [35] and [49] use a dependentonallpreviouslysampledtokensandtrajectories
two-stage process with a learned tokenizer to map images arerolledoutsequentially(Fig.2).
to discrete tokens and an autoregressive model to predict LetS representtheinputdataforagivenscenario. This
the discrete tokens given the text prompt. For audio gen- may include context such as roadgraph elements, traffic
eration, WaveNet [46] applies a µ-law transformation be- lightstates,aswellasfeaturesdescribingroadagents(e.g.,
fore discretizing. Borsos et al. [2] learn a hierarchical to- vehicles, cyclists, and pedestrians) and their recent histo-
kenizer/detokenizer, with the main transformer sequence ries, all provided at the current timestep t = 0. Our
.
modeloperatingontheintermediatediscretetokens. When task is to generate predictions for joint agent states Y =
t
generating polygonal meshes, Nash et al. [30] uniformly {y1,y2,...,yN}forN agentsofinterestatfuturetimesteps
t t t
quantizethecoordinatesofeachvertex. InMotionLM,we t = 1,...,T. Ratherthancompletestates,thesefuturestate
employasimpleuniformquantizationofaxis-aligneddeltas targetsaretypicallytwo-dimensionalwaypoints(i.e.,(x,y)
betweenconsecutivewaypointsofagenttrajectories. coordinates), with T waypoints forming the full ground
truthtrajectoryforanindividualagent.
3.MotionLM
We aim to model a distribution over multi-agent inter-
actions in a general manner that can be applied to distinct
2Wemakenoclaimsthatourmodeliscapableofdirectlymodeling
causalrelationships(duetothetheoreticallimitsofpurelyobservational
downstream tasks, including marginal, joint, and condi-
data and unobserved confounders). Here, we solely take care to avoid
tional forecasting. This requires an expressive generative breakingtemporalcausality.3.1.Jointprobabilisticrollouts Ourmodelissubjecttothesametheoreticallimitations
asgeneralimitationlearningframeworks(e.g.,compound-
In our modeling framework, we sample a predicted ac-
ing error [38] and self-delusions due to unobserved con-
tion for each target agent at each future timestep. These
founders [33]). However, we find that, in practice, these
actionsareformulatedasdiscretemotiontokensfromafi-
donotpreventstrongperformanceonforecastingtasks.
nite vocabulary, as described later in Section 3.2.2. Let
an represent the target action (derived from the ground 3.2.Modelimplementation
t .
truth waypoints) for the nth agent at time t, with A =
t
{a1,a2,...,aN}representingthesetoftargetactionsforall Our model consists of two main networks, an encoder
t t t
which processes initial scene elements followed by a tra-
agentsattimet.
jectorydecoderwhichperformsbothcross-attentiontothe
scene encodings and self-attention along agent motion to-
Factorization. Wefactorizethedistributionoverjointfu- kens,followingatransformerarchitecture[48].
tureactionsequencesasaproductofconditionals:
3.2.1 Sceneencoder
T
(cid:89)
p θ(A 1,A 2,...A T |S)= p θ(A t |A <t,S), (1) Thesceneencoder(Fig.2,left)istaskedwithprocessingin-
t=1 formationfromseveralinputmodalities,includingtheroad-
graph,trafficlightstates,andhistoryofsurroundingagents’
N trajectories. Here,wefollowthedesignoftheearlyfusion
(cid:89)
p θ(A t |A <t,S)= p θ(an t |A <t,S). (2) networkproposedby[31]asthesceneencodingbackbone
n=1 ofourmodel.Earlyfusionisparticularlychosenbecauseof
Similarto[36,43],Eq.(2)representsthefactthatwetreat itsflexibilitytoprocessallmodalitiestogetherwithminimal
agentactionsas conditionallyindependentattime t, given inductivebias.
thepreviousactionsandscenecontext. Thisalignsempiri- The features above are extracted with respect to each
callywithreal-worlddrivingovershorttimeintervals;e.g., modeledagent’sframeofreference. Inputtensorsarethen
non-impaired human drivers generally require at least 500 fedtoastackofself-attentionlayersthatexchangeinforma-
mstoreleasetheacceleratorinresponsetoavehiclebraking tion across all past timesteps and agents. In the first layer,
ahead([13]). Inourexperiments,wefind2Hzreactionsto latentqueriescross-attendtotheoriginalinputsinorderto
besufficienttosurpassstate-of-the-artjointpredictionmod- reduce the set of vectors being processed to a manageable
els. number,similarto[23,19]. Foradditionaldetails,see[31].
We note that our model’s factorization is entirely latent
variablefree;multimodalpredictionsstempurelyfromcat- 3.2.2 Jointtrajectorydecoder
egoricaltokensamplingateachrollouttimestep.
Ourtrajectorydecoder(Fig.2,center)istaskedwithgener-
atingsequencesofmotiontokensformultipleagents.
Trainingobjective. MotionLMisformulatedasagener-
ative model trained to match the joint distribution of ob- Discrete motion tokens. We elect to transform trajecto-
servedagentbehavior. Specifically,wefollowamaximum
ries comprised of continuous waypoints into sequences of
likelihooodobjectiveovermulti-agentactionsequences:
discretetokens. Thisenablestreatingsamplingpurelyasa
classificationtaskateachtimestep,implementedviaastan-
T
(cid:89) dard softmax layer. Discretizing continuous targets in this
argmax p (A |A ,S) (3)
θ t <t mannerhasproveneffectiveinotherinherentlycontinuous
θ
t=1
domains, e.g., in audio generation [46] and mesh genera-
tion[30]. Wesuspectthatdiscretemotiontokensalsonatu-
Similartothetypicaltrainingsetupofmodernlanguage
rallyhidesomeprecisionfromthemodel,possiblymitigat-
models,weutilize“teacher-forcing”wherepreviousground
ingcompoundingerroreffectsthatcouldarisefromimper-
truth (not predicted) tokens are provided at each timestep,
fectcontinuousvalueprediction. Likewise,wedidnotfind
which tends to increase stability and avoids sampling dur-
it necessary to manually add any noise to the ground truth
ingtraining. Wenotethatthisappliestoalltargetagents;in
teacher-forcedtrajectories(e.g.,asisdonein[26]).
training,eachtargetagentisexposedtogroundtruthaction
sequence prefixes for all target agents prior to the current
timestep. Thisnaturallyallowsfortemporalparallelization Quantization. To extract target discrete tokens, we be-
when using modern attention-based architectures such as gin by normalizing each agent’s ground truth trajectory
transformers[48]. with respect to the position and heading of the agent atFigure3.DisplayedarethetoptwopredictedjointrolloutmodesforthreeWOMDscenes.Colorgradientsindicatetimeprogressionfrom
t=0stot=8s,withthegreatestprobabilityjointmodetransitioningfromgreentoblueandthesecondaryjointmodetransitioningfrom
orangetopurple.Threetypesofinteractionsareobserved:anagentintheadjacentlaneyieldstothelane-changingagentaccordingtothe
timingofthelanechange(left),apedestrianwalksbehindthepassingvehicleaccordingtotheprogressofthevehicle(center),theturning
vehicleeitheryieldstothecrossingcyclist(mostprobablemode)orturnsbeforethecyclistapproaches(secondarymode)(right).
time t = 0 of the scenario. We then parameterize a uni- T foreachofN agents,weperformself-attentionoverNT
formly quantized (∆x,∆y) vocabulary according to a to- elements. While this does mean that these self-attended
talnumberofper-coordinatebinsaswellasmaximumand sequences grow linearly in the number of jointly modeled
minimum delta values. A continuous, single-coordinate agents, we note that the absolute sequence length here is
delta action can then be mapped to a corresponding index stillquitesmall(length32fortheWOMDinteractivesplit
∈ [0,num bins − 1], resulting in two indices for a com- – 8 sec. prediction at 2 Hz for 2 agents). Separate passes
plete (∆x,∆y) action per step. In order to extract actions offactorizedagentandtimeattentionarealsopossible[32],
thataccuratelyreconstructanentiretrajectory,weemploya butweuseasinglepasshereforsimplicity.
greedy search, sequentially selecting the quantized actions
that reconstruct the next waypoint coordinates with mini- Egoagentreferenceframes. Tofacilitatecross-attention
mumerror. to the agent-centric feature encodings (Section 3.2.1), we
We wrap the delta actions with a “Verlet” step where represent the flattened token sequence once for each mod-
a zero action indicates that the same delta index should eledagent.Eachmodeledagentistreatedasthe“ego”agent
be used as the previous step (as [36] does for continuous once,andcross-attentionisperformedonthatagent’sscene
states). As agent velocities tend to change smoothly be- features.Collapsingtheegoagentsintothebatchdimension
tweenconsecutivetimesteps,thishelpsreducethetotalvo- allowsparallelizationduringtrainingandinference.
cabularysize,simplifyingthedynamicsoftraining.Finally,
to maintain only T sequential predictions, we collapse the 3.3.Enforcingtemporalcausality
per-coordinateactionstoasingleintegerindexingintotheir
Our autoregressive factorization naturally respects tem-
Cartesian product. In practice, for the models presented
poraldependenciesduringjointrollouts;motiontokensam-
here,weuse13tokenspercoordinatewith132 =169total
pling for any particular agent is affected only by past to-
discretetokensavailableinthevocabulary(seeAppendixA
kens(fromanyagent)andunaffectedbyfutureones. When
forfurtherdetails).
training,werequireamasktoensurethattheself-attention
Wecomputealearnedvalueembeddingandtwolearned
operationonlyupdatesrepresentationsateachstepaccord-
positionalembeddings(representingthetimestepandagent
ingly. As shown in Fig. 8 (appendix), this attention mask
identity) for each discrete motion token, which are com-
exhibitsablocked,staircasepattern,exposingallagentsto
bined via an element-wise sum prior to being input to the
eachother’shistoriesonlyuptotheprecedingstep.
transformerdecoder.
Temporallycausalconditioning. Asdescribedearlier,a
Flattenedagent-timeself-attention. Weelecttoinclude particularbenefitofthisfactorizationistheabilitytoquery
asingleself-attentionmechanisminthedecoderthatoper- for temporally causal conditional rollouts (Fig. 4). In this
ates along flattened sequences of all modeled agents’ mo- setting, we fix a query agent to take some sequence of ac-
tiontokensovertime. So,givenatargetsequenceoflength tionsandonlyrollouttheotheragents.… …… ……
x=1 x=2 x=T x=1 x=2 x=T x=1 x=2 x=T
S S S
… … …
y=1 y=2 y=T y=1 y=2 y=T y=1 y=2 y=T
Conditioned Agent Causal conditioning
Sampling Agent Acausal conditioning
(a)CausalBayesianNetworkforjointrollouts. (b)Post-InterventionCausalBayesianNetwork. (c)Acausalconditioning.
Figure 4. A Causal Bayesian network representation for joint rollouts (left), post-intervention Causal Bayesian network (center), and
acausal conditioning (right). Solid lines indicate temporally causal dependencies while dashed lines indicate acausal information flow.
Modelswithouttemporaldependencyconstraintswillsupportacausalconditioningbutnottemporallycausalconditioning,whichcanbe
problematicwhenattemptingtopredictagentreactions.
We may view this as an approximation of computing tionsandconductablationsofourmodelingchoices.
causalinterventions[34]intheabsenceofconfounders;in-
terventionscannotbelearnedpurelythroughobservational 4.1.Datasets
dataingeneral(duetothepossiblepresenceofunobserved
WaymoOpenMotionDataset(WOMD). WOMD[14]
confounders), but our model’s factorization at least elim-
is a collection of 103k 20-second scenarios collected from
inates certain spurious correlations arising from breaking
real-world driving in urban and suburban environments.
temporalcausality.
Segments are divided into 1.1M examples consisting of 9-
InFig.4(a),weshowanexampleofaCausalBayesian
second windows of interest, where the first second serves
networkgoverningjointrollouts. Applyinganintervention
as input context and the remaining 8-seconds are the pre-
to nodes x = 1,...T, by deleting their incoming edges,
dictiontarget. Mapfeaturessuchastrafficsignalstatesand
results in a post-intervention Bayesian network depicted
lane features are provided along with agent states such as
inFig.4(b),whichobeystemporalcausality. Ontheother
position,velocity,acceleration,andboundingboxes.
hand,acausalconditioning(Fig.4(c))resultsinnon-causal
informationflow,wherenodex=iaffectsourbeliefabout
nodey =j fori≥j.
Marginalandinteractivepredictionchallenges. Forthe
3.4.Rolloutaggregation marginalpredictionchallenge,sixtrajectoriesmustbeout-
put by the model for each target agent, along with likeli-
Joint motion forecasting benchmark tasks like
hoodsofeachmode. Fortheinteractivechallenge,twoin-
WOMD [14] require a compact representation of the
teracting agents are labeled in each test example. In this
joint future distribution in the form of a small number of
case,themodelmustoutputsixweightedjointtrajectories.
joint “modes”. Each mode is assigned a probability and
might correspond to a specific homotopic outcome (e.g.,
4.2.Metrics
pass/yield) or more subtle differences in speed/geometry.
Here, we aggregate rollouts to achieve two primary goals: Theprimaryevaluationmetricsforthemarginalandin-
1) uncover the underlying modes of the distribution and teractive prediction challenges are soft mAP and mAP, re-
2) estimate the probability of each mode. Specifically, we spectively,withmissrateasthesecondarymetric. Distance
follow the non-maximum suppression (NMS) aggregation metricsminADEandminFDEprovideadditionalsignalon
scheme described in [47], but extend it to the joint setting predictionquality. Fortheinteractivepredictionchallenge,
by ensuring that all agent predictions reside within a thesemetricsrefertoscene-leveljointcalculations.Wealso
given distance threshold to the corresponding cluster. In useacustompredictionoverlapmetric(similarto[29])to
addition, we leverage model ensembling to account for assess scene-level consistency for joint models. See Ap-
epistemicuncertaintyandfurtherimprovethequalityofthe pendixCfordetailsonthesemetrics.
predictions,combiningrolloutsfromindependentlytrained
replicaspriortotheaggregationstep. 4.3.Modelconfiguration
Weexperimentwithupto8modelreplicasand512roll-
4.Experiments
outsperreplica,assessingperformanceatvariousconfigu-
We evaluate MotionLM on marginal and joint motion rations. Forcompleteactionspaceandmodelhyperparam-
forecasting benchmarks, examine its conditional predic- eterdetails,seeAppendicesAandB.Model minADE(↓) minFDE(↓) MissRate(↓) SoftmAP(↑)
HDGT[20] 0.7676 1.1077 0.1325 0.3709
MPA[22] 0.5913 1.2507 0.1603 0.3930
MTR[40] 0.6050 1.2207 0.1351 0.4216
Wayformerfactorized[31] 0.5447 1.1255 0.1229 0.4260
Wayformermulti-axis[31] 0.5454 1.1280 0.1228 0.4335
MTR-A[40] 0.5640 1.1344 0.1160 0.4594
MotionLM(Ours) 0.5509 1.1199 0.1058 0.4507
Table1.MarginalpredictionperformanceonWOMDtestset.Wedisplaymetricsaveragedovertimesteps(3,5,and8seconds)andagent
types(vehicles,pedestrians,andcyclists).Greyedcolumnsindicatetheofficialrankingmetricsforthemarginalpredictionchallenge.
Model minADE(↓) minFDE(↓) MissRate(↓) mAP(↑)
SceneTransformer(J)[32] 0.9774 2.1892 0.4942 0.1192
M2I[42] 1.3506 2.8325 0.5538 0.1239
DenseTNT[28] 1.1417 2.4904 0.5350 0.1647
MTR[40] 0.9181 2.0633 0.4411 0.2037
JFP[29] 0.8817 1.9905 0.4233 0.2050
MotionLM(Ours) 0.8911 2.0067 0.4115 0.2178
Table2. JointpredictionperformanceonWOMDinteractivetestset. Wedisplayscene-leveljointmetricsaveragedovertimesteps(3,5,
and8seconds)andagenttypes(vehicles,pedestrians,andcyclists).Greyedcolumnsindicatetheofficialrankingmetricsforthechallenge.
4.4.Quantitativeresults Model PredictionOverlap(↓)
Marginal motion prediction. As shown in Table 1, our LSTMBaseline[14] 0.07462
model is competitive with the state-of-the-art on WOMD Test SceneTransformer[32] 0.04336
JFP[29] 0.02671
marginal motion prediction (independent per agent). For
MotionLM(joint) 0.02607
themainrankingmetricofsoftmAP,ourmodelrankssec-
ond,lessthan2%behindthescoreachievedbyMTRA[40]. Val MotionLM(marginal) 0.0404
MotionLM(joint) 0.0292
Inaddition,ourmodelattainsasubstantiallyimprovedmiss
rateoverallpriorworks,witharelative9%reductioncom-
Table 3. Prediction overlap rates. Displayed is the custom pre-
pared to the previous state-of-the-art. The autoregressive
diction overlap metric for various model configurations on the
rolloutsareabletoadequatelycapturethediversityofmul-
WOMDinteractivetestandvalidationsets.
timodal future behavior without reliance on trajectory an-
chors[8]orstaticintentionpoints[40].
across the modeled agents during both training and infer-
Interactive motion prediction. Our model achieves ence rollouts, while the joint version performs 2 Hz inter-
state-of-the-art results for the interactive prediction chal- activeattention. Weseethatthemarginalversionobtainsa
lenge on WOMD, attaining a 6% relative improvement in relative38%higheroverlapratethanthejointversion. The
mAPand3%relativeimprovementinmissrate(thetwoof- interactiveattentioninthejointmodelallowstheagentsto
ficial ranking metrics) over the previous top scoring entry, moreappropriatelyreacttooneanother.
JFP[29](seeTable2).IncontrasttoJFP,ourapproachdoes
4.5.Ablationstudies
notscorepairsofpreviouslyconstructedmarginaltrajecto-
ries. but generates joint rollouts directly. Fig. 3 displays Interactive attention frequency. To assess the impor-
exampleinteractionspredictedbyourmodel. tance of inter-agent reactivity during the joint rollouts, we
Table 3 displays prediction overlap rates for various vary the frequency of the interactive attention operation
models on the WOMD interactive test and validation sets while keeping other architecture details constant. For our
(see metric details in Appendix C.2). We obtain test set leaderboard results, we utilize the greatest frequency stud-
predictions from the authors of [14, 32, 29]. MotionLM ied here, 2 Hz. At the low end of the spectrum, 0.125 Hz
obtains the lowest prediction overlap rate, an indication of correspondstotheagentsonlyobservingeachother’sinitial
scene-consistent predictions. In addition, on the valida- states,andthenproceedingwiththeentire8-secondrollout
tion set we evaluate two versions of our model: marginal withoutcommunicatingagain(i.e.,marginalrollouts).
andjoint. Themarginalversiondoesnotperformattention Performance metrics generally improve as agents are0.125 Hz 2 Hz
No interactive attention 16 interaction iterations
0.910
0.905
0.900
0.895
0.890
0.885
0.125 0.25 0.5 1 2
Interactive attention frequency (Hz)
Figure5.Visualizationofthetopjointrolloutmodeatthetwoex-
tremesoftheinteractiveattentionfrequenciesstudiedhere. With
nointeractiveattention(left),thetwomodeledagentsonlyattend
to each other once at the beginning of the 8-second rollout and
neveragain,incontrastto16totaltimesfor2Hzattention(right).
Theindependentrolloutsresultingfromzerointeractiveattention
canresultinscene-inconsistentoverlap;e.g.,aturningvehiclefails
toaccommodateacrossingpedestrian(topleft)oryieldappropri-
atelytoacrossingvehicle(bottomleft).
permitted to interact more frequently (Fig. 6 top, Table 5
in appendix). Greater interactive attention frequencies not
only lead to more accurate joint predictions, but also re-
duce implausible overlaps (i.e., collisions) between differ-
entagents’predictions. Fig.5displaysexampleswherethe
marginal predictions lead to implausible overlap between
agentswhilethejointpredictionsleadtoappropriatelysep-
aratedtrajectories. Seesupplementaryforanimatedvisual-
izations.
Numberofrollouts. Ourrolloutaggregationrequiresthat
wegenerateasufficientnumberofsamplesfromthemodel
in order to faithfully represent the multimodal future dis-
tribution. Forthisablation,wevarythenumberofrollouts
generated,butalwaysclusterdowntok =6modesforeval-
uation. Ingeneral,weseeperformancemetricsimproveas
additional rollouts are utilized (Fig. 6, bottom and Table 6
inappendix). Forourfinalleaderboardresults,weuse512
rolloutsperreplica,although32rolloutsissufficienttosur-
passtheprevioustopentryonjointmAP.
EDAnim
2.060
2.050
2.040
minADE 2.030
2.020
2.010
2.000
1.990
1.980
EDFnim
minFDE
0.200
0.190
0.180
0.170
0.160
0.150
1 2 4 8 16 32
Number of rollouts per replica
PAm
0.540
0.520
0.500
mAP
0.480
0.460
0.440
0.420
etar
ssiM
Miss rate
Figure 6. Joint prediction performance across varying interac-
tiveattentionfrequencies(top)andnumbersofrolloutsperreplica
(bottom) on the WOMD interactive validation set. Vertical axes
displayjoint(scene-level)metricsforan8-replicaensemble. See
Tables5and6intheappendixforfullparameterrangesandmet-
rics.
4.6.Conditionalrollouts
As described in Section 3.3, our model naturally sup-
ports “temporally causal” conditioning, similarly to previ-
ous autoregressive efforts such as [36, 43]. In this setting,
wefixonequeryagenttofollowaspecifiedtrajectoryand
stochastically roll out the target agent. However, we can
alsomodifythemodeltoleakthequeryagent’sfulltrajec-
tory,acausallyexposingitsfuturetothetargetagentduring
conditioning. This resembles the approach to conditional
predictionin,e.g.,[44],wherethisacausalconditionaldis-
tributionismodeleddirectly,or[29],wherethisdistribution
isaccessedviainferenceinanenergy-basedmodel.
Here, we assess predictions from our model across
threesettings: marginal,temporallycausalconditional,and
acausalconditional(Fig.4).Quantitatively,weobservethat
bothtypesofconditioningleadtomoreaccuratepredictions
for the target agent (Table 4, Fig. 7). Additionally, we see
thatacausalconditioningleadstogreaterimprovementthan
temporallycausalconditioningrelativetomarginalpredic-
tionsacrossallmetrics,e.g.,8.2%increaseinsoftmAPfor
acausalvs. 3.7%increasefortemporallycausal.Predictionsetting minADE(↓) minFDE(↓) MissRate(↓) SoftmAP(↑)
Marginal 0.6069 1.2236 0.1406 0.3951
Temporallycausalconditional 0.5997 1.2034 0.1377 0.4096
Acausalconditional 0.5899 1.1804 0.1338 0.4274
Table4.Conditionalpredictionperformance.Displayedaremarginal(single-agent)metricsacrossthreepredictionsettingsforourmodel
ontheWOMDinteractivevalidationset:marginal,temporallycausalconditional,andacausalconditional.
Intuitively, the greater improvement for acausal condi- Marginal Conditional
tioning makes sense as it exposes more information to the
model. However, the better quantitative scores are largely
duetopredictionsthatwouldbedeemednonsensicalifin-
terpretedaspredictedreactionstothequeryagent.
This can be illustrated in examples where one agent is
followinganother,wheretypicallytheleadagent’sbehavior
is causing the trailing agent’s reaction, and not vice versa,
but this directionality would not be captured with acausal
conditoning. Thistemporallycausalmodelingisespecially
importantwhenutilizingtheconditionalpredictionstoeval-
uatesafetyforanautonomousvehicle’sproposedplans. In Figure7. Visualizationofthemostlikelypredictedfutureforthe
pedestrianinthemarginalsetting(left)andtemporallycausalcon-
ascenariowhereanautonomousvehicle(AV)isstoppedbe-
ditionalsetting(right). Whenconsideringthepedestrianindepen-
hindanotheragent,planningtomoveforwardintotheother
dently,themodelassignsgreatestprobabilitytoatrajectorywhich
agent’s current position could be viewed as a safe maneu-
crosses the road. When conditioned on the the vehicle’s ground
verwithacausalconditioning,astheotheragentalsomov-
truthturn(magenta),thepedestrianisinsteadpredictedtoyield.
ing forward is correlated with (but not caused by) the AV
proceeding. However,itistypicallytheleadagentmoving
forwardthatcausesthetrailingAVtoproceed,andtheAV Acknowledgements. We would like to thank David
movingforwardonitsownwouldsimplyrear-endthelead Weiss,PaulCovington,AshishVenugopal,PiotrFidkowski,
agent. andMinfaWangfordiscussionsonautoregressivebehavior
In the supplementary, we compare examples of predic- predictors;ColeGulinoandBrandynWhiteforadvisingon
tionsinvariousscenariosforthecausalandacausalcondi- interactivemodeling;CheolPark,WenjieLuo,andScottEt-
tioning schemes. Models that ignore temporal dependen- tingerforassistancewithevaluation;DragoAnguelov,Kyr-
cies during conditioning (e.g., [44, 29]) may succumb to iacos Shiarlis, and anonymous reviewers for helpful feed-
thesameincorrectreasoningthattheacausalversionofour backonthemanuscript.
modeldoes.
5.Conclusionandfuturework
Inthiswork,weintroducedamethodforinteractivemo-
tion forecasting leveraging multi-agent rollouts over dis-
crete motion tokens, capturing the joint distribution over
multimodal futures. The proposed model establishes new
state-of-the-artperformanceontheWOMDinteractivepre-
dictionchallenge.
Avenues for future work include leveraging the trained
model in model-based planning frameworks, allowing a
searchtreetobeformedoverthemulti-agentactionrollouts,
or learning amortized value functions from large datasets
of scene rollouts. In addition, we plan to explore distilla-
tionstrategiesfromlargeautoregressiveteachers, enabling
fasterstudentmodelstobedeployedinlatency-criticalset-
tings.References [12] Henggang Cui, Vladan Radosavljevic, Fang-Chieh Chou,
Tsung-HanLin,ThiNguyen,Tzu-KuoHuang,JeffSchnei-
[1] Elmira Amirloo, Amir Rasouli, Peter Lakner, Mohsen Ro-
der,andNemanjaDjuric. Multimodaltrajectorypredictions
hani,andJunLuo. LatentFormer: Multi-agenttransformer-
forautonomousdrivingusingdeepconvolutionalnetworks.
basedinteractionmodelingandtrajectoryprediction,2022.
In2019InternationalConferenceonRoboticsandAutoma-
tion(ICRA),pages2090–2096.IEEE,2019.
[2] Zala´n Borsos, Raphae¨l Marinier, Damien Vincent, Eugene
Kharitonov,OlivierPietquin,MattSharifi,DominikRoblek,
[13] Johan Engstro¨m, Shu-Yuan Liu, Azadeh Dinparastdjadid,
Olivier Teboul, David Grangier, Marco Tagliasacchi, and
and Camelia Simoiu. Modeling road user response timing
NeilZeghidour. AudioLM:alanguagemodelingapproach
in naturalistic settings: a surprise-based framework, 2022.
toaudiogeneration,2023.
URLhttps://arxiv.org/abs/2208.08651.
[3] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Sub-
[14] Scott Ettinger, Shuyang Cheng, Benjamin Caine, Chenxi
biah,JaredDKaplan,PrafullaDhariwal,ArvindNeelakan-
Liu, Hang Zhao, Sabeek Pradhan, Yuning Chai, Ben-
tan,PranavShyam,GirishSastry,AmandaAskell,etal.Lan-
jamin Sapp, Charles R. Qi, Yin Zhou, Zoey Yang, Au-
guagemodelsarefew-shotlearners. Advancesinneuralin-
relien Chouard, Pei Sun, Jiquan Ngiam, Vijay Vasude-
formationprocessingsystems,33:1877–1901,2020.
van, Alexander McCauley, Jonathon Shlens, and Dragomir
Anguelov. Largescaleinteractivemotionforecastingforau-
[4] HolgerCaesar,VarunBankiti,AlexH.Lang,SourabhVora,
tonomousdriving:Thewaymoopenmotiondataset. CoRR,
VeniceErinLiong,QiangXu,AnushKrishnan,YuPan,Gi-
abs/2104.10133,2021.
ancarlo Baldan, and Oscar Beijbom. nuscenes: A multi-
modaldatasetforautonomousdriving. InCVPR,2020.
[15] Roger Girgis, Florian Golemo, Felipe Codevilla, Martin
[5] Sergio Casas, Wenjie Luo, and Raquel Urtasun. Intentnet: Weiss, JimAldonD’Souza, SamiraEbrahimiKahou, Felix
Learningtopredictintentionfromrawsensordata. InCon- Heide, and Christopher Pal. Latent variable sequential set
ferenceonRobotLearning,pages947–956,2018. transformersforjointmulti-agentmotionprediction. InIn-
ternationalConferenceonLearningRepresentations,2021.
[6] Sergio Casas, Cole Gulino, Renjie Liao, and Raquel Urta-
sun. Spagnn: Spatially-aware graph neural networks for [16] AriHoltzman,JanBuys,LiDu,MaxwellForbes,andYejin
relational behavior forecasting from sensor data. In 2020 Choi. Thecuriouscaseofneuraltextdegeneration. InInter-
IEEEInternationalConferenceonRoboticsandAutomation nationalConferenceonLearningRepresentations,2020.
(ICRA),pages9491–9497.IEEE,2020.
[17] JoeyHong,BenjaminSapp,andJamesPhilbin. Rulesofthe
[7] Sergio Casas, Cole Gulino, Simon Suo, Katie Luo, Renjie road:Predictingdrivingbehaviorwithaconvolutionalmodel
Liao,andRaquelUrtasun. Implicitlatentvariablemodelfor of semantic interactions. In Proceedings of the IEEE/CVF
scene-consistent motion forecasting. In European Confer- Conference on Computer Vision and Pattern Recognition,
enceonComputerVision,pages624–641.Springer,2020. pages8454–8462,2019.
[8] YuningChai,BenjaminSapp,MayankBansal,andDragomir [18] Maximilian Igl, Daewoo Kim, Alex Kuefler, Paul Mougin,
Anguelov. Multipath: Multipleprobabilisticanchortrajec- Punit Shah, Kyriacos Shiarlis, Dragomir Anguelov, Mark
tory hypotheses for behavior prediction. In Conference on Palatucci, Brandyn White, and Shimon Whiteson. Sym-
RobotLearning,pages86–99.PMLR,2020. phony:Learningrealisticanddiverseagentsforautonomous
drivingsimulation. InICRA,2022.
[9] Ming-Fang Chang, John Lambert, Patsorn Sangkloy, Jag-
jeetSingh,SlawomirBak,AndrewHartnett,DeWang,Peter [19] AndrewJaegle,FelixGimeno,AndrewBrock,AndrewZis-
Carr, Simon Lucey, Deva Ramanan, et al. Argoverse: 3d serman,OriolVinyals,andJoa˜oCarreira.Perceiver:General
trackingandforecastingwithrichmaps. InProceedingsof perceptionwithiterativeattention. InICML,2021.
theIEEE/CVFConferenceonComputerVisionandPattern
Recognition,pages8748–8757,2019. [20] XiaosongJia,PenghaoWu,LiChen,HongyangLi,YuLiu,
andJunchiYan. Hdgt: Heterogeneousdrivinggraphtrans-
[10] Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, formerformulti-agenttrajectorypredictionviasceneencod-
Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul ing. CoRL,2022.
Barham, Hyung Won Chung, Charles Sutton, Sebastian
Gehrmann, Parker Schuh, et al. PaLM: Scaling language [21] Siddhesh Khandelwal, William Qi, Jagjeet Singh, Andrew
modelingwithpathways,2022. Hartnett, and Deva Ramanan. What-if motion prediction
forautonomousdriving. arXivpreprintarXiv:2008.10587,
[11] Alexander Cui, Sergio Casas, Abbas Sadat, Renjie Liao, 2020.
andRaquelUrtasun. Lookout: Diversemulti-futurepredic-
tion and planning for self-driving. In Proceedings of the [22] StepanKonev.Mpa:Multipath++basedarchitectureformo-
IEEE/CVF International Conference on Computer Vision, tionprediction,2022.URLhttps://arxiv.org/abs/
pages16107–16116,2021. 2206.10041.[23] Juho Lee, Yoonho Lee, Jungtaek Kim, Adam R. Kosiorek, [34] JudeaPearl. Causality. CambridgeUniversityPress,2edi-
Seungjin Choi, and Yee Whye Teh. Set transformer: A tion,2009.
framework for attention-based permutation-invariant neural
networks. InICML,2019. [35] Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray,
ChelseaVoss,AlecRadford,MarkChen,andIlyaSutskever.
[24] NamhoonLee,WongunChoi,PaulVernaza,ChristopherB Zero-shottext-to-imagegeneration.InInternationalConfer-
Choy,PhilipHSTorr,andManmohanChandraker. Desire: enceonMachineLearning,pages8821–8831.PMLR,2021.
Distantfuturepredictionindynamicsceneswithinteracting
agents. InProceedingsoftheIEEEconferenceoncomputer [36] Nicholas Rhinehart, Rowan McAllister, Kris Kitani, and
visionandpatternrecognition,pages336–345,2017. SergeyLevine. PRECOG:Predictionconditionedongoals
invisualmulti-agentsettings. InICCV,2019.
[25] MingLiang,BinYang,RuiHu,YunChen,RenjieLiao,Song
Feng,andRaquelUrtasun. Learninglanegraphrepresenta- [37] A Robicquet, A Sadeghian, A Alahi, and S Savarese.
tions for motion forecasting. In European Conference on Learning social etiquette: Human trajectory prediction in
ComputerVision,pages541–556.Springer,2020. crowdedscenes. InEuropeanConferenceonComputerVi-
sion(ECCV),volume2,2020.
[26] JerryLiu,WenyuanZeng,RaquelUrtasun,andErsinYumer.
Deep structured reactive planning. In ArXiv, volume [38] StephaneRoss,GeoffreyGordon,andDrewBagnell. Are-
abs/2101.06832,2021. ductionofimitationlearningandstructuredpredictiontono-
regretonlinelearning. InAISTATS,2011.
[27] Qiujing Lu, Weiqiao Han, Jeffrey Ling, Minfa Wang,
Haoyu Chen, Balakrishnan Varadarajan, and Paul Coving-
[39] Tim Salzmann, Boris Ivanovic, Punarjay Chakravarty, and
ton. Kemp: Keyframe-based hierarchical end-to-end deep
Marco Pavone. Trajectron++: Multi-agent generative tra-
model for long-term trajectory prediction. In 2022 Inter-
jectoryforecastingwithheterogeneousdataforcontrol. In
national Conference on Robotics and Automation (ICRA),
ECCV,2020.
2022.
[40] Shaoshuai Shi, Li Jiang, Dengxin Dai, and Bernt Schiele.
[28] Ruikang Luo, Yaofeng Song, Han Zhao, Yicheng Zhang,
Motiontransformerwithglobalintentionlocalizationandlo-
YiZhang,NanbinZhao,LipingHuang,andRongSu.Dense-
calmovementrefinement. AdvancesinNeuralInformation
tnt:efficientvehicletypeclassificationneuralnetworkusing
ProcessingSystems,2022.
satelliteimagery. ICCV,2021.
[41] HaoranSong,WenchaoDing,YuxuanChen,ShaojieShen,
[29] WenjieLuo, CheolPark, AndreCornman, BenjaminSapp,
Michael Yu Wang, and Qifeng Chen. PiP: Planning-
andDragomirAnguelov. JFP:Structuredmulti-agentinter-
informed trajectory prediction for autonomous driving. In
activetrajectoriesforecastingforautonomousdriving.In6th
ECCV,2020.
AnnualConferenceonRobotLearning,2022.
[30] Charlie Nash, Yaroslav Ganin, S. M. Ali Eslami, and Pe- [42] QiaoSun,XinHuang,JunruGu,BrianWilliams,andHang
ter W. Battaglia. PolyGen: An autoregressive generative Zhao. M2I: From factored marginal trajectory prediction
modelof3Dmeshes. InternationalConferenceonMachine to interactive prediction. In Proceedings of the IEEE/CVF
Learning,2020. Conference on Computer Vision and Pattern Recognition
(CVPR),2022.
[31] NigamaaNayakanti,RamiAl-Rfou,AurickZhou,Kratarth
Goel, Khaled S. Refaat, and Benjamin Sapp. Wayformer: [43] YichuanCharlieTangandRuslanSalakhutdinov. Multiple
Motionforecastingviasimple&efficientattentionnetworks. futuresprediction. InAdvancesinneuralinformationpro-
ArXiv,abs/2207.05844,2022. cessingsystems,2019.
[32] Jiquan Ngiam, Vijay Vasudevan, Benjamin Caine, Zheng- [44] Ekaterina Tolstaya, Reza Mahjourian, Carlton Downey,
dong Zhang, Hao-Tien Lewis Chiang, Jeffrey Ling, Re- Balakrishnan Vadarajan, Benjamin Sapp, and Dragomir
becca Roelofs, Alex Bewley, Chenxi Liu, Ashish Venu- Anguelov. Identifyingdriverinteractionsviaconditionalbe-
gopal,DavidJWeiss,BenSapp,ZhifengChen,andJonathon havior prediction. In 2021 IEEE International Conference
Shlens. Scene transformer: A unified architecture for pre- onRoboticsandAutomation(ICRA),2021.
dictingfuturetrajectoriesofmultipleagents.InInternational
ConferenceonLearningRepresentations,2022. [45] Aa¨ron van den Oord, Nal Kalchbrenner, and Koray
Kavukcuoglu. Pixelrecurrentneuralnetworks. InInterna-
[33] PedroA.Ortega,MarkusKunesch,Gre´goireDele´tang,Tim tional conference on machine learning, pages 1747–1756.
Genewein, Jordi Grau-Moya, Joel Veness, Jonas Buchli, PMLR,2016.
Jonas Degrave, Bilal Piot, Julien Pe´rolat, Tom Everitt,
Corentin Tallec, Emilio Parisotto, Tom Erez, Yutian Chen, [46] Aa¨ron van den Oord, Sander Dieleman, Heiga Zen, Karen
ScottE.Reed,MarcusHutter,NandodeFreitas,andShane Simonyan,OriolVinyals,AlexanderGraves,NalKalchbren-
Legg. Shakingthefoundations:delusionsinsequencemod- ner,AndrewSenior,andKorayKavukcuoglu. WaveNet: A
elsforinteractionandcontrol.ArXiv,abs/2110.10819,2021. generativemodelforrawaudio. InArXiv,2016.[47] Balakrishnan Varadarajan, Ahmed Hefny, Avikalp Sri-
vastava, Khaled S. Refaat, Nigamaa Nayakanti, Andre
Cornman, Kan Chen, Bertrand Douillard, Chi-Pang Lam,
DragomirAnguelov,andBenjaminSapp. Multipath++: Ef-
ficientinformationfusionandtrajectoryaggregationforbe-
haviorprediction. InICRA,2022.
[48] AshishVaswani,NoamShazeer,NikiParmar,JakobUszko-
reit,LlionJones,AidanNGomez,ŁukaszKaiser,andIllia
Polosukhin.Attentionisallyouneed.InAdvancesinNeural
InformationProcessingSystems,2017.
[49] JiahuiYu,YuanzhongXu,JingYuKoh,ThangLuong,Gun-
janBaid,ZiruiWang,VijayVasudevan,AlexanderKu,Yin-
fei Yang, Burcu Karagol Ayan, Ben Hutchinson, Wei Han,
Zarana Parekh, Xin Li, Han Zhang, Jason Baldridge, and
YonghuiWu.Scalingautoregressivemodelsforcontent-rich
text-to-imagegeneration. TransactionsonMachineLearn-
ingResearch,2022.
[50] Ye Yuan, Xinshuo Weng, Yanglan Ou, and Kris M Kitani.
Agentformer: Agent-awaretransformersforsocio-temporal
multi-agent forecasting. In Proceedings of the IEEE/CVF
InternationalConferenceonComputerVision,pages9813–
9823,2021.
[51] Wei Zhan, Liting Sun, Di Wang, Haojie Shi, Aubrey
Clausse, Maximilian Naumann, Julius Kummerle, Hendrik
Konigshof, Christoph Stiller, Arnaud de La Fortelle, et al.
Interactiondataset: Aninternational, adversarialandcoop-
erative motion dataset in interactive driving scenarios with
semanticmaps. arXivpreprintarXiv:1910.03088,2019.
[52] Hang Zhao, Jiyang Gao, Tian Lan, Chen Sun, Ben Sapp,
BalakrishnanVaradarajan,YueShen,YiShen,YuningChai,
Cordelia Schmid, et al. Tnt: Target-driven trajectory pre-
diction. InConferenceonRobotLearning,pages895–904.
PMLR,2021.A.Motiontokenvocabulary N agents
Delta action space. The models presented in this paper
usethefollowingparametersforthediscretizeddeltaaction
space:
• Stepfrequency: 2Hz
• Deltainterval(perstep): [-18.0m,18.0m]
• Numberofbins: 128
At 2 Hz prediction, a maximum delta magnitude of 18 m
coversaxis-alignedspeedsupto36m/s(∼80mph),>99%
oftheWOMDdataset.
Verlet-wrapped action space. Once the above delta ac-
tion space has the Verlet wrapper applied, we only require
13 bins for each coordinate. This results in a total of
132 = 169totaldiscretemotiontokensthatthemodelcan
select from the Cartesian product comprising the final vo-
cabulary.
Sequence lengths. For 8-second futures, the model out-
puts 16 motion tokens for each agent (note that WOMD
evaluates predictions at 2 Hz). For the two-agent interac-
tive split, our flattened agent-time token sequences (Sec-
tion3.2.2)havelength2×16=32.
B.Implementationdetails
B.1.Sceneencoder
We follow the design of the early fusion network pro-
posedby[31]asthesceneencodingbackboneofourmodel.
Thefollowinghyperparametersareused:
• Numberoflayers: 4
• Hiddensize: 256
• Feed-forwardnetworkintermediatesize: 1024
• Numberofattentionheads: 4
• Numberoflatentqueries: 92
• Activation: ReLU
B.2.Trajectorydecoder
Toautoregressivelydecodemotiontokensequences,we
utilizeacausaltransformerdecoderthattakesinthemotion
tokensasqueries, andthesceneencodingsascontext. We
usethefollowingmodelhyperparameters:
• Numberoflayers: 4
• Hiddensize: 256
T
timesteps
Figure 8. Masked causal attention between two agents dur-
ing training. We flatten the agent and time axes, leading to an
NT ×NT attentionmask.Theagentsmayattendtoeachother’s
previousmotiontokens(solidsquares)butnofuturetokens(empty
squares).
• Feed-forwardnetworkintermediatesize: 1024
• Numberofattentionheads: 4
• Activation: ReLU
B.3.Optimization
We train our model to maximize the likelihood of the
ground truth motion token sequences via teacher forcing.
Weusethefollowingtraininghyperparameters:
• Numberoftrainingsteps: 600000
• Batchsize: 256
• Learningrateschedule: Lineardecay
• Initiallearningrate: 0.0006
• Finallearningrate: 0.0
• Optimizer: AdamW
• Weightdecay: 0.6
B.4.Inference
We found nucleus sampling [16], commonly used with
languagemodels,tobehelpfulforimprovingsamplequal-
ity while maintaining diversity. Here we set the top-p pa-
rameterto0.95.
C.Metricsdescriptions
C.1.WOMDmetrics
All metrics for the two WOMD [14] benchmarks are
evaluated at three time steps (3, 5, and 8 seconds) and are
averagedoverallobjecttypestoobtainthefinalvalue. For
jointmetrics,asceneisattributedtoanobjectclass(vehicle,
pedestrian, or cyclist) according to the least common type
ofagentthatispresentinthatinteraction,withcyclistbeingtherarestobjectclassandvehiclesbeingthemostcommon. the bounding boxes of two predicted agents collide at any
Up to 6 trajectories are produced by the models for each timestep in a scene, that counts as an overlap/collision for
target agent in each scene, which are then used for metric thatscene. Thefinalpredictionoverlaprateiscalculatedas
evaluation. thesumofper-sceneoverlaps,averagedacrossthedataset.
D.Additionalevaluation
mAP & Soft mAP mAP measures precision of predic-
tionlikelihoodsandiscalculatedbyfirstbucketingground
Ablations. Tables5and6displayjointpredictionperfor-
truthfuturesofobjectsintoeightdiscreteclassesofintent:
mance across varying interactive attention frequencies and
straight, straight-left, straight-right, left, right, left u-turn,
numbersofrollouts,respectively. Inadditiontotheensem-
rightu-turn,andstationary.
bledmodelperformance,singlereplicaperformanceiseval-
For marginal predictions, a prediction trajectory is con-
uated. Standard deviations are computed for each metric
sidereda“miss”ifitexceedsalateralorlongitudinalerror
over8independentlytrainedreplicas.
thresholdataspecifiedtimestepT. Similarlyforjointpre-
dictions, apredictionisconsidereda“miss”ifnoneofthe
Scaling analysis. Table 7 displays the performance of
kjointpredictionscontainstrajectoriesforallpredictedob-
different model sizes on the WOMD interactive split, all
jectswithinagivenlateralandlongitudinalerrorthreshold,
trained with the same optimization hyperparameters. We
withrespecttothegroundtruthtrajectoriesforeachagent.
varythenumberoflayers,hiddensize,andnumberofatten-
Trajectory predictions classified as a miss are labeled as a
tionheadsintheencoderanddecoderproportionally. Due
false positive. In the event of multiple predictions satisfy-
to external constraints, in this study we only train a single
ingthemisscriteria, consistentwithobjectdetectionmAP
replicaforeachparametercount. Weobservethatamodel
metrics,onlyonetruepositiveisallowedforeachscene,as-
with 27M parameters overfits while 300K underfits. Both
signed to the highest confidence prediction. All other pre-
the1Mand9Mmodelsperformdecently. Inthispaper,our
dictionsfortheobjectareassignedafalsepositive.
mainresultsuse9M-parameterreplicas.
To compute the mAP metric, bucket entries are sorted
and a P/R curve is computed for each bucket, averaging
Latency analysis. Table 8 provides inference latency on
precision values over various likelihood thresholds for all
the latest generation of GPUs across different numbers of
intent buckets results in the final mAP value. Soft mAP
rollouts. These were measured for a single-replica joint
differsonlyinthefactthatadditionalmatchingpredictions
modelrollingouttwoagents.
(otherthanthemostlikelymatch)areignoredinsteadofbe-
ingassignedafalsepositive,andsoarenotpenalizedinthe
E.Visualizations
metriccomputation.
Inthesupplementaryzipfile,wehaveincludedGIFan-
Missrate Usingthesamedefinitionofa“miss”described imations of the model’s greatest-probability predictions in
aboveforeithermarginalorjointpredictions,missrateisa various scenes. Each example below displays the associ-
measure of what fraction of scenarios fail to generate any atedsceneID,whichisalsocontainedinthecorresponding
predictionswithinthelateralandlongitudinalerrorthresh- GIFfilename. Wedescribetheexampleshere.
olds,relativetothegroundtruthfuture.
E.1.Marginalvs. Joint
minADE & minFDE minADE measures the Euclidean • Scene ID: 286a65c777726df3
distance error averaged over all timesteps for the closest Marginal: The turning vehicle and crossing cyclist
prediction, relative to ground truth. In contrast, minFDE collide.
considers only the distance error at the final timestep. For Joint: Thevehicleyieldstothecyclistbeforeturning.
joint predictions, minADE and minFDE are calculated as
• Scene ID: 440bbf422d08f4c0
theaveragevalueoverbothagents.
Marginal:Theturningvehiclecollideswiththecross-
C.2.Predictionoverlap ingvehicleinthemiddleoftheintersection.
Joint: The turning vehicle yields and collision is
As described in [29], the WOMD [14] overlap met-
avoided.
riconlyconsidersoverlapbetweenpredictionsandground
truth. Here we use a prediction overlap metric to assess • Scene ID: 38899bce1e306fb1
scene-levelconsistencyforjointmodels. Ourimplementa- Marginal: Thelane-changingvehiclegetsrear-ended
tion is similar to [29], except we follow the convention of bythevehicleintheadjacentlane.
the WOMD challenge of only requiring models to gener- Joint: The adjacent vehicle slows down to allow the
ate (x,y) waypoints; headings are inferred as in [14]. If lane-changingvehicletocompletethemaneuver.Ensemble SingleReplica
Freq.(Hz) minADE(↓) minFDE(↓) MR(↓) mAP(↑) minADE(↓) minFDE(↓) MR(↓) mAP(↑)
0.125 0.9120 2.0634 0.4222 0.2007 1.0681(0.011) 2.4783(0.025) 0.5112(0.007) 0.1558(0.007)
0.25 0.9083 2.0466 0.4241 0.1983 1.0630(0.009) 2.4510(0.025) 0.5094(0.006) 0.1551(0.006)
0.5 0.8931 2.0073 0.4173 0.2077 1.0512(0.009) 2.4263(0.022) 0.5039(0.006) 0.1588(0.004)
1 0.8842 1.9898 0.4117 0.2040 1.0419(0.014) 2.4062(0.032) 0.5005(0.008) 0.1639(0.005)
2 0.8831 1.9825 0.4092 0.2150 1.0345(0.012) 2.3886(0.031) 0.4943(0.006) 0.1687(0.004)
Table5. JointpredictionperformanceacrossvaryinginteractiveattentionfrequenciesontheWOMDinteractivevalidationset. Displayed
arescene-leveljointevaluationmetrics.Forthesinglereplicametrics,weincludethestandarddeviation(across8replicas)inparentheses.
Ensemble SingleReplica
#Rollouts minADE(↓) minFDE(↓) MR(↓) mAP(↑) minADE(↓) minFDE(↓) MR(↓) mAP(↑)
1 1.0534 2.3526 0.5370 0.1524 1.9827(0.018) 4.7958(0.054) 0.8182(0.003) 0.0578(0.004)
2 0.9952 2.2172 0.4921 0.1721 1.6142(0.011) 3.8479(0.032) 0.7410(0.003) 0.0827(0.004)
4 0.9449 2.1100 0.4561 0.1869 1.3655(0.012) 3.2060(0.035) 0.6671(0.003) 0.1083(0.003)
8 0.9158 2.0495 0.4339 0.1934 1.2039(0.013) 2.7848(0.035) 0.5994(0.004) 0.1324(0.003)
16 0.9010 2.0163 0.4196 0.2024 1.1254(0.012) 2.5893(0.031) 0.5555(0.005) 0.1457(0.003)
32 0.8940 2.0041 0.4141 0.2065 1.0837(0.013) 2.4945(0.035) 0.5272(0.005) 0.1538(0.004)
64 0.8881 1.9888 0.4095 0.2051 1.0585(0.012) 2.4411(0.033) 0.5114(0.005) 0.1585(0.004)
128 0.8851 1.9893 0.4103 0.2074 1.0456(0.012) 2.4131(0.033) 0.5020(0.006) 0.1625(0.004)
256 0.8856 1.9893 0.4078 0.2137 1.0385(0.012) 2.3984(0.031) 0.4972(0.007) 0.1663(0.005)
512 0.8831 1.9825 0.4092 0.2150 1.0345(0.012) 2.3886(0.031) 0.4943(0.006) 0.1687(0.004)
Table6. JointpredictionperformanceacrossvaryingnumbersofrolloutsperreplicaontheWOMDinteractivevalidationset. Displayed
arescene-leveljointevaluationmetrics.Forthesinglereplicametrics,weincludethestandarddeviation(across8replicas)inparentheses.
Parametercount MissRate(↓) mAP(↑) Numberofrollouts Latency(ms)
300K 0.6047 0.1054 16 19.9(0.19)
1M 0.5037 0.1713 32 27.5(0.25)
9M 0.4972 0.1663 64 43.8(0.26)
27M 0.6072 0.1376 128 75.8(0.23)
256 137.7(0.19)
Table7. Jointpredictionperformanceacrossvaryingmodelsizes
on the WOMD interactive validation set. Displayed are scene- Table8. InferencelatencyoncurrentgenerationofGPUsfordif-
leveljointmAPandmissratefor256rolloutsforasinglemodel ferentnumbersofrolloutsofthejointmodel.Wedisplaythemean
replica(exceptfor9Mwhichdisplaysthemeanperformanceof8 and standard deviation (in parentheses) of the latency measure-
replicas). mentsforeachsetting.
E.2.Marginalvs. Conditional
• Scene ID: 2ea76e74b5025ec7
Marginal: The cyclist crosses in front of the vehicle “Conditional”herereferstotemporallycausalcondition-
leadingtoacollision. ingasdescribedinthemaintext.
Joint: Thecyclistwaitsforthevehicletoproceedbe-
foreturning. • Scene ID: 5ebba77f351358e2
Marginal: Thepedestriancrossesthestreetasavehi-
cleisturning,leadingtoacollision.
• Scene ID: 55b5fe989aa4644b
Conditional: When conditioning on the vehicle’s
Marginal: Thecyclistlanechangesinfrontofthead-
turningtrajectoryasaquery, the pedestrian isinstead
jacentvehicle,leadingtocollision.
predictedtoremainstationary.
Joint:Thecyclistremainsintheirlanefortheduration
ofthescene,avoidingcollision. • Scene ID: d557eee96705c822Marginal: Themodeledvehiclecollideswiththelead
vehicle.
Conditional:Whenconditioningontheleadvehicle’s
query trajectory, which remains stationary for a bit,
themodeledvehicleinsteadcomestoaanappropriate
stop.
• Scene ID: 9410e72c551f0aec
Marginal: Themodeledvehicletakestheturnslowly,
unawareofthelastturningvehicle’sprogress.
Conditional: When conditioning on the query vehi-
cle’sturnprogress,themodeledagentlikewisemakes
moreprogress.
• Scene ID: c204982298bda1a1
Marginal: Themodeledvehicleproceedsslowly,un-
awareofthemergingvehicle’sprogress.
Conditional: When conditioning on the query vehi-
cle’s merge progress, the modeled agent accelerates
behind.
E.3.TemporallyCausalvs. AcausalConditioning
• Scene ID: 4f39d4eb35a4c07c
Jointprediction: Thetwomodeledvehiclesmaintain
speedforthedurationofthescene.
Conditioningontrailingagent:
- Temporally causal: The lead vehicle is indifferent
tothequerytrailingvehicledeceleratingtoastop,pro-
ceedingalongataconstantspeed.
- Acausal: The lead vehicle is “influenced” by the
queryvehicledecelerating.Itlikewisecomestoastop.
Intuitively, this is an incorrect direction of influence
thattheacausalmodelhaslearned.
Conditioningonleadagent:
-Temporallycausal:Whenconditioningonthequery
lead vehicle decelerating to a stop, the modeled trail-
ingvehicleislikewisepredictedtostop.
-Acausal: In this case, the acausal conditional pre-
dictionissimilartothetemporallycausalconditional.
The trailing vehicle is predicted to stop behind the
queryleadvehicle.