======================== RELEVANT AGENT METRICS FILTER ========================

This is the Relevant Agent Metrics Filter (aka the RAM Filter), a
Region-Of-Interest-based track filter that uses Perception, Planner, and
Prediction code to identify a set of input tracks (from a variety of different
sources) that interact with a given ROI "envelope".

This filter is designed to be somewhat of a compartmentalized/black-box system
that has a fairly limited set of straightforward APIs. The key feature is that
the filter is highly configurable, allowing for many different potential input
sources, ROI envelopes, and forms of track position prediction.

Additionally the RAM Filter is being deployed with a new CI metrics target,
RAM-CI. RAM-CI is built in a lightweight way on top of Perception-CI, so as to
leverage the existing metrics suites/algorithms already in use. However, this CI
pipeline is by-and-large meant to serve as an example of how to deploy one of
the RAM filters in an arbitrary piece of code.

There are 3 types of RAM Filter, largely separated by the type of position
prediction that the filter can do:

  1. UnstructuredRamFilter - Uses instantaneous 3d velocity of each track to
     predict future track positions.
  2. StructuredRamFilter - Uses a PredictionClient engine to predict the
     state/position of each track at some future time.
  3. PerfectRamFilter - Uses "perfect" knowledge of future track positions
     instead of predicting.

Depending on the filter class being used, there are different potential input
sources (though it should be fairly easy to extend any of the filters with a new
input source provided it has all of the necessary data), any of which serve
simply as a container for track boxes and velocities:

For UnstructuredRamFilter, acceptable inputs are:
  - prediction::PredictionInternalState (e.g. from
    "/prediction/prediction_internal_state")
  - common::proto::PerceptionObstacles2Proto (e.g. from
    "/driving/PerceptionObstacles2")
  - map of {track_id : (track_box_2d, track_vel_2d)} (in local_utm)

For StructuredRamFilter, acceptable inputs are:
  - prediction::PredictionInternalState (e.g. from
    "/prediction/prediction_internal_state")
  - common::proto::PerceptionObstacles2Proto (e.g. from
    "/driving/PerceptionObstacles2")
  [NB: Cannot feed the simple map in as per UnstructuredRamFilter since you need
  much more information to run Prediction].

For PerfectRamFilter, acceptable inputs are:
  - LMDB containing a payload map of one of:
    - {std::string (ts) : lidar::annotation::Spin}
    - {std::string (ts) : lidar::annotation::Frame}
    - {std::string (ts) : common::proto::PerceptionObstacles2Proto}
  - Run/meta id and timestamp range

(These filters should be fairly easy to extend with new datatypes though!)

Building a RamFilter is extremely straightforward, since most of the cross-team
linkages are handled under the hood. To build a RamFilter, all you need is a
Region (NB: THE REGION MUST BE CONSTRUCTED PROPERLY WITH THE OFFSET OTHERWISE
THERE WILL BE A CONSISTENT POSITIONAL BIAS BETWEEN THE ENVELOPES AND THE
TRACKS) and an envelope configuration (either in the form of a proto object, or
one of the pbtxt proto files can be loaded).

Running the filter depends on the desired filter type as well as the desired
output. Each filter has different filtering options depending on the envelope
configuration, the input data, and the function called. For clarity, the
simplest one-line control flow is described below. For further information about
what each one-line call is doing, please refer directly to the .cpp
implementations in each filter.

UnstructuredRamFilter:
  ```
  cas::proto::RamEnvelopeProto envelope_config;
  std::shared_ptr<roads::Region> region;
  UnstructuredRamFilter ram_filter(envelope_config, region);
  // e.g. from "/driving/PerceptionObstacles2"
  common::proto::PerceptionObstacles2Proto po2_proto;
  // e.g. from "/planner/hero_state"
  common::proto::HeroState hero_state_proto;
  // e.g. from "/planner/mission"
  common::proto::MissionRoute hero_route_proto;
  const auto& frame_relevant_track_map = ram_filter.getFilteredTracks(
      po2_proto,
      hero_state_proto,
      hero_route_proto);
  ```

  This will instantiate an UnstructuredRamFilter, build the relevant track map
  corresponding to the frame encapsulated by the PO2, HeroState, and
  MissionRoute protos, and return the track relevance map for the given frame.

StructuredRamFilter:
  ```
  cas::proto::RamEnvelopeProto envelope_config;
  std::shared_ptr<roads::Region> region;
  StructuredRamFilter ram_filter(envelope_config, region);
  // e.g. from "/driving/PerceptionObstacles2"
  common::proto::PerceptionObstacles2Proto po2_proto;
  // e.g. from "/planner/hero_state"
  common::proto::HeroState hero_state_proto;
  // e.g. from "/planner/mission"
  common::proto::MissionRoute hero_route_proto;
  const auto& frame_relevant_track_map = ram_filter.getFilteredTracks(
      po2_proto,
      hero_state_proto,
      hero_route_proto);
  ```

  This will instantiate a StructuredRamFilter, build the relevant track map
  corresponding to the frame encapsulated by the PO2, HeroState, and
  MissionRoute protos, and return the track relevance map for the given frame.

PerfectRamFilter:
  ```
  cas::proto::RamEnvelopeProto envelope_config;
  std::shared_ptr<roads::Region> region;
  PerfectRamFilter ram_filter(envelope_config, region);
  std::string meta_id = "xxx-KITTxx";
  int64_t start_ts = xxx;
  int64_t end_ts = yyy;
  ram_filter.generateRelevantTrackMap(meta_id, start_ts, end_ts);
  const auto& relevant_track_map = ram_filter.getFilteredTracks();
  ```

  This will instantiate a PerfectRamFilter, build the relevant track map for the
  entire run (within the ts range specified by [start_ts, end_ts]), and grabs
  the populated map (so that the user only needs to generate the map once). Note
  that this gets the track relevance map for the entire log. For a similar
  interface to the Unstructured/Structured filters, PerfectRamFilter also
  provides an alternate interface that takes a timestamp lookup and returns a
  map that mirrors the other filters' return values:

  ```
  ...
  ram_filter.generateRelevantTrackMap(meta_id, start_ts, end_ts);
  const double lookup_ts = xxx;
  const auto& frame_relevant_track_map =
      ram_filter.getFilteredTracksFrom(lookup_ts);
  ```

  PerfectRamFilter::getFilteredTracksFrom() pretends as if the caller is in the
  frame keyed by "lookup_ts" and returns a similar map to the other filters as a
  result (see the filter implementations for more explanation if this is not
  clear). There is one final interface called
  PerfectRamFilter::getFilteredTracksNear() which assumes that the "lookup_ts"
  is not one of the frame ts values populated internally and simply starts at
  the closest frame to the lookup time.

=============================== RAM-CI PIPELINE ===============================

The main feature that leverages RelevantAgentMetrics filters in the v1 release
is a new CI pipeline! The "RAM-CI" pipeline is built in a fairly streamlined way
on top of Perception-CI such that a set of PerfectRamFilters looks through the
generated metrics matches and prunes matches that don't satisfy certain
criteria. Downstream of the RAM Filters, the metrics that are calculated using
the PO2<->GT matches will then only use matches deemed to be "relevant" by the
filters (i.e. they contain either a Perception or GT track that was deemed
relevant by RAM).

The goal of the RAM-CI pipeline is to start to narrow down metrics results in a
way that targets CI metrics towards specific purposes (instead of a "general"
set of metrics on all tracks seen in Perception). To do this, the RAM-CI
pipeline maintains two PerfectRamFilters for every log: one to generate a track
relevance map for the PO2 output, and one to generate a track relevance map for
the groundtruth output. The pipeline runs the standard Perception-CI tracking
metrics matcher to get the PO2<->GT matches, and then runs the set of matches
through the filters. If a match DOES NOT contain either a "relevant" GT id nor a
"relevant" PO2 id for that frame, then that match is removed altogether. All of
the downstream metrics that are generated from these matches are unmodified from
the original Perception-CI pipeline.

The important part of configuring the RAM-CI pipeline then simply comes down to
the envelope that the user is filtering with. This is straightforward to use as
well, as there is an optional field that the user can specify in
"lidar/metrics/data/tracking_metrics_params_default.prototxt" called
"ram_envelope_config". If the user creates an envelope config pbtxt file, this
flag (defined in the proto from "lidar/metrics/proto/tracking_metrics.proto")
can be set with either a hard path to the pbtxt file, or just the pbtxt file
name (as long as the pbtxt file is placed in "lidar/metrics/cas_pcp/data/").

Finally, there is a new script target to run the RAM-CI pipeline:

  $ZOOX_WORKSPACE_ROOT/lidar/metrics/run_ram_ci [--email xxx@zoox.com --slack]

This script is pretty much exactly the same as run_perception_ci, but calls a
target that runs Perception-CI with an additional argument flag (`ram_ci`).
The Perception-CI pipeline then knows how to instantiate the PerfectRamFilters
properly and use them to filter results. This method keeps the actual
infrastructure changes to Perception-CI extremely minimal, however if
drastically different metrics are needed then this most likely will need to have
its own "RamCiRunner" (analogous to the PerceptionCiRunner).

NB/ADDENDUM: One of the things RAM Filters need to run is the published hero
mission routes from Planner. If this message is not published in the log, the
RAM Filter can't rebuild the hero trajectories and therefore we can't build the
interaction envelopes. Since we can't have one log without Planner bringing down
the entire RAM-CI pipeline, there is another tool in this directory
(`build_ram_ci_dataset.py`) that will take the weekly Perception-CI dataset and
filter out any logs that do not have a message coming over "/planner/mission".
The script then saves that dataset file to
`/mnt/nautilus_rw/3dbt_data/perception_ci/ram_ci/ram_ci_data_set.pbtxt`, where
the perception_ci_runner can then find it with a helper method.