microduck/mjlab
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Import upstream snapshot c19f713c415a699a79d71cd96aa13c3104a05047
nightly / Test against latest dependencies (py3.10) (push) Has been cancelled
nightly / Test against latest dependencies (py3.13) (push) Has been cancelled
tests / tests (3.13, locked) (push) Has been cancelled
tests / tests (3.13, unlocked) (push) Has been cancelled
tests / pyright (3.10) (push) Has been cancelled
tests / lint-format (push) Has been cancelled
tests / tests (3.10, locked) (push) Has been cancelled
tests / tests (3.11, locked) (push) Has been cancelled
tests / tests (3.12, locked) (push) Has been cancelled
tests / pyright (3.11) (push) Has been cancelled
tests / pyright (3.12) (push) Has been cancelled
tests / pyright (3.13) (push) Has been cancelled
tests / ty-check (3.10) (push) Has been cancelled
tests / ty-check (3.11) (push) Has been cancelled
tests / ty-check (3.12) (push) Has been cancelled
tests / ty-check (3.13) (push) Has been cancelled
tests / stubs (push) Has been cancelled
tests / smoke-test (push) Has been cancelled
Docker / check_paths (push) Has been cancelled
Docker / build (push) Has been cancelled
docs / build (push) Has been cancelled

Browse Source 
Upstream: https://github.com/michaelgillett/mjlab
Upstream-Commit: c19f713c415a699a79d71cd96aa13c3104a05047
Upstream-Branch: main
This commit is contained in:
Upstream Snapshot
2026-08-28 15:42:17 +08:00
commit 32a241c28f
513 changed files with 120771 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/source/sensors/index.rst
+455
View File
@@ -0,0 +1,455 @@
.. _sensors:
Sensors
=======
As described in :ref:`entity`, sensors sit between ``EntityData`` and
raw simulation arrays in mjlab's data access hierarchy. At their
simplest, they wrap MuJoCo sensor primitives with a clean interface
that maps to real robot hardware. Beyond wrapping, they are a general
abstraction for transforming simulation data into structured outputs:
``ContactSensor`` aggregates contact pairs with reduction and air time
tracking, ``RayCastSensor`` performs GPU-accelerated terrain scanning,
``CameraSensor`` renders RGB and depth images on the GPU, and the base
``Sensor`` class can be subclassed for custom measurement logic.
Sensors are configured at the **scene level**, not on individual entities. A
sensor can reference an entity element (a contact sensor on the robot's
feet, an accelerometer attached to a body site), but it can also be
independent of any entity entirely. This is why sensors live in
``SceneCfg`` rather than ``EntityCfg``.
.. code-block:: python
from mjlab.sensor import (
BuiltinSensorCfg, ContactSensorCfg, ContactMatch, ObjRef,
)
# A robot with an IMU accelerometer and foot contact detection.
scene_cfg = SceneCfg(
entities={"robot": robot_cfg},
sensors=(
BuiltinSensorCfg(
name="imu_acc",
sensor_type="accelerometer",
obj=ObjRef(type="site", name="imu_site", entity="robot"),
),
ContactSensorCfg(
name="feet_contact",
primary=ContactMatch(
mode="geom", pattern=r".*_foot$", entity="robot",
),
secondary=ContactMatch(mode="body", pattern="terrain"),
fields=("found", "force"),
),
),
)
# Access at runtime.
imu = env.scene["robot/imu_acc"].data # [B, 3] acceleration
feet = env.scene["feet_contact"].data # ContactData
feet.found # [B, P] contact count per foot
feet.force # [B, P, 3] contact force per foot
mjlab provides four sensor types: ``BuiltinSensor`` for native MuJoCo
measurements, ``ContactSensor`` for structured contact detection,
``RayCastSensor`` for GPU-accelerated raycasting, and ``CameraSensor``
for RGB-D rendering. The base ``Sensor`` class can be subclassed for
custom measurement logic; see `Extending: custom sensors`_ below.
BuiltinSensor
-------------
``BuiltinSensor`` wraps MuJoCo's native sensor types. Each sensor is
attached to a MuJoCo element (site, joint, body, etc.) via ``ObjRef``
and returns a ``torch.Tensor`` with shape ``[num_envs, dim]`` where
``dim`` depends on the sensor type (3 for vectors, 4 for quaternions,
1 for scalars).
+-----------+----------------------------------------------------------------------------------------------------------------------------------------------------+
| Category | Available Sensors |
+===========+====================================================================================================================================================+
| **Site** | ``accelerometer``, ``velocimeter``, ``gyro``, ``force``, ``torque``, ``magnetometer``, ``rangefinder`` |
+-----------+----------------------------------------------------------------------------------------------------------------------------------------------------+
| **Joint** | ``jointpos``, ``jointvel``, ``jointlimitpos``, ``jointlimitvel``, ``jointlimitfrc``, ``jointactuatorfrc`` |
+-----------+----------------------------------------------------------------------------------------------------------------------------------------------------+
| **Frame** | ``framepos``, ``framequat``, ``framexaxis``, ``frameyaxis``, ``framezaxis``, ``framelinvel``, ``frameangvel``, ``framelinacc``, ``frameangacc`` |
+-----------+----------------------------------------------------------------------------------------------------------------------------------------------------+
| **Other** | ``actuatorpos``, ``actuatorvel``, ``actuatorfrc``, ``subtreecom``, ``subtreelinvel``, ``subtreeangmom``, ``clock``, ``e_potential``, ``e_kinetic`` |
+-----------+----------------------------------------------------------------------------------------------------------------------------------------------------+
``ObjRef`` identifies which MuJoCo element the sensor attaches to. The
``entity`` field scopes the lookup to a specific entity's namespace, and
the sensor name is auto-prefixed accordingly (e.g., ``"imu_acc"`` on
entity ``"robot"`` becomes ``"robot/imu_acc"``).
.. code-block:: python
from mjlab.sensor import BuiltinSensorCfg, ObjRef
# Accelerometer attached to a site.
BuiltinSensorCfg(
name="imu_acc",
sensor_type="accelerometer",
obj=ObjRef(type="site", name="imu_site", entity="robot"),
)
# Joint limit sensor with output clamping.
BuiltinSensorCfg(
name="knee_limit",
sensor_type="jointlimitpos",
obj=ObjRef(type="joint", name="knee_joint", entity="robot"),
cutoff=0.1,
)
# Relative frame position (end-effector w.r.t. base).
BuiltinSensorCfg(
name="ee_pos",
sensor_type="framepos",
obj=ObjRef(type="body", name="end_effector", entity="robot"),
ref=ObjRef(type="body", name="base", entity="robot"),
)
Auto-discovery
^^^^^^^^^^^^^^
Sensors already defined in an entity's XML are automatically discovered
during scene composition and prefixed with the entity name. There is no
need to create a ``BuiltinSensorCfg`` for these.
.. code-block:: xml
<!-- In robot.xml -->
<sensor>
<accelerometer name="trunk_imu" site="imu_site"/>
<jointpos name="hip_sensor" joint="hip_joint"/>
</sensor>
.. code-block:: python
# Access by prefixed name.
imu = env.scene["robot/trunk_imu"]
hip = env.scene["robot/hip_sensor"]
ContactSensor
-------------
Each physics step, MuJoCo produces a flat, unstructured list of contact
pairs across the entire scene. A single foot geom might generate several
simultaneous contacts with the ground, interleaved with contacts from
other entities. ``ContactSensor`` filters this raw list to the pairs you
care about, reduces multiple contacts per element down to a fixed count,
and packages the result into clean, batched tensors your policy can
consume directly. It builds on MuJoCo's native
`contact sensor <https://mujoco.readthedocs.io/en/stable/XMLreference.html#sensor-contact>`_.
Primary and secondary
^^^^^^^^^^^^^^^^^^^^^
Contacts are pairwise: you typically want to know "did the robot's feet
touch the terrain?", not just "did something touch something."
``primary`` defines the elements you are measuring (the feet).
``secondary`` optionally restricts what they are contacting (the
terrain). When ``secondary`` is ``None``, any contact with a primary
element counts.
Each side is specified with a ``ContactMatch``. The ``mode`` selects the
MuJoCo element type (``"geom"``, ``"body"``, or ``"subtree"``) and the
``pattern`` accepts a regex or tuple of regexes matched against element
names within the entity.
.. code-block:: python
from mjlab.sensor import ContactSensorCfg, ContactMatch
# Foot geoms contacting the terrain body.
ContactSensorCfg(
name="feet_ground",
primary=ContactMatch(
mode="geom", pattern=r".*_foot$", entity="robot",
),
secondary=ContactMatch(mode="body", pattern="terrain"),
fields=("found", "force"),
)
# Self-collision: pelvis subtree against itself.
ContactSensorCfg(
name="self_collision",
primary=ContactMatch(
mode="subtree", pattern="pelvis", entity="robot",
),
secondary=ContactMatch(
mode="subtree", pattern="pelvis", entity="robot",
),
fields=("found",),
)
Output shape
^^^^^^^^^^^^
A pattern like ``r".*_foot$"`` resolves to ``P`` primary elements (e.g.
four feet on a quadruped). Each primary becomes one column on the
per-contact axis of the output tensors:
.. list-table::
:header-rows: 1
:widths: 35 25 40
* - Field group
- Shape
- Notes
* - Per-contact
(``found``, ``force``, ``torque``, ``dist``, ``pos``, ``normal``,
``tangent``)
- ``[B, P * num_slots, ...]``
- Primary-major: indices
``[i * num_slots : (i + 1) * num_slots]`` belong to primary ``i``.
* - Per-primary
(``current_air_time``, ``last_air_time``,
``current_contact_time``, ``last_contact_time``)
- ``[B, P]``
- Air-time fields are accumulated per primary, reducing across
slots (any slot in contact counts as the primary in contact).
With the default ``num_slots=1`` the two shape families coincide
(``N == P``), which is why most code can treat both as ``[B, P, ...]``.
Use :attr:`mjlab.sensor.contact_sensor.ContactSensor.primary_names` to
recover the index-to-name mapping after pattern expansion:
.. code-block:: python
sensor = env.scene["feet_contact"]
sensor.primary_names # ["FR_foot", "FL_foot", "RR_foot", "RL_foot"]
sensor.data.current_air_time[:, 0] # air time for FR_foot
Reduction
^^^^^^^^^
A single primary element can have many simultaneous contacts with the
secondary (e.g. a flat foot resting on rough terrain has multiple
contact points). The ``reduce`` mode collapses those raw contacts down
to ``num_slots`` representative contacts:
.. list-table::
:header-rows: 1
:widths: 20 80
* - Mode
- Behavior
* - ``"none"``
- Fast, non-deterministic selection of up to ``num_slots`` contacts.
* - ``"mindist"``
- Keep the deepest ``num_slots`` contacts.
* - ``"maxforce"``
- Keep the strongest ``num_slots`` contacts by force magnitude.
* - ``"netforce"``
- Sum all contacts into a single net wrench at the force-weighted
centroid. Always emits one slot per primary regardless of
``num_slots``.
When to set ``num_slots > 1``
"""""""""""""""""""""""""""""
Almost all configurations leave ``num_slots`` at its default of ``1``,
because pattern expansion already produces one column per element of
interest (one per foot, one per finger, one per body link). Increase
``num_slots`` only when a single primary may have several physically
distinct contact points that you want to inspect separately, for
example:
- Computing a center of pressure on a flat foot from its corner
contacts.
- Reasoning about grasp quality from multiple fingertip-to-object
contact points.
- Detecting tipping by watching whether one corner of a contact patch
loses contact.
In those cases pair ``num_slots`` with ``reduce`` in
``{"mindist", "maxforce", "none"}``. With ``reduce="netforce"`` it has
no effect.
.. note::
``num_slots`` is a ceiling on what the sensor will store, not a
guarantee that MuJoCo will produce that many contacts. The collision
detector caps the number of contact points generated for each
geom-pair according to the geom types involved. For example a
sphere-vs-plane pair produces at most one contact, and a box-vs-box
pair produces at most four. Setting ``num_slots=8`` on a sphere
primary against a plane secondary therefore leaves seven slots
permanently zero. Check
`MuJoCo's collision documentation <https://mujoco.readthedocs.io/en/stable/computation/index.html#collision-detection>`_
for the per-pair limits.
Fields
^^^^^^
The ``fields`` tuple selects which contact quantities to extract. Only
requested fields are allocated; the rest are ``None`` on the output
dataclass. Available fields are ``"found"``, ``"force"``,
``"torque"``, ``"dist"``, ``"pos"``, ``"normal"``, and ``"tangent"``.
.. note::
``torque`` and the friction-tangent component of ``force`` are zero
unless the contact pair has friction enabled, which requires
``condim >= 3`` on at least one geom in the pair. With ``condim=1``
(frictionless), the contact only produces a normal force. This is a
physics property of the contact, not a sensor limitation.
Air time tracking
^^^^^^^^^^^^^^^^^
Locomotion tasks often need to know when feet land and take off for
gait rewards. Setting ``track_air_time=True`` enables per-primary
timing. The sensor maintains four additional tensors on
``ContactData``, each shaped ``[B, P]``: ``current_air_time``,
``last_air_time``, ``current_contact_time``, and
``last_contact_time``. Two helper methods provide edge detection for
transition events:
.. code-block:: python
sensor = env.scene["feet_air"]
first_contact = sensor.compute_first_contact(dt) # [B, P], True for primaries that just landed
first_air = sensor.compute_first_air(dt) # [B, P], True for primaries that just took off
Air time is per primary even when ``num_slots > 1``: the sensor reduces
``found`` across slots so that any slot in contact counts as the
primary being in contact.
.. _contact-sensor-history:
History (decimation safe contacts)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
When using decimation (multiple physics substeps per policy step), a
brief collision can occur and resolve entirely within the substep loop.
By the time the policy reads the sensor, the contact is gone and
``found`` reports zero. Setting ``history_length`` on the sensor config
tells the sensor to keep a rolling buffer of the last *N* substeps for
force, torque, and distance fields. The policy can then inspect the
full history and decide whether a real contact occurred.
Set ``history_length`` equal to your decimation value so the buffer
covers exactly one policy step:
.. code-block:: python
ContactSensorCfg(
name="self_collision",
primary=ContactMatch(mode="subtree", pattern="pelvis", entity="robot"),
secondary=ContactMatch(mode="subtree", pattern="pelvis", entity="robot"),
fields=("found", "force"),
history_length=4, # matches decimation=4
)
The history tensors live on ``ContactData`` alongside the regular
fields:
.. code-block:: python
data = sensor.data
data.force_history # [B, N, H, 3] (H = history_length)
data.torque_history # [B, N, H, 3]
data.dist_history # [B, N, H]
Index 0 is the most recent substep. To check whether any substep had
a contact force above a threshold:
.. code-block:: python
force_mag = torch.norm(data.force_history, dim=-1) # [B, N, H]
had_contact = (force_mag > 10.0).any(dim=1).any(dim=-1) # [B]
.. note::
``track_air_time=True`` already accumulates contact state across
substeps for gait rewards, so feet ground sensors typically do not
need ``history_length``. Use history for sensors where you need to
detect brief collisions that would otherwise be missed (self
collisions, illegal contact terminations).
Output
^^^^^^
``ContactData`` is a dataclass whose fields correspond to the
``fields`` tuple on the config. Unrequested fields are ``None``.
.. code-block:: python
@dataclass
class ContactData:
found: Tensor | None # [B, N] contact count
force: Tensor | None # [B, N, 3]
torque: Tensor | None # [B, N, 3]
dist: Tensor | None # [B, N] penetration depth
pos: Tensor | None # [B, N, 3] contact position
normal: Tensor | None # [B, N, 3] surface normal
tangent: Tensor | None # [B, N, 3]
# With track_air_time=True.
current_air_time: Tensor | None
last_air_time: Tensor | None
current_contact_time: Tensor | None
last_contact_time: Tensor | None
RayCastSensor
-------------
``RayCastSensor`` provides GPU-accelerated raycasting for terrain
scanning and depth sensing. It supports grid and pinhole camera ray
patterns with configurable alignment modes. See :ref:`raycast_sensor`
for full documentation.
RGB-D Camera
------------
``CameraSensor`` renders RGB and depth images from MuJoCo cameras. See
:ref:`rgbd_camera` for full documentation.
Extending: custom sensors
-------------------------
All sensors inherit from ``Sensor[T]``, a generic base class where
``T`` is the data type returned by the ``data`` property (e.g.,
``torch.Tensor`` for ``BuiltinSensor``, ``ContactData`` for
``ContactSensor``).
The base class provides automatic per-step caching. The ``data``
property calls ``_compute_data()`` on first access each step and
caches the result. The cache is invalidated automatically when
``update()`` or ``reset()`` is called, so multiple reads within the
same step (from different observation or reward terms) pay the
computation cost only once.
**Lifecycle methods:**
- ``edit_spec``: Add sensor elements to the MjSpec during scene
construction.
- ``initialize``: Post-compilation setup. Cache sensor indices,
allocate buffers, resolve references.
- ``update``: Called each physics step. Invalidates the data cache.
Override to maintain per-step state (e.g., air time counters).
- ``reset``: Called on environment reset. Invalidates the data cache.
Override to clear per-environment state.
- ``_compute_data``: Compute and return the sensor output. Called
lazily by the ``data`` property when the cache is stale.
``ContactSensor`` and ``RayCastSensor`` are the most complete
reference implementations for custom sensor development.
.. toctree::
:maxdepth: 1
:hidden:
raycast_sensor
rgbd_camera
docs/source/sensors/raycast_sensor.rst
+299
View File
@@ -0,0 +1,299 @@
.. _raycast_sensor:
RayCast Sensor
==============
``RayCastSensor`` provides GPU-accelerated raycasting for terrain
scanning, obstacle detection, and depth sensing. Rays are emitted from
a frame attached to a body, site, or geom in the scene, and the sensor
reports hit distances, world-space hit positions, and surface normals.
.. raw:: html
<video controls style="display: block; margin: 0 auto; max-width: 100%; height: auto;">
<source src="../../_static/raycast_demo.mp4" type="video/mp4">
</video>
Quick start
-----------
.. code-block:: python
from mjlab.sensor import RayCastSensorCfg, GridPatternCfg, ObjRef
# Downward-facing grid for terrain height scanning.
raycast_cfg = RayCastSensorCfg(
name="terrain_scan",
frame=ObjRef(type="body", name="base", entity="robot"),
pattern=GridPatternCfg(size=(1.0, 1.0), resolution=0.1),
max_distance=5.0,
)
scene_cfg = SceneCfg(
entities={"robot": robot_cfg},
sensors=(raycast_cfg,),
)
# Access at runtime.
data = env.scene["terrain_scan"].data
data.distances # [B, N] distance to hit, -1 if miss
data.hit_pos_w # [B, N, 3] world-space hit positions
data.normals_w # [B, N, 3] surface normals
Ray patterns
------------
Ray patterns define the spatial distribution and direction of rays
emitted from the sensor frame.
.. grid:: 2
.. grid-item-card:: Grid pattern
Parallel rays in a 2D grid with fixed spatial resolution. The
ground footprint does not change with sensor height because ray
spacing is defined in world units (meters). The natural choice for
height maps and terrain scanning.
.. raw:: html
<video autoplay loop muted playsinline style="width: 100%; height: auto;">
<source src="../../_static/pattern_grid.mp4" type="video/mp4">
</video>
.. grid-item-card:: Pinhole camera pattern
Diverging rays emitted from a single origin, analogous to a depth
camera. The ground coverage increases with sensor
height because the field of view is fixed in angular units.
.. raw:: html
<video autoplay loop muted playsinline style="width: 100%; height: auto;">
<source src="../../_static/pattern_pinhole.mp4" type="video/mp4">
</video>
.. code-block:: python
from mjlab.sensor import GridPatternCfg, PinholeCameraPatternCfg
# Parallel grid: fixed footprint, height-invariant.
grid = GridPatternCfg(
size=(1.0, 1.0), # Grid dimensions in meters
resolution=0.1, # Spacing between rays
direction=(0.0, 0.0, -1.0), # Ray direction (down)
)
# Pinhole: perspective projection, diverging rays.
pinhole = PinholeCameraPatternCfg(
width=16,
height=12,
fovy=45.0, # Vertical FOV in degrees
)
# Pinhole from a MuJoCo camera definition.
pinhole = PinholeCameraPatternCfg.from_mujoco_camera("robot/depth_cam")
# Pinhole from an intrinsic matrix.
pinhole = PinholeCameraPatternCfg.from_intrinsic_matrix(
intrinsic_matrix=[500, 0, 320, 0, 500, 240, 0, 0, 1],
width=640,
height=480,
)
Pattern comparison
^^^^^^^^^^^^^^^^^^
.. list-table::
:header-rows: 1
:widths: 20 40 40
* - Aspect
- Grid
- Pinhole
* - Ray direction
- Parallel
- Diverging
* - Spacing unit
- Meters
- Degrees (FOV)
* - Height affects coverage
- No
- Yes
* - Projection model
- Orthographic
- Perspective
Frame attachment
----------------
Rays are emitted from a frame in the scene specified via ``ObjRef``.
The frame can be a body, site, or geom on any entity.
.. code-block:: python
frame = ObjRef(type="body", name="base", entity="robot")
frame = ObjRef(type="site", name="scan_site", entity="robot")
frame = ObjRef(type="geom", name="sensor_mount", entity="robot")
``exclude_parent_body`` (default ``True``) prevents rays from hitting
the body to which the sensor is attached.
Ray alignment
-------------
The ``ray_alignment`` setting controls how rays orient relative to the
attached frame when the body rotates.
.. raw:: html
<video autoplay loop muted playsinline
style="display: block; margin: 0 auto; max-width: 100%; height: auto;">
<source src="../../_static/ray_alignment_comparison.mp4" type="video/mp4">
</video>
.. list-table::
:header-rows: 1
:widths: 15 45 40
* - Mode
- Description
- Use case
* - ``"base"``
- Full position and rotation tracking
- Body-mounted sensors
* - ``"yaw"``
- Follows yaw, ignores pitch and roll
- Terrain height maps
* - ``"world"``
- Fixed world-frame direction
- Gravity-aligned sensing
.. code-block:: python
RayCastSensorCfg(
name="height_scan",
frame=ObjRef(type="body", name="base", entity="robot"),
pattern=GridPatternCfg(size=(1.0, 1.0), resolution=0.1),
ray_alignment="yaw",
)
Geom group filtering
--------------------
MuJoCo assigns geoms to groups 0 through 5. Use
``include_geom_groups`` to restrict which geoms rays can hit. This is
useful for ignoring visual-only geoms or isolating terrain geometry.
.. code-block:: python
RayCastSensorCfg(
name="terrain_only",
frame=ObjRef(type="body", name="base", entity="robot"),
pattern=GridPatternCfg(),
include_geom_groups=(0, 1),
)
Output
------
``RayCastData`` is a dataclass with shape annotations relative to
``B`` (number of environments) and ``N`` (number of rays).
.. code-block:: python
@dataclass
class RayCastData:
distances: Tensor # [B, N] distance to hit, -1 if miss
hit_pos_w: Tensor # [B, N, 3] world-space hit positions
normals_w: Tensor # [B, N, 3] surface normals
pos_w: Tensor # [B, 3] sensor frame position
quat_w: Tensor # [B, 4] sensor frame orientation (w, x, y, z)
.. note::
Set ``debug_vis=True`` on the config to visualize ray hits at
runtime.
Examples
--------
.. code-block:: python
from mjlab.sensor import (
RayCastSensorCfg, GridPatternCfg, PinholeCameraPatternCfg, ObjRef,
)
# Dense height map for terrain-aware locomotion.
height_scan = RayCastSensorCfg(
name="height_scan",
frame=ObjRef(type="body", name="base", entity="robot"),
pattern=GridPatternCfg(
size=(1.6, 1.0),
resolution=0.1,
direction=(0.0, 0.0, -1.0),
),
ray_alignment="yaw",
max_distance=2.0,
)
# Simulated depth camera using pinhole projection.
depth_cam = RayCastSensorCfg(
name="depth",
frame=ObjRef(type="site", name="camera_site", entity="robot"),
pattern=PinholeCameraPatternCfg.from_mujoco_camera("robot/depth_cam"),
max_distance=10.0,
)
# Forward-facing obstacle scan.
obstacle_scan = RayCastSensorCfg(
name="obstacle",
frame=ObjRef(type="body", name="head", entity="robot"),
pattern=GridPatternCfg(
size=(0.5, 0.3),
resolution=0.1,
direction=(-1.0, 0.0, 0.0),
),
max_distance=3.0,
include_geom_groups=(0,),
)
TerrainHeightSensor
-------------------
``TerrainHeightSensor`` is a thin ``RayCastSensor`` subclass that adds
per-frame vertical clearance to the sensor data. It computes
``frame_z - hit_z`` for each ray, replaces misses with ``max_distance``,
and reduces across rays per frame.
.. code-block:: python
from mjlab.sensor import TerrainHeightSensorCfg, RingPatternCfg, ObjRef
cfg = TerrainHeightSensorCfg(
name="foot_height",
frame=(
ObjRef(type="site", name="left_foot", entity="robot"),
ObjRef(type="site", name="right_foot", entity="robot"),
),
pattern=RingPatternCfg.single_ring(radius=0.04, num_samples=4),
max_distance=1.0,
include_geom_groups=(0,),
)
# At runtime:
sensor = env.scene["foot_height"]
sensor.data.heights # [B, F] vertical clearance per foot
sensor.data.distances # [B, N] raw ray distances (inherited)
The ``reduction`` config field controls how rays are aggregated within
each frame: ``"min"`` (default), ``"max"``, or ``"mean"``.
docs/source/sensors/rgbd_camera.rst
+219
View File
@@ -0,0 +1,219 @@
.. _rgbd_camera:
RGB-D Camera
============
``CameraSensor`` renders RGB and depth images on the GPU using MuJoCo
Warp's ray-traced rendering pipeline. It can either wrap an existing
MuJoCo camera defined in your XML or create a new one programmatically.
Quick start
-----------
.. code-block:: python
from mjlab.sensor import CameraSensorCfg
# Wrap an existing MuJoCo camera from the robot's XML.
cam = CameraSensorCfg(
name="wrist_cam",
camera_name="robot/wrist_camera",
data_types=("rgb", "depth"),
width=160,
height=120,
)
scene_cfg = SceneCfg(
entities={"robot": robot_cfg},
sensors=(cam,),
)
# Access at runtime.
data = env.scene["wrist_cam"].data
data.rgb # [B, 120, 160, 3] uint8
data.depth # [B, 120, 160, 1] float32
Creating vs wrapping cameras
-----------------------------
There are two ways to set up a camera sensor.
**Wrap an existing camera.** If your MJCF model already defines a
camera, pass its name via ``camera_name``. The sensor uses the
camera's position, orientation, and field of view from the model.
You can optionally override ``fovy`` or switch to orthographic
projection.
.. code-block:: python
# Wrap the camera named "front_cam" in the robot's XML.
CameraSensorCfg(
name="front",
camera_name="robot/front_cam",
data_types=("rgb",),
)
**Create a new camera.** When ``camera_name`` is ``None`` (the
default), the sensor adds a new camera to the MjSpec during scene
construction. Specify ``pos``, ``quat``, and optionally ``fovy`` to
place it.
.. code-block:: python
# Fixed overhead camera on the worldbody.
CameraSensorCfg(
name="overhead",
pos=(0.0, 0.0, 2.0),
quat=(0.0, 0.707, 0.707, 0.0),
fovy=60.0,
width=320,
height=240,
data_types=("rgb", "depth"),
)
Camera parameterization
-----------------------
MuJoCo supports two ways to define a camera's projection, and both
work with ``CameraSensor``. See the `MuJoCo camera documentation
<https://mujoco.readthedocs.io/en/stable/XMLreference.html#body-camera>`_
for full details.
**FOV-based.** The simpler approach. A single ``fovy`` (vertical field
of view in degrees) combined with the image resolution defines the
projection. This is the default when creating cameras
programmatically via ``CameraSensorCfg``.
**Intrinsic-based.** For matching real camera hardware, MuJoCo cameras
can be parameterized with ``sensorsize``, ``focal`` (or
``focalpixel``), and ``principal`` (or ``principalpixel``). These
fields are set in the MJCF XML and provide direct control over the
intrinsic matrix. When intrinsic parameters are present, ``fovy`` is
ignored by MuJoCo.
When wrapping an existing camera, the sensor inherits whichever
parameterization the XML defines. When creating a new camera, the
sensor uses ``fovy``. To use intrinsic parameters for a new camera,
define it in your XML and wrap it with ``camera_name``.
.. note::
If you plan to randomize the field of view with domain
randomization, use ``dr.cam_fovy`` for FOV-based cameras or
``dr.cam_intrinsic`` for intrinsic-based cameras. Randomizing
``cam_fovy`` has no effect on cameras that use intrinsic parameters.
Body-mounted cameras
--------------------
Set ``parent_body`` to attach a new camera to a specific body rather
than the worldbody. The ``pos`` and ``quat`` are then relative to the
parent body frame, so the camera moves with the body.
.. code-block:: python
# Camera mounted on the robot's end-effector.
CameraSensorCfg(
name="ee_cam",
parent_body="robot/link_6",
pos=(0.0, 0.0, 0.05),
quat=(1.0, 0.0, 0.0, 0.0),
fovy=45.0,
width=160,
height=120,
data_types=("rgb", "depth"),
)
Data types
----------
The ``data_types`` tuple selects which image modalities to render.
Only requested types are allocated; the other field on
``CameraSensorData`` is ``None``.
.. list-table::
:header-rows: 1
:widths: 15 20 65
* - Type
- Shape
- Description
* - ``"rgb"``
- ``[B, H, W, 3]`` uint8
- Color image. Rendered as packed ABGR uint32 by MuJoCo Warp,
then unpacked to RGB channels on the GPU.
* - ``"depth"``
- ``[B, H, W, 1]`` float32
- Depth image. Values are distances from the camera plane.
* - ``"segmentation"``
- ``[B, H, W, 2]`` int32
- Typed segmentation. Channel 0 stores object IDs and channel 1 stores
MuJoCo object types. Background pixels are ``(-1, -1)``.
Render settings
---------------
All camera sensors in a scene must share identical values for
``use_textures``, ``use_shadows``, and ``enabled_geom_groups``. This
is a constraint of the underlying MuJoCo Warp rendering system, which
uses a single ``RenderContext`` for all cameras. Mismatched settings
raise a ``ValueError`` at scene construction.
.. code-block:: python
# These two cameras must agree on render settings.
cam_a = CameraSensorCfg(
name="cam_a",
camera_name="robot/front_cam",
use_textures=True,
use_shadows=False,
enabled_geom_groups=(0, 1, 2),
data_types=("rgb",),
)
cam_b = CameraSensorCfg(
name="cam_b",
camera_name="robot/wrist_cam",
use_textures=True, # Must match cam_a
use_shadows=False, # Must match cam_a
enabled_geom_groups=(0, 1, 2), # Must match cam_a
data_types=("depth",),
)
Output
------
``CameraSensorData`` is a dataclass with one field per data type.
.. code-block:: python
@dataclass
class CameraSensorData:
rgb: Tensor | None # [B, H, W, 3] uint8
depth: Tensor | None # [B, H, W, 1] float32
segmentation: Tensor | None # [B, H, W, 2] int32
By default, the returned tensors are zero-copy views into the render
buffer. Set ``clone_data=True`` on the config if you modify them in
place, to avoid corrupting the shared buffer.
Visualization in Viser
----------------------
The Viser viewer automatically discovers all ``CameraSensor`` instances
in the scene and displays their RGB and depth outputs as live image
panels in the GUI sidebar. A camera frustum is rendered in the 3D
viewport showing the camera's position, orientation, and field of view.
Depth images include an interactive scale slider for adjusting the
visualization range.
.. image:: ../_static/viser_camera_pane.png
:align: center
:alt: Viser viewer showing camera image panels and frustum visualization