Import upstream snapshot c19f713c415a699a79d71cd96aa13c3104a05047
nightly / Test against latest dependencies (py3.10) (push) Has been cancelled
nightly / Test against latest dependencies (py3.13) (push) Has been cancelled
tests / tests (3.13, locked) (push) Has been cancelled
tests / tests (3.13, unlocked) (push) Has been cancelled
tests / pyright (3.10) (push) Has been cancelled
tests / lint-format (push) Has been cancelled
tests / tests (3.10, locked) (push) Has been cancelled
tests / tests (3.11, locked) (push) Has been cancelled
tests / tests (3.12, locked) (push) Has been cancelled
tests / pyright (3.11) (push) Has been cancelled
tests / pyright (3.12) (push) Has been cancelled
tests / pyright (3.13) (push) Has been cancelled
tests / ty-check (3.10) (push) Has been cancelled
tests / ty-check (3.11) (push) Has been cancelled
tests / ty-check (3.12) (push) Has been cancelled
tests / ty-check (3.13) (push) Has been cancelled
tests / stubs (push) Has been cancelled
tests / smoke-test (push) Has been cancelled
Docker / check_paths (push) Has been cancelled
Docker / build (push) Has been cancelled
docs / build (push) Has been cancelled

Upstream: https://github.com/michaelgillett/mjlab
Upstream-Commit: c19f713c415a699a79d71cd96aa13c3104a05047
Upstream-Branch: main
This commit is contained in:
Upstream Snapshot
2026-08-28 15:42:17 +08:00
commit 32a241c28f
513 changed files with 120771 additions and 0 deletions
+507
View File
@@ -0,0 +1,507 @@
.. _entity_data:
Entity Data
===========
This page is the property reference for ``EntityData``. For an overview
of how ``entity.data`` fits into the broader data access story, see
:ref:`entity`.
All properties are PyTorch tensors backed by MuJoCo Warp's GPU buffers
with no copy overhead. The first dimension is always ``num_envs``, the
number of parallel simulation worlds.
.. warning::
Read properties reflect the state after ``sim.forward()`` is called.
If you write simulation state and then read a derived property in the
same event term, call ``sim.forward()`` between the write and the
read. The environment step sequence already does this; the warning
applies only when writing custom event terms that mix reads and
writes. See the :ref:`FAQ <faq-sim-forward>` for a detailed
explanation.
Reference: root state
---------------------
Root properties describe the position, orientation, and velocity of the
entity's root body. Properties ending in ``_w`` are expressed in the world
frame. Properties ending in ``_b`` are expressed in the entity's base frame.
See :ref:`frame-conventions` for details.
Each entity has two root reference points: the **link origin** (the body
frame origin defined in the MJCF) and the **center of mass (COM)**.
Which one is relevant depends on the task.
.. admonition:: MuJoCo's mixed-frame ``qvel``
For floating-base entities, the free joint stores 6 DOFs in
``qvel``. MuJoCo expresses the **linear** components in the
**world frame** but the **angular** components in the **local body
frame**. EntityData avoids this pitfall: all ``_w`` velocity
properties are computed from ``cvel`` (see
:ref:`cvel-section` below) and are fully world-frame. If you
read ``env.sim.data.qvel`` directly, be aware of the mixed
convention.
.. rubric:: Root link properties
.. list-table::
:header-rows: 1
:widths: 35 20 15 30
* - Property
- Shape
- Frame
- Description
* - ``root_link_pose_w``
- ``[num_envs, 7]``
- world
- Root link position (3) and quaternion (4) concatenated
* - ``root_link_pos_w``
- ``[num_envs, 3]``
- world
- Root link position
* - ``root_link_quat_w``
- ``[num_envs, 4]``
- world
- Root link orientation as quaternion (w, x, y, z)
* - ``root_link_vel_w``
- ``[num_envs, 6]``
- world
- Root link linear (3) and angular (3) velocity concatenated
* - ``root_link_lin_vel_w``
- ``[num_envs, 3]``
- world
- Root link linear velocity
* - ``root_link_ang_vel_w``
- ``[num_envs, 3]``
- world
- Root link angular velocity
* - ``root_link_lin_vel_b``
- ``[num_envs, 3]``
- body
- Root link linear velocity in base frame
* - ``root_link_ang_vel_b``
- ``[num_envs, 3]``
- body
- Root link angular velocity in base frame
.. rubric:: Root COM properties
.. list-table::
:header-rows: 1
:widths: 35 20 15 30
* - Property
- Shape
- Frame
- Description
* - ``root_com_pose_w``
- ``[num_envs, 7]``
- world
- Root COM position (3) and quaternion (4) concatenated
* - ``root_com_pos_w``
- ``[num_envs, 3]``
- world
- Root COM position
* - ``root_com_quat_w``
- ``[num_envs, 4]``
- world
- Root COM orientation as quaternion (w, x, y, z)
* - ``root_com_vel_w``
- ``[num_envs, 6]``
- world
- Root COM linear (3) and angular (3) velocity concatenated
* - ``root_com_lin_vel_w``
- ``[num_envs, 3]``
- world
- Root COM linear velocity
* - ``root_com_ang_vel_w``
- ``[num_envs, 3]``
- world
- Root COM angular velocity
* - ``root_com_lin_vel_b``
- ``[num_envs, 3]``
- body
- Root COM linear velocity in base frame
* - ``root_com_ang_vel_b``
- ``[num_envs, 3]``
- body
- Root COM angular velocity in base frame
.. rubric:: Derived root properties
.. list-table::
:header-rows: 1
:widths: 35 20 15 30
* - Property
- Shape
- Frame
- Description
* - ``projected_gravity_b``
- ``[num_envs, 3]``
- body
- Gravity vector (0, 0, -1) rotated into the base frame. Used to measure
tilt: a perfectly upright robot reads ``[0, 0, -1]``.
* - ``heading_w``
- ``[num_envs]``
- world
- Heading angle (radians) of the root body's forward axis projected onto
the XY plane.
Reference: body state
---------------------
Body properties give per-body kinematic state for all bodies belonging to the
entity. The second dimension is ``num_bodies``, which counts all non-world
bodies in the entity's kinematic tree.
.. list-table::
:header-rows: 1
:widths: 35 25 15 25
* - Property
- Shape
- Frame
- Description
* - ``body_link_pose_w``
- ``[num_envs, num_bodies, 7]``
- world
- Per-body link position (3) and quaternion (4)
* - ``body_link_pos_w``
- ``[num_envs, num_bodies, 3]``
- world
- Per-body link positions
* - ``body_link_quat_w``
- ``[num_envs, num_bodies, 4]``
- world
- Per-body link orientations
* - ``body_link_vel_w``
- ``[num_envs, num_bodies, 6]``
- world
- Per-body link linear (3) and angular (3) velocity
* - ``body_link_lin_vel_w``
- ``[num_envs, num_bodies, 3]``
- world
- Per-body link linear velocities
* - ``body_link_ang_vel_w``
- ``[num_envs, num_bodies, 3]``
- world
- Per-body link angular velocities
* - ``body_com_pose_w``
- ``[num_envs, num_bodies, 7]``
- world
- Per-body COM position (3) and quaternion (4)
* - ``body_com_pos_w``
- ``[num_envs, num_bodies, 3]``
- world
- Per-body COM positions
* - ``body_com_quat_w``
- ``[num_envs, num_bodies, 4]``
- world
- Per-body COM orientations
* - ``body_com_vel_w``
- ``[num_envs, num_bodies, 6]``
- world
- Per-body COM linear (3) and angular (3) velocity
* - ``body_com_lin_vel_w``
- ``[num_envs, num_bodies, 3]``
- world
- Per-body COM linear velocities
* - ``body_com_ang_vel_w``
- ``[num_envs, num_bodies, 3]``
- world
- Per-body COM angular velocities
* - ``body_external_wrench``
- ``[num_envs, num_bodies, 6]``
- world
- External force (3) and torque (3) applied to each body
* - ``body_external_force``
- ``[num_envs, num_bodies, 3]``
- world
- External forces applied to each body
* - ``body_external_torque``
- ``[num_envs, num_bodies, 3]``
- world
- External torques applied to each body
Reference: joint state
----------------------
Joint properties cover 1-DOF revolute and prismatic joints. The free joint
(root floating-base DOF) is excluded; use root state properties for that.
.. list-table::
:header-rows: 1
:widths: 35 25 40
* - Property
- Shape
- Description
* - ``joint_pos``
- ``[num_envs, num_joints]``
- Joint positions in radians (revolute) or metres (prismatic)
* - ``joint_pos_biased``
- ``[num_envs, num_joints]``
- Joint positions with encoder bias added. Used when simulating
encoder calibration errors via domain randomization.
* - ``joint_vel``
- ``[num_envs, num_joints]``
- Joint velocities in rad/s or m/s
* - ``joint_acc``
- ``[num_envs, num_joints]``
- Joint accelerations in rad/s² or m/s²
* - ``actuator_force``
- ``[num_envs, num_actuators]``
- Scalar actuator output in actuation space (per actuator). This is
the force before projection through the transmission Jacobian. For
actuator forces in joint space, use ``qfrc_actuator`` instead.
.. _generalized-forces:
Reference: generalized forces
-----------------------------
These properties expose selected components of MuJoCo's generalized
force decomposition, sliced to this entity's articulated joint DOFs.
Free joint DOFs are excluded. All shapes are ``[num_envs, nv]`` where
``nv`` is the number of articulated DOFs belonging to this entity.
.. list-table::
:header-rows: 1
:widths: 30 70
* - Property
- Description
* - ``qfrc_actuator``
- Forces produced by all actuators, mapped into joint space. For
motors this is the commanded torque times the gear ratio. For
position and velocity actuators this is the force computed by
the internal PD law. When ``actuatorgravcomp`` is enabled on a
joint, the gravity compensation force is included here.
* - ``qfrc_external``
- Forces on joints due to Cartesian wrenches applied to bodies
via ``xfrc_applied``. This is the :math:`J^\top F` mapping.
MuJoCo does not store this term separately; the property
recovers it from other force components after ``forward()``.
Reference: geom and site state
-------------------------------
.. list-table::
:header-rows: 1
:widths: 35 25 40
* - Property
- Shape
- Description
* - ``geom_pose_w``
- ``[num_envs, num_geoms, 7]``
- Per-geom position (3) and quaternion (4) in world frame
* - ``geom_pos_w``
- ``[num_envs, num_geoms, 3]``
- Per-geom positions in world frame
* - ``geom_quat_w``
- ``[num_envs, num_geoms, 4]``
- Per-geom orientations in world frame
* - ``geom_vel_w``
- ``[num_envs, num_geoms, 6]``
- Per-geom linear (3) and angular (3) velocity in world frame
* - ``geom_lin_vel_w``
- ``[num_envs, num_geoms, 3]``
- Per-geom linear velocities in world frame
* - ``geom_ang_vel_w``
- ``[num_envs, num_geoms, 3]``
- Per-geom angular velocities in world frame
* - ``site_pose_w``
- ``[num_envs, num_sites, 7]``
- Per-site position (3) and quaternion (4) in world frame
* - ``site_pos_w``
- ``[num_envs, num_sites, 3]``
- Per-site positions in world frame
* - ``site_quat_w``
- ``[num_envs, num_sites, 4]``
- Per-site orientations in world frame
* - ``site_vel_w``
- ``[num_envs, num_sites, 6]``
- Per-site linear (3) and angular (3) velocity in world frame
* - ``site_lin_vel_w``
- ``[num_envs, num_sites, 3]``
- Per-site linear velocities in world frame
* - ``site_ang_vel_w``
- ``[num_envs, num_sites, 3]``
- Per-site angular velocities in world frame
Reference: tendon state
-----------------------
Tendon properties are only populated for entities that have tendon-driven
actuators.
.. list-table::
:header-rows: 1
:widths: 35 25 40
* - Property
- Shape
- Description
* - ``tendon_len``
- ``[num_envs, num_tendons]``
- Tendon lengths
* - ``tendon_vel``
- ``[num_envs, num_tendons]``
- Tendon velocities
.. _frame-conventions:
Frame conventions
-----------------
Property names encode their reference frame with a suffix.
``_w`` (world frame)
A fixed global frame. The origin is typically at the scene origin and
its axes are constant throughout the episode. World-frame quantities are
useful when you need absolute position, such as checking whether the
robot has fallen below a height threshold.
``_b`` (body frame / base frame)
The entity's root body frame. It translates and rotates with the robot.
Most observation terms use body-frame quantities because they are
invariant to the robot's heading direction. A velocity expressed in the
body frame reads the same whether the robot faces north or south, which
makes it easier for the policy to generalize.
``projected_gravity_b`` is a good example of why the frame suffix
matters. It takes the world-frame gravity vector ``[0, 0, -1]`` and
rotates it into the base frame. When the robot is upright the result is
``[0, 0, -1]``; as the robot tilts, the x and y components grow,
giving the policy a direct signal for orientation correction.
Quaternion convention
^^^^^^^^^^^^^^^^^^^^^
All quaternions use the ``(w, x, y, z)`` convention, matching MuJoCo.
Reduced state vs. derived quantities
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
EntityData properties fall into two categories that behave differently
with respect to ``sim.forward()``:
**Reduced state.** ``joint_pos`` and ``joint_vel`` read directly from
MuJoCo's ``qpos`` and ``qvel`` arrays. Write methods such as
``write_joint_state_to_sim()`` modify these arrays directly, so reads
are always current.
**Derived quantities.** All pose and velocity properties (``*_pose_w``,
``*_vel_w``, ``*_vel_b``) are computed from MuJoCo's internal arrays
(``xpos``, ``xquat``, ``cvel``, ``subtree_com``, etc.) which are only
updated when ``sim.forward()`` runs. If you write to ``qpos``/``qvel``
and then read a derived property without an intervening ``forward()``,
the read will return stale values.
The environment step sequence calls ``forward()`` at the right time, so
this only matters if you write custom event terms that both write and
read in the same function. See the :ref:`FAQ <faq-sim-forward>` for
details.
.. _cvel-section:
How velocity properties are computed from ``cvel``
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
MuJoCo does not store world-frame linear velocities directly. Instead,
it stores a 6D spatial velocity per body called ``cvel`` (com-based
velocity), laid out as ``(angular[3], linear[3])``. This vector is
expressed in the **c-frame**: a frame centered at ``subtree_com`` (the
center of mass of the body's kinematic subtree) and oriented like the
world frame. MuJoCo uses this representation to improve numerical
precision for mechanisms far from the world origin. See
`c-frame variables <https://mujoco.readthedocs.io/en/stable/APIreference/APItypes.html#c-frame-variables>`_
and Featherstone's
`Spatial Algebra <http://royfeatherstone.org/spatial/>`_ for background.
To recover the world-frame linear velocity at an arbitrary point
:math:`\mathbf{p}` on a rigid body, we apply the standard rigid-body
velocity transfer formula. Let :math:`\boldsymbol{\omega}` and
:math:`\mathbf{v}_c` denote the angular and linear components of
``cvel``, and let :math:`\mathbf{c}` denote ``subtree_com``. Because
the c-frame is world-aligned, :math:`\boldsymbol{\omega}` is already in
the world frame. The linear velocity at :math:`\mathbf{p}` is:
.. math::
\mathbf{v}_p
= \mathbf{v}_c
- \boldsymbol{\omega} \times (\mathbf{c} - \mathbf{p})
EntityData applies this formula in ``compute_velocity_from_cvel()``:
.. code-block:: python
def compute_velocity_from_cvel(pos, subtree_com, cvel):
lin_vel_c = cvel[..., 3:6]
ang_vel_c = cvel[..., 0:3]
offset = subtree_com - pos
lin_vel_w = lin_vel_c - torch.cross(ang_vel_c, offset, dim=-1)
ang_vel_w = ang_vel_c
return torch.cat([lin_vel_w, ang_vel_w], dim=-1)
Every velocity property in EntityData (``root_link_vel_w``,
``body_link_vel_w``, ``geom_vel_w``, ``site_vel_w``, and their COM
variants) uses this function, substituting the appropriate point:
- **Link velocities** use ``xpos`` (body frame origin).
- **COM velocities** use ``xipos`` (body center of mass).
- **Geom/site velocities** use ``geom_xpos``/``site_xpos``, with
``cvel`` looked up from the parent body.
Default pose and relative quantities
--------------------------------------
``entity.data.default_joint_pos`` holds the joint positions from the entity's
initial-state configuration (the ``init_state.joint_pos`` field of
``EntityCfg``). It has shape ``[num_envs, num_joints]`` and is replicated
across all environments at initialization time.
The relative joint position is the deviation of the current joint position
from this default:
.. code-block:: python
joint_pos_rel = joint_pos - default_joint_pos
This is what the ``joint_pos_rel`` observation function computes:
.. code-block:: python
def joint_pos_rel(env, asset_cfg):
asset = env.scene[asset_cfg.name]
jnt_ids = asset_cfg.joint_ids
return (
asset.data.joint_pos[:, jnt_ids]
- asset.data.default_joint_pos[:, jnt_ids]
)
Relative joint positions give the policy a compact representation of posture
deviation. When the robot is at its default pose, every element is zero.
Similarly, ``default_joint_vel`` is used by the ``joint_vel_rel`` observation
function. For most configurations the default velocity is zero, so
``joint_vel_rel`` is identical to ``joint_vel``. The indirection exists to
allow non-zero reference velocities in tasks such as motion imitation.
The ``use_default_offset=True`` option in joint position action configs uses
``default_joint_pos`` as the zero point for the action space, so a network
output of zero commands the robot to its default pose. This is the standard
configuration for locomotion tasks.
+355
View File
@@ -0,0 +1,355 @@
.. _entity:
Entity
======
An ``Entity`` represents a physical object in the simulation: a robot, a
manipulated object, or a fixed fixture like a table. It is the central
abstraction in mjlab's physics layer.
A single ``Entity`` class covers all variants (contrast Isaac Lab, which
splits this across ``Articulation``, ``RigidObject``, and several other
subclasses of ``AssetBase``). Two orthogonal boolean properties classify
each instance:
**Base type.**
A *fixed-base* entity is welded to the world and has no free joint. A
*floating-base* entity has a free joint giving it 6-DOF movement.
**Articulation.**
An *articulated* entity has internal joints (revolute, prismatic, etc.).
A *non-articulated* entity has none beyond a possible free joint.
.. list-table::
:header-rows: 1
:widths: 30 25 15 15 15
* - Type
- Example
- ``is_fixed_base``
- ``is_articulated``
- ``is_actuated``
* - Fixed non-articulated
- Table, wall
- True
- False
- False
* - Fixed articulated
- Robot arm, door
- True
- True
- True/False
* - Floating non-articulated
- Box, ball, mug
- False
- False
- False
* - Floating articulated
- Humanoid, quadruped
- False
- True
- True/False
.. note::
mjlab automatically wraps every fixed-base entity in a
`mocap body <https://mujoco.readthedocs.io/en/stable/modeling.html#mocap-bodies>`_
so that each parallel environment can place the entity at a different
position. Without this wrapping, all fixed-base entities would be
welded to the world origin. The wrapping is transparent, but
**positioning only happens when a reset event runs**. You must
include a reset event such as ``reset_root_state_uniform`` in your
event config; without one, every fixed-base entity will remain at
the origin. See the :ref:`FAQ <faq>` for a full example. Mocap
entities can also be repositioned at runtime via
``entity.write_mocap_pose_to_sim()``.
Configuring an entity
---------------------
Every entity is described by an ``EntityCfg``. Only ``spec_fn`` is
required in practice; all other fields have sensible defaults. A passive
floating object needs nothing more than:
.. code-block:: python
from mjlab.entity import EntityCfg
cube_cfg = EntityCfg(spec_fn=get_cube_spec)
An actuated robot uses more of the interface:
.. code-block:: python
from mjlab.entity import EntityCfg, EntityArticulationInfoCfg
from mjlab.actuator import IdealPDActuatorCfg
robot_cfg = EntityCfg(
spec_fn=get_spec,
init_state=EntityCfg.InitialStateCfg(
pos=(0.0, 0.0, 0.8),
joint_pos={".*_hip_.*": 0.5, ".*": 0.0},
),
articulation=EntityArticulationInfoCfg(
actuators=(
IdealPDActuatorCfg(
target_names_expr=(".*",),
stiffness={".*": 50.0},
damping={".*": 5.0},
),
),
),
collisions=(my_collision_cfg,),
)
The following sections describe each field.
``spec_fn``
^^^^^^^^^^^
A callable that returns an ``mujoco.MjSpec``. The scene calls it during
composition, attaches the returned spec with a name prefix, and compiles
everything into a shared ``MjModel``.
For simple cases a lambda suffices:
.. code-block:: python
spec_fn = lambda: mujoco.MjSpec.from_file("robot.xml")
For anything more involved, use a regular function. MuJoCo resolves mesh
assets from disk automatically, so ``get_spec`` only needs to load the
XML:
.. code-block:: python
def get_spec() -> mujoco.MjSpec:
return mujoco.MjSpec.from_file(str(ROBOT_XML))
Because ``spec_fn`` is an arbitrary callable, you can perform any
`MjSpec edits <https://mujoco.readthedocs.io/en/stable/python.html#spec>`_
before returning: add bodies, change joint limits, swap materials,
or build the entire model programmatically without an XML file at all.
``init_state``
^^^^^^^^^^^^^^
Default root pose, root velocity, and joint positions/velocities. These
values are stored as a MuJoCo keyframe and used by reset events to
return the entity to its initial configuration.
``joint_pos`` and ``joint_vel`` are dicts mapping regex patterns to
values. Patterns are matched against joint names in order, so later
entries override earlier ones for any joint that matches both:
.. code-block:: python
init_state = EntityCfg.InitialStateCfg(
pos=(0.0, 0.0, 0.8), # root position
rot=(1.0, 0.0, 0.0, 0.0), # root quaternion (w, x, y, z)
joint_pos={
".*": 0.0, # all joints to zero
".*_hip_.*": 0.5, # then override hips to 0.5
},
)
Set ``joint_pos=None`` to use an existing keyframe from the MJCF model
instead of defining values here.
``articulation``
^^^^^^^^^^^^^^^^
Actuator configuration. Only needed for entities that have actuated
joints. Passive objects (boxes, tables, walls) can omit this field
entirely. See :ref:`actuators` for details on actuator types.
``soft_joint_pos_limit_factor`` (default 1.0) shrinks the joint range
used by soft-limit penalty rewards, so the policy is penalized before
reaching the physical hard stop. This does not modify the actual joint
limits in the MuJoCo model.
Spec editors
^^^^^^^^^^^^
The remaining fields are optional tuples of spec editor configs that
modify the ``MjSpec`` before compilation:
.. list-table::
:header-rows: 1
:widths: 20 80
* - Field
- Purpose
* - ``collisions``
- Replace the entity's collision structure: which geoms collide, and
with what contact parameters.
* - ``lights``
- Add lights to specific bodies.
* - ``cameras``
- Add cameras to specific bodies.
* - ``textures``
- Add procedural textures (checker, gradient, etc.).
* - ``materials``
- Add materials and optionally assign them to geoms by regex.
* - ``geoms``
- Patch attributes of existing geoms (visualization group, collision
attributes). Unset attributes are left untouched.
Each editor accepts regex patterns to target specific elements. For
example, a ``CollisionCfg`` with ``geom_names_expr=(".*_foot.*",)``
sets contact parameters only on foot geoms. See the asset zoo
(``mjlab.asset_zoo.robots``) for complete examples.
``geoms`` and ``collisions`` both write geom attributes but with
different semantics. A ``GeomCfg`` is a sparse *patch*: every attribute
defaults to ``None``, and only attributes you set are written. A
``CollisionCfg`` is a *policy*: ``contype``, ``conaffinity``,
``condim``, and ``priority`` are required and always written to every
matched geom, and non-matched geoms have collision disabled by default,
so the entity's contact behavior is fully determined by the config
regardless of the source XML. Collision configs are applied after geom
configs; mjlab warns if a ``GeomCfg`` sets a collision attribute that a
``CollisionCfg`` then overwrites.
Heterogeneous worlds
^^^^^^^^^^^^^^^^^^^^
For scenes that need different mesh assets in different parallel worlds
(for example, training a manipulation policy that generalizes across
object shapes), use ``VariantEntityCfg`` instead of ``EntityCfg``. Each
world is assigned a variant proportional to a configurable weight, and
mesh-dependent compiled constants (collision bounds, body inertials,
subtree mass) are stored as per-world arrays so domain randomization and
viewers stay consistent. See :ref:`heterogeneous_worlds`.
Subclassing Entity
^^^^^^^^^^^^^^^^^^
``Entity`` and ``EntityCfg`` can be subclassed for specialized behavior.
mjlab itself does this for terrain: ``TerrainEntity`` extends ``Entity``
with procedural terrain generation and per-environment origin
computation, and ``TerrainEntityCfg`` adds fields like
``terrain_type``, ``env_spacing``, and ``terrain_generator``. The same
pattern works for any domain-specific entity that needs logic beyond
what ``EntityCfg`` and spec editors provide.
Finding elements
^^^^^^^^^^^^^^^^
Entity provides ``find_*`` methods that accept regex patterns and return
matched element indices and names:
.. code-block:: python
ids, names = entity.find_joints((".*_hip_.*", ".*_knee_.*"))
ids, names = entity.find_geoms((".*foot.*",))
ids, names = entity.find_bodies((".*",))
Available methods: ``find_bodies()``, ``find_joints()``,
``find_geoms()``, ``find_sites()``, ``find_tendons()``.
These are used internally during scene construction and manager
initialization. In reward and observation terms, prefer
``SceneEntityCfg`` with name patterns as described below.
Reading runtime state
---------------------
Once entities are added to a ``SceneCfg`` and the environment is
constructed, their state is accessible through three interfaces at
decreasing levels of abstraction.
EntityData
^^^^^^^^^^
``entity.data`` is the primary interface for reward, observation, and
termination functions. It exposes kinematic state (poses, velocities, accelerations), actuator forces,
generalized forces, and derived body-frame quantities such as projected
gravity, all as PyTorch tensors with
shape ``(num_envs, ...)``. See :ref:`entity_data` for the full property
reference.
``SceneEntityCfg`` selects which entity and which elements within it a
term operates on. Regex patterns in ``joint_names``, ``body_names``,
``site_names``, etc. are resolved to integer indices once at manager
initialization, so there is no regex overhead at runtime:
.. code-block:: python
from mjlab.managers.scene_entity_config import SceneEntityCfg
def flat_orientation_l2(
env,
asset_cfg: SceneEntityCfg = SceneEntityCfg("robot"),
) -> torch.Tensor:
"""Penalize non-flat base orientation using projected gravity."""
asset = env.scene[asset_cfg.name]
return torch.sum(
torch.square(asset.data.projected_gravity_b[:, :2]), dim=1
)
``SceneEntityCfg`` also supports regex element selection through
``joint_names``, ``body_names``, ``site_names``, etc. The resolved
integer indices (e.g., ``asset_cfg.joint_ids``) make the runtime read a
single tensor slice with no regex overhead.
Sensors
^^^^^^^
Sensors are configured on the **scene**, not on individual entities.
A sensor can reference an entity element (e.g., a contact sensor on the
robot's feet, an accelerometer attached to a body site), but it can also
be independent of any entity. This is why sensors live in ``SceneCfg``
rather than ``EntityCfg``.
At runtime, sensors are accessed by name through ``env.scene``, the same
way entities are:
.. code-block:: python
def angular_momentum_penalty(env, sensor_name: str) -> torch.Tensor:
sensor = env.scene[sensor_name]
return torch.sum(torch.square(sensor.data), dim=-1)
Builtin sensors wrap MuJoCo sensor types (accelerometer, gyro, framepos,
subtreeangmom, etc.). ``ContactSensor``, ``RayCastSensor``, and
``CameraSensor`` provide higher-level abstractions for contact detection,
terrain scanning, and RGB-D rendering. See :ref:`sensors` for details.
Raw simulation data
^^^^^^^^^^^^^^^^^^^
For anything not covered by ``EntityData`` or sensors, the underlying
MuJoCo Warp arrays are accessible through ``env.sim.data`` and
``env.sim.model``. These expose the full ``mjData`` and ``mjModel``
fields as PyTorch tensors (zero-copy), indexed by global MuJoCo IDs
rather than per-entity IDs:
.. code-block:: python
# Global joint positions across all entities.
qpos = env.sim.data.qpos # (num_envs, nq)
# All body positions.
xpos = env.sim.data.xpos # (num_envs, nbody, 3)
# Model-level constants.
body_mass = env.sim.model.body_mass # (nbody,)
This is useful for low-level operations or when you need quantities
that span multiple entities.
.. note::
The main limitation of raw sim data is that you must manage global
MuJoCo indices yourself. In the future, we plan to support MuJoCo's
`bind <https://mujoco.readthedocs.io/en/latest/python.html#relationship-to-pymjcf-and-bind>`_
functionality, which will allow binding spec elements directly to
their corresponding data views without manual index bookkeeping.
.. toctree::
:maxdepth: 1
entity_data
per_world_mesh
+490
View File
@@ -0,0 +1,490 @@
.. _heterogeneous_worlds:
Heterogeneous Worlds
====================
mjlab can run a single batched simulation in which different parallel
worlds use different mesh assets for the same logical entity. World 0
may simulate a cube, world 1 a sphere, world 2 a bowl. All worlds
share the same compiled scene and the same body and joint structure;
only the meshes and the per-geom attributes that travel with them
(friction, contact bits, mass, density, and a few more) differ across
worlds. Articulated props work too (you can have a hinge or slide
below the variant's root), as long as the joint topology matches
across variants. The feature is exposed through ``VariantEntityCfg``.
The full breakdown of what can and cannot vary across variants is in
the next section.
Quickstart
----------
Say you want some parallel worlds to hold a sphere and others to hold
a cone, with a single shared scene running both at once. Define each
variant as a function that returns an ``MjSpec``, then group them
under one ``VariantEntityCfg``:
.. code-block:: python
import mujoco
from mjlab.entity import EntityCfg, VariantEntityCfg
def make_sphere_spec() -> mujoco.MjSpec:
spec = mujoco.MjSpec()
mesh = spec.add_mesh(name="visual")
mesh.make_sphere(subdivision=3)
mesh.scale[:] = (0.05,) * 3
body = spec.worldbody.add_body(name="prop")
body.add_freejoint()
body.add_geom(type=mujoco.mjtGeom.mjGEOM_MESH, meshname="visual")
return spec
def make_cone_spec() -> mujoco.MjSpec:
spec = mujoco.MjSpec()
mesh = spec.add_mesh(name="visual")
mesh.make_cone(nedge=16, radius=0.04)
body = spec.worldbody.add_body(name="prop")
body.add_freejoint()
body.add_geom(type=mujoco.mjtGeom.mjGEOM_MESH, meshname="visual")
return spec
object_cfg = VariantEntityCfg(
variants={
"sphere": make_sphere_spec,
"cone": make_cone_spec,
},
assignment={"cone": 2.0}, # twice as many cones as spheres
init_state=EntityCfg.InitialStateCfg(pos=(0.0, 0.0, 0.2)),
)
Plug the variant entity into a :ref:`scene` exactly like a regular
``EntityCfg``:
.. code-block:: python
from mjlab.scene import SceneCfg
scene_cfg = SceneCfg(
num_envs=4096,
entities={"object": object_cfg},
)
Twice as many worlds will hold a cone as a sphere. Variants not listed
in the ``assignment`` dict default to weight 1.0; omit ``assignment``
entirely for uniform allocation across all variants.
What variants can differ in
---------------------------
**Free to vary across variants:** the mesh asset assigned to each
slot, the number of mesh geoms per ``(body, role)`` bucket on the
variant body (one variant can have more collision meshes than
another), the per-mesh-geom attributes that travel with the mesh
(friction, contact bits, mass, density, ``condim``, and a handful of
others), and explicit body inertial values within whichever single
inertial mode the variants agree on per body.
**Must match across variants:** the body tree, joint topology,
primitive (non-mesh) geoms, and any actuators / sensors / tendons /
equalities. Variants must also agree on the inertial representation
per body (mesh-derived, diagonal, or fullinertia), and may not use the
reserved ``mjlab/pad/`` name prefix on any element. Variant entities
must also be floating-base: the root body declares a freejoint.
The validator runs at entity build time and raises ``ValueError``
naming the offending variant and the exact mismatch.
How variants are assembled
--------------------------
mjlab merges every variant's mesh assets into a single ``MjSpec`` and
gives the variant body enough mesh-geom *slots* to cover the maximum
mesh count any variant uses for each ``(body, role)`` bucket. A slot
is identified by ``(body_path, role, ordinal)``. ``role`` is "visual"
or "collision", derived from ``contype``/``conaffinity``;
mujoco_warp's ``geom_contype``/``geom_conaffinity`` are 1D shared
(not per-world), so a slot's role is fixed across worlds by
construction.
A worked example
~~~~~~~~~~~~~~~~
Say variant ``sphere`` has 1 visual mesh geom and 2 collision mesh
geoms on the prop body, and variant ``cone`` has 1 visual mesh geom
and 4 collision mesh geoms on the same body.
.. code-block:: text
sphere variant body cone variant body
------------------- -------------------
prop body prop body
[visual] sphere_vis [visual] cone_vis
[coll] sphere_col_0 [coll] cone_col_0
[coll] sphere_col_1 [coll] cone_col_1
[coll] cone_col_2
[coll] cone_col_3
mjlab walks each variant's body tree, buckets mesh geoms by
``(body_path, role)``, and lays the union out as slots:
.. list-table::
:header-rows: 1
:widths: 8 18 8 12 27 27
* - Slot
- body_path
- role
- ordinal
- sphere fills with
- cone fills with
* - 0
- /prop
- visual
- 0
- sphere_vis
- cone_vis
* - 1
- /prop
- collision
- 0
- sphere_col_0
- cone_col_0
* - 2
- /prop
- collision
- 1
- sphere_col_1
- cone_col_1
* - 3
- /prop
- collision
- 2
- *(unfilled)*
- cone_col_2
* - 4
- /prop
- collision
- 3
- *(unfilled)*
- cone_col_3
Five slots total. The merged scene's prop body has five mesh geoms:
slot 0 plus four collision slots (the union of sphere's two and
cone's four). At merge time, every variant's mesh asset is added to
the merged spec under a unique name (e.g.
``sphere/sphere_vis``, ``cone/cone_col_2``).
The merged scene compiles once into a single canonical ``MjModel``
that every world in the batch agrees on layout-wise: same nbody,
ngeom, same body and geom IDs. mjlab's per-world overrides on top of
that one model are what make worlds heterogeneous.
What each world sees at runtime
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Worlds where ``sphere`` is active see only its three meshes; the two
extra collision slots are disabled via per-world ``geom_dataid = -1``,
and mujoco_warp skips them. Worlds where ``cone`` is active see all
five meshes wired up.
.. list-table::
:header-rows: 1
:widths: 14 14 12 12 12 12 12
* - World
- variant
- slot 0
- slot 1
- slot 2
- slot 3
- slot 4
* - 0
- sphere
- sphere_vis
- sphere_col_0
- sphere_col_1
- **off (-1)**
- **off (-1)**
* - 1
- cone
- cone_vis
- cone_col_0
- cone_col_1
- cone_col_2
- cone_col_3
Three categories of per-world override carry the variation:
* **geom_dataid** is a ``(num_envs, ngeom)`` table. Its row for
world W picks which compiled mesh each slot points at. ``-1`` is
the "skip me" sentinel mujoco_warp already understands.
* **Mesh-derived fields** (``geom_size``, ``geom_rbound``,
``geom_aabb``, ``geom_pos``, ``geom_quat``, ``body_mass``,
``body_subtreemass``, ``body_inertia``, ``body_invweight0``,
``body_ipos``, ``body_iquat``) are stored as ``(num_envs, ...)``
arrays. The values for sphere worlds reflect a sphere-shaped
inertia tensor and sphere-sized AABBs; the values for cone worlds
reflect the cone. The full list is in
``mjlab.entity.variants.VARIANT_DEPENDENT_FIELDS``.
* **Per-mesh-geom attributes** (contact bits, friction, mass,
density, condim, group, priority, rgba, solref, solimp, margin,
gap) are captured per variant in ``VariantGeomSpec`` at merge time
and restored verbatim on the slot geom during the per-variant
reference compile. So if sphere's collision geoms have
``friction=0.5`` and cone's have ``friction=1.2``, world W's
per-step friction reflects the assigned variant's source value.
The one exception is ``material``, which is not propagated across
variants; if you need per-world appearance variation use DR on
``geom_rgba`` / ``mat_rgba``.
If ``sphere`` adds a body that ``cone`` lacks (or vice versa), the
validator rejects the configuration before any of the merge logic
runs. The slot mechanism only flexes mesh geom counts within
matching bodies; everything structural above the geom level must
agree.
.. note::
**Doesn't compiling the merged scene ruin the prop body's
inertia?**
No, but it's worth understanding why, because the naive intuition
says it should. If you stuck every variant's mesh geoms on the
prop body and called ``spec.compile()``, MuJoCo would sum each
geom's inertial contribution, and you would get a body whose mass
and inertia tensor are a meaningless mix of every variant's shape.
mjlab avoids this in two layers:
* **The merged scene does not stick every variant's geoms on the
body.** The prop body in the merged spec carries variant 0's
mesh geoms (with their original mass and density) plus, for any
slot variant 0 doesn't fill, a synthesized padding geom that has
``mass = 0`` and ``density = 0``. Padding contributes nothing to
body inertia. Other variants' meshes are present in the merged
spec only as **mesh assets** (in the assets section, not as geoms
on any body). They get wired in at runtime via per-world
``geom_dataid`` and never affect the host compile's inertial
sums.
* **Per-world overrides come from per-variant source compiles.**
Even with the above, the merged-scene compile's prop body inertia
is only correct for variant 0. For every other variant, mjlab
compiles that variant's original source spec in isolation (one
body, one variant's worth of meshes), reads the resulting
``body_mass``, ``body_inertia``, ``body_ipos``, ``body_iquat``,
``body_invweight0``, and ``body_subtreemass``, and writes them
into the per-world arrays at the prop body's index.
Net result: world W's prop body inertia is byte-equal to what you
would get by compiling variant W's source spec on its own. There
is a regression test
(``test_visual_collision_split_inertia_matches_independent_compile``
in ``tests/test_variants.py``) that asserts exactly this against
independent per-variant compiles.
World assignment
----------------
How worlds get mapped to variants is controlled by the ``assignment``
field on ``VariantEntityCfg``. It accepts three shapes:
* ``None`` (default): uniform allocation across variants.
* ``dict[str, float]``: per-variant weights. Variants not listed
default to weight 1.0.
* ``Callable[[int], Sequence[int]]``: an explicit assignment function
called with ``num_envs`` at simulation init.
Both the ``None`` and dict cases use the
`largest remainder method
<https://en.wikipedia.org/wiki/Largest_remainder_method>`_. Each
variant's quota is ``q_i = (w_i / sum(w)) * num_envs``; each variant
first receives ``floor(q_i)`` worlds, and the remaining
``num_envs - sum(floors)`` worlds go to the variants with the largest
fractional remainders, with ties broken by declaration order. For
``num_envs = 10`` and weights ``(1.0, 2.0, 1.0)`` this gives
``(3, 5, 2)`` worlds per variant. Weights are normalized internally,
so ``{"a": 1, "b": 2, "c": 1}`` and ``{"a": 0.25, "b": 0.5, "c": 0.25}``
produce identical assignments. A weight of zero is allowed and
produces zero worlds for that variant; at least one variant must end
up with positive weight.
The default and dict paths are purely deterministic given
``(assignment, num_envs)``. With ``assignment={"a": 1, "b": 1}`` and
``num_envs = 8`` you always get ``[0, 0, 0, 0, 1, 1, 1, 1]``. There is
no seed involved; rerunning the same config produces the same
partition every time. Note that the partition's *boundaries* depend
on ``num_envs``, so world W's variant is not necessarily stable when
you change ``num_envs``. If you need explicit per-world stability
across batch sizes (e.g. "world 0 is always variant 0, world 1 is
always variant 1, regardless of how many envs I launch"), use a
callable assignment as below.
Variant assignment is fixed at ``Simulation`` initialization and does
not resample on episode reset. The intended use is heterogeneous
training across the batch, not per-episode mesh randomization.
Read the resolved assignment from user code via
``env.sim.world_to_variant``:
.. code-block:: python
>>> env.sim.world_to_variant["object"]
tensor([0, 0, 0, 1, 1, 1, 1, 1, 1, 1])
The mapping is keyed by entity name (without trailing slash) and
returns a ``(num_envs,)`` tensor of variant indices in the order
variants were declared in ``VariantEntityCfg.variants``. The dict is
empty for non-variant scenes.
Custom assignment with a callable
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
When the weighted default is not what you want, pass a callable to
``assignment``. The callable receives ``num_envs`` and must return a
length-``num_envs`` sequence of variant indices in
``[0, len(variants))``. The returned sequence's length and bounds are
validated at sim init; mismatches raise a ``ValueError`` naming the
offending entity.
A few patterns:
**Round-robin** - cycle through variants by world index.
.. code-block:: python
cfg = VariantEntityCfg(
variants={"a": make_a, "b": make_b, "c": make_c},
assignment=lambda n: [w % 3 for w in range(n)],
)
**Stratified halves** - first half is variant 0, second half is
variant 1.
.. code-block:: python
cfg = VariantEntityCfg(
variants={"easy": make_easy, "hard": make_hard},
assignment=lambda n: [0] * (n // 2) + [1] * (n - n // 2),
)
Domain randomization
--------------------
Domain randomization on variant scenes preserves per-variant baselines
automatically. When the simulation initializes, mjlab snapshots the
variant-dependent fields as ``(num_envs, ...)`` tensors and registers
them in ``sim.per_world_default_fields``. DR operations that read
defaults (scale, additive offsets) detect this registration and index
the per-world default array by environment, so a 10% mass scale
applied across a batch containing a 100 g sphere variant and a 1 kg
cube variant produces 10% perturbations *around each variant's own
mass*, not 10% of a shared template mass.
Fields that are not variant-dependent (``geom_friction``,
``dof_armature``, ``dof_damping``, and so on) behave identically on
variant and non-variant scenes.
For inertial randomization the recommended path is
``dr.pseudo_inertia``, which jointly randomizes mass, COM offset,
principal moments of inertia, and principal frame orientation through
the pseudo-inertia matrix factorization of `Rucker and Wensing (2022)
<https://par.nsf.gov/servlets/purl/10347458>`_. It is exact for any
perturbation magnitude and remains physically consistent across
variants of different scale. ``dr.body_mass`` modifies ``body_mass``
without touching the inertia tensor and emits a ``UserWarning`` when
called; it is appropriate only for modeling a point mass added at the
COM, not for density-like randomization. The distinction matters more
on variant scenes than on single-asset scenes because variants often
differ in mass by an order of magnitude.
Viewers
-------
The native viewer, offscreen renderer, and Viser viewer all sync the
selected environment's per-world fields into the host ``MjModel``
before rendering, so the rendered geometry matches the variant
assigned to the viewed environment. Switching environments in the
native viewer (the ``,`` and ``.`` keys) updates the displayed mesh
accordingly.
Viser bakes mesh data into batched handles and cannot rely on a live
view of ``geom_dataid``. It groups worlds by visual fingerprint (mesh
selection, local geom frames, baked appearance) and builds one batched
handle per group, with each environment assigned to its handle. A
scene with N variants typically produces up to N handles per body.
Convex hull visualization is computed per variant from the variant's
mesh vertices.
Performance
-----------
**Per-step cost is unaffected by variant count.** Variant-dependent
fields are stored as per-world arrays accessed by world index in the
existing kernels, with no branching or dispatch on variant.
**Construction cost is linear in the total variant count.** mjlab
compiles the merged scene once to produce the canonical ``MjModel``,
then compiles each variant's original (un-merged) source spec in
isolation to recover that variant's per-body and per-geom mesh-derived
fields. Each per-variant compile sees only that variant's single body
and mesh, so its cost is independent of the total number of variants
in the scene.
For a scene with one variant entity declaring k variants, construction
runs ``1 + k`` compiles. With multiple variant entities, compiles
decouple across entities: two variant entities of 5 variants each cost
``1 + 5 + 5 = 11`` compiles, not ``1 + 5 * 5 = 26``. As an order of
magnitude on CPU with typical procedural meshes, each per-variant
compile takes around 1-2 ms, so a scene with 100 variants pays a few
hundred milliseconds at startup and a scene with 1000 variants pays
roughly two seconds.
The merged spec contains every variant's mesh assets simultaneously,
so memory at scene-build time scales with the total mesh vertex /
face count across all variants. This is paid once at startup and does
not affect training throughput.
Limitations
-----------
**Floating-base only.** Each variant's root body must declare a free
joint. Fixed-base variants are rejected; mocap auto-wrapping that
applies to non-variant entities is not applied here.
**Material assets are not propagated.** Each variant's ``contype``,
``conaffinity``, ``condim``, ``friction``, ``mass``, ``density``,
``group``, ``priority``, ``rgba``, ``solref``, ``solimp``, ``margin``,
and ``gap`` are restored per-world during compile, but the
``material`` reference on slot geoms inherits whichever material the
template variant set. Use DR on ``geom_rgba`` / ``mat_rgba`` for
per-world appearance variation.
**Assignment is fixed at sim init.** There is no API to swap a world
to a different variant on episode reset. World W's mesh asset is
whatever it was assigned at init for the lifetime of the simulation.
Per-episode mesh randomization is not supported today; DR can vary
scalar properties (mass, friction, color, scale) on a fixed variant
but cannot swap one mesh for another.
**No support for per-world differing kinematic topology.** Variants
must share the same body tree, joints, and actuator/sensor counts,
so you cannot configure things like:
* a different number of objects per world (world 0 has two props on
the table, world 1 has three);
* different articulation per world (world 0's prop is an articulated
drawer with a slider joint, world 1's prop is a rigid block).
True heterogeneous topology requires upstream support in mujoco_warp
that does not currently exist.