[Bug Report] Error using MultiAssetSpawnerCfg

[chandramohanjagadeesan]

Describe the bug

I used MultiAssetSpawnerCfg to import objects from usd file. When the environment gets reset in super()._reset_idx(env_ids). there is a following error. This error occurs in reset in directRL environment

Steps to reproduce

Cfg in the interactive sceneCfg

object: RigidObjectCfg = RigidObjectCfg(
prim_path="/World/envs/env_.*/Object",
spawn=sim_utils.MultiAssetSpawnerCfg(
assets_cfg=[
sim_utils.UsdFileCfg(
usd_path=f"{dir_path}/Assets/Objects/USD/object_0.usd",
scale = [1.0, 1.0, 1.0],
),
sim_utils.UsdFileCfg(
usd_path=f"{dir_path}/Assets/Objects/USD/object_1.usd",
scale = [1.0, 1.0, 1.0],
),
sim_utils.UsdFileCfg(
usd_path=f"{dir_path}/Assets/Objects/USD/object_2.usd",
scale = [1.0, 1.0, 1.0],
),
sim_utils.UsdFileCfg(
usd_path=f"{dir_path}/Assets/Objects/USD/object_3.usd",
scale = [1.0, 1.0, 1.0],
)
],
random_choice=True,
rigid_props=sim_utils.RigidBodyPropertiesCfg(
solver_position_iteration_count=4, solver_velocity_iteration_count=0
),
mass_props=sim_utils.MassPropertiesCfg(mass=1.0),
collision_props=sim_utils.CollisionPropertiesCfg(),
),
init_state=RigidObjectCfg.InitialStateCfg(pos=(0.5, 0.0, 0.0), rot=(0.0, -0.0, 0.0, -0.0)),
)

Error code

Exception has occurred: RuntimeError
CUDA error: device-side assert triggered
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1
Compile with TORCH_USE_CUDA_DSA to enable device-side assertions.
File "/media/team_omd/DATA/omniverse/pkg/IsaacLab/IsaacLab/source/extensions/omni.isaac.lab/omni/isaac/lab/assets/rigid_object/rigid_object.py", line 104, in reset
self._external_torque_b[env_ids] = 0.0
File "/media/team_omd/DATA/omniverse/pkg/IsaacLab/IsaacLab/source/extensions/omni.isaac.lab/omni/isaac/lab/scene/interactive_scene.py", line 351, in reset
rigid_object.reset(env_ids)
File "/media/team_omd/DATA/omniverse/pkg/IsaacLab/IsaacLab/source/extensions/omni.isaac.lab/omni/isaac/lab/envs/direct_rl_env.py", line 537, in _reset_idx
self.scene.reset(env_ids)
File "/home/team_omd/Projects/chandra_ma/continual_nbv/NBVPlanning/Environment/NBVEnv.py", line 430, in _reset_idx
super()._reset_idx(env_ids)
File "/media/team_omd/DATA/omniverse/pkg/IsaacLab/IsaacLab/source/extensions/omni.isaac.lab/omni/isaac/lab/envs/direct_rl_env.py", line 271, in reset
self._reset_idx(indices)
File "/media/team_omd/DATA/omniverse/pkg/IsaacLab/IsaacLab/source/extensions/omni.isaac.lab_tasks/omni/isaac/lab_tasks/utils/wrappers/sb3.py", line 207, in reset
obs_dict, _ = self.env.reset()
File "/home/team_omd/Projects/chandra_ma/continual_nbv/NBVPlanning/test.py", line 106, in main
obs = env.reset()
File "/home/team_omd/Projects/chandra_ma/continual_nbv/NBVPlanning/test.py", line 138, in
main()
RuntimeError: CUDA error: device-side assert triggered
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1
Compile with TORCH_USE_CUDA_DSA to enable device-side assertions.

System Info

Describe the characteristic of your environment:

Additional context

Checklist

• I have checked that there is no similar issue in the repo (required)
• I have checked that the issue is not in running Isaac Sim itself and is related to the repo

Acceptance Criteria

Add the criteria for which this task is considered done. If not known at issue creation time, you can add this once the issue is assigned.

[TuluhanAkbulut]

I am getting the same error with the ManagerbasedRL environments too. Here, if I spawn using the following path
usd_path=[

# self.asset_config_path + "Blocks/DexCube/dex_cube_instanceable.usd",
# self.asset_config_path + "Blocks/blue_block.usd",
# self.asset_config_path + "Blocks/green_block.usd",
# self.asset_config_path + "Blocks/red_block.usd",
# self.asset_config_path + "Blocks/yellow_block.usd",
# self.asset_config_path + "Mugs/SM_Mug_A2.usd",
# self.asset_config_path + "Mugs/SM_Mug_B1.usd",

],

If usd path consists of red, blue, green, and yellow blocks, it works fine. If dex_cube is alone in the list, it works fine too. However, if it consists dex_cube with other blocks it throws the CUDA error. For SM_Mugs it is saying that AttributeError: 'RigidObject' object has no attribute '_external_force_b' which may be unrelated.

[chandramohanjagadeesan]

Hi @TuluhanAkbulut

The issue is in reset method in RigidObject.py file.

It is caused by how self._external_force_b is indexed with env_ids to set value 0.0. I found a work around to make it work. You can change the lines

reset external wrench

    self._external_force_b[env_ids] = 0.0
    self._external_torque_b[env_ids] = 0.0

to

reset external wrench

    self._external_force_b[0,0,env_ids] = 0.0
    self._external_torque_b[0,0,env_ids] = 0.0

This makes it work but I have to test if this change has any implications on other functions.

Error occurs because env_ids is of shape tensor([3]) and self._external_force_b[env_ids] is of shape tensor[1,1,3]. Hope this helps till the actual solution comes up.

For note : @Mayankm96. I am tagging you because I found that you developed this feature from the past issues and I would like to know if this solution would work. Thanks in advance for your inputs.

[TuluhanAkbulut]

Thanks for the reply, it did not work for me. It actually made it worse by throwing the same error even for only blue_block in the list, which was not happening before. Here, the related code and error message:

I am trying this method with Omniverse-nucleus objects at the moment:

self.asset_config_path = http://omniverse-content-production.s3-us-west-2.amazonaws.com/Assets/Isaac/4.2/Isaac/Props/

self.scene.object2 = RigidObjectCfg(
    prim_path="{ENV_REGEX_NS}/Object2",
    spawn=sim_utils.MultiAssetSpawnerCfg(
    assets_cfg=[
        # sim_utils.UsdFileCfg(
        # usd_path=self.asset_config_path + "Blocks/DexCube/dex_cube_instanceable.usd",
        # scale = [0.8, 0.8, 0.8],
        # ),
        sim_utils.UsdFileCfg(
        usd_path=self.asset_config_path + "Blocks/blue_block.usd",
        scale = [1.0, 1.0, 1.0],
        ),
        # sim_utils.UsdFileCfg(
        # usd_path=self.asset_config_path + "Blocks/red_block.usd",
        # scale = [0.7, 0.7, 0.7],
        # ),
    ],
    random_choice=False,
    rigid_props=sim_utils.RigidBodyPropertiesCfg(
        solver_position_iteration_count=4, solver_velocity_iteration_count=0
    ),
    mass_props=sim_utils.MassPropertiesCfg(mass=1.0),
    collision_props=sim_utils.CollisionPropertiesCfg(),
),
init_state=RigidObjectCfg.InitialStateCfg(pos=[0.8, 0.2, 0.1], rot=[1, 0, 0, 0]),
)

Error message:

../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [0,0,0], thread: [3,0,0] Assertion -sizes[i] <= index && index < sizes[i] && "index out of bounds" failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [0,0,0], thread: [4,0,0] Assertion -sizes[i] <= index && index < sizes[i] && "index out of bounds" failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [0,0,0], thread: [5,0,0] Assertion -sizes[i] <= index && index < sizes[i] && "index out of bounds" failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [0,0,0], thread: [6,0,0] Assertion -sizes[i] <= index && index < sizes[i] && "index out of bounds" failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [0,0,0], thread: [7,0,0] Assertion -sizes[i] <= index && index < sizes[i] && "index out of bounds" failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [0,0,0], thread: [8,0,0] Assertion -sizes[i] <= index && index < sizes[i] && "index out of bounds" failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [0,0,0], thread: [9,0,0] Assertion -sizes[i] <= index && index < sizes[i] && "index out of bounds" failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [0,0,0], thread: [10,0,0] Assertion -sizes[i] <= index && index < sizes[i] && "index out of bounds" failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [0,0,0], thread: [11,0,0] Assertion -sizes[i] <= index && index < sizes[i] && "index out of bounds" failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [0,0,0], thread: [12,0,0] Assertion -sizes[i] <= index && index < sizes[i] && "index out of bounds" failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [0,0,0], thread: [13,0,0] Assertion -sizes[i] <= index && index < sizes[i] && "index out of bounds" failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [0,0,0], thread: [14,0,0] Assertion -sizes[i] <= index && index < sizes[i] && "index out of bounds" failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [0,0,0], thread: [15,0,0] Assertion -sizes[i] <= index && index < sizes[i] && "index out of bounds" failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [0,0,0], thread: [16,0,0] Assertion -sizes[i] <= index && index < sizes[i] && "index out of bounds" failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [0,0,0], thread: [17,0,0] Assertion -sizes[i] <= index && index < sizes[i] && "index out of bounds" failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [0,0,0], thread: [18,0,0] Assertion -sizes[i] <= index && index < sizes[i] && "index out of bounds" failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [0,0,0], thread: [19,0,0] Assertion -sizes[i] <= index && index < sizes[i] && "index out of bounds" failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [0,0,0], thread: [20,0,0] Assertion -sizes[i] <= index && index < sizes[i] && "index out of bounds" failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [0,0,0], thread: [21,0,0] Assertion -sizes[i] <= index && index < sizes[i] && "index out of bounds" failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [0,0,0], thread: [22,0,0] Assertion -sizes[i] <= index && index < sizes[i] && "index out of bounds" failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [0,0,0], thread: [23,0,0] Assertion -sizes[i] <= index && index < sizes[i] && "index out of bounds" failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [0,0,0], thread: [24,0,0] Assertion -sizes[i] <= index && index < sizes[i] && "index out of bounds" failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [0,0,0], thread: [25,0,0] Assertion -sizes[i] <= index && index < sizes[i] && "index out of bounds" failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [0,0,0], thread: [26,0,0] Assertion -sizes[i] <= index && index < sizes[i] && "index out of bounds" failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [0,0,0], thread: [27,0,0] Assertion -sizes[i] <= index && index < sizes[i] && "index out of bounds" failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [0,0,0], thread: [28,0,0] Assertion -sizes[i] <= index && index < sizes[i] && "index out of bounds" failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [0,0,0], thread: [29,0,0] Assertion -sizes[i] <= index && index < sizes[i] && "index out of bounds" failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [0,0,0], thread: [30,0,0] Assertion -sizes[i] <= index && index < sizes[i] && "index out of bounds" failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [0,0,0], thread: [31,0,0] Assertion -sizes[i] <= index && index < sizes[i] && "index out of bounds" failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [0,0,0], thread: [32,0,0] Assertion -sizes[i] <= index && index < sizes[i] && "index out of bounds" failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [0,0,0], thread: [33,0,0] Assertion -sizes[i] <= index && index < sizes[i] && "index out of bounds" failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [0,0,0], thread: [34,0,0] Assertion -sizes[i] <= index && index < sizes[i] && "index out of bounds" failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [0,0,0], thread: [35,0,0] Assertion -sizes[i] <= index && index < sizes[i] && "index out of bounds" failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [0,0,0], thread: [36,0,0] Assertion -sizes[i] <= index && index < sizes[i] && "index out of bounds" failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [0,0,0], thread: [37,0,0] Assertion -sizes[i] <= index && index < sizes[i] && "index out of bounds" failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [0,0,0], thread: [38,0,0] Assertion -sizes[i] <= index && index < sizes[i] && "index out of bounds" failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [0,0,0], thread: [39,0,0] Assertion -sizes[i] <= index && index < sizes[i] && "index out of bounds" failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [0,0,0], thread: [40,0,0] Assertion -sizes[i] <= index && index < sizes[i] && "index out of bounds" failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [0,0,0], thread: [41,0,0] Assertion -sizes[i] <= index && index < sizes[i] && "index out of bounds" failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [0,0,0], thread: [42,0,0] Assertion -sizes[i] <= index && index < sizes[i] && "index out of bounds" failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [0,0,0], thread: [43,0,0] Assertion -sizes[i] <= index && index < sizes[i] && "index out of bounds" failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [0,0,0], thread: [44,0,0] Assertion -sizes[i] <= index && index < sizes[i] && "index out of bounds" failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [0,0,0], thread: [45,0,0] Assertion -sizes[i] <= index && index < sizes[i] && "index out of bounds" failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [0,0,0], thread: [46,0,0] Assertion -sizes[i] <= index && index < sizes[i] && "index out of bounds" failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [0,0,0], thread: [47,0,0] Assertion -sizes[i] <= index && index < sizes[i] && "index out of bounds" failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [0,0,0], thread: [48,0,0] Assertion -sizes[i] <= index && index < sizes[i] && "index out of bounds" failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [0,0,0], thread: [49,0,0] Assertion -sizes[i] <= index && index < sizes[i] && "index out of bounds" failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [0,0,0], thread: [50,0,0] Assertion -sizes[i] <= index && index < sizes[i] && "index out of bounds" failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [0,0,0], thread: [51,0,0] Assertion -sizes[i] <= index && index < sizes[i] && "index out of bounds" failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [0,0,0], thread: [52,0,0] Assertion -sizes[i] <= index && index < sizes[i] && "index out of bounds" failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [0,0,0], thread: [53,0,0] Assertion -sizes[i] <= index && index < sizes[i] && "index out of bounds" failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [0,0,0], thread: [54,0,0] Assertion -sizes[i] <= index && index < sizes[i] && "index out of bounds" failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [0,0,0], thread: [55,0,0] Assertion -sizes[i] <= index && index < sizes[i] && "index out of bounds" failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [0,0,0], thread: [56,0,0] Assertion -sizes[i] <= index && index < sizes[i] && "index out of bounds" failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [0,0,0], thread: [57,0,0] Assertion -sizes[i] <= index && index < sizes[i] && "index out of bounds" failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [0,0,0], thread: [58,0,0] Assertion -sizes[i] <= index && index < sizes[i] && "index out of bounds" failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [0,0,0], thread: [59,0,0] Assertion -sizes[i] <= index && index < sizes[i] && "index out of bounds" failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [0,0,0], thread: [60,0,0] Assertion -sizes[i] <= index && index < sizes[i] && "index out of bounds" failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [0,0,0], thread: [61,0,0] Assertion -sizes[i] <= index && index < sizes[i] && "index out of bounds" failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [0,0,0], thread: [62,0,0] Assertion -sizes[i] <= index && index < sizes[i] && "index out of bounds" failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [0,0,0], thread: [63,0,0] Assertion -sizes[i] <= index && index < sizes[i] && "index out of bounds" failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [0,0,0], thread: [64,0,0] Assertion -sizes[i] <= index && index < sizes[i] && "index out of bounds" failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [0,0,0], thread: [65,0,0] Assertion -sizes[i] <= index && index < sizes[i] && "index out of bounds" failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [0,0,0], thread: [66,0,0] Assertion -sizes[i] <= index && index < sizes[i] && "index out of bounds" failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [0,0,0], thread: [67,0,0] Assertion -sizes[i] <= index && index < sizes[i] && "index out of bounds" failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [0,0,0], thread: [68,0,0] Assertion -sizes[i] <= index && index < sizes[i] && "index out of bounds" failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [0,0,0], thread: [69,0,0] Assertion -sizes[i] <= index && index < sizes[i] && "index out of bounds" failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [0,0,0], thread: [70,0,0] Assertion -sizes[i] <= index && index < sizes[i] && "index out of bounds" failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [0,0,0], thread: [71,0,0] Assertion -sizes[i] <= index && index < sizes[i] && "index out of bounds" failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [0,0,0], thread: [72,0,0] Assertion -sizes[i] <= index && index < sizes[i] && "index out of bounds" failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [0,0,0], thread: [73,0,0] Assertion -sizes[i] <= index && index < sizes[i] && "index out of bounds" failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [0,0,0], thread: [74,0,0] Assertion -sizes[i] <= index && index < sizes[i] && "index out of bounds" failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [0,0,0], thread: [75,0,0] Assertion -sizes[i] <= index && index < sizes[i] && "index out of bounds" failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [0,0,0], thread: [76,0,0] Assertion -sizes[i] <= index && index < sizes[i] && "index out of bounds" failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [0,0,0], thread: [77,0,0] Assertion -sizes[i] <= index && index < sizes[i] && "index out of bounds" failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [0,0,0], thread: [78,0,0] Assertion -sizes[i] <= index && index < sizes[i] && "index out of bounds" failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [0,0,0], thread: [79,0,0] Assertion -sizes[i] <= index && index < sizes[i] && "index out of bounds" failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [0,0,0], thread: [80,0,0] Assertion -sizes[i] <= index && index < sizes[i] && "index out of bounds" failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [0,0,0], thread: [81,0,0] Assertion -sizes[i] <= index && index < sizes[i] && "index out of bounds" failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [0,0,0], thread: [82,0,0] Assertion -sizes[i] <= index && index < sizes[i] && "index out of bounds" failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [0,0,0], thread: [83,0,0] Assertion -sizes[i] <= index && index < sizes[i] && "index out of bounds" failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [0,0,0], thread: [84,0,0] Assertion -sizes[i] <= index && index < sizes[i] && "index out of bounds" failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [0,0,0], thread: [85,0,0] Assertion -sizes[i] <= index && index < sizes[i] && "index out of bounds" failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [0,0,0], thread: [86,0,0] Assertion -sizes[i] <= index && index < sizes[i] && "index out of bounds" failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [0,0,0], thread: [87,0,0] Assertion -sizes[i] <= index && index < sizes[i] && "index out of bounds" failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [0,0,0], thread: [88,0,0] Assertion -sizes[i] <= index && index < sizes[i] && "index out of bounds" failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [0,0,0], thread: [89,0,0] Assertion -sizes[i] <= index && index < sizes[i] && "index out of bounds" failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [0,0,0], thread: [90,0,0] Assertion -sizes[i] <= index && index < sizes[i] && "index out of bounds" failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [0,0,0], thread: [91,0,0] Assertion -sizes[i] <= index && index < sizes[i] && "index out of bounds" failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [0,0,0], thread: [92,0,0] Assertion -sizes[i] <= index && index < sizes[i] && "index out of bounds" failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [0,0,0], thread: [93,0,0] Assertion -sizes[i] <= index && index < sizes[i] && "index out of bounds" failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [0,0,0], thread: [94,0,0] Assertion -sizes[i] <= index && index < sizes[i] && "index out of bounds" failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [0,0,0], thread: [95,0,0] Assertion -sizes[i] <= index && index < sizes[i] && "index out of bounds" failed.
Traceback (most recent call last):
File "/home/tuluhan/Desktop/Concept/isaac_ws/simple.py", line 102, in
main()
File "/home/tuluhan/Desktop/Concept/isaac_ws/simple.py", line 86, in main
env.reset()
File "/home/tuluhan/miniconda3/envs/isaac_env/lib/python3.10/site-packages/gymnasium/wrappers/order_enforcing.py", line 61, in reset
return self.env.reset(**kwargs)
File "/home/tuluhan/Desktop/IsaacLab/source/extensions/omni.isaac.lab/omni/isaac/lab/envs/manager_based_env.py", line 252, in reset
self._reset_idx(indices)
File "/home/tuluhan/Desktop/IsaacLab/source/extensions/omni.isaac.lab/omni/isaac/lab/envs/manager_based_rl_env.py", line 325, in _reset_idx
self.scene.reset(env_ids)
File "/home/tuluhan/Desktop/IsaacLab/source/extensions/omni.isaac.lab/omni/isaac/lab/scene/interactive_scene.py", line 351, in reset
rigid_object.reset(env_ids)
File "/home/tuluhan/Desktop/IsaacLab/source/extensions/omni.isaac.lab/omni/isaac/lab/assets/rigid_object/rigid_object.py", line 103, in reset
self._external_force_b[0, env_ids] = 0.0
RuntimeError: CUDA error: device-side assert triggered
Compile with TORCH_USE_CUDA_DSA to enable device-side assertions.

2024-10-21 01:51:00 [6,277ms] [Error] [omni.physx.fabric.plugin] CUDA error: device-side assert triggered: ../../../extensions/runtime/source/omni.physx.fabric/plugins/DirectGpuHelper.cpp: 328
2024-10-21 01:51:00 [6,277ms] [Error] [omni.physx.fabric.plugin] CUDA error: device-side assert triggered: ../../../extensions/runtime/source/omni.physx.fabric/plugins/DirectGpuHelper.cpp: 331
2024-10-21 01:51:00 [6,277ms] [Error] [omni.physx.fabric.plugin] CUDA error: device-side assert triggered: ../../../extensions/runtime/source/omni.physx.fabric/plugins/DirectGpuHelper.cpp: 334
2024-10-21 01:51:00 [6,277ms] [Error] [omni.physx.fabric.plugin] CUDA error: device-side assert triggered: ../../../extensions/runtime/source/omni.physx.fabric/plugins/DirectGpuHelper.cpp: 337
2024-10-21 01:51:00 [6,277ms] [Error] [omni.physx.fabric.plugin] CUDA error: device-side assert triggered: ../../../extensions/runtime/source/omni.physx.fabric/plugins/DirectGpuHelper.cpp: 340
2024-10-21 01:51:00 [6,278ms] [Warning] [omni.physx.plugin] PhysX warning: /builds/omniverse/physics/physx/source/gpucommon/src/PxgCudaMemoryAllocator.cpp, FILE /builds/omniverse/physics/physx/source/gpucommon/src/PxgCudaMemoryAllocator.cpp, LINE 167
2024-10-21 01:51:00 [6,279ms] [Warning] [omni.physx.plugin] PhysX warning: /builds/omniverse/physics/physx/source/gpucommon/src/PxgCudaMemoryAllocator.cpp, FILE /builds/omniverse/physics/physx/source/gpucommon/src/PxgCudaMemoryAllocator.cpp, LINE 167
2024-10-21 01:51:00 [6,279ms] [Warning] [omni.physx.plugin] PhysX warning: /builds/omniverse/physics/physx/source/gpucommon/src/PxgCudaMemoryAllocator.cpp, FILE /builds/omniverse/physics/physx/source/gpucommon/src/PxgCudaMemoryAllocator.cpp, LINE 167
2024-10-21 01:51:00 [6,279ms] [Warning] [omni.physx.plugin] PhysX warning: /builds/omniverse/physics/physx/source/gpucommon/src/PxgCudaMemoryAllocator.cpp, FILE /builds/omniverse/physics/physx/source/gpucommon/src/PxgCudaMemoryAllocator.cpp, LINE 167
2024-10-21 01:51:00 [6,279ms] [Warning] [omni.physx.plugin] PhysX warning: /builds/omniverse/physics/physx/source/gpucommon/src/PxgCudaMemoryAllocator.cpp, FILE /builds/omniverse/physics/physx/source/gpucommon/src/PxgCudaMemoryAllocator.cpp, LINE 167
2024-10-21 01:51:00 [6,279ms] [Warning] [omni.physx.plugin] PhysX warning: /builds/omniverse/physics/physx/source/gpucommon/src/PxgCudaMemoryAllocator.cpp, FILE /builds/omniverse/physics/physx/source/gpucommon/src/PxgCudaMemoryAllocator.cpp, LINE 167
2024-10-21 01:51:00 [6,281ms] [Error] [omni.physx.plugin] PhysX error: Failed to unload CUDA module data, returned 710., FILE /builds/omniverse/physics/physx/source/cudamanager/src/CudaContextManager.cpp, LINE 817
2024-10-21 01:51:00 [6,281ms] [Error] [omni.physx.plugin] PhysX error: Failed to unload CUDA module data, returned 710., FILE /builds/omniverse/physics/physx/source/cudamanager/src/CudaContextManager.cpp, LINE 817
2024-10-21 01:51:00 [6,281ms] [Error] [omni.physx.plugin] PhysX error: Failed to unload CUDA module data, returned 710., FILE /builds/omniverse/physics/physx/source/cudamanager/src/CudaContextManager.cpp, LINE 817
2024-10-21 01:51:00 [6,281ms] [Error] [omni.physx.plugin] PhysX error: Failed to unload CUDA module data, returned 710., FILE /builds/omniverse/physics/physx/source/cudamanager/src/CudaContextManager.cpp, LINE 817
2024-10-21 01:51:00 [6,281ms] [Error] [omni.physx.plugin] PhysX error: Failed to unload CUDA module data, returned 710., FILE /builds/omniverse/physics/physx/source/cudamanager/src/CudaContextManager.cpp, LINE 817
2024-10-21 01:51:00 [6,281ms] [Error] [omni.physx.plugin] PhysX error: Failed to unload CUDA module data, returned 710., FILE /builds/omniverse/physics/physx/source/cudamanager/src/CudaContextManager.cpp, LINE 817
2024-10-21 01:51:00 [6,281ms] [Error] [omni.physx.plugin] PhysX error: Failed to unload CUDA module data, returned 710., FILE /builds/omniverse/physics/physx/source/cudamanager/src/CudaContextManager.cpp, LINE 817
2024-10-21 01:51:00 [6,282ms] [Error] [omni.physx.plugin] PhysX error: Failed to unload CUDA module data, returned 710., FILE /builds/omniverse/physics/physx/source/cudamanager/src/CudaContextManager.cpp, LINE 817
2024-10-21 01:51:00 [6,282ms] [Error] [omni.physx.plugin] PhysX error: Failed to unload CUDA module data, returned 710., FILE /builds/omniverse/physics/physx/source/cudamanager/src/CudaContextManager.cpp, LINE 817
2024-10-21 01:51:00 [6,282ms] [Error] [omni.physx.plugin] PhysX error: Failed to unload CUDA module data, returned 710., FILE /builds/omniverse/physics/physx/source/cudamanager/src/CudaContextManager.cpp, LINE 817
2024-10-21 01:51:00 [6,282ms] [Error] [omni.physx.plugin] PhysX error: Failed to unload CUDA module data, returned 710., FILE /builds/omniverse/physics/physx/source/cudamanager/src/CudaContextManager.cpp, LINE 817
2024-10-21 01:51:00 [6,282ms] [Error] [omni.physx.plugin] PhysX error: Failed to unload CUDA module data, returned 710., FILE /builds/omniverse/physics/physx/source/cudamanager/src/CudaContextManager.cpp, LINE 817
2024-10-21 01:51:00 [6,282ms] [Error] [omni.physx.plugin] PhysX error: Failed to unload CUDA module data, returned 710., FILE /builds/omniverse/physics/physx/source/cudamanager/src/CudaContextManager.cpp, LINE 817
2024-10-21 01:51:00 [6,282ms] [Error] [omni.physx.plugin] PhysX error: Failed to unload CUDA module data, returned 710., FILE /builds/omniverse/physics/physx/source/cudamanager/src/CudaContextManager.cpp, LINE 817
2024-10-21 01:51:00 [6,282ms] [Error] [omni.physx.plugin] PhysX error: Failed to unload CUDA module data, returned 710., FILE /builds/omniverse/physics/physx/source/cudamanager/src/CudaContextManager.cpp, LINE 817
2024-10-21 01:51:00 [6,282ms] [Error] [omni.physx.plugin] PhysX error: Failed to unload CUDA module data, returned 710., FILE /builds/omniverse/physics/physx/source/cudamanager/src/CudaContextManager.cpp, LINE 817
2024-10-21 01:51:00 [6,283ms] [Error] [omni.physx.plugin] PhysX error: Failed to unload CUDA module data, returned 710., FILE /builds/omniverse/physics/physx/source/cudamanager/src/CudaContextManager.cpp, LINE 817
2024-10-21 01:51:00 [6,283ms] [Error] [omni.physx.plugin] PhysX error: Failed to unload CUDA module data, returned 710., FILE /builds/omniverse/physics/physx/source/cudamanager/src/CudaContextManager.cpp, LINE 817
2024-10-21 01:51:00 [6,283ms] [Error] [omni.physx.plugin] PhysX error: Failed to unload CUDA module data, returned 710., FILE /builds/omniverse/physics/physx/source/cudamanager/src/CudaContextManager.cpp, LINE 817
2024-10-21 01:51:00 [6,283ms] [Error] [omni.physx.plugin] PhysX error: Failed to unload CUDA module data, returned 710., FILE /builds/omniverse/physics/physx/source/cudamanager/src/CudaContextManager.cpp, LINE 817
2024-10-21 01:51:00 [6,283ms] [Error] [omni.physx.plugin] PhysX error: Failed to unload CUDA module data, returned 710., FILE /builds/omniverse/physics/physx/source/cudamanager/src/CudaContextManager.cpp, LINE 817
2024-10-21 01:51:00 [6,283ms] [Error] [omni.physx.plugin] PhysX error: Failed to unload CUDA module data, returned 710., FILE /builds/omniverse/physics/physx/source/cudamanager/src/CudaContextManager.cpp, LINE 817
2024-10-21 01:51:00 [6,284ms] [Error] [omni.physx.plugin] PhysX error: Failed to unload CUDA module data, returned 710., FILE /builds/omniverse/physics/physx/source/cudamanager/src/CudaContextManager.cpp, LINE 817
2024-10-21 01:51:00 [6,284ms] [Error] [omni.physx.plugin] PhysX error: Failed to unload CUDA module data, returned 710., FILE /builds/omniverse/physics/physx/source/cudamanager/src/CudaContextManager.cpp, LINE 817
2024-10-21 01:51:00 [6,284ms] [Error] [omni.physx.plugin] PhysX error: Failed to unload CUDA module data, returned 710., FILE /builds/omniverse/physics/physx/source/cudamanager/src/CudaContextManager.cpp, LINE 817
2024-10-21 01:51:00 [6,284ms] [Error] [omni.physx.plugin] PhysX error: Failed to unload CUDA module data, returned 710., FILE /builds/omniverse/physics/physx/source/cudamanager/src/CudaContextManager.cpp, LINE 817
2024-10-21 01:51:00 [6,284ms] [Error] [omni.physx.plugin] PhysX error: Failed to unload CUDA module data, returned 710., FILE /builds/omniverse/physics/physx/source/cudamanager/src/CudaContextManager.cpp, LINE 817
2024-10-21 01:51:00 [6,284ms] [Error] [omni.physx.plugin] PhysX error: Failed to unload CUDA module data, returned 710., FILE /builds/omniverse/physics/physx/source/cudamanager/src/CudaContextManager.cpp, LINE 817
2024-10-21 01:51:00 [6,284ms] [Error] [omni.physx.plugin] PhysX error: Failed to unload CUDA module data, returned 710., FILE /builds/omniverse/physics/physx/source/cudamanager/src/CudaContextManager.cpp, LINE 817
2024-10-21 01:51:00 [6,284ms] [Error] [omni.physx.plugin] PhysX error: Failed to unload CUDA module data, returned 710., FILE /builds/omniverse/physics/physx/source/cudamanager/src/CudaContextManager.cpp, LINE 817
2024-10-21 01:51:00 [6,284ms] [Error] [omni.physx.plugin] PhysX error: Failed to unload CUDA module data, returned 710., FILE /builds/omniverse/physics/physx/source/cudamanager/src/CudaContextManager.cpp, LINE 817
2024-10-21 01:51:00 [6,285ms] [Error] [omni.physx.plugin] PhysX error: Failed to unload CUDA module data, returned 710., FILE /builds/omniverse/physics/physx/source/cudamanager/src/CudaContextManager.cpp, LINE 817
2024-10-21 01:51:00 [6,285ms] [Error] [omni.physx.plugin] PhysX error: Failed to unload CUDA module data, returned 710., FILE /builds/omniverse/physics/physx/source/cudamanager/src/CudaContextManager.cpp, LINE 817
2024-10-21 01:51:00 [6,285ms] [Error] [omni.physx.plugin] PhysX error: Failed to unload CUDA module data, returned 710., FILE /builds/omniverse/physics/physx/source/cudamanager/src/CudaContextManager.cpp, LINE 817
2024-10-21 01:51:00 [6,285ms] [Error] [omni.physx.plugin] PhysX error: Failed to unload CUDA module data, returned 710., FILE /builds/omniverse/physics/physx/source/cudamanager/src/CudaContextManager.cpp, LINE 817
2024-10-21 01:51:00 [6,285ms] [Error] [omni.physx.plugin] PhysX error: Failed to unload CUDA module data, returned 710., FILE /builds/omniverse/physics/physx/source/cudamanager/src/CudaContextManager.cpp, LINE 817
2024-10-21 01:51:00 [6,285ms] [Error] [omni.physx.plugin] PhysX error: Failed to unload CUDA module data, returned 710., FILE /builds/omniverse/physics/physx/source/cudamanager/src/CudaContextManager.cpp, LINE 817
2024-10-21 01:51:00 [6,285ms] [Error] [omni.physx.plugin] PhysX error: Failed to unload CUDA module data, returned 710., FILE /builds/omniverse/physics/physx/source/cudamanager/src/CudaContextManager.cpp, LINE 817
2024-10-21 01:51:00 [6,285ms] [Error] [omni.physx.plugin] PhysX error: Failed to unload CUDA module data, returned 710., FILE /builds/omniverse/physics/physx/source/cudamanager/src/CudaContextManager.cpp, LINE 817
2024-10-21 01:51:00 [6,285ms] [Error] [omni.physx.plugin] PhysX error: Failed to unload CUDA module data, returned 710., FILE /builds/omniverse/physics/physx/source/cudamanager/src/CudaContextManager.cpp, LINE 817
2024-10-21 01:51:00 [6,286ms] [Error] [omni.physx.plugin] PhysX error: Failed to unload CUDA module data, returned 710., FILE /builds/omniverse/physics/physx/source/cudamanager/src/CudaContextManager.cpp, LINE 817
2024-10-21 01:51:00 [6,286ms] [Error] [omni.physx.plugin] PhysX error: Failed to unload CUDA module data, returned 710., FILE /builds/omniverse/physics/physx/source/cudamanager/src/CudaContextManager.cpp, LINE 817
2024-10-21 01:51:00 [6,286ms] [Error] [omni.physx.plugin] PhysX error: Failed to unload CUDA module data, returned 710., FILE /builds/omniverse/physics/physx/source/cudamanager/src/CudaContextManager.cpp, LINE 817
2024-10-21 01:51:00 [6,286ms] [Error] [omni.physx.plugin] PhysX error: Failed to unload CUDA module data, returned 710., FILE /builds/omniverse/physics/physx/source/cudamanager/src/CudaContextManager.cpp, LINE 817
2024-10-21 01:51:00 [6,286ms] [Error] [omni.physx.plugin] PhysX error: Failed to unload CUDA module data, returned 710., FILE /builds/omniverse/physics/physx/source/cudamanager/src/CudaContextManager.cpp, LINE 817
2024-10-21 01:51:00 [6,286ms] [Error] [omni.physx.plugin] PhysX error: Failed to unload CUDA module data, returned 710., FILE /builds/omniverse/physics/physx/source/cudamanager/src/CudaContextManager.cpp, LINE 817
2024-10-21 01:51:00 [6,286ms] [Error] [omni.physx.plugin] PhysX error: Failed to unload CUDA module data, returned 710., FILE /builds/omniverse/physics/physx/source/cudamanager/src/CudaContextManager.cpp, LINE 817
2024-10-21 01:51:00 [6,286ms] [Error] [omni.physx.plugin] PhysX error: Failed to unload CUDA module data, returned 710., FILE /builds/omniverse/physics/physx/source/cudamanager/src/CudaContextManager.cpp, LINE 817
2024-10-21 01:51:00 [6,287ms] [Error] [omni.physx.plugin] PhysX error: Failed to unload CUDA module data, returned 710., FILE /builds/omniverse/physics/physx/source/cudamanager/src/CudaContextManager.cpp, LINE 817
2024-10-21 01:51:00 [6,287ms] [Error] [omni.physx.plugin] PhysX error: Failed to unload CUDA module data, returned 710., FILE /builds/omniverse/physics/physx/source/cudamanager/src/CudaContextManager.cpp, LINE 817
2024-10-21 01:51:00 [6,287ms] [Error] [omni.physx.plugin] PhysX error: Failed to unload CUDA module data, returned 710., FILE /builds/omniverse/physics/physx/source/cudamanager/src/CudaContextManager.cpp, LINE 817
2024-10-21 01:51:00 [6,287ms] [Error] [omni.physx.plugin] PhysX error: Failed to unload CUDA module data, returned 710., FILE /builds/omniverse/physics/physx/source/cudamanager/src/CudaContextManager.cpp, LINE 817
2024-10-21 01:51:00 [6,287ms] [Error] [omni.physx.plugin] PhysX error: Failed to unload CUDA module data, returned 710., FILE /builds/omniverse/physics/physx/source/cudamanager/src/CudaContextManager.cpp, LINE 817
2024-10-21 01:51:00 [6,287ms] [Error] [omni.physx.plugin] PhysX error: Failed to unload CUDA module data, returned 710., FILE /builds/omniverse/physics/physx/source/cudamanager/src/CudaContextManager.cpp, LINE 817
2024-10-21 01:51:00 [6,287ms] [Error] [omni.physx.plugin] PhysX error: Failed to unload CUDA module data, returned 710., FILE /builds/omniverse/physics/physx/source/cudamanager/src/CudaContextManager.cpp, LINE 817
2024-10-21 01:51:00 [6,288ms] [Error] [omni.physx.plugin] PhysX error: Failed to unload CUDA module data, returned 710., FILE /builds/omniverse/physics/physx/source/cudamanager/src/CudaContextManager.cpp, LINE 817
2024-10-21 01:51:00 [6,288ms] [Error] [omni.physx.plugin] PhysX error: Failed to unload CUDA module data, returned 710., FILE /builds/omniverse/physics/physx/source/cudamanager/src/CudaContextManager.cpp, LINE 817
2024-10-21 01:51:00 [6,288ms] [Error] [omni.physx.plugin] PhysX error: Failed to unload CUDA module data, returned 710., FILE /builds/omniverse/physics/physx/source/cudamanager/src/CudaContextManager.cpp, LINE 817
2024-10-21 01:51:00 [6,288ms] [Error] [omni.physx.plugin] PhysX error: Failed to unload CUDA module data, returned 710., FILE /builds/omniverse/physics/physx/source/cudamanager/src/CudaContextManager.cpp, LINE 817
2024-10-21 01:51:00 [6,288ms] [Error] [omni.physx.plugin] PhysX error: Failed to unload CUDA module data, returned 710., FILE /builds/omniverse/physics/physx/source/cudamanager/src/CudaContextManager.cpp, LINE 817
2024-10-21 01:51:00 [6,289ms] [Error] [omni.physx.plugin] PhysX error: Failed to unload CUDA module data, returned 710., FILE /builds/omniverse/physics/physx/source/cudamanager/src/CudaContextManager.cpp, LINE 817
2024-10-21 01:51:00 [6,289ms] [Error] [omni.physx.plugin] PhysX error: Failed to unload CUDA module data, returned 710., FILE /builds/omniverse/physics/physx/source/cudamanager/src/CudaContextManager.cpp, LINE 817