TDLambdaEstimator

class torchrl.objectives.value.TDLambdaEstimator(*args, **kwargs)

TD(\(\lambda\)) 的優勢函式估計。

引數:

• gamma (scalar) – exponential mean discount.

• lmbda (scalar) – trajectory discount.

• value_network (TensorDictModule) – 用於檢索值估計的值運算子。

• average_rewards (bool, 可選) – 如果為 True，則在計算 TD 之前會對獎勵進行標準化。

• differentiable (bool, optional) –

  if True, gradients are propagated through the computation of the value function. Default is False.

  注意

  The proper way to make the function call non-differentiable is to decorate it in a torch.no_grad() context manager/decorator or pass detached parameters for functional modules.

• vectorized (bool, optional) – 是否使用 lambda 迴歸的向量化版本。預設為 True。

• skip_existing (bool, optional) – 如果設定為 True，值網路將跳過輸出已存在於 tensordict 中的模組。預設為 None，即 tensordict.nn.skip_existing() 的值不受影響。

• advantage_key (str or tuple of str, optional) – [Deprecated] the key of the advantage entry. Defaults to "advantage".

• value_target_key (str or tuple of str, optional) – [已棄用] 優勢項的鍵。預設為 "value_target"。

• value_key (str or tuple of str, optional) – [已棄用] 從輸入 tensordict 讀取的值鍵。預設為 "state_value"。

• shifted (bool, optional) – 如果設定為 True，值和下一個值將透過對值網路的單次呼叫來估計。這更快，但僅在以下情況下有效：(1) "next" 值僅偏移一步（例如，對於多步值估計則不適用），並且 (2) 在時間 t 和 t+1 使用的引數相同（在使用目標引數時則不適用）。預設為 False。

• device (torch.device, optional) – 緩衝區將被例項化的裝置。預設為 torch.get_default_device()。

• time_dim (int, optional) – 輸入 tensordict 中對應於時間的維度。如果未提供，則預設為標記為 "time" 名稱的維度（如果存在），否則預設為最後一個維度。可以在呼叫 value_estimate() 時覆蓋。

• deactivate_vmap (bool, 可選) – 是否停用 vmap 呼叫並用普通 for 迴圈替換它們。預設為 False。

forward(tensordict=None, *, params: list[Tensor] | None = None, target_params: list[Tensor] | None = None)

根據 tensordict 中的資料計算 TD(\(\lambda\)) 優勢。

If a functional module is provided, a nested TensorDict containing the parameters (and if relevant the target parameters) can be passed to the module.

引數:

tensordict (TensorDictBase) – 一個 TensorDict，包含計算值估計和 TDLambdaEstimate 所需的資料（一個觀察鍵、"action"、("next", "reward")、("next", "done")、("next", "terminated") 和 "next" tensordict 狀態，如環境返回）。傳遞給此模組的資料應結構化為 [*B, T, *F]，其中 B 是批大小，T 是時間維度，F 是特徵維度。tensordict 的形狀必須為 [*B, T]。

關鍵字引數:

• params (TensorDictBase, optional) – A nested TensorDict containing the params to be passed to the functional value network module.

• target_params (TensorDictBase, optional) – A nested TensorDict containing the target params to be passed to the functional value network module.

返回:

An updated TensorDict with an advantage and a value_error keys as defined in the constructor.

示例

>>> from tensordict import TensorDict
>>> value_net = TensorDictModule(
...     nn.Linear(3, 1), in_keys=["obs"], out_keys=["state_value"]
... )
>>> module = TDLambdaEstimator(
...     gamma=0.98,
...     lmbda=0.94,
...     value_network=value_net,
... )
>>> obs, next_obs = torch.randn(2, 1, 10, 3)
>>> reward = torch.randn(1, 10, 1)
>>> done = torch.zeros(1, 10, 1, dtype=torch.bool)
>>> terminated = torch.zeros(1, 10, 1, dtype=torch.bool)
>>> tensordict = TensorDict({"obs": obs, "next": {"obs": next_obs, "done": done, "reward": reward, "terminated": terminated}}, [1, 10])
>>> _ = module(tensordict)
>>> assert "advantage" in tensordict.keys()

The module supports non-tensordict (i.e. unpacked tensordict) inputs too

示例

>>> value_net = TensorDictModule(
...     nn.Linear(3, 1), in_keys=["obs"], out_keys=["state_value"]
... )
>>> module = TDLambdaEstimator(
...     gamma=0.98,
...     lmbda=0.94,
...     value_network=value_net,
... )
>>> obs, next_obs = torch.randn(2, 1, 10, 3)
>>> reward = torch.randn(1, 10, 1)
>>> done = torch.zeros(1, 10, 1, dtype=torch.bool)
>>> terminated = torch.zeros(1, 10, 1, dtype=torch.bool)
>>> advantage, value_target = module(obs=obs, next_reward=reward, next_done=done, next_obs=next_obs, next_terminated=terminated)

value_estimate(tensordict, target_params: TensorDictBase | None = None, next_value: torch.Tensor | None = None, time_dim: int | None = None, **kwargs)

Gets a value estimate, usually used as a target value for the value network.

如果狀態值鍵存在於 tensordict.get(("next", self.tensor_keys.value)) 下，則將使用此值，而無需呼叫值網路。

引數:

• tensordict (TensorDictBase) – the tensordict containing the data to read.

• target_params (TensorDictBase, optional) – A nested TensorDict containing the target params to be passed to the functional value network module.

• next_value (torch.Tensor, optional) – 下一個狀態或狀態-動作對的值。與 target_params 互斥。

• **kwargs – the keyword arguments to be passed to the value network.

Returns: a tensor corresponding to the state value.