SIDM-Structural Information Decision-Making Framework for Reinforcement Learning

gpt generated notes of learning the hsd3 repository

2023-05-09T15:58:56+00:00

this is my gpt generated notes of learning the hsd3 repository.

Prelude: learn by homework (not GPT genrated)

I create a hsd3_learn dir, that contains the commented version of the original code. I make a try to let gpt4 turn the code into a programming homework (the one in Stanford CSw224 GNN) usng the promtps like

act as a Stanford CS TA for RL course who is humourous and good at 引导和启发 students. Please design a programming home work in a .py file : provide information and background like the purpose of function, variable meaning, library function usage and syntax etc. in the format comment, ask student to complete the function according to the prompt. Now i will give you functions or class from real codebase one by one, turn them into the homework.py. don’t skip, ensuring every line is the same with the original codes.

you need to make students understand the whole database by home work. in what order should you transform the code into homework, for better understanding Considering the complexity of the codebase and the interconnectedness of the various components, a good strategy would be to start with the fundamental building blocks and gradually move towards the more complex functionalities. Here’s a recommended order to transform the code into homework assignments

add comment, provide information and background like the purpose of function, variable meaning, library function usage and syntax etc. in the format comment, don’t skip, ensuring every line is the same with the original code

after finishing this helpful and interesting tutorial, one should be able to run the hsd3 and have a good understanding of the codebase.

to my surprise now GPT4 is capable of reading 300+line of code

Outline

prompts: you need to make students understand the whole database by home work. in what order should you transform the code into homework, for better understanding Considering the complexity of the codebase and the interconnectedness of the various components, a good strategy would be to start with the fundamental building blocks and gradually move towards the more complex functionalities. Here’s a recommended order to transform the code into homework assignments:

Exploring the codebase: Before diving into specific files, you could assign a task for students to explore the codebase. Ask them to document the purpose of each directory and file. This will give them a broad understanding of the codebase structure.
Requirements and Setup: Students should understand the software dependencies of the project. Assign them to study the requirements.txt file, install the necessary packages, and set up the project for development.
Configuration Files: Start with the config directory. Understanding these files will provide students with insights into how different parts of the system are set up, such as the agent and environment configurations.
Utilities: The hucc/utils.py file seems to contain utility functions that are likely used throughout the project. Understanding these will provide a good foundation.
Models: Move on to the hucc/models directory. Understanding the underlying models (e.g., bandits, blocks, traced) is key to comprehending the training process.
Agents: Next, tackle the hucc/agents directory. Each file appears to define a different type of agent, and understanding how these agents are defined and operate is crucial.
Environments: Understanding the environment in which the agents operate is also crucial. Students should therefore study the hucc/envs directory next.
Replay Buffer and Hash Count: The hucc/replaybuffer.py and hucc/hashcount.py files seem to implement important aspects of the agent’s memory and state tracking. These should be covered next.
Rendering and Spaces: The hucc/render.py and hucc/spaces.py files probably deal with the visualization and the state/action spaces of the environments. Understanding these will help students visualize the agent’s actions and the environment.
Training and Pretraining: Finally, students should study the train.py and pretrain.py scripts, as well as the corresponding scripts in the learn_hsd3 directory. These scripts will tie together everything the students have learned so far and show them how all these components interact during the training process.

Remember, each of these steps could itself be broken down into several smaller tasks to make the process more manageable. Also, encourage students to constantly test their understanding by running the code and experimenting with it. This hands-on experience will greatly facilitate their learning process.

Exploring the codebase

prompts: **Exploring the codebase according the given tree . outpuut **: Before diving into specific files, explore the codebase document the purpose of each directory and file. give me a broad understanding of the codebase structure. (explain uncommon names like hucc) Sure, based on the structure you provided, here’s an overview of the directories and files:

CODE_OF_CONDUCT.md, CONTRIBUTING.md, LICENSE, and README.md are standard files found in many repositories that provide guidance on how to interact with the project.
config: This directory contains configuration files (in YAML format), which specify parameters for different parts of the system. The agent subdirectory contains configurations for different types of agents. common.yaml might contain settings common to all configurations.
hydra: Hydra is a framework for elegantly configuring complex applications. This directory might contain the setup for hydra and configurations related to it. The launcher subdirectory probably contains configurations for launching jobs, for example, on a SLURM cluster.
format, nohup.out: These files could contain output logs or formatting instructions, though without more context it’s hard to say.
hucc: This directory seems to be the core of the project, containing several modules and files related to different aspects of the codebase. Based on the information and the code snippets you’ve provided, “HUCC” could potentially stand for “Hierarchical Unsupervised Control/Controller” or “Hierarchical Unsupervised Continuous Control” given the context of the research paper “Hierarchical Skills for Efficient Exploration”. The paper proposes a hierarchical skill learning framework for reinforcement learning in continuous control tasks, which could explain the usage of “HUCC”.
- agents: This module probably contains implementations of different types of learning agents.
- envs: This module likely contains different environments in which the agents can operate.
- hashcount.py: This might implement a data structure for counting occurrences of different items.
- models: This module may contain different models used by the agents.
- render.py: This file might be responsible for rendering or visualizing the agent’s behavior.
- replaybuffer.py: This file likely implements a replay buffer, which is a common component in reinforcement learning for storing past experiences.
- spaces.py: This file might define different types of spaces (like action or observation spaces).
- utils.py: This file likely contains utility functions used across the codebase.
learn_hsd3: This directory might contain scripts and instructions for learning a specific model or task (hsd3).
pretrain.py, train.py: These scripts are likely the entry points for pre-training and training the agents.
requirements.txt: This file usually lists the Python dependencies required by the project.
scripts: This directory might contain auxiliary scripts for tasks like data preprocessing or analysis.

Please note that these are educated guesses based on the names and typical structure of machine learning projects. For precise information, you should refer to the project’s documentation or the comments and docstrings in the code.

Utilities:

The hucc/utils.py file seems to contain utility functions that are likely used throughout the project. Understanding these will provide a good foundation.

Optimizers in PyTorch: PyTorch provides several optimization algorithms that you can use to train your models. These are implemented in the torch.optim module. To create an optimizer, you need to pass the parameters of the model that the optimizer should update, and the learning rate.
Signal handling in Python: Python’s signal module provides mechanisms to handle various types of system signals, including the SIGUSR1 signal. You can use the signal function from this module to register a handler function that will be called when a specific signal is received.
Tensor indexing in PyTorch: PyTorch provides several ways to index tensors. The simplest way is to use square brackets and indices. However, if you want to perform more complex indexing, you can use the index_select or gather methods. You will need to use one of these methods in the dim_select function.
Sorting strings naturally: Python’s built-in sorted function sorts strings lexicographically, which might not be what you want if your strings contain numbers. The sorted_nicely and sorted_nicely_sep functions should sort strings in a way that humans would expect, e.g., “2” should come before “10”.
Recursive functions: A recursive function is a function that calls itself during its execution. This allows the function to be written in a more readable and elegant manner, at the expense of potentially higher memory usage. You will need to write a recursive function to handle the potentially nested optimizers in the make_optim function.

Models:

Move on to the hucc/models directory. Understanding the underlying models (e.g., bandits, blocks, traced) is key to comprehending the training process. The bandits, blocks, and traced files are all part of the codebase for a reinforcement learning system. They are grouped under the “models” directory because they collectively provide the components necessary for defining various aspects of the models used in reinforcement learning.

Here’s an overview of what each of these components does:

Bandits 老虎机: In the context of reinforcement learning, a bandit is a type of problem where an agent must choose between a set of actions, each with an uncertain reward. The goal of the agent is to find the action with the highest expected reward. The term comes from the idea of a gambler at a row of slot machines (sometimes known as “one-armed bandits”) who has to decide which machines to play, how many times to play each machine, and in which order to play them, to maximize their return. The bandits.py file likely contains code for various bandit algorithms.
Blocks 预定义的GNN模块: Blocks usually refer to pre-defined building blocks that can be used to construct neural networks. In this context, each “block” is a specific arrangement of layers in a neural network. Blocks can be combined and stacked to create complex network architectures. The blocks.py file probably contains code defining these building blocks.
Traced 模型跟踪: The traced.py file likely contains code related to the tracing functionality provided by PyTorch’s JIT (Just-In-Time) compiler. Tracing allows you to convert your PyTorch model into a form that can be more efficiently executed in C++, while also allowing the model to be saved and loaded in different environments. The TracedModule class, which you provided in the code snippet, is an example of this. It wraps a given PyTorch module and provides tracing functionality to it.

Why are they under the models directory?

In machine learning projects, it’s common to organize code into directories based on their role in the system. The models directory typically contains all code related to model definition, training, and inference. Since bandit algorithms, network building blocks, and model tracing are all directly related to models, it makes sense to place them in the models directory. This organization helps other developers and researchers understand and navigate the codebase.

bandit

prompts: again, turn it into homework

Homework: Multi-Armed Bandit Algorithms in Reinforcement Learning

The goal of this assignment is to understand the concept of multi-armed bandit problems and various strategies for solving them using reinforcement learning techniques.

The provided code defines a Bandit class and several subclasses, each of which implements a different strategy for solving the multi-armed bandit problem:

UniformBandit
EpsGreedyBandit
UCBBandit
DiscountedUCBBandit
SlidingWindowUCBBandit
Agent57Bandit
ThompsonBandit

Your task is to complete the following:

Task 1: Understanding Bandit Strategies

For each of the seven bandit strategies, provide a brief description of how the strategy works, and under what circumstances it might be a good choice. Use resources like textbooks, lecture notes, or online resources to aid your understanding.

Task 2: Implement New Bandit Strategy

Create a new class, YourBandit, that inherits from the Bandit class and implements a different strategy for the multi-armed bandit problem. Explain why you chose this strategy and in which circumstances it could be superior to the existing ones.

python

class YourBandit(Bandit):
    def __init__(self, action_space: gym.Space):
        super().__init__(action_space)
        # TODO: Add any initialization code here

    def update(self, action, r):
        # TODO: Update estimates of action values
        pass

    def sample_action(self):
        # TODO: Decide on an action to take
        pass

    def best_action(self):
        # TODO: Return the current best action
        pass

    def dist(self):
        # TODO: Return the current distribution over actions
        pass

Task 3: Test Your Bandit Strategy

Test your bandit strategy on a multi-armed bandit problem of your choice. Compare its performance to the performance of the other bandit strategies. You can use the gym environment for creating the multi-armed bandit problem.

python

# TODO: Code for testing bandit strategies

Task 4: Discussion

Discuss the results of your experiments. Did your bandit strategy perform as expected? Why or why not? How did the different bandit strategies compare?

Submit your completed assignment as a Python script or Jupyter notebook.

Hints:

You may find the PyTorch documentation helpful, particularly the sections on distributions (torch.distributions) and operations (torch.nn.functional).
The gym library is used for the action spaces. Be sure to understand how it works before proceeding with the assignment.
For Task 3, remember to average results over multiple runs to get reliable comparisons. The bandit problem has inherent randomness, so a single run may not be representative.

blocks

The provided code is a skeleton of some classes and functions that are commonly used in the domain of Reinforcement Learning (RL), particularly in policy gradient methods and model-based RL. These classes are typically used for different types of network architectures and the implementation of different types of policy and value functions.

Please note that without the full implementation details, I can only provide a high-level summary of the potential purpose of each class and function.

TransformedDistribution: This class could be a subclass of D.TransformedDistribution, a PyTorch class that creates a distribution by applying a sequence of transformations to a base distribution.
TransformDistribution: Likely a custom class to define a distribution transformed according to some specific rule. It’s a subclass of nn.Module, suggesting that it might be a part of a neural network model.
GaussianFromEmbedding: This class might be used to generate a Gaussian distribution from an embedding vector, often used in policy networks where the output is a continuous action.
GaussianFromMulogstd: This class is probably used to generate a Gaussian distribution from a mean and log standard deviation.
CategoricalFromEmbedding: This class might be used to create a categorical distribution from an embedding vector, often used in policy networks where the output is a discrete action.
MHGaussianFromEmbedding: This class could be used for generating a multivariate Gaussian distribution from an embedding. The “MH” prefix might refer to “Metropolis-Hastings”, a famous algorithm in sampling.
EmbedDiscreteSkill: This could be a class to create an embedding for a discrete skill, often used in hierarchical RL where an agent learns to perform a variety of “skills”.
SplitTaskInput: A class that might be used to split task input into several parts.
EinsumBilinear: This class might be an extension of nn.Bilinear layer that utilizes Einstein summation convention for a specific operation.
SkipNetwork, SkipDoubleNetwork, BilinearSkipNetwork: These classes likely represent different types of neural network architectures with skip connections, which provide shortcuts between layers to help gradient flow during backpropagation and mitigate the vanishing gradient problem.
GroupedLinear: This class might represent a linear layer where inputs are grouped and each group is processed separately.
Expandtasksubgoal: Likely a class to process task and subgoal information in an RL context, possibly for a hierarchical RL setup.
FlattenSpace: This class might take a complex state or action space and flatten it into a simpler space, which can be useful for preprocessing complex environments.
Parallel: This class might be used to apply the same operation to multiple inputs in parallel.
SqueezeLastDim: This class might remove the last dimension of the input, a common operation in tensor manipulation.
shorthand: This function is likely a helper function to create a shorthand notation for something. Without more context, it’s hard to say exactly what it does.
FactoryType and Factory: These are related to the Factory design pattern, which is used to create objects without specifying the exact class of object that will be created. This can make code more flexible, modular, and scalable.
make_model: This function probably creates an instance of a model. Without more context, it’s hard to say exactly what kind of model it creates.

Remember that these are general descriptions based on typical use cases in reinforcement learning and the names of classes and functions. The actual implementation in your code may differ. It’s always best to refer to the specific documentation or comments provided with the code.

traced

The TracedModule class in the provided code is a subclass of PyTorch’s nn.Module. It’s used to convert a PyTorch model into TorchScript via tracing, which is a way to create a serializable and optimizable version of your model.

The main advantage of TorchScript is that it allows you to serialize your models, so you can save them to disk and load them in another environment (e.g., in a server doing inference). It also allows you to optimize your models and run them in a high-performance, strongly typed language.

Here is what each part of the TracedModule class does:

__init__: This is the constructor method that gets called when you create an instance of TracedModule. It takes as input an nn.Module object m, initializes the parent nn.Module with super().__init__() and assigns the input module to the instance variable self.m. It also sets self._traced to None. This variable will later hold the traced version of the model.
forward: This method is called when you pass input data through the module. If self._traced is None (meaning the model has not yet been traced), it uses the trace function from torch.jit to trace the model self.m with the input data inp and assigns the traced model to self._traced. Then it applies the traced model to the input data and returns the result.

The forward method ensures that the model is only traced the first time forward is called, and the traced model is reused for subsequent calls. This allows you to use TracedModule just like any other PyTorch module, but under the hood it automatically converts the model to TorchScript the first time it’s run.

Keep in mind that tracing has limitations: it can’t handle models with control flow (like if statements or loops) that depend on input data or model parameters, because it only captures the operations performed with the example input data it’s given. If your model has such control flow, you should use scripting instead of tracing.

Also note that the forward method traces the model every time it’s run if self._traced is None. In a multi-threaded environment, this could potentially result in the model being traced multiple times simultaneously. You might want to add thread-safety measures if you’re using this in a multi-threaded context.

Agents:

Next, tackle the hucc/agents directory. Each file appears to define a different type of agent, and understanding how these agents are defined and operate is crucial.

hsd3

prompts( it changes the code): add comment, provide information and background like the purpose of function, variable meaning, library function usage and syntax etc. in the format comment, don’t skip, ensuring every line is the same with the original codes.

This completes the _update_hi method which is responsible for updating the high-level policy. The method performs updates on the Q-function, the policy, and the temperature parameter. These updates are done based on the policy gradient approach. The logs of the losses are then returned. The target network is also updated after each iteration to maintain stability in learning

HiToLoInterface

In this code, HiToLo refers to a High-Level to Low-Level interface. It represents an interface between high-level and low-level sub-agents, likely within a Hierarchical Reinforcement Learning (HRL) context.

HRL is a type of reinforcement learning architecture that allows for learning and decision-making at various levels of abstraction. It typically involves high-level policies (or sub-agents) making more abstract, strategic decisions, and low-level policies carrying out the specific actions to achieve those strategic goals.

In this case, the HiToLoInterface class seems to be handling the translation between the high-level and low-level sub-agents, including parsing information from the high-level agent and translating it into a form that the low-level agent can use, and vice versa.

Here is the explanation of the Python code snippet you’ve provided:

This script defines a class HiToLoInterface which is a part of a Hierarchical Reinforcement Learning (HRL) system. The class represents the interface between a high-level and a low-level policy.

The main role of this class is to manage the transition between high-level policies (which decide on abstract goals) and low-level policies (which execute specific actions to achieve these goals).

Key methods and elements of the HiToLoInterface class:

__init__(self, env, cfg: DictConfig): The constructor of the class that takes in the environment and a configuration dictionary as arguments. It prepares all the necessary variables, configurations, and spaces based on the input configuration. It also sets up goal spaces and task maps.
parse_lo_info(cfg): This static method loads low-level information from a provided configuration. It returns subsets and task maps.
action_mask_hi(self): This method generates a mask for high-level actions based on subsets of features in the goal space.
gs_obs(self, obs): This method returns the goal space observations from the overall observations.
translate(self, gs_obs, task, subgoal, delta_gs_obs=None): This method translates the goal space observations, task and subgoal into the low-level policy’s action space.
update_bp_subgoal(self, gs_obs, next_gs_obs, action_hi): This method updates the backprojected subgoal based on the current and next goal space observations, and the high-level action.
observation_lo(self, o_obs, action_hi): This method constructs a low-level observation from the overall observation and the high-level action.
dist_lo(self, gs_obs, task, subgoal): This method calculates the distance between the projected current state and the subgoal in the low-level policy’s action space.
reward_lo(self, gs_obs, next_gs_obs, task, subgoal): This method calculates a potential-based reward for the low-level policy.
log_goal_spaces(self): This method logs the information about the goal spaces.

This class heavily relies on PyTorch and the gym library for handling tensors and constructing spaces, respectively. It also uses some custom classes like CtrlgsPreTrainingEnv for certain operations.

Environments:

Understanding the environment in which the agents operate is also crucial. Students should therefore study the hucc/envs directory next.

add comment in a code review paradigm: provide information and background like the purpose of function, variable meaning, library function usage and syntax etc. in the format comment, make sure every line is the same with the original codes. when i say continue, directly generate the continuous upcomming content.

Replay Buffer and Hash Count: The hucc/replaybuffer.py and hucc/hashcount.py files seem to implement important aspects of the agent’s memory and state tracking. These should be covered next.
Rendering and Spaces: The hucc/render.py and hucc/spaces.py files probably deal with the visualization and the state/action spaces of the environments. Understanding these will help students visualize the agent’s actions and the environment.

configure

Hydra is a powerful open-source library for developing applications that have complex configurations. It simplifies the handling of configurations for Python applications, allowing you to manage and organize your configurations efficiently. Hydra uses a structured configuration approach which enables easier configuration management and validation.

common.yml

Here is a brief introduction of some Hydra syntax and functionalities shown in the configuration file:

hydra.run.dir: This parameter defines the directory in which the application will be run. The . signifies that the application will run in the current directory.
hydra.sweep.dir: This is used when running sweep jobs (multiple runs of the application with different configurations). The ${now:%Y}-style syntax is used to interpolate the current date and time into the directory path. The ${oc.env:USER} syntax retrieves the current user’s name from the environment variables.
slurm: This section is specific to running the application on a Slurm workload manager, typically found on high-performance computing clusters. Parameters like partition, cpus_per_task, gpus_per_task, etc., define the resources requested for the Slurm job.
seed, device, log_level: These parameters specify the random seed, the device to use for computation (CPU or GPU), and the logging level for the application, respectively.
visdom: Visdom is a visualization tool for PyTorch. Parameters under this key configure the Visdom server.
checkpoint_path, keep_all_checkpoints: These parameters define where to save the model checkpoints and whether to keep all checkpoints or just the most recent one.
env: This section configures the training and evaluation environments.
eval: This section contains parameters related to the evaluation of the model, such as the interval between evaluations and the metrics to compute.
video: This section contains parameters for recording videos of the training and evaluation sessions.

Hydra uses the YAML syntax for its configuration files. YAML is a human-readable data serialization standard that’s ideal for configuration files and data exchange between languages with different data structures. In the provided YAML file, keys and values are separated by a : and nested parameters are represented by indentation.

The ${...} syntax in Hydra is used for interpolation, where the value of one config parameter is used in another. For example, in ${env.args}, the value of env.args will be inserted in place of the entire ${env.args} string.

Finally, null in YAML signifies the absence of a value or that the value is not applicable in this context.

For a more comprehensive understanding, the official Hydra documentation is a great resource: https://hydra.cc/docs/intro/.

humanoid.yml

hsd3

Here’s a brief explanation of the parameters in the YAML configuration file:

defaults: This field includes references to other configuration files, essentially saying “include the settings defined in these files as a starting point”. The - common and - agent: hsd3 lines mean that the configuration defined in the common.yaml and hsd3.yaml files (under the agent directory) are loaded as defaults. If the same parameter is defined in this configuration file and in one of the default configuration files, the value in this configuration file will be used.
max_steps: This is the maximum number of training steps.
robot: Specifies the type of robot to be used in the environment.
goal_space: Specifies the goal space (or the representation of the goals) for the task.
subsets: This could be a parameter to specify which subsets of the data to use.
action_interval: The number of environment steps between two actions.
eps: This is typically a parameter related to the exploration strategy (epsilon-greedy, for instance), where a random action is selected with probability eps.
auto_adapt: Flag for automatic adaptation of the configuration based on the environment.
eval: This section configures the evaluation of the model, such as the interval between evaluations and the metrics to compute.
video: This parameter controls the video recording of the training and evaluation sessions. null means no video recording.
visdom.offline: This parameter decides whether to use the offline mode of visdom for visualization.
env: This section configures the training and evaluation environments.
agent: This section contains parameters specific to the agent, such as the batch size, replay buffer size, number of updates, goal space, etc.
model: This section defines the models to be used for the different components of the agent.
optim: This section specifies the optimizers to be used for training the different components of the agent, along with their learning rates.

In the provided YAML file, keys and values are separated by a : and nested parameters are represented by indentation. The ${...} syntax in Hydra is used for interpolation, where the value of one config parameter is used in another. For example, in ${robot}, the value of robot will be inserted in place of the entire ${robot} string.

pretrain

Sure, here’s a brief explanation of the additional parameters in this YAML configuration file:

horizon: This could refer to the planning horizon, which is the number of steps into the future the agent should consider when making decisions.
idle_steps: This might be the number of initial steps for which the agent takes no action.
features: This specifies the type of features to be used for the task.
feature_dims, feature_rank, feature_rank_max: These parameters could be related to feature selection or dimensionality reduction.
ctrl_eps: This could be a parameter for controlling the agent’s exploration strategy.
precision: This could be related to the precision of the goal representation or the precision with which the agent should achieve its goals.
max_new_tasks, task_weighting, downrank, lp_new_task, lp_eps: These parameters appear to be related to the task generation and selection process.
eval_mode: This could be a mode for the evaluation process. In this case, reachability could mean that the agent is evaluated based on its ability to reach different goals.
combine_after_steps: This might be the number of steps after which certain operations (like merging of tasks or updates) should be performed.
backprop_fallover: This flag could control whether to use a specific technique (like fallover) during backpropagation.
estimate_joint_spaces: This could be a flag for estimating the joint spaces of the robot. null means this option is not used.
init_model_from, checkpoint_path: These parameters could be used to initialize the agent’s model from a checkpoint.
distributed: This section configures the distributed training setup, like the number of actors and learners and the path for storing rendezvous files.
agent.flatten_obs: This parameter could decide whether to flatten the observation before feeding it into the agent’s model.
agent.per_task_alpha: This could be a flag for using a different alpha parameter (which usually controls the trade-off between exploration and exploitation) for each task.
model.reachability: This could specify the model to be used for estimating the reachability of different goals.

The ${...} syntax in Hydra is used for interpolation, where the value of one config parameter is used in another. For example, in ${robot}, the value of robot will be inserted in place of the entire ${robot} string.

Remember, these explanations are based on general machine learning concepts and some educated guesses. The actual meaning of the parameters could be different and could be understood by reading the documentation or the code of the specific project.

Train & pretrain

use GPT to add comment is the most common use.

here we explore usage of using GPT-4 to learn codes

copy in blank pretrain.py

generate comment and the hide code, ask user to copy and paste according to the prompt.

train.py

Hierarchical Skill Learning for Bipedal Robots: Concepts and Examples

2023-05-08T14:58:56+00:00

Concepts

Pretrained skills (预训练技能, yùxùnliàn jìnéng): In reinforcement learning, pretrained skills refer to abilities or behaviors that an agent has already learned before tackling a new task. These skills can be acquired through prior training on related tasks or through unsupervised learning. For example, a bipedal robot may have pretrained skills for walking, jumping, or balancing that can be used in various new tasks.
Balance between generality and specificity (通用性与特异性的平衡, tōngyòngxìng yǔ tèyìxìng de píng héng): This refers to the trade-off between designing skills that are broadly applicable across many tasks (generality) and skills that are tailored to a specific task (specificity). For instance, a general skill for a robot could be walking, while a specific skill might be walking on a tightrope. General skills can be useful for many tasks, but may require more fine-tuning for optimal performance. Specific skills can lead to faster learning, but are limited in their applicability.
Continuous control (连续控制, liánxù kòngzhì): Continuous control is a type of reinforcement learning problem where the agent’s actions are continuous rather than discrete. For example, a robot’s joint angles or motor torques are continuous variables. Continuous control is often more challenging than discrete control due to the infinite number of possible actions.
Locomotion (运动, yùndòng): In the context of robotics and reinforcement learning, locomotion refers to the ability of an agent, like a bipedal robot, to move from one place to another. Locomotion skills can include walking, running, jumping, or crawling.
Sparse-reward (稀疏奖励, xīshū jiǎnglì): Sparse-reward refers to reinforcement learning tasks where the agent receives feedback (rewards or penalties) infrequently. This can make it difficult for the agent to learn, as it has to explore the environment extensively to discover which actions lead to rewards. For example, a robot trying to find a hidden object in a room might only receive a reward once it successfully locates the object, without any intermediate feedback.

The paper presents a hierarchical skill learning framework that allows the agent to learn skills of varying complexity without prior knowledge of the task. This framework helps the agent effectively balance generality and specificity, leading to better performance in diverse, sparse-reward tasks for bipedal robots.

advanced concepts

Imagine a bipedal robot learning to navigate through a complex environment filled with obstacles. The robot needs to learn various skills, such as walking, turning, and climbing, to efficiently explore and complete tasks in the environment.

Skill policy (技能策略, jìnéng cèlüè): Individual skill policies control specific aspects of the robot’s behavior. For example, one skill policy might control the movement of the left leg, while another controls the right leg.
Shared policy (共享策略, gòngxiǎng cèlüè): The shared policy is used to represent multiple skills or behaviors with a single policy. In this example, the shared policy could represent both the left and right leg movements, as well as more complex behaviors like turning or climbing.
Hierarchy of skills (技能层次, jìnéng céngcì): Skills are organized into a hierarchy based on their complexity. In this case, simpler skills like moving individual legs form the lower levels of the hierarchy, while more complex skills like turning and climbing are at higher levels.
High-level control signal (高级控制信号, gāojí kòngzhì xìnhào): The high-level control signal is provided by a higher-level policy that guides the robot’s overall behavior. In our example, the high-level control signal might instruct the robot to move towards a specific target or avoid an obstacle, which then influences the low-level skill policies, such as leg movements and turning.
Robot configuration (机器人配置, jīqìrén pèizhì): The robot’s configuration refers to its current state, including positions, orientations, and joint angles. In this scenario, the robot’s configuration might include the positions of its legs, torso, and other body parts.

method

personal (GPT4’s) notes

The authors propose an approach that first acquires low-level policies capable of carrying out a useful set of skills in an unsupervised manner, meaning without reward signals from the main tasks of interest. These skills are then used in a hierarchical reinforcement learning setting.

The pre-training environment consists only of the robot, while in downstream tasks, additional objects may be present. These objects could be fixed (e.g., obstacles) or only indirectly controllable (such as a ball).

Low-Level Skills and Goal Spaces

The usefulness of a skill policy comes from providing additional structure for effective exploration. To provide benefits across a wide range of tasks, skills need to support both fast, directed exploration (e.g., locomotion) and precise movements (e.g., lift the left foot while bending down). The authors propose short-horizon, goal-directed low-level policies trained to achieve target configurations of robot-level state features, such as the position or orientation of its torso or relative limb positions.

This approach allows defining a hierarchy of skills by controlling single features and their combinations, resulting in varying amounts of control exposed to high-level policies. Each skill is trained to reach goals in a goal space defined over a set of features, yielding policies without task-specific rewards. These policies rely solely on state features related to the specific robot and the prescribed hierarchy of goal spaces.

A feature, in the context of this work, refers to a specific property or characteristic of the robot’s state. State features can include the position, orientation, or relative limb positions of the robot. These features help describe the current situation or configuration of the robot in the environment.

In this approach, low-level policies are trained to achieve target configurations of robot-level state features, such as reaching a certain position or orientation. The authors define a hierarchy of skills by controlling single features and their combinations, which results in different levels of control exposed to high-level policies.

A goal space is defined over a set of features. In this case, the goal space represents a collection of possible goals that can be achieved by manipulating these specific features. For instance, a goal space could be defined over the position and orientation features, meaning that the goals within this space would be related to reaching different positions and orientations.

These low-level policies (skills) do not rely on task-specific rewards; instead, they are trained based on the state features of the robot and the prescribed hierarchy of goal spaces. By focusing on state features and goal spaces, the approach enables the robot to learn a variety of skills that can be useful across a wide range of tasks. The high-level policy then selects and combines these low-level skills to accomplish the specific tasks it is given.

High-Level Policy and Task-Specific Features

On downstream tasks, high-level policies operate with a combinatorial action space. State spaces are enriched with task-specific features containing information regarding additional objects in the downstream task. This enriched state space is only available to the high-level policy. The high-level policy is modeled hierarchically with two policies that prescribe a goal space and a goal, respectively.

“Prescribe” here means that the high-level policy determines or selects the goal space and goal to be used by the low-level policies. It chooses the appropriate goal space and goal based on the current state and the task requirements.

Action space refers to the set of all possible actions that an agent (in this case, the robot) can take in a given environment. In the context of this work, “operate with a combinatorial action space” means that the high-level policy chooses a combination of low-level skills (goal space) and goals from the available options. This allows the high-level policy to create diverse and complex actions by combining the skills and goals it has access to.

The high-level policy can be seen as a decision-making mechanism that directs the low-level policies (skills) to achieve specific tasks. It is modeled hierarchically, with two components: one that selects a goal space (which skill to use) and another that selects a specific goal within that goal space. By doing so, the high-level policy can dynamically switch between different skills and goals as needed for the given task, while the low-level policies focus on executing the chosen skill to reach the desired goal. The high-level policy’s primary function is to adapt and combine the low-level skills in a way that allows the robot to effectively perform various tasks.

Temporal Abstraction and HSD-3

With a task- and state-dependent policy, it is possible to not only select the required set of skills for a given environment but also switch between them within an episode. In this three-level hierarchy, the higher-level policy explores the hierarchy of goal spaces and dynamically trades off between generality and specificity. Temporal abstraction is obtained by selecting new high-level actions at regular intervals but with a reduced frequency compared to low-level actions. The resulting learning algorithm is called HSD-3, emphasizing the three different levels of control obtained after the hierarchical skill discovery phase.

“Switch between them within an episode” refers to the ability of the high-level policy to change the selected skill and goal as needed during a single episode of the task being performed. An episode typically refers to a single run or attempt at solving a task, starting from an initial state and ending when the task is completed or a termination condition is met. By switching between different skills and goals during an episode, the high-level policy can adapt to changing requirements of the task and environment.

“Temporal abstraction” is a concept in hierarchical reinforcement learning where decisions can be made at different time scales. In the context of this work, it means that the high-level policy selects new actions (which skill to use and which goal to pursue) at a slower frequency compared to the low-level policies. This allows the high-level policy to make long-term decisions while the low-level policies focus on the shorter-term execution of those decisions. By having different levels of control operating at different time scales, the system can achieve more efficient learning and better performance in complex tasks.

formulization

In the original text, the following mathematical symbols are used:

1. $$\left\{\pi_F^{\text {lo }}: \mathcal{S}^{\mathrm{p}} \times \mathcal{G}^F \rightarrow A\right\}_{F \in \mathcal{F}}$$: This notation represents a set of low-level goal-directed skill policies ($\pi_F^{\text {lo }}$) for each feature set $F$ in the collection of feature sets $\mathcal{F}$. Each policy takes as input the proprioceptive observations ($\mathcal{S}^{\mathrm{p}}$) and a goal ($\mathcal{G}^F$) from the goal space associated with the specific feature set $F$. The policy outputs an action ($A$) to be executed by the robot. 2. $\pi^{\mathrm{f}}: S \rightarrow \mathcal{F}$: This notation defines the high-level policy that takes the current state ($S$) as input and selects a goal space from the collection of goal spaces ($\mathcal{F}$). The output is a specific feature set $F$ that the policy determines to be the most appropriate for the current state. 3. $\pi^{\mathrm{g}}: S \times \mathcal{F} \rightarrow \mathcal{G}^F$: This notation defines the second part of the high-level policy that takes the current state ($S$) and the selected feature set $F$ as input and outputs a specific goal ($g$) from the goal space associated with that feature set ($\mathcal{G}^F$). These mathematical symbols help to define the structure of the high-level and low-level policies and their relationships in a concise manner. They indicate the inputs and outputs of each policy and demonstrate how the policies interact in the hierarchical reinforcement learning framework.

在这部分中，作者回顾了与层次化强化学习相关的工作。下面是对这部分内容的梳理和分类：

经典规划系统中的宏观操作符和抽象
- 该部分讨论了经典规划系统中宏观操作符和抽象对层次化强化学习的影响。
层次化强化学习方法
- 介绍了层次化强化学习方法的发展历程，以及为现代神经网络基础学习系统带来的主要优势，如改进的探索能力。
低级策略（选项；技能）的获取
- 回顾了从随机游走、互信息目标、代理或专家轨迹数据集、运动捕捉数据以及专用预训练任务中发现低级原语的各种方法。
通用性和特异性的权衡
- 讨论了低级技能在实践中如何在通用性（广泛适用性）和特异性（特定任务的益处）之间产生权衡。
选项发现和层次化强化学习的先前工作
- 通过测试环境的选择，例如传统的网格迷宫导航任务和类似的模拟机器人导航问题，说明了先前工作在解决通用性和特异性权衡方面的偏向性。
非导航环境的技能获取
- 介绍了针对非导航环境的技能获取方法，如使用定制的预训练任务或领域内运动捕捉数据。
与现有工作的对比
- 与现有方法相比，本文关注于在不借助额外监督或预训练任务设计的情况下学习低级技能，不强制要求针对特定行为类型的固定权衡。

通过梳理和分类这些内容，您可以更好地理解文献回顾部分，并重点关注每个部分的核心内容。

Building GitHub Pages with RVM, Jekyll, and GitHub Actions Workflow

2023-05-07T09:34:56+00:00

In this post, we will go through the process of building a GitHub Pages site using RVM, Jekyll, and a GitHub Actions Workflow. We’ll cover key concepts, step-by-step instructions, and address the bugs encountered during the process.

Key Concepts

RVM (Ruby Version Manager): A command-line tool for installing, managing, and working with multiple Ruby environments. Jekyll: A static site generator built with Ruby, used for creating GitHub Pages sites.
GitHub Actions: A CI/CD platform integrated with GitHub, allowing you to automate workflows for building, testing, and deploying projects.

Step-by-Step Guide

Install RVM: Follow the installation instructions for RVM on the official RVM website.
Install Ruby: Use RVM to install the desired version of Ruby. For example, to install Ruby 2.7.4, run rvm install 2.7.4 in your terminal.
Set up Jekyll: Install Jekyll by running gem install jekyll bundler. Create a new Jekyll site: Run jekyll new my-site to create a new Jekyll site in the my-site directory. Replace my-site with your desired directory name.
Build and serve the site locally: Navigate to the site directory and run bundle exec jekyll serve. Your site should be available at http://localhost:4000.
Create a GitHub repository: Create a new GitHub repository and push your Jekyll site to the main branch.
Set up GitHub Pages: In your repository settings, enable GitHub Pages by selecting the main branch as the source.
Create a GitHub Actions Workflow (this is automatically down following the official tutorial): In your repository, create a .github/workflows/main.yml file and set up the workflow to build and deploy your site. Here’s a sample workflow: ```yaml on: push: branches: [ “main” ] pull_request: branches: [ “main” ]

jobs: build:

  runs-on: ubuntu-latest

  steps:
  - uses: actions/checkout@v3
  - name: Set up Ruby
    uses: ruby/setup-ruby@v1
    with:
      ruby-version: 2.7.4
  - name: Install dependencies
    run: bundle install
  - name: Build the site
    run: bundle exec jekyll build
  - name: Deploy to GitHub Pages
    uses: peaceiris/actions-gh-pages@v3
    with:
      github_token: $
      publish_dir: ./_site   ``` 8. Commit and push: Commit and push the changes to the main branch. Your site will be built and deployed using the GitHub Actions Workflow.

Bugs Encountered

CSS not loading: The CSS files were not loading correctly due to an incorrect baseurl. This was fixed by updating the head.html file to include /SIDM in the stylesheet link.
Missing navigation links: The navigation menu was not displaying the desired links. This was resolved by adding the site.baseurl in the header.html file, which fixed the navigation links. (remember add --- layout title ---)

SIDM-Structural Information Decision-Making Framework for Reinforcement Learning

gpt generated notes of learning the hsd3 repository

Prelude: learn by homework (not GPT genrated)

Outline

Exploring the codebase

Utilities:

Models:

bandit

Task 1: Understanding Bandit Strategies

Task 2: Implement New Bandit Strategy

Task 3: Test Your Bandit Strategy

Task 4: Discussion

blocks

traced

Agents:

hsd3

HiToLoInterface

Environments:

configure

common.yml

humanoid.yml

hsd3

pretrain

Train & pretrain

copy in blank pretrain.py

train.py

Hierarchical Skill Learning for Bipedal Robots: Concepts and Examples

Concepts

advanced concepts

method

personal (GPT4’s) notes

Low-Level Skills and Goal Spaces

High-Level Policy and Task-Specific Features

Temporal Abstraction and HSD-3

formulization

related work

Building GitHub Pages with RVM, Jekyll, and GitHub Actions Workflow

Key Concepts

Step-by-Step Guide

Bugs Encountered