Robotic Arm Manipulation with Behavior Cloning

Franka Arm Manipulation using Humans Demos in Kitchen Environment by me

This project enhances robotic arm manipulation by integrating human demonstrations using modified soft-actor-critic method, enabling robots to perform complex tasks like opening cabinets more effectively.

Soft Actor-Critic: The Big Picture

SAC is a reinforcement learning algorithm that trains an agent to act optimally in continuous action spaces, such as controlling a robot arm or navigating a drone. In the code:

• The environment is FrankaKitchen-v1, where the agent completes tasks like opening a cabinet.
• The agent optimizes its policy using the Soft Actor-Critic (SAC) algorithm.
• The algorithm prioritizes reward maximization while encouraging exploration via entropy.

---

How SAC Works in This Code

SAC involves three key networks:

1. Actor (Policy): Learns which actions to take in a given state to maximize reward.
2. Critics (Q-value estimators): Evaluate how good a given action is in a particular state.
3. Target Critic: Provides stable Q-value targets for training the critics.

The overall flow can be broken into three phases.

---

Phase 1: Initialization

1. Set Up Environment:

   • The environment is created (gym.make), and a wrapper processes observations for compatibility.

2. Agent Initialization:

   • Actor:
     • Learns a policy represented as a probability distribution.
     • Outputs:
       • Mean and log standard deviation of action distributions.
       • Ensures exploration via stochastic sampling.
   • Critics:
     • Two independent networks (Q1 and Q2) estimate action values for stability (avoids overestimation bias).
   • Target Critic:
     • Initially copies the weights of the Critic and updates slowly to ensure stable targets.

3. Replay Buffer:

   • Stores past experiences (state, action, reward, next_state, done).
   • Enables efficient learning by reusing past experiences.

4. Loading Expert Data:

   • In Phase 1, the agent leverages human demonstration data (human_memory.npz) to jumpstart training.

---

Phase 2: Training Loop

The core training happens in three stages with decreasing reliance on expert data:

Step 1: Interaction with the Environment

• The agent uses the Actor to:
  • Sample an action based on the current policy.
  • Observe the resulting next state, reward, and whether the episode ends.
• The transition (state, action, reward, next_state) is stored in the Replay Buffer.

Step 2: Sampling from the Replay Buffer

• The agent randomly samples a batch of transitions to train itself, ensuring diverse learning.

Step 3: Critic Updates

• The Critics learn to predict Q-values, which represent the expected reward for a state-action pair.
• Target Q-value computation:
  • Uses the Target Critic to estimate future rewards for next_state.
  • Incorporates the current reward and a discount factor (gamma) to compute the target: $Q_{\text{target}} = r + \gamma \cdot (1 - \text{done}) \cdot \min(Q_1', Q_2') - \alpha \cdot \text{log\_prob}$
  • The entropy term ($\alpha \cdot \text{log\_prob}$) encourages exploration by penalizing deterministic policies.
• Critic Loss:
  • Compares the predicted Q-values ($Q_1, Q_2$) to the computed target Q-value using Mean Squared Error.

Step 4: Actor Updates

• The Actor improves its policy to maximize the Q-values predicted by the critics.
• Actor Loss:
  • Encourages actions that:
    • Maximize Q-values ($\min(Q_1, Q_2)$).
    • Maintain high entropy (exploration).

Step 5: Target Critic Updates

• The Target Critic's weights are soft-updated: $$\theta_{\text{target}} \gets \tau \cdot \theta + (1 - \tau) \cdot \theta_{\text{target}}$$
• Ensures smoother, more stable training.

Step 6: Logging and Checkpointing

• TensorBoard logs:
  • Critic loss, Actor loss, and rewards.
• Saves checkpoints to allow resuming training later.

---

Phase 3: Gradual Transition to Full Autonomy

The agent is trained in three phases:

1. Phase 1: High reliance on expert data:
   • Expert data ratio = 50%.
   • Balances learning from the replay buffer and human-provided data.
2. Phase 2: Reduced expert reliance:
   • Expert data ratio = 25%.
   • Encourages the agent to learn more from its own exploration.
3. Phase 3: Full autonomy:
   • Expert data ratio = 0%.
   • The agent learns purely from its own experience.

---

Why Each Component Matters

1. Actor (Policy):

   • Learns how to act optimally by maximizing rewards while maintaining exploration.

2. Critics (Q-Values):

   • Evaluate the quality of actions taken by the policy.
   • Two critics reduce overestimation bias.

3. Replay Buffer:

   • Ensures sample efficiency by reusing past experiences.
   • Decorrelation: Helps prevent learning from sequentially correlated data.

4. Entropy Regularization:

   • Encourages exploration, preventing premature convergence to suboptimal strategies.

5. Target Networks:

   • Provide stable targets for critic training, avoiding instability caused by rapidly changing Q-values.

6. Expert Data:

   • Jumpstarts training by introducing good behaviors early on, especially useful in complex tasks like robotics.

---

Summary of Training Flow

1. Initialize environment, agent, and replay buffer.
2. Phase 1 (Exploration with Expert Data):
   • Train using a mix of expert and self-collected data.
3. Phase 2 (Reduced Expert Reliance):
   • Gradually shift focus to agent-collected experiences.
4. Phase 3 (Full Autonomy):
   • Train entirely on self-collected experiences.
5. For Each Episode:
   • Interact with the environment.
   • Store experiences in the replay buffer.
   • Periodically sample experiences to:
     • Update Critics using target Q-values.
     • Update Actor using learned Q-values and entropy regularization.
6. Log metrics and save model checkpoints.

---

Environment setup:

• MacOS Sequoia 15.1.1
• Python 3.11.9
• required installation is mentioned in requirements.txt