Standard scalar rewards often trigger reward hacking, where a model maximizes a score by ignoring safety constraints. To prevent this, engineers can shift from simple reward functions to Constrained Markov Decision Processes (CMDPs). By using Lagrangian multipliers, the system penalizes specific states. If the agent approaches a critical threshold—such as overloading a transformer in a power grid—the penalty overrides any potential gain.
Adding a KL-divergence penalty keeps the agent's policy close to a safe baseline. This stops the model from drifting into erratic, high-risk mathematical territories. Another option is reward shaping with a "safety budget." Once the agent exhausts its budget on risky actions, the reward drops to zero regardless of the outcome.
Some developers employ adversarial training. A "red team" agent searches for destructive paths, and the primary model learns to avoid them. This turns a mathematical blind spot into a hard constraint. These modifications force the AI to optimize within a rigid safety envelope rather than chasing a theoretical maximum at any cost.
