CCNets Causal GPT-RL Training Algorithm

This release updates CCNets Causal GPT-RL Training Algorithm to SageMaker Algorithm version cgrl-training-algorithm-v0-0-10 and provides updated training and inference containers for GPT-style offline reinforcement learning.

This release supports fixed-shape Gymnasium observation and action schemas built from one-dimensional Box, Discrete, MultiDiscrete, and MultiBinary leaf spaces. Dict and Tuple containers may be nested and may combine continuous and categorical fields within the same observation or action structure.

Verified exported-bundle combinations include Dict observations that combine Box and Discrete fields, and Tuple or Dict action spaces that combine continuous Box branches with Discrete branches. The same schema system also supports structured Unity ML-Agents datasets such as Tuple observations with Discrete or MultiDiscrete action branches, including egocentric Dict wrappers used by multi-agent environments.

Unity ML-Agents integration follows an offline training workflow. Users collect trajectories from a Unity ML-Agents environment, package the trajectories as Minari-compatible datasets, upload the datasets to Amazon S3, and train the policy through SageMaker. The SageMaker training container does not launch the Unity Editor or execute the customer's Unity build during training.

The public repository includes Unity trajectory collection examples and an ONNX deployment example for running a trained policy against a Unity ML-Agents build. ONNX deployment is a downstream integration example and is separate from the primary SageMaker model.tar.gz output.

Public benchmark resources are available for MuJoCo policy bundles and a Unity ML-Agents Pyramids ONNX policy. Benchmark results are environment-, runtime-, and protocol-specific and should be used as reproducible references rather than guarantees for customer datasets or environments.

The default training configuration is designed to provide a practical starting point for common MuJoCo and Unity ML-Agents workflows. Additional adjustment may still be useful depending on dataset quality, reward scale, trajectory coverage, observation and action design, and environment-specific behavior.

This release adds real-time policy delivery through the SageMaker checkpoint S3 prefix. Complete inference bundles are exported and synchronized to S3 while training is still running, allowing buyers to load and evaluate intermediate policies in their own game engine, simulator, robotics environment, Unity build, or other target environment before the final model artifact is produced.

Intermediate policies are delivered in two checkpoint series. archive/ stores five automatically scheduled points distributed across max_steps and up to ten additional buyer-requested points supplied through archive_steps. Archive points are identified by training step and are preserved permanently. improvements/ stores a rotating ten-slot best-so-far track whenever offline_eval/action_nll reaches a new minimum.

The archive series is the recommended source for measuring policy-performance trends and making early-stopping decisions. offline_eval/action_nll measures how well the model predicts held-out dataset actions and does not guarantee task performance. A lack of new improvement bundles does not mean that environment performance has stopped improving.

Each delivered bundles/*/ directory is a complete inference bundle that can be loaded directly with the public causal-gpt-rl runtime. The paired .pt files contain training, optimizer, and scheduler state for resume or retraining and are not required for normal inference.

The final model artifact remains model.tar.gz and contains reports/summary.json and the canonical inference bundle under <run_namespace>/bundle/. Intermediate archive and improvement bundles are synchronized separately through the configured checkpoint S3 prefix and are not duplicated inside model.tar.gz.

The canonical bundle is selected using offline_eval/action_nll and should be treated as a default policy rather than a guarantee that it is the best policy in the buyer's target environment. Buyers may evaluate preserved archive bundles and deploy the point that performs best under their own evaluation protocol.

CloudWatch Logs report checkpoint and bundle export paths, training steps, export reasons, the checkpoint metric, and metric direction. CloudWatch also displays offline evaluation metrics such as action negative log-likelihood. These metrics are training progress indicators measured against held-out dataset actions and do not guarantee final performance in the buyer's environment.

Real-time policy delivery refers to policy-bundle availability during an active training job. It does not mean that the training container runs the buyer's environment or that every S3 object becomes visible immediately; delivery depends on the SageMaker checkpoint synchronization completing.

The trained policy can be used through a reset, act, and observe rollout loop. The workflow supports state-based behavior control, prompt-like control through buyer-provided instruction vectors, batched rollout handling with per-episode resets, and stable episode boundary handling for cached inference.