Character.AI’s Kaiju: Scaling Conversational Models with Efficiency and Safety

Character.AI is making strides in the field of conversational AI with its Kaiju models, which are designed to handle millions of interactions daily while prioritizing safety and engagement. According to the Character.AI Blog, the Kaiju models are part of a family of in-house large language models (LLMs) that leverage advanced architectural efficiencies.

Architectural Innovations

Kaiju models are built with a dense transformer architecture and incorporate several efficiency optimizations. Notably, these models utilize int8 quantization to enhance processing speed and efficiency. The models are available in three sizes—Small (13 billion parameters), Medium (34 billion), and Large (110 billion)—and are designed to maintain a balance between performance and resource utilization.

Multiquery and Sliding Window Attention

One of the defining features of Kaiju models is the use of Multiquery Attention (MQA), which reduces the per-token key-value cache size, thus improving inference efficiency. While MQA can negatively impact some artificial general intelligence (AGI) benchmarks, its efficiency gains outweigh the drawbacks for Character.AI’s specific use cases.

The models also employ sliding window attention to decrease the computational load, especially in scenarios involving long-context processing. This approach ensures that the models remain efficient without sacrificing quality in long-context retrieval tasks.

Quantization Aware Training

Kaiju models are trained using Quantization Aware Training (QAT), which helps maintain high accuracy levels while speeding up the training process significantly. This method allows the models to achieve bf16-level accuracy while training up to 30% faster.

Safety and Alignment

Safety is a critical component of the Kaiju models. Before deployment, each model undergoes a rigorous multi-phase safety and alignment process, which includes supervised fine-tuning and reinforcement learning based on user feedback. Additionally, the models feature an optional classifier head that evaluates the safety of inputs, enhancing the robustness of the conversational AI.

Future Directions

As Character.AI continues to innovate, the focus remains on enhancing the deployment efficiency, engagement, and safety of its models. The team is committed to advancing open-source large language models (LLMs) and is actively seeking engineers and researchers to join their efforts in creating more dynamic and human-centered AI systems.

Image source: Shutterstock