GPU Performance Comparison for Large Language Models

TOPS (Tera Operations Per Second) and FLOPS (Floating Point Operations Per Second) are two important metrics for characterizing GPU performance, but they relate to different types of computational operations, especially when running and training LLMs (Large Language Models).

TOPS (Tera Operations Per Second)

• TOPS generally measures the performance of integer operations (INT8, INT16, INT32, etc.).
• It is typically used for AI accelerators (e.g., Tensor Cores, NPUs, TPUs) because LLM inference (output generation, prediction) often employs fixed-point operations, which are more efficient than floating-point calculations.
• For inference, INT8 or INT4 operations are used because they reduce computational and memory requirements without significantly degrading model performance. Therefore, the advertised performance of AI accelerators is often specified in TOPS.
• Example: A GPU might have a performance of 200 TOPS for INT8 operations, meaning it can perform 200 trillion integer operations per second.

FLOPS (Floating Point Operations Per Second)

• FLOPS measures the execution speed of floating-point operations (FP16, FP32, FP64).
• It is crucial for LLM training because large models require FP16 or FP32 precision for accurate weight and gradient calculations.
• Example: A modern GPU might have 20 TFLOPS (TeraFLOPS) FP32 performance, meaning it can perform 20 trillion floating-point operations per second.
• For very large models (e.g., GPT-4 or Gemini), FP16 (half-precision floating-point numbers) and bfloat16 (BF16) operations are also used because they are faster while still being sufficiently accurate for training.