C3PO: Optimized Large Language Model Cascades with Probabilistic Cost Constraints for Reasoning
Large language models (LLMs) have achieved impressive results on complex reasoning tasks, but their high inference cost remains a major barrier to real-world deployment. A promising solution is to use cascaded inference, where small, cheap models handle easy queries, and only the hardest examples ar…