Bhuvanashree Murugadoss

Evaluating the Evaluator: Measuring LLMs’ Adherence to Task Evaluation Instructions

LLMs-as-a-judge is a recently popularized method which replaces human judgements in task evaluation with automatic evaluation using LLMs. Due to widespread use of RLHF (Reinforcement Learning from Human Feedback), state-of-the-art LLMs like GPT4 and Llama3 are expected to have strong alignment with…