Yes, it is possible to develop metrics that assess truthfulness without a fixed dataset by shifting from absolute truth to consistency and logical coherence. Since LLMs are stochastic, the goal is often to measure how reliably a model adheres to factual logic rather than matching a specific string of text.
One primary approach is Self-Consistency. This method involves prompting the model multiple times to generate different reasoning paths for the same query. If the model converges on the same factual conclusion despite stochastic variance, the likelihood of truthfulness increases. This treats the model's internal consensus as a proxy for reliability.
Another method is Cross-Model Verification or Multi-Agent Debate. In this setup, multiple independent LLMs act as critics to evaluate a claim. If several diverse models independently validate a statement, the probability of accuracy is higher. Furthermore, RAG (Retrieval-Augmented Generation) systems allow for dynamic truthfulness checking by comparing outputs against real-time, retrieved external knowledge rather than a static training dataset. These methods move the evaluation from simple pattern matching to evaluating logical stability and external alignment.
