Evaluating Quality of Results from AI SaaS Apps Built on Fine Tuned LLMs

1. Domain-Specific Evaluation: Use case-specific LLMs are trained to excel in a particular domain or task, such as legal document analysis or medical diagnosis. Evaluating their quality requires domain-specific evaluation criteria, which might be different from generic language understanding benchmarks. It can be challenging to define and measure these domain-specific metrics accurately.

2. Subjectivity and Human Judgment: Some use cases involve subjective judgments, such as sentiment analysis or opinion mining. Evaluating the quality of results in these cases requires human judgment and subjective assessments. It can be challenging to establish a consistent evaluation framework and mitigate evaluator bias.

3. Limited Availability of High-Quality Evaluation Data: In some specialized domains, obtaining high-quality evaluation data can be challenging. Annotated or labeled data specific to the use case might be scarce or expensive to obtain. The limited availability of evaluation data can hinder the comprehensive assessment of the model's quality.

4. Lack of Ground Truth: Use case-specific LLMs often operate in scenarios where ground truth or definitive answers might not exist. For example, in legal case analysis, multiple legal interpretations or outcomes may be valid. Evaluating the quality of the model's results becomes challenging when there is no single correct answer to compare against.

5. Performance on Edge Cases: Use case-specific LLMs might perform well on typical or well-defined scenarios but struggle with edge cases or ambiguous inputs. Evaluating the model's behavior in such situations requires careful examination of its performance and understanding its limitations.

6. Integration Challenges: Evaluating the quality of results for a use case-specific LLM involves considering its integration into existing systems or workflows. Challenges can arise in integrating the model's outputs, handling errors or uncertainties, and assessing the impact on overall system performance.

The field of use-case specific AI applications is still in its early stages of evolution, and we continue to make progress in overcoming the challenges mentioned above. Fortunately, the open-source community is constantly expanding its knowledge base, providing valuable insights each day. 

3 main steps to evaluate results from LLMs

• Test data creation: involves designing input prompts to evaluate different aspects of the LLM's performance and to cover various use cases.

• Oracle generation refers to defining the expected correct responses for each input prompt. This can be done manually or by using alternative models for comparison (see frameworks below).

• Result evaluation entails measuring the LLM's output against the oracles and identifying potential errors or biases.

And, 3 strategies for Testing

1. Random testing involves generating random inputs to examine the model's behavior.

2. Adversarial testing aims to uncover vulnerabilities or biases in the LLM by intentionally crafting inputs that could cause it to fail.

3. Domain-specific testing focuses on evaluating the model's performance within specific domains or use cases, example Golden queries.

None of the following frameworks are comprehensive enough to be self-sufficient for domain-specific testing so far.

Some of the frameworks for evaluating 'generic' use-cases are

1. OpenAI Evals https://github.com/openai/evals 

2. SuperGLUE Benchmark https://super.gluebenchmark.com/ 

3. LAMBADA https://zenodo.org/record/2630551#.ZFUKS-zML0p

4. Big Bench https://github.com/google/BIG-bench 

5. MMLU https://github.com/hendrycks/test 

6. Adversarial NLI (ANLI) https://github.com/facebookresearch/anli 

7. LIT (Language Interpretability Tool) https://pair-code.github.io/lit/ 

8. ParlAI https://github.com/facebookresearch/ParlAI 

9. HellaSwag https://rowanzellers.com/hellaswag/ 

10. LogiQA https://github.com/lgw863/LogiQA-dataset 

Conclusion

Unfortunately there are no well tested frameworks for domain-specific testing yet for use-case specific AI SaaS apps, contrary to generic summarisation use-cases where several popular frameworks are available. A custom combination of (above mentioned) frameworks combined with human testing could be a potential solution until further research is available.

Some references:

• https://towardsdatascience.com/testing-large-language-models-like-we-test-software-92745d28a359

• https://www.analyticsvidhya.com/blog/2023/05/how-to-evaluate-a-large-language-model-llm/ 

• https://arxiv.org/pdf/2307.03109v3.pdf 

• https://ekzhu.medium.com/human-aligned-text-to-sql-evaluation-399123fa0a64