Detection & evals
LMArena
LMArena (Chatbot Arena) is a community-driven open-source benchmarking platform designed to evaluate the capabilities of Large Language Models (LLMs) through crowdsourcing.
What is LMArena?
LMArena (Chatbot Arena) is a community-driven open-source benchmarking platform designed to evaluate the capabilities of Large Language Models (LLMs) through crowdsourcing.Instead of relying on static datasets, the platform employs a 'blind test' mechanism where users converse with two anonymous models and vote for the better response without knowing their identities.
As a key reference tool in AI research, LMArena provides a dynamic leaderboard based on the Elo rating system.This human-preference-based evaluation method more accurately reflects actual user experience in complex instruction following, logical reasoning, and creative writing, offering valuable feedback data for developers and researchers.
Key features
- Anonymous BlindUsers compare models without knowing their names, eliminating brand bias and ensuring objective evaluation.
- Dynamic Elo LeadUses the Elo rating system common in chess to update model rankings in real-time based on win/loss outcomes, reflecting relative strength changes.
- Multi-turn EvaluSupports multi-turn interaction testing, assessing not just single-response quality but also coherence and memory in long contexts.
- Broad Model CoveThe platform integrates numerous mainstream open-source and proprietary LLMs, providing a wide range of horizontal comparisons.
- Community-DrivenAggregates real-world usage data from global users to form a representative public preference benchmark.
How to use it
- Visit the official LMArena website and enter the chat interface.
- Type your question or prompt into the input box; the system will send requests to two anonymous models simultaneously.
- Review the responses generated by both models, carefully comparing their accuracy, logic, and usefulness.
- Click the vote button to select the model you believe performed better (or choose a tie).
- After voting, the system typically reveals the models' identities and updates the leaderboard data.
Where it fits
- Model SelectionDevelopers check the leaderboard to understand performance differences between models as perceived by real users when choosing base models.
- Academic ResearcResearchers use crowdsourced data to study the relationship between human preferences and model output quality, optimizing alignment algorithms.
- Daily ExperienceGeneral users discover the best AI assistant for their specific needs (such as coding or creative writing) through blind testing.
Pricing and free quota
Pricing and any free quota follow the official site.
Who it is for
- AI Researchers
- LLM Developers
- Tech Enthusiasts
- Product Managers
FAQ
How is the LMArena ranking calculated?
Rankings are based on the Elo rating system, a method for calculating the relative skill levels of players.Each vote is treated as a match, where the winner gains points from the loser.
Can I know which model I am talking to?
Model identities are hidden before voting to ensure fairness.After you cast your vote, the system typically reveals the names of the models you interacted with.
What languages does the platform support?
LMArena primarily supports English, but it is gradually adding support for other multilingual models, depending on the capabilities of the integrated models.