GitHub · #573

EleutherAI/lm-evaluation-harness

A framework for few-shot evaluation of language models. On Linux: git clone, python3 -m venv .venv, then pip install -e . 14,259 GitHub stars (#573 in this directory).

Repository
https://github.com/EleutherAI/lm-evaluation-harness
Homepage
https://www.eleuther.ai
Owner
EleutherAI (Organization)
Stars
14,259
Forks
3,692
Open issues
1,201
Language
Python
License
MIT
Created
2020-08-28
Last push
2026-09-14
Topics
evaluation-framework, language-model, transformer
Snapshot
2026-10-18 · weekly refresh

Try it on Shadowfetch Linux.

Shadowfetch Linux is a Debian testing desktop with KDE Plasma, local AI tools, and ShadowCode. Clone this repository there, follow the steps for its primary language, and ask ShadowCode to finish the build.

Clone and set up a Python environment.

git clone https://github.com/EleutherAI/lm-evaluation-harness.git
cd lm-evaluation-harness
python3 -m venv .venv
source .venv/bin/activate
pip install -e .

If the project has no installable package, use the README’s requirements file or conda notes instead of pip install -e .

ShadowCode prompt.

Paste this into ShadowCode after you open the project.

Clone https://github.com/EleutherAI/lm-evaluation-harness into ~/Workspaces/lm-evaluation-harness.
Read the README.
On this Shadowfetch Linux desktop, create a Python virtualenv, install from the README, then run or import the project.
Ask before installing packages or using the network.

Download Shadowfetch Linux — ISO, checksum and signature.

Back to the GitHub directory · Methodology