Rank #03 · Daily · DevTools
SkillEvaluator: NVIDIA's Multi-Tier Framework for Evaluating Agent Skills
Check whether an agent skill is well-formed and safe, then measure whether adding it actually changes how the agent performs a task.
What it does
SkillEvaluator is a multi-tier evaluation framework from NVIDIA for agent skills and plugins. It is an evaluation tool, not a skills or prompt collection. GitHub description: "Multi-tier framework for evaluating AI agent skills and plugins with quality gates, semantic overlap detection, synthetic evaluation dataset generation, and live agent evaluation that measures how skills affect agent behavior." Docs: docs.nvidia.com/skills/skillevaluator. PyPI: skillevaluator. Paper: arXiv:2608.20614. NVIDIA Developer blog: Evaluating AI Agent Skill Performance with NVIDIA SkillEvaluator.
The repository's root skills/ directory holds the project's own skill (skills/skillevaluator) so a coding agent can run the evaluator. It is not a dump of skills to install as content.
README claims (as verified):
- Agent skills, in this project's wording, are folders of instructions and supporting files as defined by the Agent Skills specification. The README places SkillEvaluator in the NVIDIA Verified Skills pipeline; this write-up covers the evaluator only, not a skills catalog.
- Three independent tiers. Tier 1 (Validation) asks whether a skill is safe, well-formed, and harness-compatible. A keyless path (
skillevaluator validate ./my-skill --tiers 1, orquality-check) produces a deterministic quality score and findings; the default passing threshold is 70. Full static coverage can involve external scanners (the README names SkillSpector and Gitleaks; Semgrep is optional). A configured model enables LLM-backed rubric checks. Tier 2 (Deduplication) looks for repeated or overlapping guidance, including optional inter-skill comparison against a catalog. Tier 3 (Live Evaluation) asks whether the skill helps the agent: it needs a supported agent and a running sandbox (Docker by default). Harbor powers the sandboxed agent runs. - Three ways in, one CLI. Install with
pip install skillevaluator(oruv tool install skillevaluator) and run the CLI; add repository checks withskillevaluator ci init(GitHub and GitLab are documented); or wire the project's own skill into an agent (npx skills add NVIDIA/SkillEvaluator) and ask whether a given skill is any good. - Combined
validatecan run all tiers with autopilot for skills. Tier 1 gatesvalidate. Tier 2 gates unless--no-block-on-dedupis set. Tier 3 gates only with--block-on-agent-eval.doctorchecks readiness, not inference success. Dataset generation and refinement flags are documented for Tier 3; review generated cases before treating them as a complete benchmark. - Supported Python versions listed on the README badge: 3.12, 3.13, and 3.14. Support level is Experimental: community-supported, best-effort, with no SLA or NVIDIA enterprise support entitlement.
License: Apache-2.0 (LICENSE file — Copyright 2026 NVIDIA CORPORATION & AFFILIATES — and GitHub SPDX agree). The README notes that the project may download additional third-party open-source software; review those licenses before use.
Who it's for
- Teams that author or review agent skills and want quality gates, overlap checks, and a way to measure whether a skill changes task results.
- Groups that want those checks in CI without maintaining a custom evaluation harness.
- Researchers or practitioners who want to follow the published SkillEvaluator paper and CLI, not a prompt pack.
Less ideal if you are looking for a library of skills to install (this repository does not ship one for that purpose), need NVIDIA enterprise support or an SLA, or only want a one-off linter and cannot set up scanners, a model provider, or a live-evaluation sandbox.
Stars / language / license
2,696 stars · Python · Apache-2.0
Data verified: 2026-11-17 08:55 Asia/Shanghai (official Daily Top10 gate; language/license confirmed via GitHub API get_repository + README + LICENSE; docs, PyPI, and arXiv links verified from the README)
Trending note: +873 stars today (captured 08:55 Asia/Shanghai)
About the license: Apache-2.0 applies to this repository as published. Third-party scanners and extras may carry their own terms. This is a summary, not legal advice.
Category tags
DevTools · agent-evaluation · skill-evaluation · Python · Apache-2.0 · CLI · CI · NVIDIA
How it differs from similar projects
Compared with skills or prompt packs (including filtered trending repos). Those distribute instructions for an agent to follow. SkillEvaluator evaluates a skill you already have: quality, overlap, and live impact. Do not treat the root skills/ folder as a content pack.
Compared with generic agent eval harnesses (including Harbor on its own). Harbor is documented here as the sandbox runner for Tier 3. SkillEvaluator adds skill-specific quality gates, semantic overlap detection, optional synthetic eval-dataset generation, and a combined CLI/CI path around that live run.
Compared with static scanners alone (Semgrep, Gitleaks, and similar). Those can be part of Tier 1. The framework's further claim is rubric scoring and a with-versus-without-skill live evaluation — capabilities that need a model and, for Tier 3, an agent sandbox. TidyHow has not audited detection rates or score quality.
Caveats. Support is Experimental. A scanner that cannot run is marked INCOMPLETE and can fail the command. Model endpoints and managed sandboxes can incur charges. The README's example scores apply to bundled reference skills, not to your skill and not to a TidyHow benchmark. No independent accuracy, coverage, or cost figures are claimed here.