Issues

Balanced few-shot sampling truncates on a falsy class label and uses the global random module

#1309

· ErenAta16 opened

on Jul 20, 2026

Sequential few-shot selection mutates the task's shared few-shot pool in place, corrupting later variance seeds

#1307

· ErenAta16 opened

on Jul 20, 2026

[EVAL] Add ArmBench-LLM Armenian evaluation suite

#1306

· bugdaryan opened

on Jul 20, 2026

avg_at_k / avg_at_k_math / maj_at_k metric regression fixtures silently no-op (metric_class name doesn't match registered Metrics enum members)

#1304

· ErenAta16 opened

on Jul 19, 2026

[BUG] Task : `wikitext:103:document_level` does not work

bug

#1299

· debasisdwivedy opened

on Jul 15, 2026

[BUG] LLM judge silently scores API error strings: litellm backend sends max_tokens as a list, provider returns 400

bug

#1296

· AbhilakshSinghReen opened

on Jul 14, 2026

hellaswag_arabic_pfn uses eval() instead of ast.literal_eval() on a dataset field, arbitrary code execution from an untrusted HF dataset

#1293

· ErenAta16 opened

on Jul 11, 2026

[BUG] transformers backend overrides the model's generation_config.eos_token_id, so chat models whose turn terminator differs from tokenizer.eos never stop (Gemma pads to max_new_tokens)

#1278

· waqaskhan137 opened

on Jul 2, 2026

[FT] Introduce a static 'lint' command to validate custom tasks before execution

#1266

· Kesav2k04 opened

on Jun 18, 2026

MMLU-Pro incorrect prompt format

bug

#1265

· RasmusHoier opened

on Jun 17, 2026

Lighteval's LCB grader fails correct solutions that read sys.stdin.buffer

bug

#1255

· RasmusHoier opened

on Jun 12, 2026

Guidance on adding REFUTE benchmark task (scientific critique & calibration)

#1253

· connerlambden opened

on Jun 5, 2026

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

All issues

Balanced few-shot sampling truncates on a falsy class label and uses the global random module

Sequential few-shot selection mutates the task's shared few-shot pool in place, corrupting later variance seeds

[EVAL] Add ArmBench-LLM Armenian evaluation suite

avg_at_k / avg_at_k_math / maj_at_k metric regression fixtures silently no-op (metric_class name doesn't match registered Metrics enum members)

[BUG] Task : `wikitext:103:document_level` does not work

[BUG] LLM judge silently scores API error strings: litellm backend sends max_tokens as a list, provider returns 400

hellaswag_arabic_pfn uses eval() instead of ast.literal_eval() on a dataset field, arbitrary code execution from an untrusted HF dataset

[BUG] transformers backend overrides the model's generation_config.eos_token_id, so chat models whose turn terminator differs from tokenizer.eos never stop (Gemma pads to max_new_tokens)

[FT] Introduce a static 'lint' command to validate custom tasks before execution

MMLU-Pro incorrect prompt format

Lighteval's LCB grader fails correct solutions that read sys.stdin.buffer

Guidance on adding REFUTE benchmark task (scientific critique & calibration)

Uh oh!

Search results