EvoStruggle Dataset

Data of video recordings of manual activities of various people performing manual tasks from a first-person perspective. Activities include origami, card shuffling, tangrams and knot tying.

Please see the paper "EvoStruggle: A Dataset Capturing the Evolution of Struggle across Activities and Skill Levels" (ICPR 2026) for more details. The EvoStruggle Dataset Promo Video is available on YouTube.

A related talk was given by Prof Walterio Mayol - Keynote: From Skill to Struggle at the ICCV 2025 SAUAFG Workshop.

Contents: Introduction · Dataset at a Glance · Download · Quick Start · Usage of the Data · Repository Structure · Citation · License · Contact

“To struggle, as defined in the dictionary (verb), is to experience difficulty and make a great effort in order to do something.”

In this work, struggle is defined as observable difficulty in completing a given activity. It may be characterized by one or more of the following indicators:

• Motor hesitation of the hands
• Repeated attempts
• Prolonged actions
• Body-gesture signs of frustration (e.g., hand and/or head movements)
• Disruptive errors and pauses

• Over 60 hours video recordings, 2,793 videos, and 5,385 annotated temporal struggle segments from 76 participants.

• Evolution of Skill: Five Attempts/Repetitions Each Task.

• Diversity: 18 Tasks Grouped into Four activities--Tying Knots, Origami, Tangram Puzzles, and Shuffling Cards.

Computed from the annotation files in annotations/. All videos are recorded at 50 fps.

The annotations and data splits are included in this repository. The videos are hosted externally:

To download from Hugging Face, e.g. with the huggingface_hub CLI:

pip install -U huggingface_hub
huggingface-cli download Shijia2025/EvoStruggle --repo-type dataset --local-dir EvoStruggle

See Downloading Datasets for other options.

Note: The original 1080p recordings are currently only available via Baidu NetDisk.

Clone this repository to get the annotations and splits:

git clone https://github.com/FELIXFENG2019/EvoStruggle.git
cd EvoStruggle

Load the struggle annotations of one activity (list-valued columns are stored as JSON strings):

import json
import pandas as pd

df = pd.read_csv("annotations/origami_tsa_full.csv")
for col in ["keyframes", "keyframes(frames)", "struggle", "struggle(frames)"]:
    df[col] = df[col].apply(json.loads)

row = df.iloc[0]
print(row["video_name"], row["duration"], row["fps"])  # 01_01_01 83.66 50
print(row["struggle"])  # [[16.184, 21.79392], [45.16892, 59.16892], [69.97771, 75.85271]]

Load a benchmark split (ActivityNet-style JSON, directly usable by temporal action localization codebases such as StruggleTAL):

import json

with open("splits/separate_attempts/Origami/Origami_sepattempt.json") as f:
    split = json.load(f)

for video_name, info in split["database"].items():
    subset = info["subset"]  # e.g. "train_attempt01", "validation"
    segments = [ann["segment"] for ann in info["annotations"]]  # [[start_sec, end_sec], ...]

This section describes the annotation format, video naming convention, and data splits used in the Struggle Temporal Action Localization (Struggle TAL) task.

There is one annotation file per activity in annotations/. Each row corresponds to one video:

Videos without any observed struggle have empty lists ([]). There is a single action class, Struggle (see annotations/category_idx.txt).

Each video follows the naming format <participant_id>_<task_index>_<attempt_id>, where

• participant_id: two-digit participant identifier (e.g. 01)
• task_index: two-digit index of the task within the corresponding activity (as listed above)
• attempt_id: repetition number of the task, ranging from 01 to 05

Example: 01_03_04 denotes participant 01 performing task 03 (e.g. Helmet, Ribbon Spread and Wave, Cyclist, or Carrick Bend, depending on the activity) on the fourth attempt.

Video names are only unique within an activity, so always use them together with the activity name.

The official code release for the Struggle Temporal Action Localization task is available at:

• GitHub Repository: StruggleTAL

This repository can be used to reproduce the experimental results reported in the paper.

We provide three types of data splits for different training and evaluation settings (see Figure 6 in the paper for a visual overview).

Note: The files in splits/ are the official splits used in the paper. Please use them as provided so that results are comparable with the paper and with other work.

All JSON split files share the same structure:

{
  "version": "...",
  "database": {
    "<video_name>": {
      "subset": "train",
      "duration": 83.66,
      "fps": 50,
      "annotations": [
        {"label": "Struggle", "segment": [16.184, 21.79392], "segment(frames)": [809, 1090], "label_id": 1}
      ]
    }
  }
}

The CSV files next to each JSON file list the videos (with their annotations and metadata) in each subset. The JSON files were generated from them with the scripts in tools/.

<Activity> is one of Origami, Shuffle_Cards, Tangram, Tying_Knots.

Directory: splits/crossdomain_generalization

These splits are used for Activity-Level Generalization experiments across the four activities: <Activity> is held out as the unseen test activity, and the train/validation splits of the other three activities are used for training and validation.

Files:

• <Activity>_crossdomain_testonvalonly.json (recommended): the test set contains only the validation split of the unseen activity
• <Activity>_crossdomain.json: the test set contains all videos of the unseen activity, grouped by task (test_subactivity<XX>)

Directory: splits/indomain_generalization

These splits support Task-Level Generalization experiments within each activity: task <XX> is held out as the unseen test task, and the remaining tasks of the same activity are split by participant into train and validation sets.

Files: <Activity>_subactivity<XX>_data.json. Use these files to load the corresponding training and testing data.

Directory: splits/separate_attempts

• Within-Activity Evaluation: Provides baseline Struggle TAL performance within the same activity (vanilla setting).
• Separate Attempts Evaluation: Investigates the effect of multiple attempts on Struggle TAL performance. Training videos are grouped by attempt (train_attempt01 … train_attempt05).

Files:

• <Activity>_sepattempt.json: use this file to run both evaluation settings.
• <Activity>_allattempts_sample0{1,2,3}.json: training sets of the same size as a single attempt, sampled evenly from all five attempts with three different random seeds.

EvoStruggle/
├── annotations/            # Struggle annotations, one CSV per activity
├── splits/
│   ├── crossdomain_generalization/   # Activity-level generalization splits
│   ├── indomain_generalization/      # Task-level generalization splits
│   └── separate_attempts/            # Within-activity / separate-attempts splits
├── tools/                  # Scripts for generating splits and extracting video features
├── data/                   # Place the downloaded videos here (see data/README.md)
├── extracted_features/     # Place extracted video features here (see extracted_features/README.md)
├── LICENSE                 # Apache-2.0 (code)
└── LICENSE-DATASET.md      # CC BY-NC 4.0 (dataset)

See tools/README.md for how to use the scripts.

• Shijia Feng
• Michael Wray
• Walterio Mayol-Cuevas

If you use this dataset, please cite our paper:

@inproceedings{feng2026evostruggle,
  title={{EvoStruggle}: A Dataset Capturing the Evolution of Struggle across Activities and Skill Levels},
  author={Feng, Shijia and Wray, Michael and Mayol-Cuevas, Walterio},
  booktitle={International Conference on Pattern Recognition (ICPR)},
  pages={96--111},
  year={2026},
  organization={Springer}
}

arXiv version:

@misc{feng2025evostruggledatasetcapturingevolution,
  title={{EvoStruggle}: A Dataset Capturing the Evolution of Struggle across Activities and Skill Levels},
  author={Shijia Feng and Michael Wray and Walterio Mayol-Cuevas},
  year={2025},
  eprint={2510.01362},
  archivePrefix={arXiv},
  primaryClass={cs.CV},
  url={https://arxiv.org/abs/2510.01362}
}

The code in this repository is released under the Apache-2.0 License (see LICENSE). The EvoStruggle dataset (videos, annotations and splits) is released under CC BY-NC 4.0 (see LICENSE-DATASET.md). If you use this dataset, please cite our paper.

For questions, bug reports, or requests, please open an issue in this repository.