Roadmap and Feedback

[sming256]

We keep this issue open to collect feature requests and feedback from users, and thus keep improving this codebase.

If you didn't find the features you need in the Road Map, please leave a message here.

Thank you!

[rixejzvdl649]

cool

[akshitac8]

Hi @sming256 Thank you for sharing the toolkit. Its really amazing, I wanted to the ask, the paper AdaTAD also has the parallel adapter approach, AdaTAD' (75.4 in paper) but when implementing it using the codebase shared I am able to achieve somewhere around 73.4 mAP, can you please share if there are any parameters are different between AdaTAD and AdaTAD'?

[sming256]

Only a few hyperparameters are changed, such as the mlp ratio in the adapter is changed from 1/4 to 1/8, and the learning rate is searched between 1e-4 to 5e-5. I will update the parallel backbone and checkpoint later.

[akshitac8]

That would be really helpful @sming256 for reproducing the results 😃.

[akshitac8]

Hi @sming256 wanted to check if you could please upload the parallel backbone code as well that would be great.

[Caspeerrr]

Are you planning to also release tridetplus in this toolkit? (https://github.com/dingfengshi/tridetplus)
thanks!

[sming256]

@Caspeerrr Thanks for your suggestion! Integrating TriDetPlus into OpenTAD seems straightforward. However, TriDetPlus only released the VideoMAEv2 feature, not the DINO2 feature. This is the only reason we haven't integrated it now.

[sming256]

Hi @akshitac8 , the side tuning model is released here, and we provide a training example here .
When implementing the side-tuning with the latest OpenTAD, we find a performance drop of around 1% on THUMOS. In our released checkpoint, we achieve 74.65% mAP using VideoMAEv2-g.

[aidevveloper]

Hi @sming256, Thank you for an excellent work. I have a question about open source datasets not yet included in this repo.

You mentioned as follows:
Support multiple TAD datasets. We support 9 TAD datasets, including ActivityNet-1.3, THUMOS-14, HACS, Ego4D-MQ, EPIC-Kitchens-100, FineAction, Multi-THUMOS, Charades, and EPIC-Sounds Detection datasets.

What about other open source datasets (e.g, AVA2, UCF24)?

[sming256]

Hi @Harry-KIT, thanks for your question. OpenTAD is designed for temporal action detection task. I think AVA2 and UCF24 are datasets of spatial-temporal action detection task, which requires additional spatial bounding boxes.

[aidevveloper]

I see. Thank you very much

[BruParis]

Hello @sming256,
Thank you for sharing this amazing tool.

I noticed the Roadmap mentions 'Feature extraction code.' Could you provide more details on this point?
Do you plan to export the outputs of the backbone (adapter/ladder), or perhaps the intermediate projection within the detectors ?

Looking forward to hearing more about it.

[sming256]

Thanks for your question @BruParis .

We plan to release the feature extraction code to extract untrimmed video features, such as I3D/SlowFast/VideoMAE/InternVideo2. People can also add their own backbone models to the code to extract new features. Thus, the AdaTAD with adapters is also supported as the backbone.

Due to some refactoring work, we plan to release it next month.

[BruParis]

Alright, great !
Cannot wait to check this out !

Thank you for your answer.