Human Action Co-occurrence in Lifestyle Vlogs using Graph Link Prediction

September 12, 2023 · Entered Twilight · 🏛 arXiv.org

Repo contents: .gitignore, README.md, action_downstream.py, data, data_analysis.ipynb, data_processing.py, environment.yml, frames_sample, img, link_prediction.py, requirements.txt, utils

Authors Oana Ignat, Santiago Castro, Weiji Li, Rada Mihalcea arXiv ID 2309.06219 Category cs.CV: Computer Vision Cross-listed cs.CL, cs.CY, cs.IR Citations 0 Venue arXiv.org Repository https://github.com/MichiganNLP/vlog_action_co-occurrence ⭐ 3 Last Checked 3 months ago

Abstract

We introduce the task of automatic human action co-occurrence identification, i.e., determine whether two human actions can co-occur in the same interval of time. We create and make publicly available the ACE (Action Co-occurrencE) dataset, consisting of a large graph of ~12k co-occurring pairs of visual actions and their corresponding video clips. We describe graph link prediction models that leverage visual and textual information to automatically infer if two actions are co-occurring. We show that graphs are particularly well suited to capture relations between human actions, and the learned graph representations are effective for our task and capture novel and relevant information across different data domains. The ACE dataset and the code introduced in this paper are publicly available at https://github.com/MichiganNLP/vlog_action_co-occurrence.