AI Knows Which Words Will Appear in Next Year's Korean CSAT

November 24, 2022 · Entered Twilight · 🏛 Information and Communication Technology Convergence

Repo contents: .DS_Store, 2023 sunung analysis, Build Contents, README.md, bigdata analysis, build_paragraphs, compare with other books, correlation analysis, dump to json, get_whitehouse_briefing, pdf parser, sentence extractor, word_meaning_extraction

Authors Byunghyun Ban, Jejong Lee, Hyeonmok Hwang arXiv ID 2211.15426 Category cs.CL: Computation & Language Citations 0 Venue Information and Communication Technology Convergence Repository https://github.com/needleworm/bigdata_voca ⭐ 2 Last Checked 3 months ago

Abstract

A text-mining-based word class categorization method and LSTM-based vocabulary pattern prediction method are introduced in this paper. A preprocessing method based on simple text appearance frequency analysis is first described. This method was developed as a data screening tool but showed 4.35 ~ 6.21 times higher than previous works. An LSTM deep learning method is also suggested for vocabulary appearance pattern prediction method. AI performs a regression with various size of data window of previous exams to predict the probabilities of word appearance in the next exam. Predicted values of AI over various data windows are processed into a single score as a weighted sum, which we call an "AI-Score", which represents the probability of word appearance in next year's exam. Suggested method showed 100% accuracy at the range 100-score area and showed only 1.7% error of prediction in the section where the scores were over 60 points. All source codes are freely available at the authors' Git Hub repository. (https://github.com/needleworm/bigdata_voca)