Querying a sign language dictionary with videos using dense vector search
- Author
- Mathieu De Coster (UGent) and Joni Dambre (UGent)
- Organization
- Project
- Abstract
- To search for an unknown sign in a sign language dictionary, users typically indicate parameters of the query, e.g., hand shape and signing location. Recent advances in sign language recognition enable video-based sign language dictionary search. In such a system, users can record an unknown sign and retrieve a list of signs that look similar, preferably including the queried sign as one of the top results. We have realized such a system by interpreting it as a dense vector search task. First, we learn a mapping (embedding) from sign videos to a vector space. The dictionary can then be searched by looking for the vectors in this space that are closest to the vector corresponding to the query. We present a proof of concept on a subset of the Flemish Sign Language dictionary. Further research is required to scale up our method to the large vocabularies of entire dictionaries.
- Keywords
- information retrieval, vector search, sign language
Downloads
-
DS660 acc.pdf
- full text (Accepted manuscript)
- |
- open access
- |
- |
- 597.53 KB
-
(...).pdf
- full text (Published version)
- |
- UGent only
- |
- |
- 647.35 KB
Citation
Please use this url to cite or link to this publication: http://hdl.handle.net/1854/LU-01H9J903EYS5FW2SGJ1XPMV55V
- MLA
- De Coster, Mathieu, and Joni Dambre. “Querying a Sign Language Dictionary with Videos Using Dense Vector Search.” 2023 IEEE INTERNATIONAL CONFERENCE ON ACOUSTICS, SPEECH, AND SIGNAL PROCESSING WORKSHOPS, ICASSPW, IEEE, 2023, doi:10.1109/icasspw59220.2023.10193531.
- APA
- De Coster, M., & Dambre, J. (2023). Querying a sign language dictionary with videos using dense vector search. 2023 IEEE INTERNATIONAL CONFERENCE ON ACOUSTICS, SPEECH, AND SIGNAL PROCESSING WORKSHOPS, ICASSPW. Presented at the ICASSPW2023, the IEEE International Conference on Acoustics, Speech, and Signal Processing Workshops, Rhodes Island, Greece. https://doi.org/10.1109/icasspw59220.2023.10193531
- Chicago author-date
- De Coster, Mathieu, and Joni Dambre. 2023. “Querying a Sign Language Dictionary with Videos Using Dense Vector Search.” In 2023 IEEE INTERNATIONAL CONFERENCE ON ACOUSTICS, SPEECH, AND SIGNAL PROCESSING WORKSHOPS, ICASSPW. IEEE. https://doi.org/10.1109/icasspw59220.2023.10193531.
- Chicago author-date (all authors)
- De Coster, Mathieu, and Joni Dambre. 2023. “Querying a Sign Language Dictionary with Videos Using Dense Vector Search.” In 2023 IEEE INTERNATIONAL CONFERENCE ON ACOUSTICS, SPEECH, AND SIGNAL PROCESSING WORKSHOPS, ICASSPW. IEEE. doi:10.1109/icasspw59220.2023.10193531.
- Vancouver
- 1.De Coster M, Dambre J. Querying a sign language dictionary with videos using dense vector search. In: 2023 IEEE INTERNATIONAL CONFERENCE ON ACOUSTICS, SPEECH, AND SIGNAL PROCESSING WORKSHOPS, ICASSPW. IEEE; 2023.
- IEEE
- [1]M. De Coster and J. Dambre, “Querying a sign language dictionary with videos using dense vector search,” in 2023 IEEE INTERNATIONAL CONFERENCE ON ACOUSTICS, SPEECH, AND SIGNAL PROCESSING WORKSHOPS, ICASSPW, Rhodes Island, Greece, 2023.
@inproceedings{01H9J903EYS5FW2SGJ1XPMV55V,
abstract = {{To search for an unknown sign in a sign language dictionary, users typically indicate parameters of the query, e.g., hand shape and signing location. Recent advances in sign language recognition enable video-based sign language dictionary search. In such a system, users can record an unknown sign and retrieve a list of signs that look similar, preferably including the queried sign as one of the top results. We have realized such a system by interpreting it as a dense vector search task. First, we learn a mapping (embedding) from sign videos to a vector space. The dictionary can then be searched by looking for the vectors in this space that are closest to the vector corresponding to the query. We present a proof of concept on a subset of the Flemish Sign Language dictionary. Further research is required to scale up our method to the large vocabularies of entire dictionaries.}},
author = {{De Coster, Mathieu and Dambre, Joni}},
booktitle = {{2023 IEEE INTERNATIONAL CONFERENCE ON ACOUSTICS, SPEECH, AND SIGNAL PROCESSING WORKSHOPS, ICASSPW}},
isbn = {{9798350302615}},
keywords = {{information retrieval,vector search,sign language}},
language = {{eng}},
location = {{Rhodes Island, Greece}},
pages = {{5}},
publisher = {{IEEE}},
title = {{Querying a sign language dictionary with videos using dense vector search}},
url = {{http://doi.org/10.1109/icasspw59220.2023.10193531}},
year = {{2023}},
}
- Altmetric
- View in Altmetric
- Web of Science
- Times cited: