Embodied and Emotional Voice Interaction with ChatGPT and QTrobot
Join the LuxAI QTrobot competition at IROS2023 and win one of three QTrobots for research!
Task
Large language models such as GPT3 can be used for different applications in human-computer interaction, and we want to explore their use in embodied interaction. The task in the competition will be to integrate ChatGPT gpt-3.5-turbo model into QTrobot by LuxAI to build a model for embodiment based on the output of the large language model. The embodiment model should employ and synchronize voice, facial expressions and gestures of the robot to produce emotional and multimodal (voice-based and embodied) dialogues. The steps can be for example:
- A user sends a textual input to ChatGPT using a web interface.
- Response from ChatGPT arrives and is analysed by your model.
- QTrobot performs what your model suggests: how to move arms, how to speak, how to show emotions.
You can find the technical description of the QTrobot here: QTrobot Documentation (luxai.com)
Imagine, you are an actor, you receive a text as input and you need to perform by using your voice, gestures, body language and face. Now imagine that QTrobot is an actor, and ChatGPT writes the text that QTrobot needs to perform. And you are the one who provides the robot with the model to do this. The model should use the text generated by ChatGPT as input and generate the embodiment as output that would be than played on the QTrobot.
Sounds exciting? We are sure it will be a lot of fun at IROS!
Awards
- Each of the three best teams receives one QTrobot for research by LuxAI.
- All finalists and their submissions will be featured in the QTrobot repository for learning accessible for the entire community.
Equality of chances
Voice-based input is of course possible, however, in a noisy environment, the accuracy of speech recognition can suffer. Therefore, we suggest that all teams use a text-based interface to interact with the QTrobot. Examples of text-based interfaces are web form, WhatsApp, Facebook Messenger, Telegram and many more. You choose which interface you use.
We will supply all teams with the following:
- Templates and a simulator for facial expressions will be provided to all registered teams.
- Prompts for development will be provided to all registered teams.
- Tests at the competition will be performed on a different but similar set of prompts.
- Language of the competition is English.
Prerequisites
OpenAI account will be needed to access the API.
Onsite participation
Because the submitted software will be deployed on physical robots at the competition venue, in-person participation is preferred. In exceptional cases (if travel is not possible because of medical or geopolitical reasons), online participation can be made possible.
Rules of Participation
Teams from public research institutions who are eligible to participate in IROS2023 are also eligible to participate in this competition. LuxAI S.A. current and former employees are not eligible for this competition.
Deadlines and steps
- 10.04.2023: Registration deadline. Register your team and declare interest in participation (mandatory). Go to the registration form.
- 14.04.2023: Kick-off online team meeting. Simulators and templates provided to teams.
- 18.05.2023: First submission deadline: mandatory minimum viable application. Selection of teams for the second round.
- 15.06.2023: Final submission deadline: fine-tuned model, documentation and short paper describing the model.
- 30.06.2023: Jury decision on finalists and author notification.
- 1-5.10.2023: Competition @IROS 2023 in person or online.
Organisers
Sviatlana Höhn (LuxAI S.A), Ali Paikan (LuxAI. S.A), Denis Kovacevic (LuxAI S.A.) Nina Hosseini-Kivanani (University of Luxembourg)
Contact
Any questions? Contact the organisers via [email protected]!
