Advancing AI that converses naturally with people — at the research frontier.

The Language Media Laboratory at Kyoto University researches, around natural language processing, dialogue systems (meeting assistants) that converse naturally with people to support meetings. As the team advanced research on LLM-based conversation systems focused on "who to speak to" and "when to speak," ANOSUPO handled the annotation of the evaluation data underpinning it.

Evaluating meeting effectiveness required large-volume, high-difficulty annotation to validate a new method that segments and scores meetings topic by topic. Crowdsourcing (e.g., Amazon Mechanical Turk) had been used before, but controlling worker quality was hard — and the harder the task, the lower the agreement between annotators. With large data volumes, handling it with lab members alone wasn't realistic.

Alongside approachable pricing, ANOSUPO handled this high-difficulty task with a system that selects suitable workers through testing. Over about two weeks, ANOSUPO co-designed the annotation specification with the team — defining details finely while keeping it at a granularity workers wouldn't shy away from. The "worker-quality control" crowdsourcing struggled with was secured through selection and testing.

Even on hard tasks where crowdsourcing agreement dropped, consistent annotation by selected workers was achieved. This built the data foundation for research comparing meeting effectiveness across human evaluation and LLM-based automatic evaluation — supporting, from the data side, the establishment of an evaluation method for meeting-assistant development.