Information Quality and Semantic Stability of AI Chatbot Responses to Complete Denture Questions: A Turkish–English Benchmarking Study

Makaleler > Tam Makale · 2026

Yazarlar: Gülcan COŞKUN AKAR

HEALTHCARE, SCI İndekslerine Giren Dergi, Uluslararası, 2026, cilt 14, ss 2797-2815

DOI: 10.3390/healthcare14172797

Özet

<p><span style="background-color: rgb(255, 255, 255); color: rgb(34, 34, 34);">Artificial&nbsp;intelligence&nbsp;(AI)-based&nbsp;chatbots&nbsp;are&nbsp;increasingly&nbsp;used&nbsp;as&nbsp;sources&nbsp;of&nbsp;health&nbsp;information,&nbsp;yet&nbsp;the&nbsp;quality&nbsp;and&nbsp;semantic&nbsp;stability&nbsp;of&nbsp;their&nbsp;responses&nbsp;may&nbsp;vary&nbsp;across&nbsp;systems,&nbsp;languages,&nbsp;repeated&nbsp;queries,&nbsp;and&nbsp;prompt&nbsp;formulations.&nbsp;This&nbsp;study&nbsp;compared&nbsp;expert-rated&nbsp;general&nbsp;information&nbsp;quality&nbsp;and&nbsp;embedding-based&nbsp;semantic&nbsp;stability&nbsp;of&nbsp;responses&nbsp;generated&nbsp;by&nbsp;seven&nbsp;AI&nbsp;chatbot&nbsp;systems&nbsp;to&nbsp;expert-derived,&nbsp;patient-oriented&nbsp;complete&nbsp;denture&nbsp;questions&nbsp;in&nbsp;Turkish&nbsp;and&nbsp;English.&nbsp;</span><strong style="background-color: rgb(255, 255, 255); color: rgb(34, 34, 34);">Methods</strong><span style="background-color: rgb(255, 255, 255); color: rgb(34, 34, 34);">:&nbsp;Twenty-two&nbsp;complete&nbsp;denture-related&nbsp;questions&nbsp;were&nbsp;submitted&nbsp;to&nbsp;seven&nbsp;chatbot&nbsp;systems&nbsp;on&nbsp;three&nbsp;study&nbsp;days&nbsp;across&nbsp;scheduled&nbsp;morning,&nbsp;afternoon,&nbsp;and&nbsp;evening&nbsp;sessions.&nbsp;Two&nbsp;additional&nbsp;semantically&nbsp;equivalent&nbsp;Turkish&nbsp;variants&nbsp;were&nbsp;generated&nbsp;for&nbsp;each&nbsp;original&nbsp;question.&nbsp;Five&nbsp;prosthodontists&nbsp;assessed&nbsp;the&nbsp;original&nbsp;Turkish&nbsp;and&nbsp;English&nbsp;responses&nbsp;using&nbsp;the&nbsp;5-point&nbsp;Global&nbsp;Quality&nbsp;Score&nbsp;(GQS).&nbsp;Semantic&nbsp;stability&nbsp;was&nbsp;quantified&nbsp;using&nbsp;the&nbsp;multilingual&nbsp;SentenceTransformer&nbsp;checkpoint&nbsp;paraphrase-multilingual-MiniLM-L12-v2&nbsp;and&nbsp;cosine&nbsp;similarity.&nbsp;To&nbsp;incorporate&nbsp;complete&nbsp;responses,&nbsp;responses&nbsp;were&nbsp;divided&nbsp;into&nbsp;non-overlapping&nbsp;token-based&nbsp;chunks,&nbsp;chunk&nbsp;embeddings&nbsp;were&nbsp;combined&nbsp;into&nbsp;one&nbsp;normalized&nbsp;full-response&nbsp;representation,&nbsp;and&nbsp;the&nbsp;36&nbsp;pairwise&nbsp;similarities&nbsp;from&nbsp;the&nbsp;nine&nbsp;repeated&nbsp;responses&nbsp;were&nbsp;averaged&nbsp;to&nbsp;one&nbsp;stability&nbsp;estimate&nbsp;per&nbsp;question–model–language/variant&nbsp;condition.&nbsp;Linear&nbsp;mixed-effects&nbsp;models&nbsp;with&nbsp;question-level&nbsp;clustering&nbsp;were&nbsp;used,&nbsp;with&nbsp;Bonferroni&nbsp;adjustment&nbsp;and&nbsp;partial&nbsp;eta&nbsp;squared&nbsp;effect&nbsp;sizes&nbsp;with&nbsp;95%&nbsp;confidence&nbsp;intervals.&nbsp;</span><strong style="background-color: rgb(255, 255, 255); color: rgb(34, 34, 34);">Results</strong><span style="background-color: rgb(255, 255, 255); color: rgb(34, 34, 34);">:&nbsp;AI&nbsp;model&nbsp;significantly&nbsp;affected&nbsp;embedding-based&nbsp;semantic&nbsp;stability&nbsp;(</span><em style="background-color: rgb(255, 255, 255); color: rgb(34, 34, 34);">p</em><span style="background-color: rgb(255, 255, 255); color: rgb(34, 34, 34);">&nbsp;&lt;&nbsp;0.001;&nbsp;ηp</span><sup style="background-color: rgb(255, 255, 255); color: rgb(34, 34, 34);">2</sup><span style="background-color: rgb(255, 255, 255); color: rgb(34, 34, 34);">&nbsp;=&nbsp;0.820,&nbsp;95%&nbsp;CI&nbsp;0.795–0.837)&nbsp;and&nbsp;GQS&nbsp;(</span><em style="background-color: rgb(255, 255, 255); color: rgb(34, 34, 34);">p</em><span style="background-color: rgb(255, 255, 255); color: rgb(34, 34, 34);">&nbsp;&lt;&nbsp;0.001;&nbsp;ηp</span><sup style="background-color: rgb(255, 255, 255); color: rgb(34, 34, 34);">2</sup><span style="background-color: rgb(255, 255, 255); color: rgb(34, 34, 34);">&nbsp;=&nbsp;0.249,&nbsp;95%&nbsp;CI&nbsp;0.152–0.316).&nbsp;Grok&nbsp;had&nbsp;the&nbsp;numerically&nbsp;highest&nbsp;overall&nbsp;semantic&nbsp;stability&nbsp;mean&nbsp;(0.952&nbsp;±&nbsp;0.015),&nbsp;whereas&nbsp;GPT-4o&nbsp;had&nbsp;the&nbsp;numerically&nbsp;highest&nbsp;overall&nbsp;GQS&nbsp;(4.48&nbsp;±&nbsp;0.85).&nbsp;English&nbsp;responses&nbsp;showed&nbsp;higher&nbsp;overall&nbsp;GQS&nbsp;than&nbsp;the&nbsp;original&nbsp;Turkish&nbsp;responses&nbsp;(3.96&nbsp;±&nbsp;0.94&nbsp;vs.&nbsp;3.66&nbsp;±&nbsp;1.23;&nbsp;</span><em style="background-color: rgb(255, 255, 255); color: rgb(34, 34, 34);">p</em><span style="background-color: rgb(255, 255, 255); color: rgb(34, 34, 34);">&nbsp;=&nbsp;0.004)&nbsp;and&nbsp;higher&nbsp;embedding-based&nbsp;similarity&nbsp;than&nbsp;the&nbsp;Turkish&nbsp;conditions&nbsp;overall&nbsp;(</span><em style="background-color: rgb(255, 255, 255); color: rgb(34, 34, 34);">p</em><span style="background-color: rgb(255, 255, 255); color: rgb(34, 34, 34);">&nbsp;&lt;&nbsp;0.001).&nbsp;No&nbsp;significant&nbsp;overall&nbsp;effect&nbsp;of&nbsp;scheduled&nbsp;study&nbsp;day&nbsp;was&nbsp;observed&nbsp;(</span><em style="background-color: rgb(255, 255, 255); color: rgb(34, 34, 34);">p</em><span style="background-color: rgb(255, 255, 255); color: rgb(34, 34, 34);">&nbsp;=&nbsp;0.504;&nbsp;ηp</span><sup style="background-color: rgb(255, 255, 255); color: rgb(34, 34, 34);">2</sup><span style="background-color: rgb(255, 255, 255); color: rgb(34, 34, 34);">&nbsp;=&nbsp;0.001),&nbsp;whereas&nbsp;semantic&nbsp;stability&nbsp;differed&nbsp;across&nbsp;scheduled&nbsp;query&nbsp;sessions&nbsp;(</span><em style="background-color: rgb(255, 255, 255); color: rgb(34, 34, 34);">p</em><span style="background-color: rgb(255, 255, 255); color: rgb(34, 34, 34);">&nbsp;&lt;&nbsp;0.001;&nbsp;ηp</span><sup style="background-color: rgb(255, 255, 255); color: rgb(34, 34, 34);">2</sup><span style="background-color: rgb(255, 255, 255); color: rgb(34, 34, 34);">&nbsp;=&nbsp;0.055),&nbsp;with&nbsp;means&nbsp;of&nbsp;0.863&nbsp;±&nbsp;0.106,&nbsp;0.874&nbsp;±&nbsp;0.092,&nbsp;and&nbsp;0.841&nbsp;±&nbsp;0.140&nbsp;for&nbsp;morning,&nbsp;afternoon,&nbsp;and&nbsp;evening&nbsp;sessions,&nbsp;respectively.&nbsp;Inter-rater&nbsp;reliability&nbsp;was&nbsp;moderate&nbsp;for&nbsp;English&nbsp;GQS&nbsp;ratings&nbsp;(ICC&nbsp;=&nbsp;0.675,&nbsp;95%&nbsp;CI&nbsp;0.581–0.752)&nbsp;and&nbsp;good&nbsp;for&nbsp;Turkish&nbsp;ratings&nbsp;(ICC&nbsp;=&nbsp;0.771,&nbsp;95%&nbsp;CI&nbsp;0.687–0.833).&nbsp;</span><strong style="background-color: rgb(255, 255, 255); color: rgb(34, 34, 34);">Conclusions</strong><span style="background-color: rgb(255, 255, 255); color: rgb(34, 34, 34);">:&nbsp;The&nbsp;evaluated&nbsp;chatbot&nbsp;systems&nbsp;differed&nbsp;in&nbsp;expert-rated&nbsp;information&nbsp;quality&nbsp;and&nbsp;full-response&nbsp;embedding-based&nbsp;semantic&nbsp;stability.&nbsp;English&nbsp;responses&nbsp;showed&nbsp;higher&nbsp;overall&nbsp;values&nbsp;than&nbsp;Turkish&nbsp;responses,&nbsp;although&nbsp;cross-language&nbsp;measurement&nbsp;effects&nbsp;of&nbsp;the&nbsp;embedding&nbsp;model&nbsp;cannot&nbsp;be&nbsp;excluded.&nbsp;Differences&nbsp;across&nbsp;scheduled&nbsp;query&nbsp;sessions&nbsp;should&nbsp;be&nbsp;interpreted&nbsp;as&nbsp;run-to-run&nbsp;output&nbsp;variability&nbsp;rather&nbsp;than&nbsp;intrinsic&nbsp;temporal&nbsp;behavior.&nbsp;These&nbsp;findings&nbsp;characterize&nbsp;comparative&nbsp;chatbot&nbsp;performance&nbsp;under&nbsp;the&nbsp;tested&nbsp;conditions&nbsp;but&nbsp;do&nbsp;not&nbsp;establish&nbsp;clinical&nbsp;accuracy,&nbsp;safety,&nbsp;or&nbsp;patient&nbsp;education&nbsp;effectiveness.</span></p>

Detay görüntüle