Full Length Article
DOI: https://doi.org/10.54216/JAIM.110206
MultiUserWiki: Managing Editorial Divergence in Collaborative LLM-Maintained Knowledge Bases
When multiple LLM-assisted editors maintain a shared knowledge base, the conflicts that arise are seldom outright factual contradictions. More often, two users reading the same source extract compatible assertions that differ in specificity, coverage, or editorial emphasis. This paper asks a prior question: when should a conflict be resolved by selecting a winner, and when should both perspectives be preserved? We study this question in a controlled single-source collaborative setting. Five role-conditioned instances of GLM-4-Flash—simulating a senior researcher, a teacher, a student, a clinician, and a conservative editor—independently extract assertions from AI-Wiki-21, a 21-page AI knowledge base. Their outputs yield 4,556 candidate conflict pairs; annotation of 138 selected pairs produces a 100-conflict gold standard. Three findings emerge, each with scope limited to the single-source, single- LLM-family, simulated-role setting studied. First, confirmed conflicts are exclusively editorial—54% Granularity and 46% Coverage—with zero Factual conflicts, suggesting that in this controlled setting the dominant challenge is editorial coordination, not truth arbitration. Second, direct LLM judgment achieves 97.5% agreement with human annotators on this editorial-conflict gold standard, while a DPO-inspired quality heuristic (DIQS) reaches 81.0% without API overhead; collective voting plateaus at 41.8%, echoing social-choice instability. Third, sensitivity analysis of the fiber split threshold across τ ∈ [0.1,0.9] identifies τ∗ = 0.5 as the stable optimum, partitioning 3.1% of conflicts while preserving diverse user perspectives. The results also raise ethical questions about automated knowledge governance.
Bailing Zhang
visibility
477
download
242