Preprint: CRAFT: Exploring AI as a Critical Partner in Design Problem Solving

Abstract

Generative AI is commonly positioned as an idea generator in design problem solving, although direct suggestions can also anchor designers or reduce their engagement with problem framing. CRAFT explores a complementary role: an AI agent that enters only after unaided ideation and challenges a provisional design through critical feedback. In a three-condition between-participants study, 30 students from 12 academic degree programs completed a time-bounded ill-structured design task involving the development of an interactive, adaptive piece of furniture and then reviewed their concepts either without support (A), with a short generic checklist (B), or with design-specific feedback from CRAFT (C; n = 10 each). Outcomes included think-aloud verbalizations, an adapted version of the Creative Product Semantic Scale (CPSS) rated blindly by two human experts and two independent AI systems, design-change ratings, workload, and a semantic-divergence pretest. The small, heterogeneous sample make this an exploratory, formative study rather than a confirmatory test. No omnibus comparison met the .05 threshold for statistical significance. Nevertheless, several descriptive patterns converged: C produced the highest mean CPSS composite for all four raters and the highest qualitative-change rating for all four raters; participants in C also verbalized more requirements and self-criticism, but fewer self-directed process questions and planning statements. Rater rank-order showed convergence in general but varied substantially. The reported observations do not establish an intervention effect. They instead identify candidate mechanisms and controls for a larger preregistered study of whether deliberately critical AI feedback can support reflective design without taking over ideation.