Beyond Self-Report: Evaluating Authentic AI Literacy in a Continuing Education Context

Research on AI literacy has predominantly relied on self-report instruments. which correlate weakly with objective performancebased measures (Zhang et al., 2026). This study examines how AI literacy can be reliably assessed among adults engaged in continuing education. Participants (N = 141) completed the AILQ (Ng et al., 2024), then performed an authentic task based on a real-world employer branding scenario. Performance was assessed using a fourdimension analytic rubric. A Disposition-Performance Gap (DPG) index quantified the misalignment between self-perceived disposition and demonstrated performance: 68.8% of participants were aligned, while 27.7% showed a Disposition Surplus. This pattern was strongly stratified by competence: all participants at the lowest performance level overestimated their competence, whereas proficient participants were almost entirely aligned (χ²(4) = 78.413, p < .001). Findings support performance-based assessment as a necessary complement to self-report instruments.