Characterizing and Detecting Navigation Inefficiencies in Smartphone Screen Reader Use

Screen readers on smartphones (e.g., TalkBack) enable people with visual impairments (PVIs) to navigate user interfaces (UIs) through sequential focus shifts, but this linear auditory structure can introduce inefficiencies and unnecessary navigation. This project investigated how PVIs actually explore and interact with smartphone UIs, with the goal of identifying structural mismatches between system-provided focus sequences and user intent. We developed an Android screen reader logger to capture PVIs-smartphone interaction and recruited 11 PVIs for a one-month in-the-wild study, complemented by semi-structured interviews. Analyses revealed three recurring inefficiency patterns: (1) unintentionally skipping initial focus targets, (2) traversing non-interactable UI elements, and (3) reversing navigation directions. Building on these findings, we designed algorithms that automatically detect these patterns in usage logs, providing a foundation for accessibility services that adapt focus ordering and offer context-aware shortcuts to reduce temporal and cognitive costs for screen reader users.