Title: Runtime Repair and Enhancement of Mobile App Accessibility
Advisor: James Fogarty
Supervisory Committee: James Fogarty (Chair), Leah Findlater (GSR, HCDE), Jennifer Mankoff, Jacob Wobbrock (iSchool), and Meredith Morris (MSR)
Abstract: Mobile devices and applications have become ubiquitous in daily life. Ensuring full access to the wealth of information and services provided by such apps is a matter of social justice. Unfortunately, many capabilities and services offered by apps remain inaccessible for people with disabilities. The built-in accessibility tools rely on correct app implementation, but app developers often fail to implement accessibility guidelines. In addition, the absence of tactile cues makes mobile touchscreens difficult to navigate for people with visual impairments.
In this dissertation, I describe the work I have done in runtime repair and enhancement of mobile app accessibility. Firstly, we explored a design space of interaction re-mapping, which provides examples of re-mapping existing inaccessible interactions into new accessible interactions. I also implemented interaction proxies, a strategy to modify an interaction at runtime without rooting the phone or accessing app source code. It enables third-party developers and researchers to repair and enhance mobile app accessibility. Secondly, I developed a system for robust annotations on mobile app interfaces to make the accessibility repairs reliable and scalable. Thirdly, I built Interactiles, a low-cost, portable, and unpowered system to enhance the tactile interaction on touchscreen phones for people with visual impairments.
The thesis of this dissertation is: An interaction remapping strategy can enable third-party developers and researchers to robustly repair and enhance the accessibility of mobile applications at runtime, while preserving the platform’s security model and accessibility infrastructure.