ProgramAlly: Creating Custom Visual Access Programs via Multi-Modal End-User Programming

OVERVIEW

ProgramAlly is a mobile platform that empowers blind and visually impaired users to create and personalize visual information filters using intuitive end-user programming techniques.

• Multi-Modal Programming: Users can build custom visual access programs via block-based interfaces, natural language, and programming by example, tailored to individual needs.
• Market Opportunity: Addresses a major gap in assistive technology by enabling robust, real-time adaptation for diverse, one-off, or repetitive scenarios that mainstream solutions cannot serve—unlocking new market potential for personalized accessibility.

BACKGROUND

Access to visual information—such as reading text, identifying objects, and navigating spaces—is a central challenge for blind or visually impaired people. Most commercial visual assistive technologies are optimized for broad, generic scenarios (like reading printed text or identifying common objects), leaving users to sift through irrelevant information or manually adapt in time-sensitive or repeated tasks (e.g., sorting mail, catching a bus, reading package labels).

DIY assistive technology has brought personalization to hardware (like 3D-printed aids), but software tools lag behind: there’s a notable gap between users’ needs for custom solutions and the ability of existing apps to support such customization. Research and market surveys show a growing demand for “long-tail” personalization—solutions that serve individuals' unique daily challenges, especially as AI becomes more ubiquitous but not yet sufficiently adaptable. Key trends include rising interest in inclusive design, user-driven innovation, and scalable user customization in digital products.

INNOVATION

ProgramAlly introduces an easy-to-use programming environment allowing non-experts to create custom filters for visual information on their smartphones (“find NUMBER on BUS”, “find ADDRESS on PACKAGE”, etc.). Users can construct these filters using three intuitive methods:

• Choose from pre-structured blocks (block-based programming) for precise control.
• Type or speak out requests in everyday language (natural language programming) for speed and convenience.
• Demonstrate tasks live by selecting features detected in the camera feed (programming by example).

The underlying system interprets real-world scenarios, maps user inputs to customizable program templates, and runs live detection using both on-device and cloud-based AI models (like YOLO for objects, native text recognition for text, and more). This eliminates barriers to customization, letting users adapt their technology to evolving needs—whether for routine efficiency or solving unique, situational challenges outside the reach of existing applications.

Unlike conventional assistive technologies, ProgramAlly does not require technical expertise or access to proprietary system settings. It allows users to build, iterate, and share personalized programs—giving them finer control over what information is surfaced, how, and when. The multi-modal approach ensures accessibility for diverse cognitive and technical backgrounds, democratizing AI customization in the accessibility space.

ADDITIONAL INFORMATION

REFERENCES:

"ProgramAlly: Creating Custom Visual Access Programs via Multi-Modal End-User Programming"