Context-based authentication has been proposed as a way to enable secure authentication with minimal or even no user interaction requirements by using sensor data to ensure the device being authenticated is in possession of the person initiating the authentication request. A key limitation of practically all context-based authentication systems is that they are vulnerable to context manipulation attacks where an attacker manipulates the environment to create a desired response in the sensor data. We contribute Chirp-Loc as a system that has been designed to improve the robustness of context-based authentication solutions against context-manipulation attacks. Chirp-Loc integrates an innovative approach based on room impulse response (RIR) to establish a location fingerprint that characterizes the physical environment instead of the ambient environment. We describe the design and development of an Android prototype of Chirp-Loc. We also conduct extensive accuracy and security analysis of Chirp-Loc by considering a multi-factor authentication solution that uses Chirp-Loc to verify the proximity of an authentication token, such as a smartphone. Through extensive experiments, we demonstrate that Chirp-Loc offers high degree of security and usability. Our work paves the way for improving the resilience of context-based authentication against attackers that manipulate the context information and offers a way to implement authentication systems that minimize user interaction demands while offering a high degree of security.