Switched Affine Systems (SAS׳s) is a class of Hybrid Systems composed of a collection of Affine Systems (AS׳s) and a switching signal that determines, at each time instant, the evolving affine subsystem. This paper is concerned with the observability and observer design for single-input single-output (SISO) SAS׳s under unknown perturbation, for the case that no information about the switching signal is available. It is firstly demonstrated that in the presence of disturbances every pair of AS׳s is always indistinguishable from the continuous output, meaning that it is not possible to infer the evolving AS by using only the information provided by the output of the SAS. Nevertheless, by taking advantage of the knowledge on the disturbance bound, new distinguishability conditions are derived, making possible to distinguish the evolving AS. By using these new distinguishability conditions, an observer scheme for SISO SAS׳s, subject to unknown switching signal and unknown perturbations, is presented. Such an observer scheme determines in finite-time the evolving AS. Furthermore, it estimates both the state of the system and the disturbance. Finally, the proposed observer scheme is effectively applied for a non-autonomous chaotic modulation application, which is an attractive method for spread-spectrum secure communication in which the message is fed as a disturbance to a chaotic SAS and the output is then transmitted through an open channel to a receiver, which is an observer algorithm that recovers the message (the disturbance) from the output signal.