Voice-first ambient interfaces (VFAIs), such as Alexa, can uniquely meet the health needs of older adults. However, inequitable technology may worsen health disparities and decrease independence, calling for participatory methods to increase the agency of older adults in the design processes of these technologies. We adapt and conduct a participatory design workshop to focus on ambient interfaces for home health with 13 diverse older adults in San Francisco's Tenderloin neighborhood. Using the prototypes they made as discussion catalyzers, participants shared different perspectives entailing stigmatized topics that can be difficult to discuss, such as drug use, sex, isolation, and dementia. They deliberated on the negative implications of VFAIs, such as a justified concern for surveillance, in conjunction with their positive implications, such as receiving always-available ''non-judgmental'' support. Similarly, the risk of leaking drug use data was considered alongside the benefits of sharing important medical information with clinicians. We synthesize our findings into design considerations, such as how we might address varying levels of trust in different stakeholders and reduce stigma that may hinder users from fully benefiting from VFAIs' capabilities.