This article discusses how to deal with secrecy and limited access in ethnographically inspired research of security fields. Drawing inspiration from recent debates about secrecy in Critical Security Research and from Franz Kafka's The Castle, we propose to treat access limitations and the secrecy we encounter as methodological tools that provide insights into social relations and power structures of security fields. We develop the argument in two steps. First, we argue for a more fine-grained taxonomy of secrecy, that allows to distinguish between mystery, concealment and the relational dimension of secrecy. Second, we apply the taxonomy to our respective fieldwork experiences in the fields of cybersecurity and refugee governance, to show how attending to different forms of secrecy produces empirical insights into the fields of study. Setting out how to work with rather than against secrecy, the article contributes to methodological debates in Critical Security Studies and Secrecy Studies, and ultimately to further cross-fertilize these fields.