The prediction of stress, strain fields or fracture patterns, along with other physical quantities as a function of boundary conditions and material microstructure, is an important objective of various physics-based simulation methods like finite element models or molecular dynamics modeling. However, such simulations may be expensive and can be costly in their execution, limiting their application to solve inverse design problems, or in the discovery of novel materials that feature a desirable performance. In other fields of studies, such as in mechanobiology, field data can often not be calculated due to a lack of models to begin with; where experimental observations may be the only source of insight. Here we introduce a data-driven method to relate input microstructures and field data using an alternative approach, enabled by deep learning, to predict complex mechanical fields directly based on input microstructure data. We report two approaches to solve said problem, first using cycle-consistent adversarial neural networks trained on unpaired images of microstructures and physical fields, and second, using a Perceiver neural networks based on an attention mechanism based on paired images, implemented in a transformer neural network. Due to the cycle consistent nature of the GAN model we can immediately make forward (from microstructure to stress field) and backward (from stress field to microstructure) predictions, offering an avenue for inverse material design. The Perceiver model, implementing an efficient transformer model that can be scaled to very large input and output data, shows strong generalizability and can be successfully applied to scenarios distinct from those in the training set. We demonstrate the model in several sample microstructures including a set of small cracks (voids) to predict both stress fields and fracture patterns, as well as in a composite microstructure of a soft matrix and rigid inclusions, and in a series of solutions to the inverse problem facilitated by the cycle consistent GAN model. The applications of the models include biomimetic biotechnology (e.g. bone-tendon-inspired attachments and gradients), armor materials, mechanobiology, soft robotics, and lightweight resilient and tough structural systems.