Recently, factorization machine and its variants have shown promising results for context-aware recommender systems (CARS), especially when combined with deep neural networks. Among them, convolutional factorization machine (CFM) is a prominent example. The key to the success of CFM is its 3D convolutional architecture for capturing complex interactions on top of embedded features. However, the resultant computational cost can also be demanding. Moreover, the feature embedding scheme of CFM and other factorization models can be potentially vulnerable to noise. To tackle these issues, in this study we propose two models, namely, the fast convolutional factorization machine (FCFM) that slims down the complete pairwise feature interaction for higher computational efficiency, and adversarial fast convolutional factorization machine (AFCFM) that further enhances the robustness of the model by introducing adversarial noise to the feature interaction image generated by the model. Experimental results on four benchmark datasets prove that the proposed FCFM is nearly five times faster than CFM with competitive performance, while AFCFM improves the performance of the state-of-the-art models by about 8\% with higher efficiency than CFM.