An Elementary View on Factorization Machines

Abstract

Factorization Machines (FMs) are a model class capable of learning pairwise (and in general higher order) feature interactions from high dimensional, sparse data. In this paper we adopt an elementary view on FMs. Specifically, we view FMs as a sum of simple surfaces - a hyperplane plus several squared hyperplanes - in the original feature space. This elementary view, although equivalent to that of low rank matrix factorization, is geometrically more intuitive and points to some interesting generalizations. Led by our intuition, we challenge our understanding of the inductive bias of FMs by showing a simple dataset where FMs counterintuitively fail to learn the weight of the interaction between two features. We discuss the reasons, and mathematically formulate and prove a form of this limitation. Also inspired by our elementary view, we propose modeling intermediate orders of interaction, such as 1.5-way FMs. Beyond the specific proposals, the goal of this paper is to expose our thoughts and ideas to the research community in an effort to take FMs to the next level.