This paper initiates the study of the structure of a new class of $p$-Banach spaces, $0<p<1$, namely the Lipschitz free $p$-spaces (alternatively called Arens-Eells $p$-spaces) $\mathcal{F}_{p}(\mathcal{M})$ over $p$-metric spaces. We systematically develop the theory and show that some results hold as in the case of $p=1$, while some new interesting phenomena appear in the case $0<p<1$ which have no analogue in the classical setting. For the former, we, e.g., show that the Lipschitz free $p$-space over a separable ultrametric space is isomorphic to $\ell_{p}$ for all $0<p\le 1$, or that $\ell_p$ isomorphically embeds into $\mathcal{F}_p(\mathcal{M})$ for any $p$-metric space $\mathcal{M}$. On the other hand, solving a problem by the first author and N. Kalton, there are metric spaces $\mathcal{N}\subset \mathcal{M}$ such that the natural embedding from $\mathcal{F}_p(\mathcal{N})$ to $\mathcal{F}_p(\mathcal{M})$ is not an isometry.