The study of very large graphs is a prominent theme in modern-day mathematics. In this paper we develop a rigorous foundation for studying the space of finite labelled graphs and their limits. These limiting objects are naturally countable graphs, and the completed graph space $\mathscr{G}(V)$ is identified with the 2-adic integers as well as the Cantor set. The goal of this paper is to develop a model for differentiation on graph space in the spirit of the Newton-Leibnitz calculus. To this end, we first study the space of all finite labelled graphs and their limiting objects, and establish analogues of left-convergence, homomorphism densities, a Counting Lemma, and a large family of topologically equivalent metrics on labelled graph space. We then establish results akin to the First and Second Derivative Tests for real-valued functions on countable graphs, and completely classify the permutation automorphisms of graph space that preserve its topological and differential structures.