Non-perturbative features of the derivative expansion of the effective action of a single D3-brane are obtained by considering scattering amplitudes of open and closed strings. This motivates expressions for the coupling constant dependence of world-volume interactions of the form $(\partial F)^4$ (where F is the Born-Infeld field strength), $(\partial^2\phi)^4$ (where $\phi$ are the normal coordinates of the D3-brane) and other interactions related by $\calN=4$ supersymmetry. These include terms that transform with non-trivial modular weight under Montonen-Olive duality. The leading D-instanton contributions that enter into these effective interactions are also shown to follow from an explicit stringy construction of the moduli space action for the D-instanton/D3-brane system in the presence of D3-brane open-string sources (but in the absence of a background antisymmetric tensor potential). Extending this action to include closed-string sources leads to a unified description of non-perturbative terms in the effective action of the form (embedding curvature)$^2$ together with open-string interactions that describe contributions of the second fundamental form.