The distributed robust optimization problem is investigated in this paper for a class of networked agent systems under stochastically switching communication graphs. Unlike most existing results on distributed optimization of networked agent systems, the dynamics of each agent in the present model are allowed to be subjected to matched linearly-parameterized unknown nonlinearities while the underlying communication graph is stochastically switching among some matrix-weighted communication graphs. The objective here is to cooperatively optimize a holistic cost function, which is a sum of all the convex local objective functions, under the mild condition that each agent may only have access to the gradient of its local objective function and the relative local information between itself and its neighbors. To solve such an outstanding optimization problem, a neuro-adaptive optimization protocol is developed based upon parameterizations of the unknown nonlinearities where the neuro-adaptive learning algorithm for estimating the weight matrices therein acts as the key role to compensate for the unknown nonlinearities. Furthermore, the signum function-based feedback law is subtly integrated into the proposed protocol to deal with the effect of the gradient term of the local objective function. Efficient criteria are provided to guide selections of proper control parameters under which the robust optimization problem is successfully resolved in the mean-square sense. Lastly, theoretical results are validated by presenting several illustrative simulation examples.