The advances in the optical space surveillance technology and the possibly more deployments of space-based optical sensors enable more and more space objects detectable and make the catalog expansion more realistic. The cataloging of an optically detected new object requires finding three or more tracklets over a few days from the object. In this process, the tracklet-to-tracklet association is a critical step, but associating short-arc optical tracklets remains a difficult task. This paper develops a novel orbit determination approach to the optical tracklet association. The main idea is to determine accurate orbit elements from two tracklets of the same object by exclusively using the Gooding angles-only initial orbit determination (IOD) method. Before applying the Gooding IOD method to two tracklets apart by a few days, the dominant secular effects on the angular observations have to be considered. The significant short-periodic effects may also need consideration if tracklet-to-tracklet IOD is to achieve high accuracy. In this paper, the secular effects from the J2 and lunisolar gravitational perturbations, as well as the short-periodic effects due to J2 and J2,2, are accounted for in the tracklet-to-tracklet IOD. Extensive tests using tracklets from low Earth orbit (LEO), high elliptical orbit (HEO), medium Earth orbit (MEO), geosynchronous Earth orbit (GEO), and Molniya orbit are performed to assess the performance of the developed method. It is shown that the true positive (TP) rate for associating two tracklets from the same object within 48 h is 98.3% for LEO objects and higher than 89.8% for HEO and Molniya objects. The TP rate is remarkably high at around 99.0% for MEO and GEO objects. The analytic approach is also computationally very efficient. The results demonstrate the strong applicability of the developed method to associate optical tracklets of uncatalogued objects of all orbit types.