Using unmanned aerial vehicles (UAVs) is an effective way to gather data from Internet of Things (IoT) devices. To reduce data gathering time and redundancy, thereby enabling the timely response of state-of-the-art systems, one can partition a network into clusters and perform aggregation within each cluster. Existing works solved the UAV trajectory planning problem, in which the energy consumption and/or flight time of the UAV is the minimization objective. The aggregation scheduling within each cluster was neglected, and they assumed that data must be ready when the UAV arrives at the cluster heads (CHs). This paper addresses the minimum time aggregation scheduling problem in duty-cycled networks with a single UAV. We propose an adaptive clustering method that takes into account the trajectory and speed of the UAV. The transmission schedule of IoT devices and the UAV departure times are jointly computed so that (1) the UAV flies continuously throughout the shortest path among the CHs to minimize the hovering time and energy consumption, and (2) data are aggregated at each CH right before the UAV arrival, to maximize the data freshness. Intensive simulation shows that the proposed scheme reduces up to 35% of the aggregation delay compared to other benchmarking methods.