Color change is the most obvious characteristic of the tomato ripening stage and an important indicator of the tomato ripening condition, which directly affects the commodity value of tomato. To visualize the color change of tomato fruit during the mature stage, this paper proposes a gated recurrent unit network with an encoder–decoder structure. This structure dynamically simulates the growth and development of tomatoes using time-dependent lines, incorporating real-time information such as tomato color and shape. Firstly, the .json file was converted into a mask.png file, the tomato mask was extracted, and the tomato was separated from the complex background environment, thus successfully constructing the tomato growth and development dataset. The experimental results showed that for the gated recurrent unit network with the encoder–decoder structure proposed, when the hidden layer number was 1 and hidden layer number was 512, a high consistency and similarity between the model predicted image sequence and the actual growth and development image sequence was realized, and the structural similarity index measure was 0.746. It was proved that when the average temperature was 24.93 °C, the average soil temperature was 24.06 °C, and the average light intensity was 11.26 Klux, the environment was the most suitable for tomato growth. The environmental data-driven tomato growth model was constructed to explore the growth status of tomato under different environmental conditions, and thus, to understand the growth status of tomato in time. This study provides a theoretical foundation for determining the optimal greenhouse environmental conditions to achieve tomato maturity and it offers recommendations for investigating the growth cycle of tomatoes, as well as technical assistance for standardized cultivation in solar greenhouses.