Maternal diet may affect human milk macronutrients, but it remains to be elucidated whether this is also influential in infant growth. This study aimed to examine (1) how maternal diet influences human milk macronutrients, and (2) to what extent the variation in milk macronutrients affects infant growth during the first month of life. In 71 Japanese lactating women, maternal dietary information was collected from the brief-type self-administered diet history questionnaire, and anthropometry of mother-infant dyads was collected from medical records. Macronutrients in milk were analyzed by a Human Milk Analyzer. Maternal retinol intake was associated with the carbohydrate content in human milk at 1-month postpartum (standardized β coefficient: 0.287; p = 0.038). Moreover, the energy content in human milk was associated with an increase in the weight standard deviation score based on the WHO growth standard at 1 month of age (standardized β coefficient: 0.399; p = 0.046). Nevertheless, the milk macronutrient was not associated with the risk of infant growth abnormalities. In conclusion, a part of the maternal diet impacts macronutrient contents in human milk, but milk macronutrients have a limited effect on infant growth only within the normal growth curve during the first month of life.