Physical activity during pregnancy is beneficial for the woman and the fetus. However, non-objective methods are often used to measure physical activity levels during pregnancy. This study aimed to evaluate objectively measured maternal early to mid-pregnancy sedentary behavior and physical activity in relation to infant well-being. This cohort study included 1153 pregnant women and was performed at Uppsala University Hospital, Uppsala, Sweden, between 2016 and 2023. Sedentary behavior and physical activity levels were measured by accelerometers during 4-7 days in early to mid-pregnancy. Outcome measures were infant birthweight standard deviation score, small-for-gestational-age, large-for-gestational-age, preterm birth (<37 weeks' gestation), spontaneous preterm birth, iatrogenic preterm birth, Apgar <7 at 5 min of age, umbilical artery pH ≤7.05, and admission to the neonatal intensive care unit (NICU). There were no associations of sedentary behavior and physical activity levels with infant birthweight standard deviation score, small-for-gestational-age, or large-for-gestational-age. After adjustment for BMI, age, smoking, parity, maternal country of birth, and a composite of pre-pregnancy disease, the most sedentary women had higher odds of preterm birth (adjusted odds ratio (AOR) 2.47, 95% confidence interval (CI) 1.17-5.24, p = 0.018), and NICU admission (AOR 1.93, CI 1.11-3.37, p = 0.021) than the least sedentary women. The most physically active women had lower adjusted odds for NICU admission (AOR 0.45, CI 0.26-0.80, p = 0.006) than the least physically active women. Objectively measured levels of sedentary behavior and physical activity in early to mid-pregnancy were not associated with standardized infant birth size. Sedentary behavior was associated with an increased likelihood of preterm birth and NICU admission, while high level of physical activity was associated with a decreased likelihood of admission to NICU.