Dissecting the evolution of memory B cells (MBCs) against SARS-CoV-2 is critical for understanding antibody recall upon secondary exposure. Here, we utilized single-cell sequencing to profile SARS-CoV-2-reactive B cell subsets in 42 COVID-19 patients. We isolated thousands of B cells in multiple distinct subsets specific to the SARS-CoV-2 spike, endemic coronavirus (HCoV) spikes, nucleoprotein (NP), and open reading frame 8 (ORF8). Spike-specific cells were enriched in the memory compartment of acutely infected and convalescent patients 1.5–5 months post-infection. With severe acute infection, we identified substantial populations of endemic HCoV-reactive antibody-secreting cells with highly mutated variable genes, indicative of preexisting immunity. Finally, MBCs exhibited maturation to NP and ORF8 over time relative to spike, especially in older patients. Monoclonal antibodies against these targets were non-neutralizing and non-protective in vivo. These findings reveal considerable antibody adaptation to non-neutralizing antigens during infection, emphasizing the importance of vaccination for inducing neutralizing spike-specific MBCs.Trial Registration Number: This clinical trial was registered at ClinicalTrials.gov with identifier NCT04340050, and clinical information for patients included in the study is detailed in Table S1–S3.Funding: This project was funded in part by the National Institute of Allergy and Infectious Disease (NIAID); National Institutes of Health (NIH) grant numbers U19AI082724 (P.C.W.), U19AI109946 (P.C.W.), U19AI057266 (P.C.W.), the NIAID Centers of Excellence for Influenza Research and Surveillance (CEIRS) grant numbers HHSN272201400005C(P.C.W.). N.W.A. was supported by the Multi-disciplinary Training program in Cancer Research (MTCR) - NIH T32 CA009594. A.J. and R.P.J were supported by federal funds from the NIAID, NIH, and Department of Health and Human Services under Contract HHSN272201700060C. F.K and F.A. were funded by the NIAID CEIRS contractHHSN272201400008C, Collaborative Influenza Vaccine Innovation Centers (CIVIC) contract 75N93019C00051 and the generous support of the JPB foundation, the Open Philanthropy Project (#2020-215611) and other philanthropic donations. Y.K. and P.H.were funded by the Research Program on Emerging and Re-emerging Infectious Disease grant (JP19fk0108113) and the Japan Program for Infectious Diseases Research and Infrastructure (JP20fk0108272) from the Japan Agency for Medical Research and Development (AMED), NIAID CEIRS contract HHSN272201400008C, and CIVIC contract 75N93019C00051. D.F., C.N, Y.D., and P.D.H, were supported by NIAID contracts HHSN272201700060C and 75N93019C00062. M.S.D. and E.S.W. were supported by NIH grants R01 AI157155 and F30 AI152327, respectively.Conflict of Interest: Several antibodies generated from this work are being used by Now Diagnostics in Springdale, AR for the development of a diagnostic test. M.S.D. is a consultant for Inbios, Vir Biotechnology, NGM Biopharmaceuticals, and Carnival Corporation, and on the Scientific Advisory Boards of Moderna and Immunome. The Diamond laboratory has received funding support in sponsored research agreements from Moderna, Vir Biotechnology, and Emergent BioSolutions.Ethical Approval: All studies were performed with the approval of the University of Chicago institutional review board IRB20-0523 and University of Chicago, University of Wisconsin-Madison, and Washington University in St. Louis institutional biosafety committees. Informed consent was obtained after the research applications and possible consequences of the studies were disclosed to study subjects.