The building sector is a significant energy consumer, making building energy optimization crucial for reducing energy demand. Automating energy optimization tasks eases the workload on engineers and hastens energy savings. More than 85% of building data is unstructured and diverse, concealing energy insights that demand laborious extraction. We propose an LLM-based multi-agent framework to explore automated tasks using these data. The framework includes three stages: building information processing, performance diagnosis, and retrofit recommendation, where LLMs injected with domain expertise act as agents for the roles of planner, researcher and advisor. We develop knowledge databases with retriever tools to inject knowledge and validate through experiments. In case studies, our framework delivered reliable results with only $5.15, effectively handling diverse inputs and tasks across cases. This demonstrates its potential to significantly reduce repetitive human labor and costs. We also discuss the potential of LLM-based multi-agent systems as trustworthy, generalized automated task solvers.