Neuromorphic photonic integrated circuits over silicon photonic platform have recently made significant progress. Photonic neural networks with a small number of neurons have demonstrated important applications in high-bandwidth, low latency machine learning (ML) type signal processing applications. Naturally an important topic is to investigate building a large scale photonic neural networks with high flexibility and scalability to potentially support ML type applications involving high-speed processing of a high volume of data. In this paper we revisited the architecture of microring resonator (MRR) -based non-spiking and spiking photonic neurons, and photonic neural networks using broadcast-and-weight scheme. We illustrate expanded neural network topologies by cascading photonic broadcast loops, to achieve scalable neural network scalability with a fixed number of wavelengths. Furthermore, we propose the adoption of wavelength selective switch (WSS) inside the broadcasting loop for wavelength-switched photonic neural network (WS-PNN). The WS-PNN architecture will find new applications of using off-chip WSS switches to interconnect groups of photonic neurons. The interconnection of WS-PNN can achieve unprecedented scalability of photonic neural networks while supporting a versatile selection of mixture of feedforward and recurrent neural network topologies.