Buckwheat hull (BWH) is a source of insoluble dietary fiber (IDF), which can be added to starch-based foods to increase their nutritional characteristics. The influence mechanism of different amounts (10, 20, 30 g/hg) of BWH at cell-scale (CBH, 10–50 μm) and tissue-scale (TBH,100–500 μm) on the physiochemical and textural properties of steamed rice cake was analyzed. With the decrease of BWH particle size and the increase of addition amount, the water retention capacity of rice flour-BWH mixture significantly (P < 0.05) increased. Compared with TBH, the addition of CBH significantly (P < 0.05) increased the viscosity of rice cake batter. The 20 g/hg and 30 g/hg addition of TBH significantly (P < 0.05) decreased the specific volume of rice cake, while that of CBH reversed the negative effects on the specific volume. Texture properties showed that the hardness of cake added with CBH was significantly (P < 0.05) lower than TBH. CBH with higher water affinity and stronger physical support enhanced the interactions of water-starch-IDF and the starch gel network of cake, thus leading to an open and fluffy cake structure. Therefore, milling the IDF to cell-scale might be a promising alternative for developing dietary fiber-enriched starch-based products.