This study was focused on the preparation of novel multi-stimuli-responsive semi-interpenetrating polymer networks (semi-IPN) hydrogels by cross-linking polymerization of N,N-dimethylaminoethyl methacrylate (DMAEM) in the presence of potato starch (PS) or anionically modified PS by using ice-templating strategy (−18°C). Cryogelation was used for this aim because it allowed decreasing the monomer concentration to 10wt.% and a very fast response at swelling of the composite gels. Furthermore, the anionically modified PS allowed the decrease of the volume phase transition temperature (VPTT) of the composite cryogels by its electrostatic interactions with the matrix. After the characterization of the composite structure by Fourier transform infrared spectroscopy and of the macroporous morphology by scanning electron microscopy analysis, the effect of the entrapped polysaccharide on the swelling properties of the composite cryogels, and their response at three stimuli (pH, temperature and ion nature and concentration) were deeply investigated. The VPTT of the composite cryogels was situated in the range 36–39°C, the particular value depending on the DMAEM concentration and on the nature and content of polysaccharide. After that, diclofenac sodium (DS), taken as a model acidic drug, was easily loaded into the composite cryogels by the solvent sorption–evaporation strategy due to the interconnected pores of these materials. The controlled delivery of DS from the novel semi-IPN composite cryogels was then optimized by the investigation of the effects of pH, temperature, and cycling changes of the release temperature. It was found that these composite cryogels are promising systems for the sustained delivery of DS in the simulated intestinal fluid, but not in the simulated gastric fluid, and this recommends them as vehicles of drugs in colon. Finally, the release mechanism of DS from the composite cryogels was discussed based on two kinetic models.