Heart rate variability (HRV) reflects the regulating effect of autonomic nerve on cardiovascular functions and is strongly influenced by respiratory rhythm. For more convenient monitoring physiological or pathological information about heart and autonomic nerves during sleep than conventional systems, a smart mattress that could measure electrocardiogram (ECG) and respiratory signals through clothes was designed. The system mainly consists of three electrodes that are made of flexible silver fiber conductive fabric, a signal preprocessing module and a user interface. To reduce the effect of surrounding electromagnetic radiation on the electrodes, two equipotential shielding layers are placed under the two active electrodes, respectively. To verify the proposed system, experiments in four sleeping postures (supine, left lateral, right lateral, and prone) were carried out. The results showed that the proposed system could successfully measure ECG signals and extract respiratory signals from them in all sleeping postures. Finally, to study respiratory sinus arrhythmia during sleep, an experiment was conducted to measure ECG and respiratory signals of 7 subjects under unconstrained sleep over night. Then, root mean square of successive differences (RMSSD) and peak-valley respiratory sinus arrhythmia (pvRSA) that can reflect the vagus nerve tension were calculated. The results showed that all subjects had varying degrees of respiratory sinus arrhythmia, with the mean and standard deviation of $176 \pm 80\textit {ms}$ and $288 \pm 126\textit {ms}$ , for the RMSSD and the pvRSA respectively. The RMSSD and the pvRSA of the subject with most obvious arrhythmia were $346\textit {ms}$ and $552\textit {ms}$ respectively. The results also indicate the potential application of the proposed system in detecting sleep diseases such as sleep apnea.