This paper presents a signal processing and coding system for processing realistic two-dimensional magnetic recording (TDMR) data generated by a grain-flipping probability model. The data set was generated at the Data Storage Institute, Singapore, and will be referred to as the DSI data. Three types of 2-D intersymbol interference (2-D-ISI) detectors with varying complexity are proposed. The first detector is a two-sided feedback (TSF) detector with known boundaries, which simultaneously detects the three tracks of received data. This detector uses only one of the two samples provided by the reader for each bit. The second detector is a joint TSF (JTSF) detector which uses both samples provided by the reader for each bit. The third detector is a 1-D state-input decision-feedback (ODSIDF) detector which detects the data based on a 2-D-ISI mask. Each of the 2-D-ISI detectors is utilized in a turbo iterative approach with an irregular repeat-accumulate (IRA) decoder in the proposed system. We use a coset coding approach in our IRA decoder for decoding the received data based on a known block of the data. A TDMR log-likelihood ratio (LLR) function is used to pass LLRs from the 2-D-ISI detector to the IRA decoder. The read head sensitivity function (2-D-ISI mask) is estimated based on the first sample alone (for TSF and ODSIDF detectors) and both samples (for JTSF detector) from the reader by using the least squares approach based on known data bits for a given set of reader outputs. The best simulation results show that the proposed signal processing and coding approach can achieve up to 1.7 Tb/in <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</sup> density at 18 nm track pitch (TP). Using a squeeze margin to account for the imperfect tracking of the track center by the read and write heads in a practical system, the proposed system achieves a realistic areal density of 1.25 Tb/in <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</sup> at TP = 18 nm.