Caller ID (caller identification) is a service provided by telephone operators where the phone number and/or the name of the caller is transmitted to inform the callee who is calling. Today, most people trust the caller ID information and some banks even use Caller ID to authenticate customers. However, with the proliferation of smartphones and VoIP, it is easy to spoof caller ID information by installing a particular application on the smartphone or by using service providers that offer Caller ID spoofing. As the phone network is fragmented between countries and companies and upgrades of old hardware is costly, no mechanism is available today to let end-users easily detect Caller ID spoofing attacks. In this article, we propose a new approach of using end-to-end caller ID verification schemes that leverage features of the existing phone network infrastructure (CallerDec). We design an SMS-based and a timing-based version of CallerDec that works with existing combinations of landlines, cellular and VoIP networks and can be deployed at the liberty of the users. We implemented both CallerDec schemes as an App for Android-based phones and validated their effectiveness in detecting spoofing attacks in various scenarios.