Using blue and fin whale calls for estimating population density of the two species from passive acoustic data is an active research topic. However, manually analyzing long-term data is extremely time consuming yet a necessary first step for such work. Blue and fin whale social calls are highly variable and quite similarity, which has resulted in challenges in developing automatic detectors for these calls. The applicability of faster region-based convolutional neural network (Faster R-CNN) method for speeding up this detection and classification task was explored. A large dataset of blue whale D calls and fin whale 40 Hz calls from southern California was used for training the network: 1378 spectrograms were created from 10-s sound clips each containing a call. The resulting images were contrast-enhanced and manually labeled with region of interest (ROI) before being applied as training images. The probability score for each ROI was modified to favor detections within previously measured frequency range of the calls. Testing shows this Faster R-CNN to have a very low miss- and false positive rate for both call types and is thus a highly promising tool for detecting and classifying these baleen whale social calls.