Despite the importance of remote detection of marine debris, nearly all published studies are focused on either controlled experiments, or Sentinel-2 data with mixed band resolutions that are subject to large uncertainties. To date, key questions such as the following have not been addressed adequately: To what extent can the various forms of marine debris be remotely detected and differentiated through satellite observations in the visible and near infrared (NIR) spectral range, and how? Here, using published reflectance spectra of various types of floating matters, I address these questions through sensitivity analyses, simulations, and spectral analyses of satellite images. While the study is by no means comprehensive, several observations can still be made. First, it appears impossible to remotely detect marine microplastics from all existing and planned optical sensors. This is simply because the contribution of these particles to the sensor signal, even when they are aggregated on the water surface at the reported maximum particle density, is at least 60 times lower than the required signal (~0.2% subpixel coverage) and 20 times lower than the sensor noise for a sensor with a signal-to-noise ratio (SNR) of 200. In contrast, detecting macroplastics and other debris is possible when they form large patches along ocean fronts or windrows. Second, assuming a SNR of 200, discriminating large patches of marine debris from floating algae is only possible with a subpixel coverage of >0.3%. These threshold values are based on the sensor SNRs only, and they represent the lower bounds of detection and discrimination, respectively. The real threshold values above which a detection or discrimination is possible also depend on the observing conditions, and therefore higher. Third, currently, Sentinel-2 MSI (Multi Spectral Instrument) sensors provide an optimal trade between resolution and coverage, yet MSI sensors have SNRs <200, and interpretation of the MSI spectra requires extra caution due to variable spatial resolutions in different bands, among other factors. From the perspective of pure spectroscopy, it is possible to discriminate floating algae from non-algae floating matters but difficult to differentiate the type of the latter (either plastic or non-plastic debris, foam, etc) because different non-algae floating matters all show relatively flat reflectance spectral shapes in the vis-NIR spectral range. Finally, based on these results, recommendations are made on algorithm designs and sensor designs, for example spectral analysis should be performed over the difference spectra to minimize the impact of variable subpixel coverage, and certain spectral bands are more important than others for the remote detection of marine debris.