Private browsing has been a popular privacy feature built into mainstream browsers since 2005. However, despite the prevalent use, the security of this feature has received little attention from the research community. To the best of our knowledge, no study has existed that systematically evaluates the security of private browsing across major browsers and from multiple angles: not only examining the memory, but also the underlying database structure on the disk and the web traffic. In this paper, we present an up-to-date and comprehensive analysis of private browsing across the four popular web browsers: IE, Firefox, Chrome and Safari. We report that all browsers under study suffer from a variety of vulnerabilities, many of which have not been reported or known before. The problems are generally caused by the following factors: a lax control of permission to allow extensions to run in the private mode with unrestricted privilege; inconsistent implementations of the underlying SQLite database between the private and usual modes; the neglect of the cross-mode interference when the two modes are run in parallel; a lack of attention to side-channel timing attacks, etc. All of the attacks have been experimentally verified with countermeasures proposed.