This article deals with Pieter de Hooch’s enigmatic painting Woman holding a basket with beans in a garden of 1661 (Kunstmuseum, Basel), and, more specifically, with the identity of the man depicted on the window shutter of the building in the foreground of the painting. The author argues that this portrait, which was hidden behind a layer of paint for a long time and only uncovered sometime between 1913-1927, represents the Emperor Charles V, whose portrait still decorates many buildings in the Netherlands. It is also argued that the building in the background of the painting can be typified as a ‘hofje’ or almshouse, and that its architecture resembles that of the Leiden Eva van Hoogeveenshofje, built by the architect Arent van ‘s-Gravesande in the 1650s. This suggests that De Hooch did not only paint locations in Delft and Amsterdam, but also in Leiden. The Dutch Republic’s seventeenth-century ‘hofjes’ were renowned, eliciting praise from foreign visitors, not only because of the way the Dutch Calvinist elite took care of the needy but also because of their superb architecture that enhanced the beauty of the Republic’s cities. Interestingly, De Hooch painted a servant girl in the garden of the almshouse, which at first may seem incongruous with an institution intended for the poor. However, as will be shown, residents of almshouses in the seventeenth-century Dutch Republic were not as poor as is generally thought. Some were even quite well-off and did indeed keep servants. In fact, some contemporaries even likened the Dutch almshouses to palaces. The portrait of the emperor on the window shutter may thus have criticised this type of ‘poor relief’ and could well have been intended to bring out the resemblance between the ‘hofje’ and an emperor’s palace. The word ‘hofje ‘ after all means ‘small court’. As such an overt criticism is unique in De Hooch’s oeuvre, it may have been the artist himself who blotted the emperor’s portrait out, thus changing the image into a straightforward genre scene. It was precisely this type of scene that was to bring De Hooch some success following his move to Amsterdam in 1660.