The first finished version of the painting included Madame Bloch and her three children, Giselle, Thierri, and Claude, with their nanny, Francine Damon (known as Douce Nounou), at far left. When a fourth child, Agnès, was born in 1928, and Vuillard was invited back to paint a second version, similar in composition to the first. In addition to including Agnès who may be seen in the lower right corner of the second version, the nanny is more visible in the second version.
Vuillard worked on the versions of the portrait for several years. In the later one the children are discernibly a little older and the youngest (a newborn baby when the first version was painted), looks about five. The style in the later work is a little less brushy, and details are established with greater precision.
Vuillard retained the first version of the portrait, which his estate sold after his death. The second version hung in Jean Bloch's Avenue Matignon office for about ten years until the invasion of France by Germany in World War II. In December 1941 Bloch was arrested by the Nazis in an infamous round-up of 743 prominent Jews—intellectuals and professionals were especially targeted. They were interned in France for a time and in 1942 were deported to Auschwitz, where they died. At the time of Bloch's arrest, his wife and children were still in Paris and sought his release, but without success. They made preparations to leave the city for the relatively greater safety of rural France, but Gilberte Bloch first visited her husband's former office, which had been commandeered by the Germans. Her daughter Claude recalls:
"My father had decorated his office in Avenue Matignon with paintings by celebrated contemporary artists. Maman steeled herself to go meet the Germans who had moved in there. . . . She had forgotten the German learned from a nanny in her childhood but, pressed by necessity, she managed to ask permission to take away the paintings. The occupants, who must not have been connoisseurs, were concerned only to have the works replaced by others, so that their removal wouldn't be noticed. . . . So we went to Montmartre, where Sunday painters sold their mediocre daubs; we bought some of these and the exchange was made. Maman had taken some risk in making the request."
The family portrait was left with friends, and Gilberte and her four children fled Paris soon thereafter. They survived the war in hiding in the countryside of France and recovered it upon their return to Paris after the Liberation.