Artificial intelligence has helped to recreate the lost pieces of Rembrandt’s famous painting, The Night Watch, and it can now be seen in its entirety for the first time in 300 years.
Image: Rijksmuseum/Reiner Gerritsen
Painted in 1642, the Militia Company of District II under the Command of Captain Frans Banninck Cocq – better known as The Night Watch – is considered Rembrandt Harmenszoon van Rijn’s most ambitious work.
The painting depicts a group of Amsterdam’s civic guard and Captain Frans Banninck Cocq, who also commissioned the painting. But it was unlike the usual Baroque military portraits, which were static and featured glum looking fellas sitting at a banquet or lined up in neat rows staring out at the viewer.
Instead, it was lively, brimming with the energy and noise of militiamen readying to march into action. It told a story – the painter’s equivalent of a snapshot – and was, in fact, the first portrait to see figures in a group actually doing something. The piece was also impressive for its size – approximately 13 ft tall by 15 ft wide – and dramatic contrast of light and shade.
When the painting was moved to what was then Amsterdam’s City Hall in 1715, it was trimmed to fit between two doors. Strips were cut from all four sides, mostly from the left side which lost 60 cm (2 ft). Another 7 cm was snipped from the right, 22 cm from the top and 12 cm from the bottom. These pieces have never been found.
Thanks to the Rijksmuseum in Amsterdam – where the original painting is displayed – the lost parts of the painting have been recreated using artificial intelligence. This is part of a large-scale research project that the museum set up called Operation Night Watch, which began in 2019 and is dedicated to the long-term preservation of the artwork.
“The fate of the missing pieces of The Night Watch remains a great mystery,” says Pieter Roelofs, Head of Paintings and Sculptures, Rijksmuseum. “Each generation has used the tools available to it to attempt to reconstruct the painting. Now we are doing the same, using the most advanced techniques currently available.”
To do this, the Rijksmuseum used high-resolution photography, machine learning, and a copy of the painting by Gerrit Lundens. On loan from the National Gallery of London, Lundens’ replica was painted before Rembrandt’s original was cut and therefore shows the work in full.
However, Lundens’ copy is one-fifth of the size of the original. As well as painting in a different style to Rembrandt, he also used different paint mixes. There are also distortions in perspective likely caused by him sitting on the left of the painting as he worked, and his version has aged differently to the original.
The Operation Night Watch team used an incredibly detailed, high-resolution image of the Rembrandt, and convolutional neural networks (CNN) – an AI algorithm that trains computers to recognise patterns and visual features in images using a multi-layer system.
“We made three types of algorithms,” says Taco Dibbits, Director of the Rijksmuseum. “One to identify perspective distortions and correct them; the second to recognise the colour scheme of the original and project it onto the missing pieces; and the third is the brushstrokes, the technique that Rembrandt used.” The result is a computer reproduction that looks as much like The Night Watch as possible.
The new panels, which are printed on canvas and now frame the original painting, reveal three additional figures on the left-hand side and a complete helmet (worn by the militiaman) on the far right. The composition has also shifted, so that the main figures are now more right of centre, rather than in the middle of the canvas.
“This project testifies to the key importance of science and modern techniques in the research being conducted into The Night Watch. It is thanks to artificial intelligence that we can so closely simulate the original painting and the impression it would have made,” says Robert Erdmann, Senior Scientist at Rijksmuseum.
This isn’t the first time that CNN has been applied to Rembrandt’s work. Back in 2016 the Dutch bank, ING Group utilised it – along with Microsoft and a team of 20 data scientists, developers, AI and 3D printing experts – to create a 3D printed Rembrandt painting, a kind of new old master, in a project called The Next Rembrandt.
The Night Watch and its once missing pieces now recreated by AI are on show at the Rijksmuseum for three months. You can watch a short video of how it was reconstructed here.