APPLICATION EXAMPLES
Authenticity Analysis
This application is based on an ensemble of five convolutional neural networks (CNNs) trained to recognize at least 40 artists from different eras. In accordance with the mission and the values of Art Intelligence GmbH, the accuracy of the individual CNNs as well as that of the ensemble are disclosed in the analysis - for all painters with whom these networks were trained.
The number of works by various artists available to Art Intelligence GmbH varies greatly. For example, fewer than three dozen works by Carel Fabritius , probably Rembrandt 's most talented pupil, have survived. The exact number is unclear as there is great disagreement among art historians as to the authorship of some of the works. By contrast, there are 1876 pictures by Vincent van Gogh in the data set.
The different number of images reflects the accuracy of the model for the respective artists. There is a logarithmic relationship here, which is shown in Figure 1.
Figure 1: Accuracy of the model in artist attribution versus the percentage of individual artist images in the data set.
For the authenticity analysis, it is not decisive whether a specific picture is assigned to a specific painter in the painter classification or not. Rather, it is decisive how strongly the model recognizes the style of the presumed artist in the picture. For this reason, Art Intelligence GmbH uses a modified evaluation procedure for checking the authenticity, which is based on the so-called SoftMax Output of the ensemble for the respective artist.
In order to understand this procedure, it is important to classify the output of the model correctly. It is a vector whose dimension corresponds to the number of painters with which the model or models were trained. Each entry of this vector is a real number between 0 and 1, calculated with the so-called SoftMax function . The higher the value, the more the model recognizes the style of the painter located at that point in the vector.
The SoftMax output should not be confused with the probability that it is actually a work by the assumed artist - even if the size is interpreted in this way by numerous AI practitioners. From the point of view of Art Intelligence GmbH, the values can only be used as an indicator of whether it is a painting by the assumed artist or not.
For the authenticity analysis, Art Intelligence GmbH not only compares the softmax output of the picture to be tested with that of other originals, but also with counterfeits or imitations of the respective artist. On this basis, the model is then evaluated - in relation to the question of how well counterfeits by the artist are recognized.
Specifically, this means: The conditional probabilities are determined that
-
the test result is positive if it is a fake, and
-
the test result is negative if it is an original.
This more in-depth evaluation of the model is shown in Figure 2 for works and imitations by Vincent van Gogh.
Figure 2: SoftMax output for originals (blue) and counterfeits or imitations (red) by the artist Vincent van Gogh.
For the assessment of an image to be tested, in which it is not clear whether it is an original or a forgery, exactly the opposite probabilities are required, i.e
the probability that the image is an original if the test result is negative, or
the probability that it is a fake if the test result is positive.
These probabilities, which are decisive for the authenticity analysis, are calculated using Bayes' theorem .
The result is then also visualized using so-called Class Activation Maps (CAMs) - both for the suspected painter and for all painters in the data set. CAMs make it possible to identify the places in an image where the model recognizes a painter's style or how strongly the model perceives a painter's style at a certain point. Examples of this can be found in Figures 3 and 4.
Figure 3: Original, heatmap and heatmap overlaid on the original for Vincent van Gogh's style (from left).
Figure 4: Overlaid heatmap for the style of other painters used to train the model. The red area at the bottom for Salvador Dali's style is an artifact resulting from the pre-processing of the image.
The redder the marked areas appear, the more the model recognizes the style of the respective painter. From Figure 4 it is clear that the model in the present test image also recognizes the style of other painters, in particular Claude Monet - although not nearly as pronounced as that of van Gogh and not so much in the central areas.
The hypothesis on which the authenticity check is based is that the style of a painter can be found in all areas of a picture. In fact, there are differences. A painter's style tends to be more pronounced in key areas, simply because artists spend more time and effort on these more important areas. The edge areas, on the other hand, are often not fully worked out.
Face Recognition
With the help of artificial intelligence, it is also possible to identify faces. Until now, Conventional Neural Networks have been trained almost exclusively on photos of people. However, Art Intelligence GmbH has extended the methods to paintings - and carried out numerous tests.
Adele Bloch-Bauer , for example, is the only person Gustav Klimt has portrayed twice. The two paintings are shown in Figure 5.
Figure 5: "Adele Bloch-Bauer I", also known as "The Golden Adele", 1907, Neue Galerie, New York (left). "Adele Bloch-Bauer II", 1912, private collection.
With a pre-trained algorithm, the faces are first extracted in the process. Then another pre-trained model is used to mark prominent recognition points on the faces. Using these recognition points, which resemble biometric features, the similarity between two faces is then determined, for example with the so-called Procrustes distance - and finally compared with 42 other faces extracted from Klimt paintings. The result is shown in Figure 6.
Figure 6: Face details including landmarks from the paintings "Adele Bloch-Bauer I" (left), "Adele Bloch-Bauer II" (middle) and from the portrait of Fritza Riedler (right).
Of the faces painted and extracted by Klimt, only one, namely the portrait of Fritza Riedler , bears a comparable resemblance to the painting "Adele Bloch-Bauer I" as does the second portrait of Adele Bloch-Bauer.
The woman in Leonardo da Vinci's famous “ Mona Lisa ” has also been painted twice. To this day it is unclear or disputed who Leonardo portrayed.
Figure 7: Leonardo da Vinci, "La Gioconda", around 1503/4, Louvre, Paris (left), Raphael Sanzio, "Pen and Ink Sketch of a young woman on a balcony", Louvre, Paris.
The sitter is probably Lisa del Giocondo , the wife of the Florentine cloth merchant Francesco di Bartolomeo di Zanobi del Giocondo . Raphael Sanzio de Urbino sketched the picture in Florence around 1504 in da Vinci's workshop, where he worked for a short time as a student. The two images can be seen in Figure 7.
Raphael's work is considered by art experts as evidence that Leonardo painted the Mona Lisa twice or significantly modified the work in later years.
The resemblance of the two faces is clearly detectable - based on both 51 and only five landmarks in the faces. For the test, all faces were extracted from the da Vinci paintings by Art Intelligence GmbH. In both cases, the landmarks in the face sections of Leonardo's and Raphael's works were not corrected. They are shown in Figure 8, along with the faces that were incorrectly detected.
Figure 8: Face sections of Leonardo da Vinci, "La Gioconda" and Raphael Sanzio, "Pen and Ink Sketch of a Young Woman on a Balcony" (1st and 2nd from left) and incorrectly recognized faces of da Vinci with five landmarks ( above) and 51 (below).
The method can also be applied to portraits of the same person by different artists. So are two representations of Dr. Narrated by Nicolaes Tulp (1593 – 1674). The Dutch surgeon and anatomist was painted by both Rembrandt and Nicolaes Eliaszoon Pickenoy . Both works are shown in Figure 9.
Figure 9: Rembrandt, "The Anatomy of Dr. Tulp', 1632, Mauritshuis, The Hague (left), Pickenoy, 'Dr. Nicolaes Tulp”, 1633, Stedelijk Museum, Amsterdam (right).
Art Intelligence GmbH had a total of 372 face details from Rembrandt, but they were often painted in very unusual positions - which is why the face orientation points were sometimes very poorly recognized by the algorithm and needed to be corrected manually.
Figure 10: Face details with landmarks for the two portraits of Dr. Nicolaes Tulp by Nicolaes Eliaszoon Pickenoy (above left) and Rembrandt (right) as well as for those erroneously identified as Dr. Tulp recognized faces from Rembrandt paintings (below).
Using the procedure described above, out of the 372 Rembrandt faces, five are mistakenly identified as Dr. Tulip. This results in a recognition accuracy of 98.6 percent. The corresponding facial details of the illustrations of Dr. Tulp and the ones erroneously referred to as Dr. Tulp are listed in Figure 10 with their landmarks.
Painter Determination
With some paintings it is known from which period they come and to which style or genre they belong, but the actual painter is not known. Sometimes here, only a few artists may be eligible. A customer of Art Intelligence GmbH owns a painting that various art historians attribute to "Rembrandt's environment, but not to Rembrandt himself". Various chemical analyzes and an examination of the wood on which the portrait is painted clearly show that the work dates from the first half of the 17th century. The image is shown in Figure 11.
Figure 11: Head study "from the environment of Rembrandt".
A plausible possibility is that the painting was made by a student of the Dutch master. Art Intelligence GmbH has therefore researched numerous originals by the five painters that the owner considered possible. With these images, the classification algorithm described above was trained and evaluated with 45 artists, and the image in question was tested. The model recognized Gerrit Dou as the author of the work.
The result was initially confirmed by training the model exclusively with the originals of the five students and the originals by Rembrandt.
Another confirmation came with a completely different approach. For this purpose, a so-called Siamese network was trained on the basis of 40 painters without originals by Rembrandt or his students. Siamese networks do not learn a painter's style - but the similarity between different input images. They generate representations of a painter's style, which then allow the similarity between an image under test and different painters to be recognized - regardless of whether the images of the painters in question were in the training data set or not. This method is particularly useful when there are very few originals by certain painters, which is the case with Carel Fabritius the case is. According to art historians, arguably Rembrandt's most talented student only 29 works can be assigned with certainty.
Typically, this approach involves calibrating a similarity threshold. Since this definition of a fixed value is often ambivalent, Art Intelligence GmbH used a different method in this case - and considered all threshold values in the interval in question. The result is shown in Figure 12.
Figure 12: Siamese network detection accuracy for different painters and all thresholds in the interval of interest.
The percentage recognition accuracy for each painter was summed up across all threshold values. This sum corresponds to the area under the colored curve for each painter. The larger the area, the stronger the resemblance to the artist's style. Here, too, it can be seen that Gerrit Dou is the most likely of the artists to be considered - even though the network was not trained with originals by this Rembrandt pupil.