AI will now be well-versed in cultural heritage thanks to the new Saint George on a Bike project coordinated by the Barcelona Supercomputing Center. The project has begun training machine learning algorithms in culture, symbols, and historical contexts with the aim of automatically generating rich metadata for hundreds of thousands of images from various European cultural heritage repositories.
To enable access to collections of cultural heritage images, platforms like Europeana require high quality metadata. However, human experts from libraries, archives and museums simply cannot produce fine-grained descriptions for all the resources that result from massive digitisation programs. To alleviate this bottleneck, researchers are looking at employing AI technology to automate at least a part of the description process.
But training AI to be aware of cultural heritage contexts is not as simple as teaching it to identify different objects in a modern picture. Saint George on a Bike is fine-tuning the algorithms so that they 'think' in context and time.
“The AI we are developing will be able to tell whether a painting shows Saint George on a horse or a bike,” said Maria Cristina Marinescu, coordinator of the project and senior researcher at the Computer Applications in Science and Engineering department (CASE) of BSC. “This is not as easy as it sounds because the shapes are similar. By training it in various cultural heritage elements, including the first appearance of the objects depicted and the period the painting is from, the AI algorithm would conclude that a 16th century painting of Saint George would logically depict a horse rather than a bike.”
Europeana will serve as the main test-bed for the metadata enrichments generated by the project. Antoine Isaac, R&D Manager at Europeana Foundation, says, "Existing AI tools tend to underperform when applied to cultural heritage collections, because they are trained on modern documents and material that is not cultural heritage based. If Saint George on a Bike manages to create computer vision tools that properly recognize our contexts, Europeana will be able to support a richer browsing and discovery experience for the cultural collections of our partners.”
Find out more
The Saint George on a Bike project is holding webinars on the 9 - 10 September to demonstrate the project results and potential to adapt automatically produced descriptions of paintings to the time period when they were created. Researchers in data mining and data science with an interest in art and culture are welcome to attend!