High Resolution Images Could Help Rebuild Notre-Dame Cathedral
by Alexis C. Madrigal

“Before the tragedy seen all around the world, flames leaping from the top of Notre-Dame Cathedral, there was a smaller one, thousands of miles away in upstate New York. Andrew Tallon, a pioneering architectural historian and father of four, died on November 16, 2018, from brain cancer. He was 49. He had dedicated his life to the study of medieval architecture, its mysteries and resonances, blending in his interest in technology to create novel ways of studying centuries-old buildings. “When you’re working on medieval buildings, it’s difficult to have the impression you can say anything new. They’ve been looked at and written about for ages,” Tallon told a documentary crew in 2015. “So I’ve been using more sophisticated technology these days to try to get new answers from the buildings.”

And so it was that in 2010, Tallon, an art professor at Vassar, took a Leica ScanStation C10 to Notre-Dame and, with the assistance of Columbia’s Paul Blaer, began to painstakingly scan every piece of the structure, inside and out.

They mounted the Leica on a tripod, put up markers throughout the space, and set the machine to work. Over five days, they positioned the scanner again and again—50 times in all—to create an unmatched record of the reality of one of the world’s most awe-inspiring buildings, represented as a series of points in space. Tallon also took high-resolution panoramic photos to map onto the three-dimensional forms that the laser scanner could create.

“Andrew was relentless at scanning full buildings,” his colleague John Ochsendorf, of MIT, told me. “He would get on top of the vaults and under the roofs to capture the geometry.” A year before he died, Tallon posted a brief tour of the upper parts of the choir as a 3-D video to YouTube, embedded above.

Now, with the building having sustained untold but very substantial damage, the data that Tallon and Blaer created could be an invaluable aid to whoever is charged with rebuilding the structure. Ochsendorf described the data as “essential for capturing [the structure] as built geometry.” (He added, however, that the cathedral, no matter what happens now, “is irreplaceable, of course.”)

“l’incendie à l’église Saint-Sulpice n’était pas accidentel

Tallon and Blaer’s laser data consist of 1 billion data points, structured as “point clouds,” which software can render into images of the three-dimensional space. Stitch them together, inside and out, map the photographs onto the precise 3-D models, and you have a full digital re-creation of incredible detail and resolution. “I saw this happening, and I had two thoughts,” Blaer told me of watching the cathedral engulfed in flames. “One thought was that I was kind of relieved that he didn’t actually have to see this happen. But on the other hand, he knew it so well and had so much information about how it’s constructed, he would have been so helpful in terms of rebuilding it.”

Laser scans like Tollen and Blaer’s represent a significant development in the restoration of historic buildings. “Having the scans is incredibly important in any kind of reconstruction,” said Raymond Pepi, the president of Building Conservation Associates, which was the restoration consultant for St. Patrick’s Cathedral in New York City. Megan Rispoli is a specialist in restoring religious buildings and ruins; she also worked on the St. Patrick’s project with her previous firm, Murphy Burnham & Buttrick.

She said the value of the laser-scanning data is that they capture the buildings as they really are. “It would capture the cathedral in all its true dimension and imperfection,” Rispoli told me. “When you draw a building, you tend to make things perfect right angles, and it helps you make sense of the building. But that isn’t actually what exists.”

For a structure such as Notre-Dame, built over hundreds of years, it’s almost certainly the case that any drawing or archival material about its construction would be incomplete or incorrect, the preservationist and architectural historian Lindsay Peterson told me. Given that, the laser data might be the ground truth in a way that nothing else is.

“Historic drawings or even modern drawings are only accurate up to a certain degree,” said Peterson, now with the preservation consultancy Higgins Quasebarth & Partners. “Laser scans get you accuracy up to the millimeter.”

“A statue of Saint John was removed from the spire of Notre-Dame cathedral by crane on Thursday.”

But there’s a problem with all that detail: That much data requires massive storage and processing power to use. So it tends to be stored quite locally, by just a small number of people. Pepi, for example, said that his firm tends to hold on to the laser data for its project partners, primarily because its clients (such as churches) just don’t have the infrastructure to use it.

The work at Notre-Dame has not yet begun, but it will be a long, long process. Rispoli thought the first step might be emergency shoring for the walls, as the roof has endured so much damage. Then the rebuilding firms will have to assess the integrity of the stone in the rest of the structure. Pepi said that they’d drill holes in the stone to get core samples, and then they’d look at them alongside known good stone from the cathedral to get a sense of “the physical condition and mineralogical properties of the stone.”

But as the cathedral officials and architectural authorities in France and around the world begin to consider what to do, they will come looking for the laser-data files that Tallon created. Blaer estimated that, despite the high resolution of the scans and panoramic photographs, the files would be roughly a terabyte, small enough to fit on a single hard drive, but unlikely to be stored in the cloud. All those data now exist on a single disk, a tiny portal into the past, which is sitting somewhere on Earth.

Blaer thought it might be in the hands of Tallon’s students at Vassar. But Ochsendorf thought the data were most likely with the rest of Tallon’s archive, in the possession of his widow, Marie, who held Andrew in her arms as he died. A cathedral calls us to consider time beyond the boundaries of one life, enclosing us in a grand view of what humanity can do that humans cannot. Andrew Tallon will not reappear among the living, but the work he put into recording stone and wood as it was built by countless hands over time may restore that creation—and embed the man into the place he venerated.”

Art historian uses lasers to understand medieval architectural masterpieces
by Rachel Hartigan Shea  /  June 22, 2015

“Thirteen million people visit the Notre Dame Cathedral in Paris every year, entering through massive wooden doors at the base of towers as solidly planted as mountains. They stand in front of walls filigreed with stained glass and gaze at a ceiling supported by delicate ribs of stone.

If its beauty and magnificence is instantly apparent, so much about Notre Dame is not. To begin with, we don’t know who built this cathedral—or how. The bishop of Paris, Maurice de Sully, commissioned the massive church complex around 1160. Yet the names of those who first constructed this masterpiece are lost to history.

They left no records—only centuries of speculation—behind. “So much ink has been spilled over that building,” says art historian Andrew Tallon. “So much of it is completely wrong.” A former composer, would-be monk, and self-described gearhead—or, as he puts it, “tacklehead”—Tallon intends to make that history right.

Frear’s Troy Cash Bazaar, 1897, scanned by Hudson Reality Capture

With the help of 21st-century laser scanners, he is teasing out clues hidden in the ancient stones of Notre Dame and other medieval structures—and revolutionizing our understanding of how these spectacular buildings were made.

St. Paul’s Church in Troy NY, “comprehensively remodeled during the period 1891-1893 by the Louis C. Tiffany Glass and Decorating Company.”

Tallon, 46, wasn’t the first to realize that laser scanners could be used to deconstruct Gothic architecture. But he was the first to use the scans to get inside medieval builders’ heads. “Every building moves,” he says. “It heaves itself out of shape when foundations move, when the sun heats up on one side.” How the building moves reveals its original design and the choices that the master builder had to make when construction didn’t go as planned.

St. John’s Church in Troy NY contains stained glass masterworks by Tiffany Glass Co”

Tracking this thought process requires precise measurements. For a long time, the tools used to measure medieval buildings were nearly as old as the buildings themselves: plumb bobs, string, rulers, and pencils. Using them was tedious, time-consuming, and error-prone. “You can’t hang from a vault and measure it by hand,” says Michael Davis, an art historian at Mount Holyoke College who once spent nine weeks surveying two churches with these primitive tools.

St Mary’s Church, Troy NY scanned and converted to AutoCAD by Hudson Virtual Tours

Laser scans, with their exquisite precision, don’t miss a thing. Mounted on a tripod, the laser beam sweeps around the choir of a cathedral, for example, and measures the distance between the scanner and every point it hits.

Cathedral of All Saints – Albany, NY, scanned by Hudson Reality Capture

Each measurement is represented by a colored dot, which cumulatively create a three-dimensional image of the cathedral. “If you’ve done your job properly,” says Tallon, the scan is “accurate to within five millimeters [.5 centimeter].”

“A laser scan of the National Cathedral in Washington, D.C. The scan is a collection of data points; a close-up view renders them see-through.”

Two researchers at Columbia University—Peter Allen, a computer scientist, and Stephen Murray, an art historian—attempted one of the first laser scans of a Gothic building in 2001 at the cathedral in Beauvais, north of Paris, which Murray had once measured by hand with steel tape and wooden calipers. Unfortunately, the scanner “actually went up in a puff of smoke. It really did emit smoke,” says Murray. “And at that point people didn’t know how to render [the data] into a three-dimensional model that was manageable.” Art historian Andrew Tallon is using laser scanning technology to discover anomalies in the structure of cathedrals such as Notre-Dame in Paris. Here’s how it’s done.

A tripod-mounted laser collected data by scanning more than 50 locations in and around Notre-Dame. The data are stitched together to form a “point cloud” containing more than a billion points, revealing structural concerns that otherwise would be hard to detect. Tallon figured out how to knit the laser scans together to make them manageable and beautiful. Each time he makes a scan, he also takes a spherical panoramic photograph from the same spot that captures the same three-dimensional space. He maps that photograph onto the laser-generated dots of the scan; each dot becomes the color of the pixel in that location in the photograph. As a result, the stunningly realistic panoramic photographs are amazingly accurate.

photos by Andrew Tallon

At Notre Dame, he took scans from more than 50 locations in and around the cathedral—collecting more than one billion points of data. Tallon, says Murray, his Ph.D. adviser at Columbia, is “able to combine that astonishing grasp of technology with the big humanistic vision that one hopes that art historians have.” Tall and lean with short-cropped hair and narrow eyeglasses, Tallon leans forward when he talks, sketching architectural features with his hands. He pops up and down from his chair, eager to show a visitor the exact spot on a projected image that demonstrates his point. His high-tech approach to Gothic architecture seemed preordained. As a kid growing up in Milwaukee, “I loved gadgets,” he says. He took apart reel-to-reel tape decks and played around with synthesizers. He was also obsessed with Notre Dame.

“I had this little guidebook and I annotated it like a nutcase,” he says, describing the year he was in fourth grade and living in Paris while his mother worked on her dissertation in theater history. “I longed to know the usual questions. Who made that thing? How did they make it? Could I ever go up in one of those passages?” The answer to that last question was yes, but it would take a long time to get there. “I have a career I don’t really tell my French colleagues about,” he admits, “because they’ll think I’m not serious.”

As an undergraduate at Princeton, he majored in music but also took every class taught by Robert Mark, an engineer who studied the structure of Gothic buildings. After graduation, his peripatetic journey began. First he went to France, where he studied medieval acoustics. Next stop was New York City, where he started a music composition studio. Then he paused in a Northern California monastery to explore the life of a monk. (The monks told him he wasn’t cut out for it.) Music led him back to Gothic cathedrals.

“Tallon laser scanned 45 historic buildings, including this cathedral in Chartres”

Murray, the Columbia University art historian, was putting together a multimedia project on Amiens Cathedral and needed someone to create “sounds that a cathedral might make.” Mark told him there was only one person he should contact—Andrew Tallon. The project was a labor of love. “I finally realized that what I really wanted to do was to be able to mix all these things together—love for medieval buildings, for art, for technology, for knowledge,” Tallon says.

Murray introduced him to laser scanning. More than a decade later, Tallon, now a tenured professor at Vassar College in the art department, has scanned some 45 historic buildings. “That very same thrill that I longed for as a kid looking up at those passageways in Notre Dame—’I want to go up there’—well, here I am up there, and it’s thrilling,” he says. Some of the thrill comes from the dizzying heights Tallon must ascend to do his job.

Reverse Engineering Bourges Cathedral with 3D Printing

At the Laon Cathedral in northern France, “I wanted to see the building right below the great eastern rose window,” he says, but his tripod filled the narrow ledge, which was more than 80 feet (25 meters) off the ground. “I had to step out into the void to go around it.” When his wife, Marie, saw the resulting image, showing the precipitous drop, she threatened to smash his camera. “You can’t really just fly over with a helicopter, scan the building, and call it a day,” says John Ochsendorf, an MIT engineer who is using Tallon’s scans to create a small-scale model of Bourges Cathedral with a 3-D printer. “You really need to get up into all the difficult places you can access: the top of the roof, the top of the vaults, inside the stairwells, all the hidden spaces people don’t normally see.” To Tallon, the father of four boys under the age of eight, the discoveries make the scrambling around worthwhile. “If I had texts at every point, I could look in the texts and try to get back into the heads of the builders,” he says. “I don’t have it, so it’s detective work for me.”

The laser scans have led to surprising new information about Notre Dame’s builders. For one thing, they sometimes took shortcuts. Even though medieval builders strove to create perfect dwelling places for the spirit of God, Tallon’s scans reveal that the western end of the cathedral is “a total mess … a train wreck.” The interior columns don’t line up and neither do some of the aisles. Rather than removing the remains of existing structures from the site, the workers appear to have built around them. That cost cutting could have been catastrophic.

Based on stylistic changes, scholars have long suspected that work on the western facade stopped for a while before the towers could be built. When Tallon scanned it, he discovered why. The Gallery of Kings—the line of statues above the three massive doorways—was almost a foot (.3 meters) out of plumb. Tallon concluded that the western facade, built on unstable soil, began leaning forward and to the north. Construction had to be halted until the builders could be confident that the ground had compressed enough to resume. After an anxious decade or so, it had.

“Tallon’s laser scans reveal that some of the columns in the nave of Paris’s Notre-Dame Cathedral don’t line up because they were built around existing structures.”

The builders were more sensible when it came to constructing flying buttresses, which some scholars have argued were added after the cathedral was built. After measuring the walls, Tallon determined that the flyers, as he calls them, were part of the cathedral’s original design. The vaults in the ceiling should push walls outward but “the upper part of the building has not moved one smidgen in 800 years.” The reason? “The flying buttresses were there from the get-go,” pushing the walls inward and creating a stable balance of forces.

Last spring, Tallon scanned the National Cathedral in Washington, D.C., which was begun in 1907, long after the Middle Ages had ended. He was looking for evidence of the influence of William Goodyear, a renowned American art historian who believed that the allure of Gothic cathedrals can be traced to imperfections that builders purposefully introduced. The architects charged with completing the National Cathedral had consulted Goodyear, and Tallon wondered whether he’d find some intentional flaws in the neo-Gothic building. He did.

Though most of the structure is perfectly plumb, the great columns at the center of the cathedral were built ever-so-slightly outwards, and the choir doesn’t align exactly with the nave. To Goodyear, imperfection “was the secret sauce,” says Tallon, “that medieval folks sprinkled on their buildings to make them beautiful.” Tallon believes the true “secret sauce” was faith. “There was a biblical, a moral imperative to build a perfect building,” he says, “because the stones of the building were directly identified with the stones of the Church”—the people who make up the body of the church. “I like to think that this laser scanning work and even some of the conventional scholarship I do is informed by that important world of spirituality,” says Tallon. “It’s such a beautiful idea.”

One of the most advanced Artificial Intelligence systems ever built

“The Time Machine will create advanced AI technologies to make sense of vast amounts of information from complex historical data sets. This will enable the transformation of fragmented data – with content ranging from medieval manuscripts and historical objects to smartphone and satellite images – into useable knowledge for industry. Considering the unprecedented scale and complexity of the data, The Time Machine’s AI even has the potential to create a strong competitive advantage for Europe in the global AI race.”

A pan-European infrastructure for digitising and processing cultural heritage

“The mirrorworld doesn’t yet fully exist, but it is coming. Someday soon, every place and thing in the real world—every street, lamppost, building, and room—will have its full-size digital twin in the mirrorworld. For now, only tiny patches of the mirrorworld are visible through AR headsets. Piece by piece, these virtual fragments are being stitched together to form a shared, persistent place that will parallel the real world. The author Jorge Luis Borges imagined a map exactly the same size as the territory it represented. “In time,” Borges wrote, “the Cartographers Guilds struck a Map of the Empire whose size was that of the Empire, and which coincided point for point with it.” We are now building such a 1:1 map of almost unimaginable scope, and this world will become the next great digital platform.” – Kevin Kelly, Wired Magazine

“On Digital Day 2019 in Brussels, 24 countries (Austria, Belgium, Cyprus, Czechia, Estonia, Finland, France, Greece, Italy, Ireland, Latvia, Lithuania, Luxembourg, Malta, the Netherlands, Norway, Poland, Portugal, Romania, Slovakia, Slovenia, Spain, Sweden and the United Kingdom) signed a landmark declaration of cooperation for advancing digitisation of cultural heritage.

This historic declaration formally recognises that “natural disasters, pollution, mass tourism, deterioration over time, terrorism and vandalism, create urgent need to make the most of digital technologies to record, document and preserve Europe’s cultural heritage and foster their accessibility to European citizens.” To face such threats, members states have decided to join forces in developing key technologies for transforming the processing of, and the access to, cultural heritage. There will be a focus on big data, artificial intelligence, augmented reality and 3D, and the development of European platforms in line with European values. Another critical element will be fostering the development of common open interoperable standards for the processing and storing of cultural heritage assets.

A sign of ever-increasing momentum, this latest step has taken place just one month after the launch of the EU-funded Time Machine CSA: a pan-European initiative for digitising and processing cultural heritage assets across Europe. The project currently involves around 300 European institutions from 32 countries including: 7 national libraries, 19 state archives, some of the most important museums in Europe (the Louvre and Rijksmuseum), as well as hundreds of other research institutes, companies and governmental bodies. Currently, Time Machine is working on two fronts: the definition and development of its distributed infrastructure, and the standardisation of operations between the project stakeholders. Joining Time Machine’s network ensures that museums, archives, libraries, monuments and other cultural heritage sites can benefit from the digitisation capabilities of all Time Machine operators.

These competencies range from robotic-based mass digitisation pipelines for documents and objects, to drone-based scanning systems. Time Machine’s infrastructure represents a pan-European platform that will be used not only for matching digitisation needs with digitisation services, but also for channelling funding opportunities to cover digitisation costs and for developing services through platforms dedicated to a variety of domains (e.g. tourism, creative industries, smart cities, land-use policies, etc.). This organisation of Europe’s digitisation market is made possible through standard metrics that permit users to compare and to align with offers and technology across European countries, be it for digitising fragile documents or an entire monument. This results in a global optimisation of the digitisation program advocated by the European Commission and the 24 countries that have signed the declaration.

For partners in Time Machine’s network, digitisation is only the first step in a long series of extraction processes. These include document segmentation and understanding, alignment of named entities and simulation of hypothetical spatiotemporal 4D reconstructions. Time Machine’s infrastructure is currently working on standardised and generic deep learning-based modules that can be composed in sequence to extract information based on state-of-the-art technology. Any document using Time Machine standards will benefit from a full series of extraction tools, including the latest handwriting recognition technology and document analysis algorithms. The commonality of these modules also allows us to envision a distributed processing solution hosted by Time Machine partners, which are deployed based on the territorial progress of the digitisation program.

Local Time Machines and the step-by-step path towards mirror worlds
There is more. Even though Time Machine’s infrastructure has been built to enable cooperation across Europe, it simultaneously creates vertical synergies between clusters of actors operating in the same area. For example, some cities can become Local Time Machines. This occurs when a number of Time Machine partners operating in close geographical proximity to one another decide to join forces in creating a full spatiotemporal representation of the area. These efforts, currently ongoing in about 20 European cities, will also become formalised in the coming months as the Time Machine Organisation comes online. Local Time Machines evolve through successive steps: from formal declaration of interest, to official partnership, and on to higher density of cross-linked information, population and urban reconstruction. During Digital Day 2019, while commenting on the declaration, Harry Verwayen, Executive Director of Europeana — one of founding members of Time Machine who is also currently collaborating with 3,700 institutions and hosting 58 million items — explained why Time Machine opens the door to something much bigger than just digitisation and interoperability. Quoting Kevin Kelly’s recent article on what he refers to as the ‘mirrorworld’, Harry helped the audience to envision the radical transformation that Time Machine is introducing.

“With a swipe of your hand, you will be able to go back in time, at any location, and see what came before. You will be able to lay a reconstructed 19th-century view right over the present reality. To visit an earlier time at a location, you simply revert to a previous version kept in the log. (…) Or you’ll scroll in the other direction: forward. Artists might create future versions of a place, in place. The verisimilitude of such crafty world-building will be revolutionary. These scroll-forward scenarios will have the heft of reality because they will be derived from a full-scale present world. In this way, the mirrorworld may be best referred to as a 4D world.” – Kevin Kelly, Wired

The Local Time Machine maturation process offers a realistic scenario to progressively move from dense information networks to actual 4D worlds. To go back in time in cities such as Venice, Paris, Amsterdam, Antwerp or Jerusalem, one would need to jump from using a dense network of information, the result of transparent interpretation processes from reliable sources, to a continuous simulation representing the multi-scale evolution of a city and its population. The hypothesis pursued by Time Machine is that such computational models with an extended temporal horizon are key resources for developing new policies and services for cities. Deploying itself both horizontally (across Europe) and vertically (in time, first for specific Local Time Machines and then progressively in the entire territory), Time Machine offers a sustainable approach not only to safeguard European cultural heritage but also to transform it into a living resource for Europe’s future.

Community of communities
Since its launch, Time Machine has received offers from hundreds of volunteers wishing to help digitise and valorise European cultural heritage or to participate in the technical development of this pan-European infrastructure. Time Machine coordination units are now developing solutions to help structure this community of communities. Local inhabitants of a city will have the possibility to help model their own neighbourhood or to bring their own private archive to the Time Machine. Genealogists will find potentially missing elements within the new stream of documents and data for reconstructing historical links between the ancestors they study. As Time Machine codes are open source, developers can also join in by upgrading elements of the infrastructure.”



Leave a Reply