In June 2022, with a very basic understanding of the SAR technology, I started to use Sentinel-1 data to report on the progression of damage in Ukraine. Shortly after, Tim Wallace, our Editor for Geography connected me with Oregon State University, Jamon Van Den Hoek and City University of New York, Corey Scher, two leading researchers in the field of InSar sensing. They have been using this technology for a long time and were also exploring its use in Ukraine shortly after the beginning of the war. They have also help us to analyze the development of destruction in Gaza more recently.

Corey and Jamon help us to process terabytes of data to flag areas where structural changes happened. The first rounds were a challenge, the data is so sensitive that it was influenced by things like vegetation, soil conditions and snow. Things like mining activity, construction areas and train stations represented a challenge too.

Somehow we needed to filter out all of those things. I was doing a roll of analyst testing the data the researchers provide with HD satellite images. I spent weeks looking at the areas flagged to find things that maybe interesting to highlight later, but I also encountered weird stuff, which is normal in early stages of data analysis.

The image below is an screenshot of one of the many QGIS files I created to explore things in different ways, the rectangles there are areas I flagged for myself to follow closer.

While exploring the data, I also noticed these long straight lines, which seemed to be the data tiles overlapping or something.

Looking down from space

I used QGIS, the Planet explorer, Maxar imagery and Google earth to crosscheck the data and be sure we were pointing at real damage related to the war and not something else.

Every corner of the country had its own story to tell, some bettern known like the case of Mariupol, but during my exploration I learned about many places I had never heard of before, many villages that were destroyed far from the media spotlight.

In the satellite image below, the town of Kamyanka, in Kharkiv, shows the remains of what were once homes. When you look closely, you can notice the multiple craters, presumably caused by explosions.

Google has being updating imagery in Ukraine, this is one of those places you can easily see with Google Earth. As I’m writing this post the buildings are still in ruins accordingly to the image from August 29, 2022.

Damaged buildings… but what type: houses? stores? military?

I used OpenStreetMap to identify some of the damaged buildings, however, even tho OSM has a lot of good data, I needed more to fill the gaps.

Around June 2023, Microsoft Bing released a great data set of millions of structures in Ukraine, so I merged it with my existing OSM records, and ended up processing +17 million of building footprints. But it was also a nightmare, I tried to use Google Cloud and a few more things to get overlapping records with the damage flags, short story… all collapsed. –I probably set it up all wrong.

Finally I found a quicker way using a script python, reducing the time of cross check from weeks to a few hours running the code in my laptop overnight.

After overcoming that hurdle, I began collecting evidence of sensitive damaged locations.

New data sets were back and forth, from that first version to a better, more precise and clearer one including additional months of data. Meanwhile I was going through hundreds of photos, fighting and shelling reports and other damaged related information. We accumulated so much data since the war started that it took me weeks to classify all of it and get a better sense of the mayor events happening in different cities.

A dosage of ‘infofails’

If you have read my blog before, you know what an infofail is. But just in case, I tend to save pretty much all the stuff I do, even if it’s just for my own understanding of the information. Here’s a little of a wonderful date set I spent a lot of time in, but we left out of the project.

I did a lot of sketches to understand the data we have from fires, some seemed to align to the front lines after plotting the data like in the image below showing clusters of fire detections in a year.

As I mentioned, we abandoned this data for editing reasons, but also because some months showed some sort of alignment with war events, and others just weren’t very clear. One idea I have was to use the large amount of photos we have showing explosions/fire in certain locations where the satellite also picked up a thermal signature.

I also rendered a 2022 country wide “calendar of fires” (see below), there’s one row per month, taller and warmer colors represent more fire activity, some months match intense battles around the country, unfortunately other seasonal fires were also blended in the data.

In the image above, January is at top, note the 3rd line were the columns are taller and warmer. That matches when Russia launched its full scale offensive over Ukraine.

But the data also slightly varies by region, so I also did 2d graphics looking a different locations, including a few more data and in different sorting of time.

Sketching data

As time went on, I continued sketching maps and other visualizations for this project. This was not full-time dedication, while working on this project, I published 17 other projects and explored about 15 more that led to nothing or are still in progress. So I had to do quick and easy things to distribute my efforts. But let’s take a look to those quick sketches.

Part of my process is to annotate screenshots, just to remember things later. Below is an area of ​​Kharkiv, highlights in red are buildings damaged during the first months of the war in 2022.

This probably turns out to be obvious, but I found it quite interesting to see how in the first versions of the damage data some patterns emerged, the damage followed the roads leading to the cities. Can you see the lines too?

I spent long hours just exploring the map, turning layers on and off, adding before and after HD images, making things jump off my screen. Most of the exploring was made using Felt, Google Earth, Google spreadsheets and QGIS, screenshots and annotations on photoshop by copy+paste portions of my screen.

I did a lot of maps reporting on different of battles in other stories, so, some places seemed more familiar to me. Somehow the shape of the damage logs matched what I saw before, and that was a relief.

But also, other things, where it seems like the data tells a story that I can’t confirm. In the timelapse below, using the ISW control areas, you can see how perhaps Ukrainian troops tried to take this city, as the front lines move away, the records also cease.

Ukraine has a huge territory, so I identified regions to look closer, sometimes because the change over time was interesting, or because the amount of damage registered. I guess my interest was too wide, I selected almost half of Ukraine, these are some quadrants I was interested in explore deeper:

Just to be clear, explore deeper mean I went on looking for more images, reading reports and blogs about what happened there… It took me months, but I did explore all of them.

I also did some sketches using 2d geometries based in the totals of area damaged per settlement.

I also sketched some of cities data through time using photoshop:

Then modeled in blender using an early version of the data, just to get a sense of a few cities and how they compare to each other. Note the colors are key in time, older records are cold (blues) while more recent are warmer.

Modeling data

We also tested some building height data, following the idea of ​​full 3D transitions from the photos we had of the Marinka town where we focused our story.

We changed the top sequence a few times, but it was always focused over Marinka. The concept was to look close to a building with meaning to the residents, then scale to a street, to the town, to the region, to the country. That setting up to the reader the scale of data we analyzed.

Here you have some of the first demos I did using the data on a larger area to eventually move the camera far enough to see the whole country.

Workflow

To get you an idea of my workflow, I started with a set of geotiffs, shapefiles and geojsons. We applied some python to filer the data and focus on the things we wanted, clustering data in different resolutions, then added all into QGIS to export tiff layers out. I used one black and white file for each feature, one for water, one for land, one for how high each polygon should be, one for color highlights and so on.

I like to use QGIS to wrap-up everything, but I feel better styling the data in photoshop, so some of the work in styles was done there, mostly color using the black/white tiffs as a mask over solid layers.

Then I exported a layer out of photoshop with colors and overall styles, plus one more with the elevation data matching the same crop from the original QGIS outputs.

Those tiffs went to blender to be rendered as 3d objects.

From blender, I moved all to After Effects. There is handled the timing, labels and transitions, a custom built script takes each layer and exports a single json file for the web component. The file includes the basic properties as position, styles and timing. The svelte component takes that and arranges the assets depending on what device you are watching on.

I tested a few different set ups changing styles and clustering, here’s one of those versions with a flight-over camera.

I did a few versions more just to find the one that feels right. Sometimes happy accidents happened like this inverted render, which is nice but not the thing I was looking for:

some more versions…

and versions…

Binary maps and storage

Many more versions later, I did a series of custom cities showing only buildings colored by damage as true/false. Red is damaged, by this point we were in the version 40 of the data, 1 terabyte of maps, photos, videos, scripts and many other assets. I bought an external drive for this project, every time I was going to move files I left my laptop running for a day or two to make sure I didn’t lose anything.

But that was just a little portion of the data we used. Corey and Jamon, the researchers you meet at the top of the long page, accumulated more than 50 times more data than me to get the right data set for this article.

Around +40 versions later, we landed in the definitive data set.

Introducing the story for readers

The story was almost there, but some small tweaks here and there, mostly in the copy side and technical descriptions, but there was yet one more thing to take care about, and that was explaining and introducing this to our readers in the home page.

I prepared still images and animations for this purpose, the video colleagues also help me prepare a reel that would be shown on the main page and on social networks. Basically me talking about the project, you can see the final version of it at The New York Times instagram account here.

Preparing the print

We usually go first on the web, then on print. So that means after we hit the publish button, I started to plan the pages for the print edition with a different team. Even tho we are using the same story as base line, the way of presenting things is different, new pieces should be produced taking into consideration the requirements of paper.

The process was sort of similar, first preparing assets in QGIS, render out black and white .tifs to be later used as color mask in photoshop, then merged in a color layer plus a black and white layer with the heights data baked in. Then those 2 layers went to blender to get the final base map. Annotations and other small details were done in Adobe Illustrator.

A few different versions went around, because we move it to the front page, some small charts of context were dropped off. Here some of those early testing versions, some even with a few errors:

The print edition was published today (June 22, 2024) on a special format displaying the large map and a few more pages inside wrapping up the whole story with adaptations of the original content. The print team did an exceptional work, they have a lot of experience in getting the best version for the readers out there.

I started exploring this project in June 2022, and today, two years later, I find myself reading the last piece of the project finally published, there’s a sense of relief to know that I fulfilled the mission of reporting on this very relevant topic with such a display for the New York Times.

Thanks for reading, and don’t forget to buy this special edition of the paper.