blogging

Visualizing organic carbon in near real time. –A step-by-step guide

Long time ago someone on twitter ask me to do an explainer on how I did the “smoke” animations for this Reuters piece. It has been a while since then, but maybe it would be useful for someone out there, even if that mean learning how NOT to do things.

Before continuing, to follow my guide and visualize organic carbon, you should be able to use your terminal window, QGIS and optional Adobe After Effects.

Earth globe showing organic carbon released into the atmosphere by wildfires in America
Organic carbon released into the atmosphere during the wildfires season in California in 2020

Let’s talk about this wonderful data first

NASA’s Global Modeling and Assimilation Office Research Site (GMAO) provides a number of models from different data sets, this is basically a collection of data from many different services processed for historical records or forecast models. This data works well for a global picture or continent level even, but maybe isn’t a good idea to use this data for a country level analysis, for those uses you may want to check other sources of the data instead of GMAO models, like MODIS for instance if you you are looking for similar data.

ORGANIC CARBON

There are a lot of different sets of products available at the GMAO servers, you can check details here, here and here. However for purposes of this practical guide, I’ll be focusing in the emissions of Organic Carbon which is stored into the tavg3_2d_aer_Nx set. That’s a GEOS5 FP 2d time-averaged primary aerosol diagnostics, which includes Organic Carbon Column mass density in the 38th band, there is some documentation available in this pdf. ( No worries if is this sounds too technical stay with me and keep going. )

A day of observations accounts for 8 files since this data is processed every 3 hours. This is great for animation because it would look smooth. Knowing that, let’s move to our guide.


Step 1. Get the data

The data is stored into this url. You can go into the folders and get all 8 files for each day manually if you like or get them with a command line using wget or curl into the terminal. You just need to know a little of the url structure:

url structure
GMAO organic carbon files and url structure
  • Create a folder to store your files with some name like data
  • Open your terminal window
  • Type cd in the terminal window
  • Drag the folder you created inside the window
A “cd folder” example stolen from the web.


Then run a short command like the following, you would get a file named 20220619_0130.nc4 in you data folder:

curl https://portal.nccs.nasa.gov/datashare/gmao/geos-fp/das/Y2022/M06/D19/GEOS.fp.asm.tavg3_2d_aer_Nx.20220619_0130.V01.nc4 -o 20220619_0130.nc4

Note that I have renamed the output ( -o ) with a shorter name. The file will go to your folder ready to use into GQIS. Of course you will need a few more files to run an animation. Remember that this data is available for every 3 hours daily, so you need to set the url and name for something like this:

01:30 AM >> 20220619_0130.V01.nc4
04:30 AM >> 20220619_0430.V01.nc4
07:30 AM >> 20220619_0730.V01.nc4
10:30 AM >> 20220619_1030.V01.nc4
01:30 PM >> 20220619_1330.V01.nc4
04:30 PM >> 20220619_1630.V01.nc4
07:30 PM >> 20220619_1930.V01.nc4
10:30 PM >> 20220619_2230.V01.nc4

Just create a text list with all the urls you need and run the script into the terminal window with the same process:

curl -O [URL1] -O [URL2]

Each file is usually about 120MB, if there’s something wrong with the data the file will be created anyway but would be an empty file of just a few KB. Do a day or two first and check, that’s 8-16 files, check them, if all looks good load a few more if you like.

Step 2. Loading the data

Once you have a nice folder with all the files you want, you can just drag and drop the .nc4 files into QGIS. We are looking for the 38th Band, OCCMASS which is our Organic Carbon Column mass:

QGIS prompt window when you drop one of the file in.

Once you have the data loaded, you want to set the data projection to WGS 84, this will enable the data layers to be re-projected later on. To do that, select all you data layers, right click on them, and select Layer CRS > Set Layer CRS > 4326. Be sure of selecting all the layers at once so you do this only one time. Otherwise you will need to doing over and over.

Data layers projection to WGS 84.

Since this is a good global data set, you may want to load a globe for reference, you can use your own custom projection, or use a plugin like globe builder:

Access Globe Builder from the plugins menu > Manage and Install > type: Globe.

Once installed, just run it from the little globe icon, or in the menu plugins > Globe builder > Build globe view. You have a few options there, play around with the center point lat/long. You will see that this data sets always have large concentrations of emissions in Africa, maybe that’s a great place to start. I’ll do a similar view to the California story for now.

Step 3. Styling your map

The color ramp is important, you want to have a data layer that can be overlayed in the base map, so you want to have white/black for the lower values and high contrast in the other end of the data, since we are working on white background I’m using white to black with yellow and brown stops. Check what are the highest values in your data set the style for on layer to something like this:

Number in the min/max will change depending on the highest values of your data and the style you want. This image is set for OCCMASS from June 19th, 2022, 4:30 pm.

Once you have the ideal color ramp for one layer, right click on that layer, go to Styles > Copy style. Then select all you carbon data layers, right click on them and select Styles > Paste Style.

Step 4. Preparing to export your map

You are almost done, by this point you can see how each data layer creates swirls in the atmosphere, maybe some evolution of it too just by toggling the layers visibility. I like to have all the layers well organized so you can quick check the data. I’m maybe a little too obsessive but I usually rename all layers and groups to something like this:

QGIS layers panel.

The name change works if you are using an automatic export of all layers, the script takes the name of the layer to save each file. But there are alternative ways to do this if you’re not as crazy as I am and don’t want to spend time manually renaming.

Step 5. Export your map

There are many ways of doing this, you can set up the time for each layer by using the temporal controller, there’s a good guide here. That way you can get a mp4 video right away from QGIS, but you need to set up each data layer time manually.

You can also use a little code to export each layer into an image, which you can then import into After Effects. To do that, the first step of course, is to get the script. Download the files HERE.

Now, go to the plugins menu at the top, there, you will see the Python console, go and click that, you will see this window popping-up:

Python console in QGIS.

Click the paper icon, then click the folder icon and select the python script you dowloaded above. Just be careful with the filePath option.

If you are on a mac, right click your output folder and hold the option key, that will allow you to copy the absolute path of you folder, paste that to replace the filePath field value (the green text in the image below). If you are on Windows, just make sure to get the absolute path and not a relative one.

I left some annotations on the script to better understand what each part is, it’s based on a script someone did with Vietnamese annotations, source and credit are in the drive link too.

Now just click the play button in the python console, seat back and look all the frames of your animation loading in the output folder you selected. You should see a file for each of your layers when the script finishes.

Step 6. Export your animation

Take all the files this into After Effects. First, add your carbon data as sequence (0001.png, 0002.png, 0003.png…), keep that in a sub-composition and use a multiply blend mode to overlay the layers, then add the countries/land and the optional halo.

Finally, in the drive folder you will see a .aep file, that’s a simple number animation to control dates, copy the text layer into your composition. You know when the data starts and when it ends, in the example is just 3 days 19-21, “June” is a different text layer, so add those numeric values to the keyframes into the text layer you have copied, and leave it at the very top:

Once you are all set, just export to media encoder to get you mp4 animation.

If any of this doesn’t make sense to you, or if you’re having trouble with a step, feel free to reach out to me on Twitter. I will be happy to hear from you.


Happy mapping!

Standard
blogging

Random Failed Map Details

Recently I have been working on maps, maps and more maps. I really like the world of cartography, although I’m not a cartographer a lot of my work includes trying to make maps for news. –My apologies to my carto-friends who actually do this properly, I’m just an enthusiastic fan with perilous initiative. 🤣

Since I moved to the NYT, I have been in a process of rebooting, adjusting myself to the new environment learning new stuff and understanding how things work in this side of the world. But as usual, while I’m executing random ideas I have left behind a bunch of un published visuals like the screengrab at the top of this entry which is a DEM of an area of eastern Ukraine.

For nerdy purposes, the image at the top and the following are SRTM elevation and Open Street Maps data processed with QGIS with a little color retouch in Photoshop.

A failed map of eastern Ukraine.

Of course these detailed images doesn’t work well for the purposes of the news story I was working on. If you have seen our Ukraine maps coverage, you’ll notice that while our maps have evolved, they also keep consistency somehow. To be honest, I made those alternate versions because I couldn’t stop thinking about how this would look in another style. You can see what I mean below, these are the same area in eastern Ukraine rendered for different purposes:

  • Alternative terrain section of eastern Ukraine including part of the Sea of Azov at the bottom
  • Screenshot of a piece published by the New York Times on the Ukraine - Russian war in the Donbas region.
  • Alternative terrain section of eastern Ukraine including part of the Sea of Azov at the bottom

Here are some closer shots of that map above, the geography of this region of Ukraine is marvelous.

There are so many of these maps, I have literally spent months looking at the progress of the war with maps, many different approaches and a heavy editing process of what takes place until the final version of the story. It is a strenuous process but super interesting at the same time. I feel very grateful to be able to see all this and be part of the search for the truth to inform the readers of the NYT.

Basic vectors

Primary roads in eastern Ukraine

There’s something with the base layers, is amazing how you can see the population density of a place just by plotting roads. Some areas with certain road layers look like leaves or some kind of vein system.
[ Click on the images to see a larger single image ]

The same thing happens looking at water features, some times you are able to see canals making geometric patterns in contrast to the organic river beds.

Since Ukraine has vast tracts of land dedicated to agriculture, those patterns are clearer in some regions, however the rivers and lakes are still fascinating as well.

As of the date of publication of this entry, I have worked on about 15 pieces with some kind of map of some region of Ukraine analyzing all kinds of approaches, such as the strategy to isolate Ukrainian forces in the east, aerial bombardment, the damage in the port city of Mariupol, and fighting reports all in a day cycle. More recently we have focused on deeper stories like the battle in the Donbas region that we just published where most of this entry’s images came from. (There are some more map stories coming soon).

About #infofails post series:
I truly believe that failure is more important than success. One doesn’t try to fail as a goal, but by embracing failure I have learned a lot in my quest to do something different, or maybe it is because I have had few successes… it depends on how you look at it. Anyway, these posts are a compendium of graphics that are never formally published. Those are maybe tons of versions of a single graphic or some floating concepts and ideas, all part of my creative process.

In short, #infofails are a summary of my creative process and extensive failures at work.

Are you liking #infofails?, have a look to previous ones:

01: Wildfires
02: Plastic bottles
03: Hong Kong protest
04: The Everest
05: Amazon gold
06: The world on fire
07: A busy 2021 kick off
08: Olympics
09: Floods
10: Doodles for news

Standard
blogging, infofails

🎵 I can see clearly now the rain is gone…

Last July was a crazy month full of flood news all over the world. I remember seeing impressive videos and images of the floods in China and Germany, and digging a little deeper I found many more reports about it from around the world. I tried to put some things together, but time and other projects played a trick and the project became material for #infofails.

Some times taking notes of things isn’t enough for me. One or two illustrator artboards with basic ideas have become the new “office whiteboard sessions” since we started remote work. Quick sketches and some data samples usually help me to organize myself better.

Sampling flood reports and daily precipitation data.

I collected some data from NASA including the PPS and MERRA-2 to visualize precipitation. It was so cool when I saw the data of total rainfall in a month over the planet. Is curious to see how dynamic our planet is isn’t?

July’s total precipitation. Data by NASA’s Precipitation Processing System (PPS)

Whenever I have a global data set, I always look at how things are for my family and friends in Costa Rica. I remember that in July I had seen videos of flooded areas in Turrialba, a region in the Atlantic region of the country. And yes, the accumulated data showed that intense blue layer near the border with Panama.

Detail of the precipitation data. NASA PPS.

Of course, there were other much worse areas that saw terrifying amounts of precipitation causing dozens of deaths, western India for example was one of those areas. I continued to explore a bit more on the map and checking against the flood reports I found to find points of interest and to highlight later in the story.

Detail of the precipitation data. NASA PPS.

The testing continued

One aspect to consider was how to visualize the data in the end. There was even a 3D spinning globe in the process… As you can imagine it was chaos displaying flood reports, animated rain data, and 3D navigation all at the same time.

However, one of my favourite pieces was not the maps. There were some small graphics to condense powerful messages had something interesting too. Within them was this simple stacked bar chart where each block showed the total precipitation each month in Zhengzhou, just by putting the amount of water they received on July 20 next to it was really impressive. This is real evidence of how extreme our planet’s climate is becoming.

BTW, there’s also a great graphic from the South China Morning Post friends explaining the huge amount of water that Zhengzhou received over the downpours [ check that story here ]

Extremes

A few years ago I was working on a graphic about extreme temperatures of the earth, it was happening the 2019 polar vortex in the US and at the same time Australia was on 40° C on the other side. In my head, the perfect title was “Earth’s Goldilocks Climate.” It sounds crazy but it is actually very common, our planet is full of those strange contrasts all the time.

In July China was having its own ‘goldilocks’ event, or kind of, because wasn’t temperature. As enormous amount of water flooded train stations and caused chaos in Henan, south of there a nine-month drought hit Fujian province.

July total precipitation in China. Data by NASA PPS

Similar situations occurred in the Middle East, in Afghanistan a long drought was worsening the already difficult situation of the Afghans. Ironically, extreme rains in the border areas also caused flash flooding, while the country as a whole has not seen any rain for months.

July total precipitation in the Middle East. Data by NASA PPS

NASA’s MODIS/Terra offers also daily and monthly averages of surface temperature. This was some other stuff I was considering for this story. It’s incredible to see how high the temperatures go in the region. There’s also an other cool data set of monthly temp. anomalies here in case you want to explore the world too.

Temperature anomaly for Feb. 2021. Red areas show were the temp. was higher in comparison with the averages of 10 years ago. Afghanistan was about 12C warmer in average according to NASA Earth Observations data. LPDAAC and MODIS.

Anyway, none of these charts, maps or data made it into a true story on Reuters, but it was fun collecting, preparing and sketching ideas for it. And of course, in the end it became an average #infofails story here. Maybe later we will take back again this story, unfortunately extreme weather events are becoming more and more frequent

About #infofails post series:
Graphics that are never formally published. Those are maybe tons of versions of a single graphic or some floating concepts and ideas, all part of my creative process. All wrapped up in #infofails, a compilation of my creative process and failures at work.

Did you like #infofails?
Have a look to other #infofails 👇

1: Wildfires
2: Plastic bottles
3: Hong Kong protest
4: The Everest
5: Amazon gold
6: The world on fire
7: A busy 2021 kick off
8:Olympics

Standard
Digital works

Nerd facts: Asian American population

US elections are coming and here in Hong Kong we want to see it from an Asian perspective. Some weeks ago I start this project to collect the data of Asians in US and visualize it.

boundaries

The map in first instance is nothing more than a choropleth map, I start from a black map as the one in the top, then one by one I manually add a grey value in five clusters, white for 0% and so on to +25% as the dark grey.

The US Census Bureau has a very specific data for Asians in US, but I use data also from the APIA and AAJC Asian organizations in US as well, the research phase takes me about a week, producing the graphics and coding it was about four days, and about an other week making edits and corroborating each small graphic and each word in the special feature. I must highlight that this graphic was not just a map, each state in the US has a detail of text and graphics so it was not ease to check, edit and revise.

screen-shot-2016-10-20-at-1-00-48-pm

Early version of the country over view. As you may note, beyond color the elevation is different, my first data set was for total population instead of percent of the population

The map was generated with Cinema 4D, based on a grey scale county map, then styled with Illustrator and photoshop and coded with Brackets io trough SVG, Jquery, JS…

This slideshow requires JavaScript.

The color code is very important because the position of camera will change in perspective, so the color make easier to visually determinate the height of each area no matter the position. In the following image the camera position affect the way we perceive the height of the counties, just because the matter of point of view, but when color is on… well, see your self:

color

Brown means Asian population over 25% in the specific county (as well the height), I call that to perceive data in multidimensional ways, but keep it understandable

The position of cameras will change for each state, as for the videos and other sections using the specific portions of the map to tell different stories.

Following the 3D idea, the complementary graphics was also generated in C4D and the layout phase begin.

screen-shot-2016-10-20-at-12-57-26-pm

Some graphics display scroll-trigged video on desktop versions, then for mobile switch for static renders because IOS don’t allow autoplay videos or video out side the native player

hi_motion

This is one of the city spinners, all of them uses an image like this, is positioned by the mouse drag final position and makes the 3d effect

Hope you as reader enjoy the project much as I enjoy to make it, please visit the special feature here http://multimedia.scmp.com/news/world/AsianAmerican/

Standard
Digital works

Pokemon invasion

The Pokemon fever runs out the whole world, here in Hong Kong the people also was infected with the Pokemon global madness, so we in the SCMP graphics want’s to offer something different to our readers, and that was a detailed map of the Hong Kong Pokemon spots and a plus graphic about the Skills and weaknesses of each class of Pokemon.

 

folder

The map was a hard work, to get all this data the folder of Pokemon project reach about 12 Gigabytes of my computer, but off course the final application on line is just about 500KB, I catch this information from an other game in which the creators was based to create the Pokemon virtual world, the whole process takes me 5 days including the new design and the development phase.

When I finish the data catch phase and clean it, the points shows a really nice pattern, following the urban areas but also reveals by him self the Islands shapes.

vector_10pc

10% map sized of the original data. In the bottom right corner the most concentrated area of Pokemon spots, corresponds with the Hong Kong Island districts and the Kowloon territory crossing the bay area.

My first design was a purple points for the spots over a grey map, to give it a high contrast, but, then turns green and blue to keep it closer to the actual game interface. Usually I wrote my own text, but as you may know my english is not the best so, the layouts and the final product changes a bit on that also.

layout_vrs2

Illustrator layouts for Pokemon project. Version 2.2

The bottom section has a graphic about some characteristics of each class of Pokemon, the dropdown menu allows readers to switch the class and display different graphics, I have to say that I don’t like the Pokemon at all, even when I was younger, but I love to create data visualizations and offer to the people an other way of stories… visual stories; so Yes, I enjoy this project much as the Pokemon lovers has enjoy it also.

pokemon_types

All the small graphics for Pokemon skills. Red for attack, blue for defence.

If you go inside a region of the map, the graphics will show you how dense are the spots in that area, how many spots are there and how big is in square km, ho! and also will allow you to zoom in just hovering the mouse in the map.

sitezoom

Area in detail, you can access by clicking areas in the main map, clicking the half circles of the top graphic, or choosing an specific region in the drop down menu.

You can find and enjoy the infographic in this link http://bit.ly/2aFU7W8

Standard
Digital works

The Hobbit: The Battle of the five armies

El Hobbit, Erebor

The gate of Erebor. 3D Model by Edgar Jiménez; Art direction by Marco Hernández and Manuel Canales.

In the eve of the final movie of the Hobbit trilogy, we decide to give to our readers at nacion.com a new perspective of the scenary. Edgar Jiménez an terrific 3D artist, Alexander Sánchez the journalist of films and myself works together to develop this special site of the Hobbit’s Lonely Mountain, the field of the Battle of the five armies.

This its part of the promotional pieces:

nacion.com post for social media

nacion.com post for social media

nacion.com post for social media

nacion.com post for social media

Screen Shot 2014-12-14 at 3.42.35 PMThis site its a scroll controlled experience that allows to explore the Lonely Mountain and the city of Dwarves, Erebor. There you will find, besides, the 3d model, short texts hidden about key points of the movie. Hope you will enjoy it!
Here’s the link: http://goo.gl/8TMVKw 

Standard
Digital works

Cuna de un sociedad antigua

1500 años de historia, este es un especial sobre los descubrimientos arqueológicos en el sitio Nuevo Corinto, publicado en nacion.com

Cuna de una sociedad antigua. Especial en nacion.com sobre los descubrimientos en el sitio arqueológico Nuevo Corinto.

Cuna de una sociedad antigua. Especial en nacion.com sobre los descubrimientos en el sitio arqueológico Nuevo Corinto.

Standard

Recent work: Macondo

Reconstruction of the imaginary town of the book “Cien Años de Soledad” by Gabriel García Márquez.

Digital works

Recent work: Macondo

Image