blogging

Visualizing organic carbon in near real time. –A step-by-step guide

Long time ago someone on twitter ask me to do an explainer on how I did the “smoke” animations for this Reuters piece. It has been a while since then, but maybe it would be useful for someone out there, even if that mean learning how NOT to do things.

Before continuing, to follow my guide and visualize organic carbon, you should be able to use your terminal window, QGIS and optional Adobe After Effects.

Earth globe showing organic carbon released into the atmosphere by wildfires in America
Organic carbon released into the atmosphere during the wildfires season in California in 2020

Let’s talk about this wonderful data first

NASA’s Global Modeling and Assimilation Office Research Site (GMAO) provides a number of models from different data sets, this is basically a collection of data from many different services processed for historical records or forecast models. This data works well for a global picture or continent level even, but maybe isn’t a good idea to use this data for a country level analysis, for those uses you may want to check other sources of the data instead of GMAO models, like MODIS for instance if you you are looking for similar data.

ORGANIC CARBON

There are a lot of different sets of products available at the GMAO servers, you can check details here, here and here. However for purposes of this practical guide, I’ll be focusing in the emissions of Organic Carbon which is stored into the tavg3_2d_aer_Nx set. That’s a GEOS5 FP 2d time-averaged primary aerosol diagnostics, which includes Organic Carbon Column mass density in the 38th band, there is some documentation available in this pdf. ( No worries if is this sounds too technical stay with me and keep going. )

A day of observations accounts for 8 files since this data is processed every 3 hours. This is great for animation because it would look smooth. Knowing that, let’s move to our guide.


Step 1. Get the data

The data is stored into this url. You can go into the folders and get all 8 files for each day manually if you like or get them with a command line using wget or curl into the terminal. You just need to know a little of the url structure:

url structure
GMAO organic carbon files and url structure
  • Create a folder to store your files with some name like data
  • Open your terminal window
  • Type cd in the terminal window
  • Drag the folder you created inside the window
A “cd folder” example stolen from the web.


Then run a short command like the following, you would get a file named 20220619_0130.nc4 in you data folder:

curl https://portal.nccs.nasa.gov/datashare/gmao/geos-fp/das/Y2022/M06/D19/GEOS.fp.asm.tavg3_2d_aer_Nx.20220619_0130.V01.nc4 -o 20220619_0130.nc4

Note that I have renamed the output ( -o ) with a shorter name. The file will go to your folder ready to use into GQIS. Of course you will need a few more files to run an animation. Remember that this data is available for every 3 hours daily, so you need to set the url and name for something like this:

01:30 AM >> 20220619_0130.V01.nc4
04:30 AM >> 20220619_0430.V01.nc4
07:30 AM >> 20220619_0730.V01.nc4
10:30 AM >> 20220619_1030.V01.nc4
01:30 PM >> 20220619_1330.V01.nc4
04:30 PM >> 20220619_1630.V01.nc4
07:30 PM >> 20220619_1930.V01.nc4
10:30 PM >> 20220619_2230.V01.nc4

Just create a text list with all the urls you need and run the script into the terminal window with the same process:

curl -O [URL1] -O [URL2]

Each file is usually about 120MB, if there’s something wrong with the data the file will be created anyway but would be an empty file of just a few KB. Do a day or two first and check, that’s 8-16 files, check them, if all looks good load a few more if you like.

Step 2. Loading the data

Once you have a nice folder with all the files you want, you can just drag and drop the .nc4 files into QGIS. We are looking for the 38th Band, OCCMASS which is our Organic Carbon Column mass:

QGIS prompt window when you drop one of the file in.

Once you have the data loaded, you want to set the data projection to WGS 84, this will enable the data layers to be re-projected later on. To do that, select all you data layers, right click on them, and select Layer CRS > Set Layer CRS > 4326. Be sure of selecting all the layers at once so you do this only one time. Otherwise you will need to doing over and over.

Data layers projection to WGS 84.

Since this is a good global data set, you may want to load a globe for reference, you can use your own custom projection, or use a plugin like globe builder:

Access Globe Builder from the plugins menu > Manage and Install > type: Globe.

Once installed, just run it from the little globe icon, or in the menu plugins > Globe builder > Build globe view. You have a few options there, play around with the center point lat/long. You will see that this data sets always have large concentrations of emissions in Africa, maybe that’s a great place to start. I’ll do a similar view to the California story for now.

Step 3. Styling your map

The color ramp is important, you want to have a data layer that can be overlayed in the base map, so you want to have white/black for the lower values and high contrast in the other end of the data, since we are working on white background I’m using white to black with yellow and brown stops. Check what are the highest values in your data set the style for on layer to something like this:

Number in the min/max will change depending on the highest values of your data and the style you want. This image is set for OCCMASS from June 19th, 2022, 4:30 pm.

Once you have the ideal color ramp for one layer, right click on that layer, go to Styles > Copy style. Then select all you carbon data layers, right click on them and select Styles > Paste Style.

Step 4. Preparing to export your map

You are almost done, by this point you can see how each data layer creates swirls in the atmosphere, maybe some evolution of it too just by toggling the layers visibility. I like to have all the layers well organized so you can quick check the data. I’m maybe a little too obsessive but I usually rename all layers and groups to something like this:

QGIS layers panel.

The name change works if you are using an automatic export of all layers, the script takes the name of the layer to save each file. But there are alternative ways to do this if you’re not as crazy as I am and don’t want to spend time manually renaming.

Step 5. Export your map

There are many ways of doing this, you can set up the time for each layer by using the temporal controller, there’s a good guide here. That way you can get a mp4 video right away from QGIS, but you need to set up each data layer time manually.

You can also use a little code to export each layer into an image, which you can then import into After Effects. To do that, the first step of course, is to get the script. Download the files HERE.

Now, go to the plugins menu at the top, there, you will see the Python console, go and click that, you will see this window popping-up:

Python console in QGIS.

Click the paper icon, then click the folder icon and select the python script you dowloaded above. Just be careful with the filePath option.

If you are on a mac, right click your output folder and hold the option key, that will allow you to copy the absolute path of you folder, paste that to replace the filePath field value (the green text in the image below). If you are on Windows, just make sure to get the absolute path and not a relative one.

I left some annotations on the script to better understand what each part is, it’s based on a script someone did with Vietnamese annotations, source and credit are in the drive link too.

Now just click the play button in the python console, seat back and look all the frames of your animation loading in the output folder you selected. You should see a file for each of your layers when the script finishes.

Step 6. Export your animation

Take all the files this into After Effects. First, add your carbon data as sequence (0001.png, 0002.png, 0003.png…), keep that in a sub-composition and use a multiply blend mode to overlay the layers, then add the countries/land and the optional halo.

Finally, in the drive folder you will see a .aep file, that’s a simple number animation to control dates, copy the text layer into your composition. You know when the data starts and when it ends, in the example is just 3 days 19-21, “June” is a different text layer, so add those numeric values to the keyframes into the text layer you have copied, and leave it at the very top:

Once you are all set, just export to media encoder to get you mp4 animation.

If any of this doesn’t make sense to you, or if you’re having trouble with a step, feel free to reach out to me on Twitter. I will be happy to hear from you.


Happy mapping!

Standard
Digital works, Motion graphics

Quick graphics for social networks

Last year (2015) I start many new projects, some of them are still progress, but one of these are the quick graphics for social media.

At the unit of infographics and datavisualization, we spend many time in daily production and in special huge projects, I believe that we have enough time and muscle to produce frequently these small graphics,  every single infographic must have a print version, desktop digital versión, mobile version but also, i like to choose some of these to develop a social media version, sometimes a gif, or an image or even a short video with a compact version of the main graphic.

So, here is some of my favorites picks:

The spooky asteroid:

In october 31, an asteroid pass near the earth (near in space terms), this asteroid was renamed “the spooky asteroid” and take the attention of the world by the time of his visit, so it was a nice opportunity for a quick graphic for social media:


Compact version for social media. Motion-Infographic by Marco Hernández.

spooky

Gif version for social media. Infographic by Marco Hernández.

pesoInfo_spooky

Desktop digital version for article. Infographic by Marco Hernández.

The MAVEN findings:

The NASA made the announcement of the Maven mission findings about what happened with the atmosphere and magnetic fields of the red planet. We are awaiting the press release with a stock of pieces to develop soon as we can a simply animation for social media:

ARTpieces

Illustrations by Dominick Baltodano and Marco Hernández.


Compact version for social media. Motion-infographic by Marco Hernández.

MAVEN-Marte_LNCIMA20151105_0084_1

Desktop version, infographic by Edgar Jiménez, Carlos Fonseca and Marco Hernández.

Hope in this 2016 have many more of these chances to create more quick graphics for social media, these are a great way to engage readers with content made exclusively for consumption on social networks.

Standard

A few days ago we released an special about the births of a century in Costa Rica, I had the chance to produce this motion for. It’s nice to get raw data and transform it in to visuals, alternative ways to see the past or future of a population, if you like to take a look in to the full site here http://bit.ly/1kpN45p you may use google chrome for translation from spanish 🙂

Digital works

WebGL power for infographics

In the last weeks I’ been integrating the WebGL Technology to take advantage of the great skills of my partners of the infographics department here at nacion.com Daniel Solano and Edgar Jiménez both of them great artist´s, they works on Cinema 4D to produce print graphics for the regular editions of the news corporation, that kind of work it’s simply awesome and it’s a shame don’t use all his digital potential.

So, here’s a pair of work that we worked on together to improve the experiences for the readers.

Lost in space for ever and ever

with Daniel Solano

This slideshow requires JavaScript.

On september 15th of 1965 Fox and CBS launch in USA the TV Show “Lost in space”, 50 years after, we wants to remember the serie, the main characters and vehicles that captivated audiences for decades beyond its premiere. Here’s an exclusive access to the 3D models, The Chariot, and The Jupiter 2.

Original feature

Original feature

It’s really nice have the chance to work in this kind of topics, science and space are one of my all time favorites. if you want to know more about the graphic the original feature at nacion.com here.

Technology the leading actor

with Edgar Jiménez

NASA´s SEV Vehicle

NASA´s SEV Vehicle

At least nine of the technologies developed by NASA towards Mars space exploration are mentioned in the book and movie The Martian, this 3D infographic collects those coincidences.

Martian tech

Martian tech

I also want to add some short animations as gif files in the graphic to give support to the information and the whole infographic, Edgar and me develop this sort animations in After Effects.

This slideshow requires JavaScript.

if you want to know more about the graphic the original feature at nacion.com here.

The original feature publication at nacion.com

The original feature publication at nacion.com

I very lucky to have co-workers with this kind of skills, and the space to make this kind of publications, I love infographics, and space…   🙂

Standard
Digital works

CTCC Parque Viva

nocturno_00

A few hours more than a full day of work takes to two co-workers and me, to release today this special feature about the V8 cars competition in the “Parque Viva” racetrack.
Animated graphics, details about the cars and restrictions and a complementary key note of Jose Luis Rodríguez fusioned together to create this digital experience.
I hope you will enjoy it!

Graphics inside the special

Graphics inside the special

Graphics inside the special

Graphics inside the special

Graphics inside the special

Graphics inside the special

Graphics inside the special

Graphics inside the special

Visite the special site of nacion.com here: http://goo.gl/gzjLYy

Standard
Digital works

The Hobbit: The Battle of the five armies

El Hobbit, Erebor

The gate of Erebor. 3D Model by Edgar Jiménez; Art direction by Marco Hernández and Manuel Canales.

In the eve of the final movie of the Hobbit trilogy, we decide to give to our readers at nacion.com a new perspective of the scenary. Edgar Jiménez an terrific 3D artist, Alexander Sánchez the journalist of films and myself works together to develop this special site of the Hobbit’s Lonely Mountain, the field of the Battle of the five armies.

This its part of the promotional pieces:

nacion.com post for social media

nacion.com post for social media

nacion.com post for social media

nacion.com post for social media

Screen Shot 2014-12-14 at 3.42.35 PMThis site its a scroll controlled experience that allows to explore the Lonely Mountain and the city of Dwarves, Erebor. There you will find, besides, the 3d model, short texts hidden about key points of the movie. Hope you will enjoy it!
Here’s the link: http://goo.gl/8TMVKw 

Standard