← Back to Project Matsu main

Matsu 'Wheel' Analytics

An efficient framework to support multiple analytics applied to a scan of all incoming satellite imagery data

 

Project Matsu is a collaboration with NASA to process Earth Observing-1 (EO-1) satellite imagery with the goal of detecting floods, fires, and natural disasters and providing relevant information to first responders. The EO-1 data are freely available from the Open Science Data Cloud (OSDC) for public download and are also available to compute over within the cloud for OSDC users.

The Project Matsu "Wheel" applies Hadoop-based analytics to multi-spectral data from Earth Observing-1's ALI and Hyperion instruments daily as the data become available from NASA. The daily summary reports and individual "scene" (image) reports generated by the Matsu Wheel scanning analytics are made available here.

Briefly, EO-1 Level 0 (raw data) images are received daily from NASA and are transformed into various Level 1 products by NASA using the Open Science Data Cloud. This transformation may not occur each day. When new Level 1G products are available, the Matsu Wheel staging system detects the new images, which are then converted to SequenceFiles and uploaded to HDFS. The Matsu Wheel analytics are then applied via MapReduce once a day to build the reports below. If the Level 1G data are not available at that time, images will be processed the next day, but not all days have images. Daily processing began in July 2013 for some analytics. Prior to that, a few select days from 2010 and 2012 were processed. We are continuing to add beta analytics.

 

Current beta "Wheel" analytics
The analytics are run, and the reports are generated, on Skidmore, an Open Science Data Cloud resource intended for computational projects. The main service provided by Skidmore is Hadoop over 25 nodes with 800 compute cores and 3200 GB of compute RAM. The usable storage available is 295 TB.

[ICO]NameLast modifiedSize
[DIR]9999/2014-04-16 19:56 -
[DIR]2016-12/2016-12-02 09:40 -
[DIR]2016-11/2016-11-02 08:36 -
[DIR]2016-10/2016-10-05 08:53 -
[DIR]2016-09/2016-09-02 08:34 -
[DIR]2016-08/2016-08-02 07:03 -
[DIR]2016-07/2016-07-05 09:07 -
[DIR]2016-06/2016-06-01 14:47 -
[DIR]2016-05/2016-05-16 21:57 -
[DIR]2016-04/2016-05-16 13:51 -
[DIR]2016-03/2016-04-19 11:40 -
[DIR]2016-02/2016-04-16 02:52 -
[DIR]2016-01/2016-04-11 18:14 -
[DIR]2015-12/2015-12-01 08:32 -
[DIR]2015-11/2015-11-02 12:20 -
[DIR]2015-10/2015-10-02 08:36 -
[DIR]2015-09/2015-09-02 08:21 -
[DIR]2015-08/2015-08-07 16:01 -
[DIR]2015-07/2015-07-22 22:35 -
[DIR]2015-06/2015-06-02 09:35 -
[DIR]2015-05/2015-05-05 11:09 -
[DIR]2015-04/2015-04-02 09:40 -
[DIR]2015-03/2015-03-01 09:26 -
[DIR]2015-02/2015-02-01 08:53 -
[DIR]2015-01/2015-01-07 11:44 -
[DIR]2014-12/2014-12-16 18:52 -
[DIR]2014-11/2014-12-01 10:35 -
[DIR]2014-10/2014-11-04 13:58 -
[DIR]2014-09/2014-09-02 16:28 -
[DIR]2014-08/2014-08-26 20:43 -
[DIR]2014-07/2014-07-24 03:37 -
[DIR]2014-06/2014-06-18 17:20 -
[DIR]2014-05/2015-02-12 19:10 -
[DIR]2014-04/2014-08-18 15:28 -
[DIR]2014-03/2014-08-18 15:27 -
[DIR]2014-02/2014-04-20 11:06 -
[DIR]2014-01/2014-10-14 01:18 -
[DIR]2013-12/2014-03-25 17:18 -
[DIR]2013-11/2014-03-25 17:18 -
[DIR]2013-10/2014-03-25 17:18 -
[DIR]2013-09/2014-03-25 17:18 -
[DIR]2013-08/2014-03-25 17:18 -
[DIR]2013-07/2014-03-25 17:18 -
[DIR]2012/2014-03-25 17:18 -
[DIR]2010/2014-03-25 17:18 -
[TXT]phpinfo.php2017-01-09 13:54 46
[   ]Normalization.README2014-03-25 17:18 698
[   ]index.html-old2017-02-03 12:30 703