The TimeSeriesMetaData class

This provides an overview of the basic functionality of the TimeSeriesMetaData class.

import astropy.units as u

import sunpy.data.sample
import sunpy.timeseries
from sunpy.net import Fido
from sunpy.net import attrs as a
from sunpy.time import TimeRange, parse_time

Search for timeseries data.

goes_res = Fido.search(a.Time("2010-11-03", "2010-11-03 23:59:59"), a.Instrument.xrs,
                       a.goes.SatelliteNumber(15))
norh_res = Fido.search(a.Time("2010-11-03", "2010-11-03 23:59:59"), a.Instrument.norh,
                       a.Wavelength(17 * u.GHz))

Download the data and load it.

goes_files = Fido.fetch(goes_res)
norh_files = Fido.fetch(norh_res)
goes_ts = sunpy.timeseries.TimeSeries(goes_files, source='XRS', concatenate=True)
norh_ts = sunpy.timeseries.TimeSeries(norh_files, source='NoRH', concatenate=True)

Combining the two series.

large_ts = goes_ts.concatenate(norh_ts)

The metadata can be easily viewed:

print(large_ts.meta)
|-------------------------------------------------------------------------------------------------|
|TimeRange                  | Columns         | Meta                                              |
|-------------------------------------------------------------------------------------------------|
|2010-11-02T22:44:51.801000 | Correlation Coe | simple: True                                      |
|            to             |                 | bitpix: 16                                        |
|2010-11-03T06:29:58.801000 |                 | naxis: 1                                          |
|                           |                 | naxis1: 27908                                     |
|                           |                 | bscale: 3.053e-07                                 |
|                           |                 | bzero: 0.0                                        |
|                           |                 | bunit: CORRELATION COEFF.                         |
|                           |                 | blank: -32768                                     |
|                           |                 | crval1: 22:44:51.801                              |
|                           |                 | startfrm: 1                                       |
|                           |                 | ...                                               |
|-------------------------------------------------------------------------------------------------|
|2010-11-03T00:00:00.851000 | xrsa            | conventions: ACDD-1.3, Spase v2.2.6               |
|            to             | xrsb            | title: GOES 1-15 L2 XRS high-resolution Irradiance|
|2010-11-03T23:59:58.881000 | xrsa_quality    | id: b' '                                          |
|                           | xrsb_quality    | summary: The X-ray flux product consists of reproc|
|                           |                 | keywords: NumericalData.MeasurementType.Irradiance|
|                           |                 | keywords_vocabulary: SPASE: Space Physics Archive |
|                           |                 | naming_authority: gov.nesdis.noaa                 |
|                           |                 | history: See algorithm information.               |
|                           |                 | source: GOES XRS counts                           |
|                           |                 | processing_level: Level 2                         |
|                           |                 | ...                                               |
|-------------------------------------------------------------------------------------------------|

You can reduce the depth of the view:

print(large_ts.meta.to_string(2))
|-------------------------------------------------------------------------------------------------|
|TimeRange                  | Columns         | Meta                                              |
|-------------------------------------------------------------------------------------------------|
|2010-11-02T22:44:51.801000 | Correlation Coe | simple: True                                      |
|2010-11-03T06:29:58.801000 |                 | bitpix: 16                                        |
|                           |                 | ...                                               |
|-------------------------------------------------------------------------------------------------|
|2010-11-03T00:00:00.851000 | xrsa            | conventions: ACDD-1.3, Spase v2.2.6               |
|2010-11-03T23:59:58.881000 | xrsb            | title: GOES 1-15 L2 XRS high-resolution Irradiance|
|                           | ...             | ...                                               |
|-------------------------------------------------------------------------------------------------|

The TimeSeriesMetaData class stores all the individual file metadata MetaDict objects as 3-tuple entries in it’s internal list with the TimeRange, list of column names and metadictionary. This list is stored in order of ascending TR.start. Access of the the list is done using the metadata property:

[(   <sunpy.time.timerange.TimeRange object at 0x7f3715979c30>
    Start: 2010-11-02 22:44:51
    End:   2010-11-03 06:29:58
    Center:2010-11-03 02:37:25
    Duration:0.3229976851851851 days or
           7.751944444444443 hours or
           465.11666666666656 minutes or
           27906.999999999996 seconds
, ['Correlation Coefficient'], MetaDict([('simple', True), ('bitpix', 16), ('naxis', 1), ('naxis1', 27908), ('bscale', 3.053e-07), ('bzero', 0.0), ('bunit', 'CORRELATION COEFF.'), ('blank', -32768), ('crval1', '22:44:51.801'), ('startfrm', 1), ('crpix1', 1.0), ('ctype1', 'TIME(SECOND)'), ('cdelt1', 1.0), ('fund-amp', 11227), ('object', 'SUN'), ('image1', 'R+L'), ('obs-freq', '17GHZ'), ('telescop', 'RADIOHELIOGRAPH'), ('date-obs', '2010-11-02'), ('jstdate', '2010-11-03'), ('jsttime', '07:44:51.801'), ('origin', 'NOBEYAMA RADIO OBS'), ('date', '****-**-**'), ('pversion', '6.00'), ('comment', 'BUNIT IS  THE MEAN  OF CORRELATION COEFFICIENTS OF THE ANTENNA BASELINELENGTH ABOVE  100*D*COS(THETA).THETA: ANGLE BETWEEN THE ANTENNA BASELINE & THE DIRECTION OF THE SUN.D    : FUNDAMENTAL BASELINE LENGTH OF 1.528 METERSSFACT: FACTOR INTRODUCED TO REDUCE THE DISCRETIZATION ERROR'), ('history', ''), ('keycomments', {'SIMPLE': 'BASIC FITS TAPE FORM', 'BITPIX': "2-BYTE TWO'S COMPLEMENT INTEGER", 'NAXIS': '1-DIMENSIONAL TEMPORAL PROF', 'NAXIS1': '# PIXEL/ROW', 'BSCALE': 'SFACT/32768, REAL = TAPE * BSCALE + BZERO', 'BZERO': 'NO BITS ADDED', 'BUNIT': 'CROSS CORRELATION COEFFICIENT', 'BLANK': 'VALUE FOR VALUE-UNDEFINED PIXEL', 'CRVAL1': 'REF POINT VALUE IN HH:MM:SS (UT)', 'STARTFRM': 'REF FRAME NUMBER', 'CRPIX1': 'REF POINT PIXEL LOCATION', 'CTYPE1': 'TYPE OF PHYSICAL COORD. ON AXIS1', 'CDELT1': 'PIXEL SIZE ON AXIS1 IN SECOND', 'FUND-AMP': 'COR_AMP. OF FUND. ANTENNA SPACING AT STARTFRAM', 'OBJECT': 'OBJECT NAME', 'OBS-FREQ': 'INTENSITY AT 17GHZ', 'TELESCOP': 'NOBEYAMA RADIO HELIOGRAPH', 'DATE-OBS': 'DATE OF DATA ACQUISITION YYYY-MM-DD (UT)', 'JSTDATE': 'DATE OF DATA ACQUISITION YYYY-MM-DD (JST)', 'JSTTIME': 'TIME OF DATA ACQUISITION HH/MM/SS (JST)', 'ORIGIN': 'TAPE WRITE INSTITUTION', 'DATE': 'DATE WHEN DATA FILE WRITTEN YYYY-MM-DD (JST)', 'PVERSION': 'CODED BY N.SHINOHARA          14.MAR.95'})])), (   <sunpy.time.timerange.TimeRange object at 0x7f371597bb50>
    Start: 2010-11-03 00:00:00
    End:   2010-11-03 23:59:58
    Center:2010-11-03 11:59:59
    Duration:0.999977199074074 days or
           23.999452777777776 hours or
           1439.9671666666666 minutes or
           86398.03 seconds
, ['xrsa', 'xrsb', 'xrsa_quality', 'xrsb_quality'], MetaDict([('conventions', 'ACDD-1.3, Spase v2.2.6'), ('title', 'GOES 1-15 L2 XRS high-resolution Irradiances'), ('id', b' '), ('summary', 'The X-ray flux product consists of reprocessed science-quality high-resolution X-ray solar measurements. The GOES X-Ray Sensor (XRS) measures in a short wavelength channel (XRS-A) with a nominal bandpass of 0.05 to 0.4 nm and in a longer wavelength channel (XRS-B) with a bandpass of 0.1 to 0.8 nm. \n\nFor the given bandpass, the XRS response is based on a flat spectrum beginning with GOES-3. A true solar spectrum is assumed for GOES 1-2.\n\nGOES 13-15 measurements are at 2-s cadence. \nGOES 1-12 measured at a 3-s cadence. \n\nFor GOES 3-12, the XRS-A response was originally determined assuming a bandpass of 0.05 to 0.3 nm.'), ('keywords', 'NumericalData.MeasurementType.Irradiance'), ('keywords_vocabulary', 'SPASE: Space Physics Archive Search and Extract Data Model version 2.2.6, GCMD: NASA Global Change Master Directory (GCMD) Earth Science Keywords version 8.5'), ('naming_authority', 'gov.nesdis.noaa'), ('history', 'See algorithm information.'), ('source', 'GOES XRS counts'), ('processing_level', 'Level 2'), ('processing_level_description', 'Derived products'), ('license', 'These data may be redistributed and used without restriction. '), ('acknowledgment', 'TBD'), ('metadata_link', b''), ('date_created', b' '), ('creator_name', 'Erika Zetterlund, Janet Machol'), ('creator_type', 'person'), ('creator_institution', 'DOC/NOAA/NCEI/CCOG/STP'), ('creator_email', 'goesr.exis@noaa.gov'), ('creator_url', 'https://www.ncei.noaa.gov/'), ('institution', 'DOC/NOAA/NESDIS> U.S. Department of Commerce, National Oceanic and Atmospheric Administration, National Environmental Satellite, Data, and Information Services'), ('publisher_name', 'National Centers for Environmental Information'), ('publisher_type', 'institution'), ('publisher_institution', 'DOC/NOAA/NESDIS/NCEI'), ('publisher_email', 'goesr.exis@noaa.gov'), ('publisher_url', 'https://www.ncei.noaa.gov/'), ('references', 'README FOR SCIENCE QUALITY GOES 13-15 XRS DATA'), ('instrument', 'GOES 1-15 X-ray Irradiance Sensor (XRS)'), ('instrument_id', b' '), ('orbital_slot', b' '), ('program', 'Geostationary Operational Environmental Satellite (GOES)'), ('project', 'Geostationary Operational Environmental Satellite (GOES) Solar Terrestrial Physics at NCEI'), ('platform', b' '), ('l1b_system_environment', b' '), ('l1b_production_site', b' '), ('time_coverage_start', b' '), ('time_coverage_end', b' '), ('time_coverage_resolution', 'PT2S'), ('processing_parameters_file', b' '), ('algorithm', b''), ('algorithm_version', b' '), ('algorithm_date', b' '), ('algorithm_parameters', 'None'), ('input_system_environments', b' '), ('input_files_first', b' '), ('input_files_last', b' '), ('input_files_total', b' '), ('filename_id', 'sci_gxrs-l2-irrad_g15_d20101103_v0-0-0.nc')]))]

The TimeSeriesMetaData class has a number of other properties, including the timerange property that returns a TimeRange for the entire metadata:

print(large_ts.meta.time_range)
<sunpy.time.timerange.TimeRange object at 0x7f371597a410>
 Start: 2010-11-02 22:44:51
 End:   2010-11-03 23:59:58
 Center:2010-11-03 11:22:25
 Duration:1.0521652777777777 days or
        25.251966666666664 hours or
        1515.118 minutes or
        90907.07999999999 seconds

Further properties can be used to get lists of details, e.g., List of the time ranges

print(large_ts.meta.timeranges)
[   <sunpy.time.timerange.TimeRange object at 0x7f3715979c30>
    Start: 2010-11-02 22:44:51
    End:   2010-11-03 06:29:58
    Center:2010-11-03 02:37:25
    Duration:0.3229976851851851 days or
           7.751944444444443 hours or
           465.11666666666656 minutes or
           27906.999999999996 seconds
,    <sunpy.time.timerange.TimeRange object at 0x7f371597bb50>
    Start: 2010-11-03 00:00:00
    End:   2010-11-03 23:59:58
    Center:2010-11-03 11:59:59
    Duration:0.999977199074074 days or
           23.999452777777776 hours or
           1439.9671666666666 minutes or
           86398.03 seconds
]

List of the column names

print(large_ts.meta.columns)
['Correlation Coefficient', 'xrsa', 'xrsa_quality', 'xrsb', 'xrsb_quality']

List of the meta dictionaries

print(large_ts.meta.metas)
[MetaDict([('simple', True), ('bitpix', 16), ('naxis', 1), ('naxis1', 27908), ('bscale', 3.053e-07), ('bzero', 0.0), ('bunit', 'CORRELATION COEFF.'), ('blank', -32768), ('crval1', '22:44:51.801'), ('startfrm', 1), ('crpix1', 1.0), ('ctype1', 'TIME(SECOND)'), ('cdelt1', 1.0), ('fund-amp', 11227), ('object', 'SUN'), ('image1', 'R+L'), ('obs-freq', '17GHZ'), ('telescop', 'RADIOHELIOGRAPH'), ('date-obs', '2010-11-02'), ('jstdate', '2010-11-03'), ('jsttime', '07:44:51.801'), ('origin', 'NOBEYAMA RADIO OBS'), ('date', '****-**-**'), ('pversion', '6.00'), ('comment', 'BUNIT IS  THE MEAN  OF CORRELATION COEFFICIENTS OF THE ANTENNA BASELINELENGTH ABOVE  100*D*COS(THETA).THETA: ANGLE BETWEEN THE ANTENNA BASELINE & THE DIRECTION OF THE SUN.D    : FUNDAMENTAL BASELINE LENGTH OF 1.528 METERSSFACT: FACTOR INTRODUCED TO REDUCE THE DISCRETIZATION ERROR'), ('history', ''), ('keycomments', {'SIMPLE': 'BASIC FITS TAPE FORM', 'BITPIX': "2-BYTE TWO'S COMPLEMENT INTEGER", 'NAXIS': '1-DIMENSIONAL TEMPORAL PROF', 'NAXIS1': '# PIXEL/ROW', 'BSCALE': 'SFACT/32768, REAL = TAPE * BSCALE + BZERO', 'BZERO': 'NO BITS ADDED', 'BUNIT': 'CROSS CORRELATION COEFFICIENT', 'BLANK': 'VALUE FOR VALUE-UNDEFINED PIXEL', 'CRVAL1': 'REF POINT VALUE IN HH:MM:SS (UT)', 'STARTFRM': 'REF FRAME NUMBER', 'CRPIX1': 'REF POINT PIXEL LOCATION', 'CTYPE1': 'TYPE OF PHYSICAL COORD. ON AXIS1', 'CDELT1': 'PIXEL SIZE ON AXIS1 IN SECOND', 'FUND-AMP': 'COR_AMP. OF FUND. ANTENNA SPACING AT STARTFRAM', 'OBJECT': 'OBJECT NAME', 'OBS-FREQ': 'INTENSITY AT 17GHZ', 'TELESCOP': 'NOBEYAMA RADIO HELIOGRAPH', 'DATE-OBS': 'DATE OF DATA ACQUISITION YYYY-MM-DD (UT)', 'JSTDATE': 'DATE OF DATA ACQUISITION YYYY-MM-DD (JST)', 'JSTTIME': 'TIME OF DATA ACQUISITION HH/MM/SS (JST)', 'ORIGIN': 'TAPE WRITE INSTITUTION', 'DATE': 'DATE WHEN DATA FILE WRITTEN YYYY-MM-DD (JST)', 'PVERSION': 'CODED BY N.SHINOHARA          14.MAR.95'})]), MetaDict([('conventions', 'ACDD-1.3, Spase v2.2.6'), ('title', 'GOES 1-15 L2 XRS high-resolution Irradiances'), ('id', b' '), ('summary', 'The X-ray flux product consists of reprocessed science-quality high-resolution X-ray solar measurements. The GOES X-Ray Sensor (XRS) measures in a short wavelength channel (XRS-A) with a nominal bandpass of 0.05 to 0.4 nm and in a longer wavelength channel (XRS-B) with a bandpass of 0.1 to 0.8 nm. \n\nFor the given bandpass, the XRS response is based on a flat spectrum beginning with GOES-3. A true solar spectrum is assumed for GOES 1-2.\n\nGOES 13-15 measurements are at 2-s cadence. \nGOES 1-12 measured at a 3-s cadence. \n\nFor GOES 3-12, the XRS-A response was originally determined assuming a bandpass of 0.05 to 0.3 nm.'), ('keywords', 'NumericalData.MeasurementType.Irradiance'), ('keywords_vocabulary', 'SPASE: Space Physics Archive Search and Extract Data Model version 2.2.6, GCMD: NASA Global Change Master Directory (GCMD) Earth Science Keywords version 8.5'), ('naming_authority', 'gov.nesdis.noaa'), ('history', 'See algorithm information.'), ('source', 'GOES XRS counts'), ('processing_level', 'Level 2'), ('processing_level_description', 'Derived products'), ('license', 'These data may be redistributed and used without restriction. '), ('acknowledgment', 'TBD'), ('metadata_link', b''), ('date_created', b' '), ('creator_name', 'Erika Zetterlund, Janet Machol'), ('creator_type', 'person'), ('creator_institution', 'DOC/NOAA/NCEI/CCOG/STP'), ('creator_email', 'goesr.exis@noaa.gov'), ('creator_url', 'https://www.ncei.noaa.gov/'), ('institution', 'DOC/NOAA/NESDIS> U.S. Department of Commerce, National Oceanic and Atmospheric Administration, National Environmental Satellite, Data, and Information Services'), ('publisher_name', 'National Centers for Environmental Information'), ('publisher_type', 'institution'), ('publisher_institution', 'DOC/NOAA/NESDIS/NCEI'), ('publisher_email', 'goesr.exis@noaa.gov'), ('publisher_url', 'https://www.ncei.noaa.gov/'), ('references', 'README FOR SCIENCE QUALITY GOES 13-15 XRS DATA'), ('instrument', 'GOES 1-15 X-ray Irradiance Sensor (XRS)'), ('instrument_id', b' '), ('orbital_slot', b' '), ('program', 'Geostationary Operational Environmental Satellite (GOES)'), ('project', 'Geostationary Operational Environmental Satellite (GOES) Solar Terrestrial Physics at NCEI'), ('platform', b' '), ('l1b_system_environment', b' '), ('l1b_production_site', b' '), ('time_coverage_start', b' '), ('time_coverage_end', b' '), ('time_coverage_resolution', 'PT2S'), ('processing_parameters_file', b' '), ('algorithm', b''), ('algorithm_version', b' '), ('algorithm_date', b' '), ('algorithm_parameters', 'None'), ('input_system_environments', b' '), ('input_files_first', b' '), ('input_files_last', b' '), ('input_files_total', b' '), ('filename_id', 'sci_gxrs-l2-irrad_g15_d20101103_v0-0-0.nc')])]

When you truncate the TimeSeries, the metadata is truncated too:

large_trunc_ts = large_ts.truncate(TimeRange('2010-11-03 13:59:57.468999',
                                             '2010-11-04 13:59:56.091999'))
print(large_trunc_ts.meta.to_string(2))
|-------------------------------------------------------------------------------------------------|
|TimeRange                  | Columns         | Meta                                              |
|-------------------------------------------------------------------------------------------------|
|2010-11-03T13:59:59.421000 | xrsa            | conventions: ACDD-1.3, Spase v2.2.6               |
|2010-11-03T23:59:58.881000 | xrsb            | title: GOES 1-15 L2 XRS high-resolution Irradiance|
|                           | ...             | ...                                               |
|-------------------------------------------------------------------------------------------------|

Finding metadata can be achieved using the find method and applying filters for time and/or colname. This returns another TimeSeriesMetaData object:

large_trunc_ts.meta.find(time=parse_time('2010-11-04 09:01:16'))
large_trunc_ts.meta.find(time='2010-11-04 09:01:16', colname='xrsb')

# You can get the time of a row a from the TimeSeries object's times:
large_trunc_ts.meta.find(time=large_trunc_ts.time[10])

# There is also a get method:
large_trunc_ts.meta.get('telescop')

# Again, filters can be used:
large_trunc_ts.meta.get('telescop', time='2010-11-04 09:01:16', colname='xrsb')

# And if we just want the values, the values method returns just a list:
large_trunc_ts.meta.get('telescop').values()
[]

You can update values similar to dictionaries, though all of the contained MetaDict objects will be updated that match your filter criteria:

large_trunc_ts.meta.update({'new_key_1': 'added to all.'})
large_trunc_ts.meta.update({'new_key_2': 'added to some.'}, colname='xrsa')
print(large_trunc_ts.meta.to_string(2))

# but you can't overwrite previous entries without setting the overwrite kwarg,
# this is to protect the integrity of the metadata:

large_trunc_ts.meta.update({'new_key_1': 'changed'}, overwrite=True)
print(large_trunc_ts.meta.to_string(2))
|-------------------------------------------------------------------------------------------------|
|TimeRange                  | Columns         | Meta                                              |
|-------------------------------------------------------------------------------------------------|
|2010-11-03T13:59:59.421000 | xrsa            | conventions: ACDD-1.3, Spase v2.2.6               |
|2010-11-03T23:59:58.881000 | xrsb            | title: GOES 1-15 L2 XRS high-resolution Irradiance|
|                           | ...             | ...                                               |
|-------------------------------------------------------------------------------------------------|

|-------------------------------------------------------------------------------------------------|
|TimeRange                  | Columns         | Meta                                              |
|-------------------------------------------------------------------------------------------------|
|2010-11-03T13:59:59.421000 | xrsa            | conventions: ACDD-1.3, Spase v2.2.6               |
|2010-11-03T23:59:58.881000 | xrsb            | title: GOES 1-15 L2 XRS high-resolution Irradiance|
|                           | ...             | ...                                               |
|-------------------------------------------------------------------------------------------------|

Total running time of the script: ( 0 minutes 7.834 seconds)

Gallery generated by Sphinx-Gallery