The TimeSeriesMetaData class

This provides an overview of the basic functionality of the TimeSeriesMetaData class.

import astropy.units as u

import sunpy.data.sample
import sunpy.timeseries
from sunpy.net import Fido
from sunpy.net import attrs as a
from sunpy.time import TimeRange, parse_time

Search for timeseries data.

goes_res = Fido.search(a.Time("2010-11-03", "2010-11-03 23:59:59"), a.Instrument.xrs,
                       a.goes.SatelliteNumber(15))
norh_res = Fido.search(a.Time("2010-11-03", "2010-11-03 23:59:59"), a.Instrument.norh,
                       a.Wavelength(17 * u.GHz))

Download the data and load it.

goes_files = Fido.fetch(goes_res)
norh_files = Fido.fetch(norh_res)
goes_ts = sunpy.timeseries.TimeSeries(goes_files, source='XRS', concatenate=True)
norh_ts = sunpy.timeseries.TimeSeries(norh_files, source='NoRH', concatenate=True)

Combining the two series.

large_ts = goes_ts.concatenate(norh_ts)

The metadata can be easily viewed:

print(large_ts.meta)

Out:

|-------------------------------------------------------------------------------------------------|
|TimeRange                  | Columns         | Meta                                              |
|-------------------------------------------------------------------------------------------------|
|2010-11-02T22:44:51.801000 | Correlation Coe | simple: True                                      |
|            to             |                 | bitpix: 16                                        |
|2010-11-03T06:29:58.801000 |                 | naxis: 1                                          |
|                           |                 | naxis1: 27908                                     |
|                           |                 | bscale: 3.053e-07                                 |
|                           |                 | bzero: 0.0                                        |
|                           |                 | bunit: CORRELATION COEFF.                         |
|                           |                 | blank: -32768                                     |
|                           |                 | crval1: 22:44:51.801                              |
|                           |                 | startfrm: 1                                       |
|                           |                 | ...                                               |
|-------------------------------------------------------------------------------------------------|
|2010-11-02T23:59:34.851082 | xrsa            | conventions: b'ACDD-1.3, Spase v2.2.6'            |
|            to             | xrsb            | title: b'GOES 1-15 L2 XRS high-resolution Irradian|
|2010-11-03T23:59:32.881082 |                 | id: b' '                                          |
|                           |                 | summary: b'The X-ray flux product consists of repr|
|                           |                 | keywords: b'NumericalData.MeasurementType.Irradian|
|                           |                 | keywords_vocabulary: b'SPASE: Space Physics Archiv|
|                           |                 | naming_authority: b'gov.nesdis.noaa'              |
|                           |                 | history: b'See algorithm information.'            |
|                           |                 | source: b'GOES XRS counts'                        |
|                           |                 | processing_level: b'Level 2'                      |
|                           |                 | ...                                               |
|-------------------------------------------------------------------------------------------------|

You can reduce the depth of the view:

print(large_ts.meta.to_string(2))

Out:

|-------------------------------------------------------------------------------------------------|
|TimeRange                  | Columns         | Meta                                              |
|-------------------------------------------------------------------------------------------------|
|2010-11-02T22:44:51.801000 | Correlation Coe | simple: True                                      |
|2010-11-03T06:29:58.801000 |                 | bitpix: 16                                        |
|                           |                 | ...                                               |
|-------------------------------------------------------------------------------------------------|
|2010-11-02T23:59:34.851082 | xrsa            | conventions: b'ACDD-1.3, Spase v2.2.6'            |
|2010-11-03T23:59:32.881082 | xrsb            | title: b'GOES 1-15 L2 XRS high-resolution Irradian|
|                           |                 | ...                                               |
|-------------------------------------------------------------------------------------------------|

The TimeSeriesMetaData class stores all the individual file metadata MetaDict objects as 3-tuple entries in it’s internal list with the TimeRange, list of column names and metadictionary. This list is stored in order of ascending TR.start. Access of the the list is done using the metadata property:

Out:

[(   <sunpy.time.timerange.TimeRange object at 0x7f22723fa250>
    Start: 2010-11-02 22:44:51
    End:   2010-11-03 06:29:58
    Center:2010-11-03 02:37:25
    Duration:0.3229976851851851 days or
           7.751944444444443 hours or
           465.11666666666656 minutes or
           27906.999999999996 seconds
, ['Correlation Coefficient'], MetaDict([('simple', True), ('bitpix', 16), ('naxis', 1), ('naxis1', 27908), ('bscale', 3.053e-07), ('bzero', 0.0), ('bunit', 'CORRELATION COEFF.'), ('blank', -32768), ('crval1', '22:44:51.801'), ('startfrm', 1), ('crpix1', 1.0), ('ctype1', 'TIME(SECOND)'), ('cdelt1', 1.0), ('fund-amp', 11227), ('object', 'SUN'), ('image1', 'R+L'), ('obs-freq', '17GHZ'), ('telescop', 'RADIOHELIOGRAPH'), ('date-obs', '2010-11-02'), ('jstdate', '2010-11-03'), ('jsttime', '07:44:51.801'), ('origin', 'NOBEYAMA RADIO OBS'), ('date', '****-**-**'), ('pversion', '6.00'), ('comment', 'BUNIT IS  THE MEAN  OF CORRELATION COEFFICIENTS OF THE ANTENNA BASELINELENGTH ABOVE  100*D*COS(THETA).THETA: ANGLE BETWEEN THE ANTENNA BASELINE & THE DIRECTION OF THE SUN.D    : FUNDAMENTAL BASELINE LENGTH OF 1.528 METERSSFACT: FACTOR INTRODUCED TO REDUCE THE DISCRETIZATION ERROR'), ('history', ''), ('keycomments', {'SIMPLE': 'BASIC FITS TAPE FORM', 'BITPIX': "2-BYTE TWO'S COMPLEMENT INTEGER", 'NAXIS': '1-DIMENSIONAL TEMPORAL PROF', 'NAXIS1': '# PIXEL/ROW', 'BSCALE': 'SFACT/32768, REAL = TAPE * BSCALE + BZERO', 'BZERO': 'NO BITS ADDED', 'BUNIT': 'CROSS CORRELATION COEFFICIENT', 'BLANK': 'VALUE FOR VALUE-UNDEFINED PIXEL', 'CRVAL1': 'REF POINT VALUE IN HH:MM:SS (UT)', 'STARTFRM': 'REF FRAME NUMBER', 'CRPIX1': 'REF POINT PIXEL LOCATION', 'CTYPE1': 'TYPE OF PHYSICAL COORD. ON AXIS1', 'CDELT1': 'PIXEL SIZE ON AXIS1 IN SECOND', 'FUND-AMP': 'COR_AMP. OF FUND. ANTENNA SPACING AT STARTFRAM', 'OBJECT': 'OBJECT NAME', 'OBS-FREQ': 'INTENSITY AT 17GHZ', 'TELESCOP': 'NOBEYAMA RADIO HELIOGRAPH', 'DATE-OBS': 'DATE OF DATA ACQUISITION YYYY-MM-DD (UT)', 'JSTDATE': 'DATE OF DATA ACQUISITION YYYY-MM-DD (JST)', 'JSTTIME': 'TIME OF DATA ACQUISITION HH/MM/SS (JST)', 'ORIGIN': 'TAPE WRITE INSTITUTION', 'DATE': 'DATE WHEN DATA FILE WRITTEN YYYY-MM-DD (JST)', 'PVERSION': 'CODED BY N.SHINOHARA          14.MAR.95'})])), (   <sunpy.time.timerange.TimeRange object at 0x7f2274e4bd30>
    Start: 2010-11-02 23:59:34
    End:   2010-11-03 23:59:32
    Center:2010-11-03 11:59:33
    Duration:0.9999771990740742 days or
           23.999452777777783 hours or
           1439.9671666666668 minutes or
           86398.03000000001 seconds
, ['xrsa', 'xrsb'], MetaDict([('conventions', b'ACDD-1.3, Spase v2.2.6'), ('title', b'GOES 1-15 L2 XRS high-resolution Irradiances'), ('id', b' '), ('summary', b'The X-ray flux product consists of reprocessed science-quality high-resolution X-ray solar measurements. The GOES X-Ray Sensor (XRS) measures in a short wavelength channel (XRS-A) with a nominal bandpass of 0.05 to 0.4 nm and in a longer wavelength channel (XRS-B) with a bandpass of 0.1 to 0.8 nm. \n\nFor the given bandpass, the XRS response is based on a flat spectrum beginning with GOES-3. A true solar spectrum is assumed for GOES 1-2.\n\nGOES 13-15 measurements are at 2-s cadence. \nGOES 1-12 measured at a 3-s cadence. \n\nFor GOES 3-12, the XRS-A response was originally determined assuming a bandpass of 0.05 to 0.3 nm.'), ('keywords', b'NumericalData.MeasurementType.Irradiance'), ('keywords_vocabulary', b'SPASE: Space Physics Archive Search and Extract Data Model version 2.2.6, GCMD: NASA Global Change Master Directory (GCMD) Earth Science Keywords version 8.5'), ('naming_authority', b'gov.nesdis.noaa'), ('history', b'See algorithm information.'), ('source', b'GOES XRS counts'), ('processing_level', b'Level 2'), ('processing_level_description', b'Derived products'), ('license', b'These data may be redistributed and used without restriction. '), ('acknowledgment', b'TBD'), ('metadata_link', b''), ('date_created', b' '), ('creator_name', b'Erika Zetterlund, Janet Machol'), ('creator_type', b'person'), ('creator_institution', b'DOC/NOAA/NCEI/CCOG/STP'), ('creator_email', b'goesr.exis@noaa.gov'), ('creator_url', b'https://www.ncei.noaa.gov/'), ('institution', b'DOC/NOAA/NESDIS> U.S. Department of Commerce, National Oceanic and Atmospheric Administration, National Environmental Satellite, Data, and Information Services'), ('publisher_name', b'National Centers for Environmental Information'), ('publisher_type', b'institution'), ('publisher_institution', b'DOC/NOAA/NESDIS/NCEI'), ('publisher_email', b'goesr.exis@noaa.gov'), ('publisher_url', b'https://www.ncei.noaa.gov/'), ('references', b'README FOR SCIENCE QUALITY GOES 13-15 XRS DATA'), ('instrument', b'GOES 1-15 X-ray Irradiance Sensor (XRS)'), ('instrument_id', b' '), ('orbital_slot', b' '), ('program', b'Geostationary Operational Environmental Satellite (GOES)'), ('project', b'Geostationary Operational Environmental Satellite (GOES) Solar Terrestrial Physics at NCEI'), ('platform', b' '), ('l1b_system_environment', b' '), ('l1b_production_site', b' '), ('time_coverage_start', b' '), ('time_coverage_end', b' '), ('time_coverage_resolution', b'PT2S'), ('processing_parameters_file', b' '), ('algorithm', b''), ('algorithm_version', b' '), ('algorithm_date', b' '), ('algorithm_parameters', b'None'), ('input_system_environments', b' '), ('input_files_first', b' '), ('input_files_last', b' '), ('input_files_total', b' ')]))]

The TimeSeriesMetaData class has a number of other properties, including the timerange property that returns a TimeRange for the entire metadata:

print(large_ts.meta.time_range)

Out:

<sunpy.time.timerange.TimeRange object at 0x7f22724bcf40>
 Start: 2010-11-02 22:44:51
 End:   2010-11-03 23:59:32
 Center:2010-11-03 11:22:12
 Duration:1.051864352800926 days or
        25.244744467222223 hours or
        1514.6846680333333 minutes or
        90881.080082 seconds

Further properties can be used to get lists of details, e.g., List of the time ranges

print(large_ts.meta.timeranges)

Out:

[   <sunpy.time.timerange.TimeRange object at 0x7f22723fa250>
    Start: 2010-11-02 22:44:51
    End:   2010-11-03 06:29:58
    Center:2010-11-03 02:37:25
    Duration:0.3229976851851851 days or
           7.751944444444443 hours or
           465.11666666666656 minutes or
           27906.999999999996 seconds
,    <sunpy.time.timerange.TimeRange object at 0x7f2274e4bd30>
    Start: 2010-11-02 23:59:34
    End:   2010-11-03 23:59:32
    Center:2010-11-03 11:59:33
    Duration:0.9999771990740742 days or
           23.999452777777783 hours or
           1439.9671666666668 minutes or
           86398.03000000001 seconds
]

List of the column names

print(large_ts.meta.columns)

Out:

['Correlation Coefficient', 'xrsa', 'xrsb']

List of the meta dictionaries

print(large_ts.meta.metas)

Out:

[MetaDict([('simple', True), ('bitpix', 16), ('naxis', 1), ('naxis1', 27908), ('bscale', 3.053e-07), ('bzero', 0.0), ('bunit', 'CORRELATION COEFF.'), ('blank', -32768), ('crval1', '22:44:51.801'), ('startfrm', 1), ('crpix1', 1.0), ('ctype1', 'TIME(SECOND)'), ('cdelt1', 1.0), ('fund-amp', 11227), ('object', 'SUN'), ('image1', 'R+L'), ('obs-freq', '17GHZ'), ('telescop', 'RADIOHELIOGRAPH'), ('date-obs', '2010-11-02'), ('jstdate', '2010-11-03'), ('jsttime', '07:44:51.801'), ('origin', 'NOBEYAMA RADIO OBS'), ('date', '****-**-**'), ('pversion', '6.00'), ('comment', 'BUNIT IS  THE MEAN  OF CORRELATION COEFFICIENTS OF THE ANTENNA BASELINELENGTH ABOVE  100*D*COS(THETA).THETA: ANGLE BETWEEN THE ANTENNA BASELINE & THE DIRECTION OF THE SUN.D    : FUNDAMENTAL BASELINE LENGTH OF 1.528 METERSSFACT: FACTOR INTRODUCED TO REDUCE THE DISCRETIZATION ERROR'), ('history', ''), ('keycomments', {'SIMPLE': 'BASIC FITS TAPE FORM', 'BITPIX': "2-BYTE TWO'S COMPLEMENT INTEGER", 'NAXIS': '1-DIMENSIONAL TEMPORAL PROF', 'NAXIS1': '# PIXEL/ROW', 'BSCALE': 'SFACT/32768, REAL = TAPE * BSCALE + BZERO', 'BZERO': 'NO BITS ADDED', 'BUNIT': 'CROSS CORRELATION COEFFICIENT', 'BLANK': 'VALUE FOR VALUE-UNDEFINED PIXEL', 'CRVAL1': 'REF POINT VALUE IN HH:MM:SS (UT)', 'STARTFRM': 'REF FRAME NUMBER', 'CRPIX1': 'REF POINT PIXEL LOCATION', 'CTYPE1': 'TYPE OF PHYSICAL COORD. ON AXIS1', 'CDELT1': 'PIXEL SIZE ON AXIS1 IN SECOND', 'FUND-AMP': 'COR_AMP. OF FUND. ANTENNA SPACING AT STARTFRAM', 'OBJECT': 'OBJECT NAME', 'OBS-FREQ': 'INTENSITY AT 17GHZ', 'TELESCOP': 'NOBEYAMA RADIO HELIOGRAPH', 'DATE-OBS': 'DATE OF DATA ACQUISITION YYYY-MM-DD (UT)', 'JSTDATE': 'DATE OF DATA ACQUISITION YYYY-MM-DD (JST)', 'JSTTIME': 'TIME OF DATA ACQUISITION HH/MM/SS (JST)', 'ORIGIN': 'TAPE WRITE INSTITUTION', 'DATE': 'DATE WHEN DATA FILE WRITTEN YYYY-MM-DD (JST)', 'PVERSION': 'CODED BY N.SHINOHARA          14.MAR.95'})]), MetaDict([('conventions', b'ACDD-1.3, Spase v2.2.6'), ('title', b'GOES 1-15 L2 XRS high-resolution Irradiances'), ('id', b' '), ('summary', b'The X-ray flux product consists of reprocessed science-quality high-resolution X-ray solar measurements. The GOES X-Ray Sensor (XRS) measures in a short wavelength channel (XRS-A) with a nominal bandpass of 0.05 to 0.4 nm and in a longer wavelength channel (XRS-B) with a bandpass of 0.1 to 0.8 nm. \n\nFor the given bandpass, the XRS response is based on a flat spectrum beginning with GOES-3. A true solar spectrum is assumed for GOES 1-2.\n\nGOES 13-15 measurements are at 2-s cadence. \nGOES 1-12 measured at a 3-s cadence. \n\nFor GOES 3-12, the XRS-A response was originally determined assuming a bandpass of 0.05 to 0.3 nm.'), ('keywords', b'NumericalData.MeasurementType.Irradiance'), ('keywords_vocabulary', b'SPASE: Space Physics Archive Search and Extract Data Model version 2.2.6, GCMD: NASA Global Change Master Directory (GCMD) Earth Science Keywords version 8.5'), ('naming_authority', b'gov.nesdis.noaa'), ('history', b'See algorithm information.'), ('source', b'GOES XRS counts'), ('processing_level', b'Level 2'), ('processing_level_description', b'Derived products'), ('license', b'These data may be redistributed and used without restriction. '), ('acknowledgment', b'TBD'), ('metadata_link', b''), ('date_created', b' '), ('creator_name', b'Erika Zetterlund, Janet Machol'), ('creator_type', b'person'), ('creator_institution', b'DOC/NOAA/NCEI/CCOG/STP'), ('creator_email', b'goesr.exis@noaa.gov'), ('creator_url', b'https://www.ncei.noaa.gov/'), ('institution', b'DOC/NOAA/NESDIS> U.S. Department of Commerce, National Oceanic and Atmospheric Administration, National Environmental Satellite, Data, and Information Services'), ('publisher_name', b'National Centers for Environmental Information'), ('publisher_type', b'institution'), ('publisher_institution', b'DOC/NOAA/NESDIS/NCEI'), ('publisher_email', b'goesr.exis@noaa.gov'), ('publisher_url', b'https://www.ncei.noaa.gov/'), ('references', b'README FOR SCIENCE QUALITY GOES 13-15 XRS DATA'), ('instrument', b'GOES 1-15 X-ray Irradiance Sensor (XRS)'), ('instrument_id', b' '), ('orbital_slot', b' '), ('program', b'Geostationary Operational Environmental Satellite (GOES)'), ('project', b'Geostationary Operational Environmental Satellite (GOES) Solar Terrestrial Physics at NCEI'), ('platform', b' '), ('l1b_system_environment', b' '), ('l1b_production_site', b' '), ('time_coverage_start', b' '), ('time_coverage_end', b' '), ('time_coverage_resolution', b'PT2S'), ('processing_parameters_file', b' '), ('algorithm', b''), ('algorithm_version', b' '), ('algorithm_date', b' '), ('algorithm_parameters', b'None'), ('input_system_environments', b' '), ('input_files_first', b' '), ('input_files_last', b' '), ('input_files_total', b' ')])]

When you truncate the TimeSeries, the metadata is truncated too:

large_trunc_ts = large_ts.truncate(TimeRange('2010-11-03 13:59:57.468999',
                                             '2010-11-04 13:59:56.091999'))
print(large_trunc_ts.meta.to_string(2))

Out:

|-------------------------------------------------------------------------------------------------|
|TimeRange                  | Columns         | Meta                                              |
|-------------------------------------------------------------------------------------------------|
|2010-11-03T13:59:57.998082 | xrsa            | conventions: b'ACDD-1.3, Spase v2.2.6'            |
|2010-11-03T23:59:32.881082 | xrsb            | title: b'GOES 1-15 L2 XRS high-resolution Irradian|
|                           |                 | ...                                               |
|-------------------------------------------------------------------------------------------------|

Finding metadata can be achieved using the find method and applying filters for time and/or colname. This returns another TimeSeriesMetaData object:

large_trunc_ts.meta.find(time=parse_time('2010-11-04 09:01:16'))
large_trunc_ts.meta.find(time='2010-11-04 09:01:16', colname='xrsb')

# You can get the time of a row a from the TimeSeries object's index:
large_trunc_ts.meta.find(time=large_trunc_ts.index[10])

# There is also a get method:
large_trunc_ts.meta.get('telescop')

# Again, filters can be used:
large_trunc_ts.meta.get('telescop', time='2010-11-04 09:01:16', colname='xrsb')

# And if we just want the values, the values method returns just a list:
large_trunc_ts.meta.get('telescop').values()

Out:

[]

You can update values similar to dictionaries, though all of the contained MetaDict objects will be updated that match your filter criteria:

large_trunc_ts.meta.update({'new_key_1': 'added to all.'})
large_trunc_ts.meta.update({'new_key_2': 'added to some.'}, colname='xrsa')
print(large_trunc_ts.meta.to_string(2))

# but you can't overwrite previous entries without setting the overwrite kwarg,
# this is to protect the integrity of the metadata:

large_trunc_ts.meta.update({'new_key_1': 'changed'}, overwrite=True)
print(large_trunc_ts.meta.to_string(2))

Out:

|-------------------------------------------------------------------------------------------------|
|TimeRange                  | Columns         | Meta                                              |
|-------------------------------------------------------------------------------------------------|
|2010-11-03T13:59:57.998082 | xrsa            | conventions: b'ACDD-1.3, Spase v2.2.6'            |
|2010-11-03T23:59:32.881082 | xrsb            | title: b'GOES 1-15 L2 XRS high-resolution Irradian|
|                           |                 | ...                                               |
|-------------------------------------------------------------------------------------------------|

|-------------------------------------------------------------------------------------------------|
|TimeRange                  | Columns         | Meta                                              |
|-------------------------------------------------------------------------------------------------|
|2010-11-03T13:59:57.998082 | xrsa            | conventions: b'ACDD-1.3, Spase v2.2.6'            |
|2010-11-03T23:59:32.881082 | xrsb            | title: b'GOES 1-15 L2 XRS high-resolution Irradian|
|                           |                 | ...                                               |
|-------------------------------------------------------------------------------------------------|

Total running time of the script: ( 0 minutes 8.294 seconds)

Gallery generated by Sphinx-Gallery