Finding and Downloading Data from JSOC#
Joint Science Operations Center (JSOC) contains data products from the Solar Dynamics Observatory, as well as certain other missions and instruments. These data are available from the JSOC database, which can be directly accessed by the online JSOC interface.
sunpy’s JSOC Client provides an easier interface to query for JSOC data and make export requests. It uses drms module as its backend.
There are two ways of downloading JSOC data.
One way is using sunpy’s unified search interface, known as Fido.
Fido supplies a single, easy and consistent way to to obtain most forms of solar physics data.
An alternative way to fetch data from JSOC is by using the underlying JSOC Client.
This option can be preferred when you need to separate the staging and downloading steps, which is not supported by Fido.
The JSOC stages data before you can download it, so a JSOC query is a three stage process.
First you query the JSOC for records and a table of these records is returned.
Then you can request these records to be staged for download and then you can download them.
Fido combines the last two stages into a single call to fetch.
Setup#
sunpy’s Fido module is in sunpy.net.
It can be imported as follows:
>>> from sunpy.net import Fido, attrs as a
The JSOC client handles the particulars of how the data from the data provider is downloaded to your computer.
Warning
You must have an email address registered with JSOC before you are allowed to make a request. See this to register your email address.
Querying the JSOC#
To search for data in JSOC, your query needs at minimum, a “Series” name and a “PrimeKey”.
>>> print(a.jsoc.Series)
sunpy.net.jsoc.attrs.Series
The JSOC Series to Download.
Attribute Name Client Full Name Description
---------------------------------- ------ ---------------------------------- --------------------------------------------------------------------------------
aia_flatfield JSOC aia.flatfield AIA flatfield
aia_lev1 JSOC aia.lev1 AIA Level 1
aia_lev1_euv_12s JSOC aia.lev1_euv_12s AIA Level 1, 12 second cadence
aia_lev1_uv_24s JSOC aia.lev1_uv_24s AIA Level 1, 24 second cadence
aia_lev1_vis_1h JSOC aia.lev1_vis_1h AIA Level 1, 3600 second cadence
aia_master_pointing3h JSOC aia.master_pointing3h Master Pointing Parameters
aia_response JSOC aia.response AIA instrument response table
aia_temperature_summary_300s JSOC aia.temperature_summary_300s Temperature Statistics from AIA Housekeeping - Thermal Packet
hmi_b_135s JSOC hmi.b_135s Full-disk Milne-Eddington inversion with the azimuth disambiguation informati...
...
Different PrimeKeys are supported by different Series, and you can find out the PrimeKeys supported in any Series by:
>>> import drms
>>> client = drms.Client()
>>> print(client.pkeys('hmi.m_720s'))
['T_REC', 'CAMERA']
The most common PrimeKey, that is supported by every Series is Time, that is denoted by T_REC or T_OBS.
Hence, Time can always be passed as an attribute while building a query.
Wavelength is another pre-defined attribute which is a PrimeKey.
Other PrimeKeys which need to be passed should be manually passed in PrimeKey.
This will be explained later in this tutorial.
Constructing a Basic Query#
Let’s start with a very simple query. We could ask for all “hmi.v_45s” series data between January 1st 2014 from 00:00 to 01:00:
>>> res = Fido.search(a.Time('2014-01-01T00:00:00', '2014-01-01T01:00:00'),
... a.jsoc.Series('hmi.v_45s'))
This returns an UnifiedResponse object containing information on the available online files which fit the criteria specified by the attrs objects in the above call.
It does not download the files.
To see a summary of results of our query, simply type the name of the variable set to the Fido search, in this case, res:
>>> res
<sunpy.net.fido_factory.UnifiedResponse object at ...>
Results from 1 Provider:
81 Results from the JSOCClient:
Source: http://jsoc.stanford.edu
T_REC TELESCOP INSTRUME WAVELNTH CAR_ROT
----------------------- -------- ---------- -------- -------
2014.01.01_00:00:45_TAI SDO/HMI HMI_FRONT2 6173.0 2145
2014.01.01_00:01:30_TAI SDO/HMI HMI_FRONT2 6173.0 2145
2014.01.01_00:02:15_TAI SDO/HMI HMI_FRONT2 6173.0 2145
2014.01.01_00:03:00_TAI SDO/HMI HMI_FRONT2 6173.0 2145
2014.01.01_00:03:45_TAI SDO/HMI HMI_FRONT2 6173.0 2145
2014.01.01_00:04:30_TAI SDO/HMI HMI_FRONT2 6173.0 2145
2014.01.01_00:05:15_TAI SDO/HMI HMI_FRONT2 6173.0 2145
2014.01.01_00:06:00_TAI SDO/HMI HMI_FRONT2 6173.0 2145
2014.01.01_00:06:45_TAI SDO/HMI HMI_FRONT2 6173.0 2145
2014.01.01_00:07:30_TAI SDO/HMI HMI_FRONT2 6173.0 2145
... ... ... ... ...
2014.01.01_00:54:00_TAI SDO/HMI HMI_FRONT2 6173.0 2145
2014.01.01_00:54:45_TAI SDO/HMI HMI_FRONT2 6173.0 2145
2014.01.01_00:55:30_TAI SDO/HMI HMI_FRONT2 6173.0 2145
2014.01.01_00:56:15_TAI SDO/HMI HMI_FRONT2 6173.0 2145
2014.01.01_00:57:00_TAI SDO/HMI HMI_FRONT2 6173.0 2145
2014.01.01_00:57:45_TAI SDO/HMI HMI_FRONT2 6173.0 2145
2014.01.01_00:58:30_TAI SDO/HMI HMI_FRONT2 6173.0 2145
2014.01.01_00:59:15_TAI SDO/HMI HMI_FRONT2 6173.0 2145
2014.01.01_01:00:00_TAI SDO/HMI HMI_FRONT2 6173.0 2145
2014.01.01_01:00:45_TAI SDO/HMI HMI_FRONT2 6173.0 2145
Length = 81 rows
Now, let’s break down the arguments of Fido.search to understand better what we’ve done.
The first argument a.Time('2014-01-01T00:00:00', '2014-01-01T01:00:00') sets the start and end times for the query (any date/time format understood by sunpy’s parse_time can be used to specify dates and time).
The Time attribute takes UTC time, as default.
If you need to pass a Time in some other time scale, such as TAI, pass an astropy.time.Time object:
>>> import astropy.time
Then, the Time attribute can be passed as:
>>> a.Time(astropy.time.Time('2014-01-01T00:00:00', scale='tai'), astropy.time.Time('2014-01-01T01:00:00', scale='tai'))
<sunpy.net.attrs.Time(2014-01-01 00:00:00.000, 2014-01-01 01:00:00.000)>
The second argument:
>>> a.jsoc.Series('hmi.v_45s')
<sunpy.net.jsoc.attrs.Series(hmi.v_45s: Dopplergrams with a cadence of 45 seconds) object ...>
sets the series we are looking for.
So what is going on here?
The notion is that a JSOC query has a set of attribute objects, imported as a.jsoc, that are specified to construct the query.
a.jsoc.Series() is compulsory to be provided in each of the jsoc queries.
Apart from this, at least one PrimeKey must be passed (generally a.Time()).
Querying with other PrimeKeys#
Other than Time, one other PrimeKey is supported with in-built attribute.
In case of AIA series, a.Wavelength() can be passed as a PrimeKey:
>>> import astropy.units as u
>>> res = Fido.search(a.Time('2014-01-01T00:00:00', '2014-01-01T01:00:00'),
... a.jsoc.Series('aia.lev1_euv_12s'),
... a.Wavelength(304*u.AA))
Note that, only Time and Wavelength are in-built attributes here. If you need to pass any other PrimeKey, it should be passed like this:
>>> a.jsoc.PrimeKey('HARPNUM', '4864')
<sunpy.net.jsoc.attrs.PrimeKey object at ...>
('HARPNUM', '4864')
If 2 or more PrimeKeys need to be passed together:
>>> a.jsoc.PrimeKey('HARPNUM', '4864') & a.jsoc.PrimeKey('CAMERA', '2')
<AttrAnd([<sunpy.net.jsoc.attrs.PrimeKey object at ...>
('HARPNUM', '4864'), <sunpy.net.jsoc.attrs.PrimeKey object at ...>
('CAMERA', '2')])>
Also, note that the pre-defined PrimeKeys, Time and Wavelength can also be passed as above, but you need to specify the exact keyword for it:
>>> a.Time('2014-01-01T00:00:00', '2014-01-01T01:00:00'), a.jsoc.PrimeKey('WAVELNTH', '161')
(<sunpy.net.attrs.Time(2014-01-01 00:00:00.000, 2014-01-01 01:00:00.000)>, <sunpy.net.jsoc.attrs.PrimeKey object at ...>
('WAVELNTH', '161'))
If the correct keyword is not specified, or the passed PrimeKey is not supported by the given series, a meaningful error will be thrown, which will give you the PrimeKeys supported by that series. Hence, by looking at the error, one can easily retry building the query with correct PrimeKeys.
Another important thing to note is that, Wavelength when passed through in-built attribute, should be passed as an astropy quantity.
Specifying spectral units in arguments is necessary or an error will be raised.
For more information on units, see units.
But, when the same is passed through PrimeKey attribute, it should be passed as a string.
All other PrimeKey values passed through PrimeKey attribute, must be passed as a string.
Manually specifying keyword data to fetch#
Upon doing Fido.search() as described above, only a limited set of keywords are returned in the response object.
These default keywords are 'DATE', 'TELESCOP', 'INSTRUME', 'T_OBS' and 'WAVELNTH'.
If you want to get a manual set of keywords in the response object, you can pass the set of keywords using show() method.
>>> res = Fido.search(a.Time('2014-01-01T00:00:00', '2014-01-01T01:00:00'),
... a.jsoc.Series('hmi.v_45s'))
>>> res.show('TELESCOP', 'INSTRUME', 'T_OBS')
<sunpy.net.fido_factory.UnifiedResponse object at ...>
Results from 1 Provider:
81 Results from the JSOCClient:
Source: http://jsoc.stanford.edu
TELESCOP INSTRUME T_OBS
-------- ---------- -----------------------
SDO/HMI HMI_FRONT2 2014.01.01_00:00:37_TAI
SDO/HMI HMI_FRONT2 2014.01.01_00:01:22_TAI
SDO/HMI HMI_FRONT2 2014.01.01_00:02:07_TAI
SDO/HMI HMI_FRONT2 2014.01.01_00:02:52_TAI
SDO/HMI HMI_FRONT2 2014.01.01_00:03:37_TAI
SDO/HMI HMI_FRONT2 2014.01.01_00:04:22_TAI
SDO/HMI HMI_FRONT2 2014.01.01_00:05:07_TAI
SDO/HMI HMI_FRONT2 2014.01.01_00:05:52_TAI
SDO/HMI HMI_FRONT2 2014.01.01_00:06:37_TAI
SDO/HMI HMI_FRONT2 2014.01.01_00:07:22_TAI
... ... ...
SDO/HMI HMI_FRONT2 2014.01.01_00:53:07_TAI
SDO/HMI HMI_FRONT2 2014.01.01_00:53:52_TAI
SDO/HMI HMI_FRONT2 2014.01.01_00:54:37_TAI
SDO/HMI HMI_FRONT2 2014.01.01_00:55:22_TAI
SDO/HMI HMI_FRONT2 2014.01.01_00:56:07_TAI
SDO/HMI HMI_FRONT2 2014.01.01_00:56:52_TAI
SDO/HMI HMI_FRONT2 2014.01.01_00:57:37_TAI
SDO/HMI HMI_FRONT2 2014.01.01_00:58:22_TAI
SDO/HMI HMI_FRONT2 2014.01.01_00:59:07_TAI
SDO/HMI HMI_FRONT2 2014.01.01_00:59:52_TAI
SDO/HMI HMI_FRONT2 2014.01.01_01:00:37_TAI
Length = 81 rows
Passing an incorrect keyword won’t throw an error, but the corresponding column in the table will not be displayed.
To display all of the columns, we can use show() without passing any arguments:
>>> res.show()
<sunpy.net.fido_factory.UnifiedResponse object at ...>
Results from 1 Provider:
81 Results from the JSOCClient:
Source: http://jsoc.stanford.edu
DATE DATE__OBS ... CALVER64
-------------------- ----------------------- ... --------
2014-01-05T17:46:02Z 2013-12-31T23:59:39.20Z ... 4370
2014-01-05T17:47:10Z 2014-01-01T00:00:24.20Z ... 4370
2014-01-05T17:48:18Z 2014-01-01T00:01:09.20Z ... 4370
2014-01-05T17:49:25Z 2014-01-01T00:01:54.20Z ... 4370
2014-01-05T17:50:34Z 2014-01-01T00:02:39.20Z ... 4370
2014-01-05T17:51:42Z 2014-01-01T00:03:24.20Z ... 4370
2014-01-05T17:52:50Z 2014-01-01T00:04:09.20Z ... 4370
2014-01-05T17:53:59Z 2014-01-01T00:04:54.20Z ... 4370
2014-01-05T17:55:08Z 2014-01-01T00:05:39.20Z ... 4370
2014-01-05T17:56:16Z 2014-01-01T00:06:24.20Z ... 4370
... ... ... ...
2014-01-05T19:05:49Z 2014-01-01T00:52:09.20Z ... 4370
2014-01-05T17:35:43Z 2014-01-01T00:52:54.20Z ... 4370
2014-01-05T17:36:54Z 2014-01-01T00:53:39.20Z ... 4370
2014-01-05T17:38:01Z 2014-01-01T00:54:24.20Z ... 4370
2014-01-05T17:39:09Z 2014-01-01T00:55:09.20Z ... 4370
2014-01-05T17:40:17Z 2014-01-01T00:55:54.20Z ... 4370
2014-01-05T17:41:25Z 2014-01-01T00:56:39.20Z ... 4370
2014-01-05T17:42:33Z 2014-01-01T00:57:24.20Z ... 4370
2014-01-05T17:43:41Z 2014-01-01T00:58:09.20Z ... 4370
2014-01-05T17:44:52Z 2014-01-01T00:58:54.20Z ... 4370
2014-01-05T17:46:03Z 2014-01-01T00:59:39.20Z ... 4370
Length = 81 rows
Using Segments#
In some cases, more than 1 file are present for the same set of query. These data are distinguished by what are called Segments. It is necessary to specify the Segment which you need to download. Providing a segment won’t have any affect on the response object returned, but this will be required later, while making an export request.
A list of supported segments of a series, say hmi.sharp_720s can be obtained by:
>>> client = drms.Client()
>>> si = client.info('hmi.sharp_720s')
>>> print(si.segments.index.values)
['magnetogram' 'bitmap' 'Dopplergram' 'continuum' 'inclination' 'azimuth'
'field' 'vlos_mag' 'dop_width' 'eta_0' 'damping' 'src_continuum'
'src_grad' 'alpha_mag' 'chisq' 'conv_flag' 'info_map' 'confid_map'
'inclination_err' 'azimuth_err' 'field_err' 'vlos_err' 'alpha_err'
'field_inclination_err' 'field_az_err' 'inclin_azimuth_err'
'field_alpha_err' 'inclination_alpha_err' 'azimuth_alpha_err' 'disambig'
'conf_disambig']
Also, if you provide an incorrect segment name, it will throw a meaningful error, specifying which segment values are supported by the given series:
>>> Fido.search(a.Time('2014-01-01T00:00:00', '2014-01-01T01:00:00'),
... a.jsoc.Series('hmi.sharp_720s'),
... a.jsoc.Segment('image'))
Traceback (most recent call last):
...
ValueError: Unexpected Segments were passed. The series hmi.sharp_720s contains the following Segments ['magnetogram', 'bitmap', 'Dopplergram', 'continuum', 'inclination', 'azimuth', 'field', 'vlos_mag', 'dop_width', 'eta_0', 'damping', 'src_continuum', 'src_grad', 'alpha_mag', 'chisq', 'conv_flag', 'info_map', 'confid_map', 'inclination_err', 'azimuth_err', 'field_err', 'vlos_err', 'alpha_err', 'field_inclination_err', 'field_az_err', 'inclin_azimuth_err', 'field_alpha_err', 'inclination_alpha_err', 'azimuth_alpha_err', 'disambig', 'conf_disambig']
To get files for more than 1 segment at the same time, chain a.jsoc.Segment() using AND operator:
>>> Fido.search(a.Time('2014-01-01T00:00:00', '2014-01-01T01:00:00'),
... a.jsoc.Series('hmi.sharp_720s'),
... a.jsoc.Segment('continuum') & a.jsoc.Segment('magnetogram'))
<sunpy.net.fido_factory.UnifiedResponse object at ...>
Results from 1 Provider:
61 Results from the JSOCClient:
Source: http://jsoc.stanford.edu
T_REC TELESCOP INSTRUME WAVELNTH CAR_ROT
----------------------- -------- --------- -------- -------
2014.01.01_00:00:00_TAI SDO/HMI HMI_SIDE1 6173.0 2145
2014.01.01_00:12:00_TAI SDO/HMI HMI_SIDE1 6173.0 2145
2014.01.01_00:24:00_TAI SDO/HMI HMI_SIDE1 6173.0 2145
2014.01.01_00:36:00_TAI SDO/HMI HMI_SIDE1 6173.0 2145
2014.01.01_00:48:00_TAI SDO/HMI HMI_SIDE1 6173.0 2145
2014.01.01_01:00:00_TAI SDO/HMI HMI_SIDE1 6173.0 2145
2014.01.01_00:00:00_TAI SDO/HMI HMI_SIDE1 6173.0 2145
2014.01.01_00:12:00_TAI SDO/HMI HMI_SIDE1 6173.0 2145
2014.01.01_00:24:00_TAI SDO/HMI HMI_SIDE1 6173.0 2145
2014.01.01_00:36:00_TAI SDO/HMI HMI_SIDE1 6173.0 2145
... ... ... ... ...
2014.01.01_00:24:00_TAI SDO/HMI HMI_SIDE1 6173.0 2145
2014.01.01_00:36:00_TAI SDO/HMI HMI_SIDE1 6173.0 2145
2014.01.01_00:48:00_TAI SDO/HMI HMI_SIDE1 6173.0 2145
2014.01.01_01:00:00_TAI SDO/HMI HMI_SIDE1 6173.0 2145
2014.01.01_00:00:00_TAI SDO/HMI HMI_SIDE1 6173.0 2145
2014.01.01_00:12:00_TAI SDO/HMI HMI_SIDE1 6173.0 2145
2014.01.01_00:24:00_TAI SDO/HMI HMI_SIDE1 6173.0 2145
2014.01.01_00:36:00_TAI SDO/HMI HMI_SIDE1 6173.0 2145
2014.01.01_00:48:00_TAI SDO/HMI HMI_SIDE1 6173.0 2145
2014.01.01_01:00:00_TAI SDO/HMI HMI_SIDE1 6173.0 2145
Length = 61 rows
Using Keywords#
In some cases, you might want to filter out files based on key metadata, also called keywords.
A list of supported keywords of a series, say hmi.sharp_720s can be obtained by:
>>> client = drms.Client()
>>> keywords = client.keys('hmi.sharp_720s')
>>> print(keywords)
['cparms_sg000', 'magnetogram_bzero', 'magnetogram_bscale', 'cparms_sg001', 'bitmap_bzero', 'bitmap_bscale', 'cparms_sg002', 'Dopplergram_bzero', 'Dopplergram_bscale', 'cparms_sg003', 'continuum_bzero', 'continuum_bscale', 'cparms_sg004', 'inclination_bzero', 'inclination_bscale', 'cparms_sg005', 'azimuth_bzero', 'azimuth_bscale', 'cparms_sg006', 'field_bzero', 'field_bscale', 'cparms_sg007', ... 'ERRJHT', 'ERRVF']
Each keyword needs to be compared to a value, e.g., a.jsoc.Keyword("bitmap_bzero") == 0 or a.jsoc.Keyword("bitmap_bzero") > 1.
An example of this is:
>>> Fido.search(a.Time('2014-01-01T00:00:00', '2014-01-01T01:00:00'),
... a.jsoc.Series('hmi.sharp_720s'),a.jsoc.Keyword('bitmap_bzero') == 0)
<sunpy.net.fido_factory.UnifiedResponse object at ...>
Results from 1 Provider:
61 Results from the JSOCClient:
Source: http://jsoc.stanford.edu
T_REC TELESCOP INSTRUME WAVELNTH CAR_ROT
----------------------- -------- --------- -------- -------
2014.01.01_00:00:00_TAI SDO/HMI HMI_SIDE1 6173.0 2145
2014.01.01_00:12:00_TAI SDO/HMI HMI_SIDE1 6173.0 2145
2014.01.01_00:24:00_TAI SDO/HMI HMI_SIDE1 6173.0 2145
2014.01.01_00:36:00_TAI SDO/HMI HMI_SIDE1 6173.0 2145
2014.01.01_00:48:00_TAI SDO/HMI HMI_SIDE1 6173.0 2145
2014.01.01_01:00:00_TAI SDO/HMI HMI_SIDE1 6173.0 2145
2014.01.01_00:00:00_TAI SDO/HMI HMI_SIDE1 6173.0 2145
2014.01.01_00:12:00_TAI SDO/HMI HMI_SIDE1 6173.0 2145
2014.01.01_00:24:00_TAI SDO/HMI HMI_SIDE1 6173.0 2145
2014.01.01_00:36:00_TAI SDO/HMI HMI_SIDE1 6173.0 2145
... ... ... ... ...
2014.01.01_00:12:00_TAI SDO/HMI HMI_SIDE1 6173.0 2145
2014.01.01_00:24:00_TAI SDO/HMI HMI_SIDE1 6173.0 2145
2014.01.01_00:36:00_TAI SDO/HMI HMI_SIDE1 6173.0 2145
2014.01.01_00:48:00_TAI SDO/HMI HMI_SIDE1 6173.0 2145
2014.01.01_01:00:00_TAI SDO/HMI HMI_SIDE1 6173.0 2145
2014.01.01_00:00:00_TAI SDO/HMI HMI_SIDE1 6173.0 2145
2014.01.01_00:12:00_TAI SDO/HMI HMI_SIDE1 6173.0 2145
2014.01.01_00:24:00_TAI SDO/HMI HMI_SIDE1 6173.0 2145
2014.01.01_00:36:00_TAI SDO/HMI HMI_SIDE1 6173.0 2145
2014.01.01_00:48:00_TAI SDO/HMI HMI_SIDE1 6173.0 2145
2014.01.01_01:00:00_TAI SDO/HMI HMI_SIDE1 6173.0 2145
Length = 61 rows
You can pass multiple keywords and they will be chained together inside the query:
>>> Fido.search(a.Time('2014-01-01T00:00:00', '2014-01-01T01:00:00'), a.jsoc.Series('hmi.sharp_720s'),
... a.jsoc.Keyword('bitmap_bzero') == 0, a.jsoc.Keyword('continuum_bscale') > 0)
<sunpy.net.fido_factory.UnifiedResponse object at ...>
Results from 1 Provider:
61 Results from the JSOCClient:
Source: http://jsoc.stanford.edu
T_REC TELESCOP INSTRUME WAVELNTH CAR_ROT
----------------------- -------- --------- -------- -------
2014.01.01_00:00:00_TAI SDO/HMI HMI_SIDE1 6173.0 2145
2014.01.01_00:12:00_TAI SDO/HMI HMI_SIDE1 6173.0 2145
2014.01.01_00:24:00_TAI SDO/HMI HMI_SIDE1 6173.0 2145
2014.01.01_00:36:00_TAI SDO/HMI HMI_SIDE1 6173.0 2145
2014.01.01_00:48:00_TAI SDO/HMI HMI_SIDE1 6173.0 2145
2014.01.01_01:00:00_TAI SDO/HMI HMI_SIDE1 6173.0 2145
2014.01.01_00:00:00_TAI SDO/HMI HMI_SIDE1 6173.0 2145
2014.01.01_00:12:00_TAI SDO/HMI HMI_SIDE1 6173.0 2145
2014.01.01_00:24:00_TAI SDO/HMI HMI_SIDE1 6173.0 2145
2014.01.01_00:36:00_TAI SDO/HMI HMI_SIDE1 6173.0 2145
... ... ... ... ...
2014.01.01_00:12:00_TAI SDO/HMI HMI_SIDE1 6173.0 2145
2014.01.01_00:24:00_TAI SDO/HMI HMI_SIDE1 6173.0 2145
2014.01.01_00:36:00_TAI SDO/HMI HMI_SIDE1 6173.0 2145
2014.01.01_00:48:00_TAI SDO/HMI HMI_SIDE1 6173.0 2145
2014.01.01_01:00:00_TAI SDO/HMI HMI_SIDE1 6173.0 2145
2014.01.01_00:00:00_TAI SDO/HMI HMI_SIDE1 6173.0 2145
2014.01.01_00:12:00_TAI SDO/HMI HMI_SIDE1 6173.0 2145
2014.01.01_00:24:00_TAI SDO/HMI HMI_SIDE1 6173.0 2145
2014.01.01_00:36:00_TAI SDO/HMI HMI_SIDE1 6173.0 2145
2014.01.01_00:48:00_TAI SDO/HMI HMI_SIDE1 6173.0 2145
2014.01.01_01:00:00_TAI SDO/HMI HMI_SIDE1 6173.0 2145
Length = 61 rows
If you provide a keyword without a comparison it will raise an error:
>>> Fido.search(a.Time('2014-01-01T00:00:00', '2014-01-01T01:00:00'),
... a.jsoc.Series('hmi.sharp_720s'),
... a.jsoc.Keyword('bitmap_bzero'))
Traceback (most recent call last):
...
ValueError: Keyword 'bitmap_bzero' needs to have a comparison to a value.
If you provide an incorrect keyword name it will also raise a error:
>>> Fido.search(a.Time('2014-01-01T00:00:00', '2014-01-01T01:00:00'),
... a.jsoc.Series('hmi.sharp_720s'),
... a.jsoc.Keyword('bac') == 0)
Traceback (most recent call last):
...
ValueError: Keyword: 'bac' is not supported by series: hmi.sharp_720s
Using Sample#
In case you need to query for data, at some interval of time, say every 10 min, you can pass it using Sample.
In other words, if you need to query for “hmi.v_45s” series data between January 1st 2014 from 00:00 to 01:00 at 10 minute intervals, you can do:
>>> Fido.search(a.Time('2014-01-01T00:00:00', '2014-01-01T01:00:00'),
... a.jsoc.Series('hmi.v_45s'), a.Sample(10*u.min))
<sunpy.net.fido_factory.UnifiedResponse object at ...>
Results from 1 Provider:
7 Results from the JSOCClient:
Source: http://jsoc.stanford.edu
T_REC TELESCOP INSTRUME WAVELNTH CAR_ROT
----------------------- -------- ---------- -------- -------
2014.01.01_00:00:45_TAI SDO/HMI HMI_FRONT2 6173.0 2145
2014.01.01_00:10:30_TAI SDO/HMI HMI_FRONT2 6173.0 2145
2014.01.01_00:20:15_TAI SDO/HMI HMI_FRONT2 6173.0 2145
2014.01.01_00:30:00_TAI SDO/HMI HMI_FRONT2 6173.0 2145
2014.01.01_00:39:45_TAI SDO/HMI HMI_FRONT2 6173.0 2145
2014.01.01_00:49:30_TAI SDO/HMI HMI_FRONT2 6173.0 2145
2014.01.01_00:59:15_TAI SDO/HMI HMI_FRONT2 6173.0 2145
Note that the argument passed in a.Sample() must be an Astropy quantity, convertible into seconds.
Constructing complex queries#
Complex queries can be built using “OR” operators. Let’s look for 2 different series data at the same time:
>>> Fido.search(a.Time('2014-01-01T00:00:00', '2014-01-01T01:00:00'),
... a.jsoc.Series('hmi.v_45s') | a.jsoc.Series('aia.lev1_euv_12s'))
<sunpy.net.fido_factory.UnifiedResponse object at ...>
Results from 2 Providers:
81 Results from the JSOCClient:
Source: http://jsoc.stanford.edu
T_REC TELESCOP INSTRUME WAVELNTH CAR_ROT
----------------------- -------- ---------- -------- -------
2014.01.01_00:00:45_TAI SDO/HMI HMI_FRONT2 6173.0 2145
2014.01.01_00:01:30_TAI SDO/HMI HMI_FRONT2 6173.0 2145
2014.01.01_00:02:15_TAI SDO/HMI HMI_FRONT2 6173.0 2145
2014.01.01_00:03:00_TAI SDO/HMI HMI_FRONT2 6173.0 2145
2014.01.01_00:03:45_TAI SDO/HMI HMI_FRONT2 6173.0 2145
2014.01.01_00:04:30_TAI SDO/HMI HMI_FRONT2 6173.0 2145
2014.01.01_00:05:15_TAI SDO/HMI HMI_FRONT2 6173.0 2145
2014.01.01_00:06:00_TAI SDO/HMI HMI_FRONT2 6173.0 2145
2014.01.01_00:06:45_TAI SDO/HMI HMI_FRONT2 6173.0 2145
2014.01.01_00:07:30_TAI SDO/HMI HMI_FRONT2 6173.0 2145
... ... ... ... ...
2014.01.01_00:54:00_TAI SDO/HMI HMI_FRONT2 6173.0 2145
2014.01.01_00:54:45_TAI SDO/HMI HMI_FRONT2 6173.0 2145
2014.01.01_00:55:30_TAI SDO/HMI HMI_FRONT2 6173.0 2145
2014.01.01_00:56:15_TAI SDO/HMI HMI_FRONT2 6173.0 2145
2014.01.01_00:57:00_TAI SDO/HMI HMI_FRONT2 6173.0 2145
2014.01.01_00:57:45_TAI SDO/HMI HMI_FRONT2 6173.0 2145
2014.01.01_00:58:30_TAI SDO/HMI HMI_FRONT2 6173.0 2145
2014.01.01_00:59:15_TAI SDO/HMI HMI_FRONT2 6173.0 2145
2014.01.01_01:00:00_TAI SDO/HMI HMI_FRONT2 6173.0 2145
2014.01.01_01:00:45_TAI SDO/HMI HMI_FRONT2 6173.0 2145
Length = 81 rows
2107 Results from the JSOCClient:
Source: http://jsoc.stanford.edu
T_REC TELESCOP INSTRUME WAVELNTH CAR_ROT
-------------------- -------- -------- -------- -------
2014-01-01T00:00:01Z SDO/AIA AIA_4 94 2145
2014-01-01T00:00:01Z SDO/AIA AIA_1 131 2145
2014-01-01T00:00:01Z SDO/AIA AIA_3 171 2145
2014-01-01T00:00:01Z SDO/AIA AIA_2 193 2145
2014-01-01T00:00:01Z SDO/AIA AIA_2 211 2145
2014-01-01T00:00:01Z SDO/AIA AIA_4 304 2145
2014-01-01T00:00:01Z SDO/AIA AIA_1 335 2145
2014-01-01T00:00:13Z SDO/AIA AIA_4 94 2145
2014-01-01T00:00:13Z SDO/AIA AIA_1 131 2145
2014-01-01T00:00:13Z SDO/AIA AIA_3 171 2145
... ... ... ... ...
2014-01-01T00:59:49Z SDO/AIA AIA_2 211 2145
2014-01-01T00:59:49Z SDO/AIA AIA_4 304 2145
2014-01-01T00:59:49Z SDO/AIA AIA_1 335 2145
2014-01-01T01:00:01Z SDO/AIA AIA_4 94 2145
2014-01-01T01:00:01Z SDO/AIA AIA_1 131 2145
2014-01-01T01:00:01Z SDO/AIA AIA_3 171 2145
2014-01-01T01:00:01Z SDO/AIA AIA_2 193 2145
2014-01-01T01:00:01Z SDO/AIA AIA_2 211 2145
2014-01-01T01:00:01Z SDO/AIA AIA_4 304 2145
2014-01-01T01:00:01Z SDO/AIA AIA_1 335 2145
Length = 2107 rows
The two series names are joined together by the operator |.
This is the “OR” operator.
Think of the above query as setting a set of conditions which get passed to the JSOC.
Let’s say you want all the “hmi.v_45s” data from two separate days:
>>> Fido.search(a.Time('2014-01-01T00:00:00', '2014-01-01T01:00:00') |
... a.Time('2014-01-02T00:00:00', '2014-01-02T01:00:00'),
... a.jsoc.Series('hmi.v_45s'))
<sunpy.net.fido_factory.UnifiedResponse object at ...>
Results from 2 Providers:
81 Results from the JSOCClient:
Source: http://jsoc.stanford.edu
T_REC TELESCOP INSTRUME WAVELNTH CAR_ROT
----------------------- -------- ---------- -------- -------
2014.01.01_00:00:45_TAI SDO/HMI HMI_FRONT2 6173.0 2145
2014.01.01_00:01:30_TAI SDO/HMI HMI_FRONT2 6173.0 2145
2014.01.01_00:02:15_TAI SDO/HMI HMI_FRONT2 6173.0 2145
2014.01.01_00:03:00_TAI SDO/HMI HMI_FRONT2 6173.0 2145
2014.01.01_00:03:45_TAI SDO/HMI HMI_FRONT2 6173.0 2145
2014.01.01_00:04:30_TAI SDO/HMI HMI_FRONT2 6173.0 2145
2014.01.01_00:05:15_TAI SDO/HMI HMI_FRONT2 6173.0 2145
2014.01.01_00:06:00_TAI SDO/HMI HMI_FRONT2 6173.0 2145
2014.01.01_00:06:45_TAI SDO/HMI HMI_FRONT2 6173.0 2145
2014.01.01_00:07:30_TAI SDO/HMI HMI_FRONT2 6173.0 2145
... ... ... ... ...
2014.01.01_00:54:00_TAI SDO/HMI HMI_FRONT2 6173.0 2145
2014.01.01_00:54:45_TAI SDO/HMI HMI_FRONT2 6173.0 2145
2014.01.01_00:55:30_TAI SDO/HMI HMI_FRONT2 6173.0 2145
2014.01.01_00:56:15_TAI SDO/HMI HMI_FRONT2 6173.0 2145
2014.01.01_00:57:00_TAI SDO/HMI HMI_FRONT2 6173.0 2145
2014.01.01_00:57:45_TAI SDO/HMI HMI_FRONT2 6173.0 2145
2014.01.01_00:58:30_TAI SDO/HMI HMI_FRONT2 6173.0 2145
2014.01.01_00:59:15_TAI SDO/HMI HMI_FRONT2 6173.0 2145
2014.01.01_01:00:00_TAI SDO/HMI HMI_FRONT2 6173.0 2145
2014.01.01_01:00:45_TAI SDO/HMI HMI_FRONT2 6173.0 2145
Length = 81 rows
81 Results from the JSOCClient:
Source: http://jsoc.stanford.edu
T_REC TELESCOP INSTRUME WAVELNTH CAR_ROT
----------------------- -------- ---------- -------- -------
2014.01.02_00:00:45_TAI SDO/HMI HMI_FRONT2 6173.0 2145
2014.01.02_00:01:30_TAI SDO/HMI HMI_FRONT2 6173.0 2145
2014.01.02_00:02:15_TAI SDO/HMI HMI_FRONT2 6173.0 2145
2014.01.02_00:03:00_TAI SDO/HMI HMI_FRONT2 6173.0 2145
2014.01.02_00:03:45_TAI SDO/HMI HMI_FRONT2 6173.0 2145
2014.01.02_00:04:30_TAI SDO/HMI HMI_FRONT2 6173.0 2145
2014.01.02_00:05:15_TAI SDO/HMI HMI_FRONT2 6173.0 2145
2014.01.02_00:06:00_TAI SDO/HMI HMI_FRONT2 6173.0 2145
2014.01.02_00:06:45_TAI SDO/HMI HMI_FRONT2 6173.0 2145
2014.01.02_00:07:30_TAI SDO/HMI HMI_FRONT2 6173.0 2145
... ... ... ... ...
2014.01.02_00:54:00_TAI SDO/HMI HMI_FRONT2 6173.0 2145
2014.01.02_00:54:45_TAI SDO/HMI HMI_FRONT2 6173.0 2145
2014.01.02_00:55:30_TAI SDO/HMI HMI_FRONT2 6173.0 2145
2014.01.02_00:56:15_TAI SDO/HMI HMI_FRONT2 6173.0 2145
2014.01.02_00:57:00_TAI SDO/HMI HMI_FRONT2 6173.0 2145
2014.01.02_00:57:45_TAI SDO/HMI HMI_FRONT2 6173.0 2145
2014.01.02_00:58:30_TAI SDO/HMI HMI_FRONT2 6173.0 2145
2014.01.02_00:59:15_TAI SDO/HMI HMI_FRONT2 6173.0 2145
2014.01.02_01:00:00_TAI SDO/HMI HMI_FRONT2 6173.0 2145
2014.01.02_01:00:45_TAI SDO/HMI HMI_FRONT2 6173.0 2145
Length = 81 rows
Each of the arguments in this query style can be thought of as setting conditions that the returned records must satisfy.
It should be noted that AND operator is supported by some of the attributes only.
The attributes which support “&” are PrimeKey and Segment.
Using “&” with any other attributes will throw an error.
Downloading data#
To download the files located by search,
you can download them by fetch:
>>> Fido.search(a.Time('2014-01-01T00:00:00', '2014-01-01T01:00:00'),
... a.jsoc.Series('hmi.v_45s') | a.jsoc.Series('aia.lev1_euv_12s'),
... a.jsoc.Notify('solar@example.com')
>>> downloaded_files = Fido.fetch(res)
To export a request for download, you must have used the sunpy.net.jsoc.attrs.Notify attribute at search time to specify your email address.
Note
Only complete searches can be downloaded from JSOC This means that no slicing operations performed on the results object will affect the number of files downloaded.
Using JSOCClient for complex usage#
Fido interface uses JSOCClient in its backend, and combines the last 2 stages the JSOC process into one.
You can directly use the JSOC client to make queries, instead of the Fido client.
This will allow you to separate the 3 stages of the JSOC process, and perform it individually, hence allowing a greater control over the whole process.
Setup#
sunpy’s JSOC module is in net.
It can be imported as follows:
>>> from sunpy.net import jsoc
>>> client = jsoc.JSOCClient()
This creates your client object.
Making a query#
Querying JSOC using the JSOC client is very similar to what we were doing with Fido.
As above, we have to make sure we have an email address registered with JSOC before you are allowed to make a request.
See this to register your email address.
We can add an email address to the search query with the sunpy.net.jsoc.attrs.Notify attribute.
Please note you can search without this but right now, you can not add the email address after the search:
>>> res = client.search(a.Time('2014-01-01T00:00:00', '2014-01-01T01:00:00'),
... a.jsoc.Series('hmi.v_45s'),
... a.jsoc.Notify('sunpy@sunpy.org'))
Apart from the function name, everything is the same.
You need to pass the same values in the search as you did in search.
Complex queries can be built in a similar way, and all other things are the same.
Staging the request#
JSOC is a 3-stage process, and after getting the query results, we need to stage a request for the data to be downloaded. Only then, can we download them. The download request can be staged like this:
>>> requests = client.request_data(res)
>>> print(requests)
<ExportRequest id="JSOC_20170713_1461", status=0>
The function request_data stages the request.
It returns a drms.client.ExportRequest object, which has many attributes.
The most important ones are id and status.
Only when the status is 0, we can move to the third step, i.e., downloading the data.
If you are making more than 1 query at a time, it will return a list of ExportRequest objects.
Hence, access the list elements accordingly.
You can get the id and status of the request (if it is not a list) by:
>>> requests.id
JSOC_20170713_1461
>>> requests.status
0
Downloading data#
Once the status code is 0 you can download the data using the get_request method:
>>> res = client.get_request(requests)
This returns a Results instance which can be used to watch the progress of the download:
>>> res.wait(progress=True)