Note
Go to the end to download the full example code.
Querying Metadata clients#
This example shows how to search and retrieve metadata using Fido.
Fido supports searching metadata from services like HEKClient,
HECClient, and JSOCClient.
In this example we will make one search for records from the JSOC and the HEK, and then download the corresponding file from the JSOC.
import os
from sunpy.net import Fido
from sunpy.net import attrs as a
We will query the HEK for all flares with a peak flux greater than 1000. We will also search JSOC for a ‘hmi.m_45s’ series.
timerange = a.Time('2010/8/1 03:40', '2010/8/1 3:40:10')
# Exporting data from the JSOC requires registering your email first.
# Please replace this with your email address once you have registered
# like so: jsoc_email = "your_email@example.com"
# See `this page <http://jsoc.stanford.edu/ajax/register_email.html>`__ for more details.
jsoc_email = os.environ["JSOC_EMAIL"]
results = Fido.search(timerange,
a.hek.FL & (a.hek.FL.PeakFlux > 1000) |
a.jsoc.Series('hmi.m_45s') & a.jsoc.Notify(jsoc_email))
results is a UnifiedResponse object that
contains records returned from querying various clients by “Fido.search”.
print(results)
Results from 2 Providers:
2 Results from the HEKClient:
gs_thumburl comment_count hpc_bbox frm_humanflag hgc_coord obs_levelnum hpc_coord event_npixels gs_imageurl ar_polarity frm_paramset hrc_coord event_starttime ar_mtwilsoncls event_type intensmin fl_fluence obs_meanwavel frm_url skel_chaincode bound_chaincode noposition active intensmax frm_versionnumber fl_halphaclass area_uncert obs_dataprepurl hpc_geom hgc_bbox intensmedian chaincodetype obs_channelid event_clippedspatial ar_noaaclass SOL_standard event_avg_rating eventtype hpc_boundcc event_mapurl frm_contact ar_penumbracls intensmean bound_ccstartc1 frm_name area_atdiskcenter frm_identifier obs_observatory event_description boundbox_c2ur obs_firstprocessingdate boundbox_c2ll frm_institute hrc_bbox refs_orig ar_mcintoshcls event_maskurl bound_ccstartc2 gs_movieurl event_score skel_startc2 skel_startc1 fl_efoldtime event_expires hrc_boundcc event_probability hpc_heliosphere_corrected intensvar frm_daterun hpc_y hpc_x search_instrument ar_numspots kb_archivdate kb_archivist intenstotal sum_overlap_scores hgs_boundcc intensskew obs_includesnrt rasterscan kb_archivid search_frm_name boundbox_c1ur ar_noaanum area_atdiskcenteruncert boundbox_c1ll event_importance_num_ratings ar_compactnesscls skel_curvature event_testflag event_c2error hrc_r skel_nsteps hgs_y obs_title hgs_x hcr_checked frm_specificid event_title obs_instrument event_c1error revision hpc_radius event_endtime event_importance search_observatory area_raw concept solar_object_locator hgc_boundcc fl_peakflux hgc_x hrc_a event_peaktime hgc_y gs_galleryid fl_goescls hgs_coord ar_zurichcls bound_ccnsteps intenskurt event_clippedtemporal hee_x hee_y hee_z fl_peakem rasterscantype search_channelid fl_peaktemp hgs_bbox obs_lastprocessingdate refs event_coord
arcsec,arcsec deg,deg arcsec,arcsec solRad,deg,solRad cm deg,deg deg,deg deg,deg arcsec,arcsec arcsec arcsec solRad,deg,solRad solRad,deg,solRad deg,deg arcsec arcsec arcsec arcsec deg,deg DN / (pix s) deg,deg deg,deg
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ ------------- ---------------------------- ------------- ----------------------------- ------------ ------------- ------------- ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ ----------- --------------------------------------------------------- --------------------------------------- ----------------------- -------------- ---------- --------- ---------- ------------- ------------------- -------------- --------------- ---------- ------ --------- ----------------- -------------- ----------- --------------- ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ ---------------------------------------------- ------------ ------------- ------------- -------------------- ------------ ------------------------------ ---------------- --------- ------------- ------------ ----------------------------------------- -------------- ---------- --------------- -------------------------------- ----------------- -------------------- --------------- ----------------- ------------- ----------------------- ------------- ------------- ------------------------------------------------------------------ --------- -------------- ------------- --------------- ----------- ------------------- ------------ ------------ ------------ ------------- ----------------- ----------------- ------------------------- --------- ------------------- ------ ----- ----------------- ----------- ------------------- ------------ ----------- ------------------ ----------- ---------- --------------- ---------- ----------------------------------------------------------------------------------------------------- -------------------------------- ------------- ---------- ----------------------- ------------- ---------------------------- ----------------- -------------- -------------- ------------- ----------------- ----------- ---------- --------- --------- ----------- -------------- ----------- -------------- ------------- -------- ----------------- ----------------------- ---------------- ------------------ -------- ------- --------------------------------- ----------- ------------ ------------------- ----------------- ----------------------- ---------- ------------ ---------- -------------------- ------------ -------------- ---------- --------------------- ----- ----- ----- --------- -------------- ---------------- ----------- -------------------------------------------- ---------------------- -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
http://sdowww.lmsal.com/sdomedia/ssw/ssw_client/data/ssw_service_100731_205448_25495028/www/EDS_FlareDetective-TriggerModule_20100801T033001-20100801T035225_AIA_171_S21W87_ssw_cutout_20100801_033013_AIA_171_S21W87_20100801_033012_context_0180.gif 0 844.8,-384.0 .. 844.8,-384.0 false 199.39180199999998,-21.228119 -- 883.2,-345.6 ——— http://sdowww.lmsal.com/sdomedia/ssw/ssw_client/data/ssw_service_100731_205448_25495028/www/EDS_FlareDetective-TriggerModule_20100801T033001-20100801T035225_AIA_171_S21W87_ssw_cutout_20100801_033013_AIA_171_S21W87_20100801_033012_context_0800.gif -- DerivativeThreshold= 2.00000e+00 EndFraction= 2.50000e-01 1.003103535969751,338.6293777306568,1.0 2010-08-01 03:40:01.000 FL ——— ——— 1.71e-06 www.cfa.harvard.edu ———,——— ———,——— false true ——— 0.5 ——— 010300000001000000050000006666666666668A4000000000000078C0CDCCCCCCCCCC8C4000000000000078C0CDCCCCCCCCCC8C4033333333333373C06666666666668A4033333333333373C06666666666668A4000000000000078C0 187.130079,-22.581195 .. 187.130079,-22.581195 ——— 171 SOL2010-08-01T03:40:01L200C111 None 9 ———,——— Paolo C. Grigis - pgrigis@cfa.harvard.edu ——— ——— Flare Detective - Trigger Module ——— Feature Finding Team SDO -307.2 -384.0 SAO 0.981493,335.55604500000004,1.0 .. 0.981493,335.55604500000004,1.0 ——— 0.13427866573283298 ——— ——— ——— ———,———,1.0 -- false ——— 2010-07-31T20:51:51 -345.6 883.2 AIA -- 2010-08-01T03:54:48 rtimmons ——— 0 ———,——— ——— ivo://helio-informatics.org/FL_FlareDetective-TriggerModule_20100801_035438_2010-08-01T03:40:01.340_1 Flare Detective - Trigger Module 921.6 -- ——— 844.8 -- false 2.0 1.003103535969751 -- -21.228119 87.464482 true AIA 2.0 2 948.4100379055465 2010-08-01 03:42:25.000 -- SDO ——— Flare SOL2010-08-01T03:40:01.34L200C111 ———,——— 1027.64 -160.60819800000002 248.6293777306568 2010-08-01 03:40:37.000 -21.228119 87.464482,-21.228119 -- ——— None None ——— 171 ——— 75.202759,-22.581195 .. 75.202759,-22.581195 {'ref_name': 'SDO Movie', 'ref_type': 'movie', 'ref_url': 'http://sdowww.lmsal.com/sdomedia/h264/2010/08/01/EDS_FlareDetective-TriggerModule_20100801T033001-20100801T035225_AIA_171_S21W87.mov'} .. {'ref_name': 'SSW Context Image', 'ref_type': 'image', 'ref_url': 'http://sdowww.lmsal.com/sdomedia/ssw/ssw_client/data/ssw_service_100731_205448_25495028/www/EDS_FlareDetective-TriggerModule_20100801T033001-20100801T035225_AIA_171_S21W87_ssw_cutout_20100801_033013_AIA_171_S21W87_20100801_033012_context_0800.gif'} <SkyCoord (Helioprojective: obstime=2010-08-01T03:40:01.000, rsun=695700.0 km, observer=<HeliographicStonyhurst Coordinate (obstime=2010-08-01T03:40:01.000, rsun=695700.0 km): (lon, lat, radius) in (deg, deg, AU)\n (0., 5.77873076, 1.01497794)>): (Tx, Ty) in arcsec\n (883.2, -345.6)>
http://sdowww.lmsal.com/sdomedia/ssw/ssw_client/data/ssw_service_100801_234037_25860951/www/EDS_FlareDetective-TriggerModule_20100801T033008-20100801T035232_AIA_193_S21W87_ssw_cutout_20100801_033020_AIA_193_S21W87_20100801_033019_context_0180.gif 0 844.8,-384.0 .. 844.8,-384.0 false 199.38612,-21.227645 -- 883.2,-345.6 ——— http://sdowww.lmsal.com/sdomedia/ssw/ssw_client/data/ssw_service_100801_234037_25860951/www/EDS_FlareDetective-TriggerModule_20100801T033008-20100801T035232_AIA_193_S21W87_ssw_cutout_20100801_033020_AIA_193_S21W87_20100801_033019_context_0800.gif -- DerivativeThreshold= 3.00000e+00 EndFraction= 2.50000e-01 1.00310352536024,338.6293777306568,1.0 2010-08-01 03:40:08.000 FL ——— ——— 1.93e-06 www.cfa.harvard.edu ———,——— ———,——— false true ——— 0.5 ——— 010300000001000000050000006666666666668A4000000000000078C0CDCCCCCCCCCC8C4000000000000078C0CDCCCCCCCCCC8C4033333333333373C06666666666668A4033333333333373C06666666666668A4000000000000078C0 187.128993,-22.581193 .. 187.128993,-22.581193 ——— 193 SOL2010-08-01T03:40:08L200C111 None 9 ———,——— Paolo C. Grigis - pgrigis@cfa.harvard.edu ——— ——— Flare Detective - Trigger Module ——— Feature Finding Team SDO -307.2 -384.0 SAO 0.981493,335.55604500000004,1.0 .. 0.981493,335.55604500000004,1.0 ——— 0.18839310914355603 ——— ——— ——— ———,———,1.0 -- false ——— 2010-08-01T23:40:09 -345.6 883.2 AIA -- 2010-08-02T06:40:38 rtimmons ——— 0 ———,——— ——— ivo://helio-informatics.org/FL_FlareDetective-TriggerModule_20100802_064031_2010-08-01T03:40:08.840_1 Flare Detective - Trigger Module 921.6 -- ——— 844.8 -- false 2.0 1.00310352536024 -- -21.227645 87.45988 true AIA 2.0 2 948.4100379055465 2010-08-01 03:42:32.000 -- SDO ——— Flare SOL2010-08-01T03:40:08.84L200C111 ———,——— 1441.78 -160.61388 248.6293777306568 2010-08-01 03:40:44.000 -21.227645 87.45988,-21.227645 -- ——— None None ——— 193 ——— 75.202753,-22.581193 .. 75.202753,-22.581193 {'ref_name': 'FRM_URL', 'ref_type': 'unknown', 'ref_url': 'http://www.cfa.harvard.edu'} .. {'ref_name': 'SSW Context Image', 'ref_type': 'image', 'ref_url': 'http://sdowww.lmsal.com/sdomedia/ssw/ssw_client/data/ssw_service_100801_234037_25860951/www/EDS_FlareDetective-TriggerModule_20100801T033008-20100801T035232_AIA_193_S21W87_ssw_cutout_20100801_033020_AIA_193_S21W87_20100801_033019_context_0800.gif'} <SkyCoord (Helioprojective: obstime=2010-08-01T03:40:08.000, rsun=695700.0 km, observer=<HeliographicStonyhurst Coordinate (obstime=2010-08-01T03:40:08.000, rsun=695700.0 km): (lon, lat, radius) in (deg, deg, AU)\n (0., 5.77873669, 1.01497793)>): (Tx, Ty) in arcsec\n (883.2, -345.6)>
1 Results from the JSOCClient:
Source: http://jsoc.stanford.edu
T_REC TELESCOP INSTRUME WAVELNTH CAR_ROT
----------------------- -------- ---------- -------- -------
2010.08.01_03:40:30_TAI SDO/HMI HMI_FRONT2 6173.0 2099
Now we will download the searched records. Since the HEK
client don’t provide files, Fido.fetch will
ignore it and only download files from JSOC.
INFO: 1 URLs found for download. Full request totaling 14MB [sunpy.net.jsoc.jsoc]
['/home/docs/sunpy/data/hmi.m_45s.20100801_034030_TAI.2.magnetogram.fits']
Now we will extract individual responses from Fido results. We can index these results using the client’s name (which is case-insensitive).
hek_results, jsoc_results = results['hek'], results['jsoc']
The results from a metadata search could have up to 100 columns.
As a result, you can use use show() to specify the column names you want to display.
hek_table = hek_results.show('event_peaktime', 'obs_instrument', 'fl_peakflux')
print(hek_table)
event_peaktime obs_instrument fl_peakflux
DN / (pix s)
----------------------- -------------- ------------
2010-08-01 03:40:37.000 AIA 1027.64
2010-08-01 03:40:44.000 AIA 1441.78
The results from JSOC have a default set of columns to show and are
['T_REC', 'TELESCOP', 'INSTRUME', 'WAVELNTH', 'CAR_ROT'].
To display all of the columns, we can use show() without passings any arguments.
print(jsoc_results)
jsoc_table = jsoc_results.show()
print(jsoc_table)
T_REC TELESCOP INSTRUME WAVELNTH CAR_ROT
----------------------- -------- ---------- -------- -------
2010.08.01_03:40:30_TAI SDO/HMI HMI_FRONT2 6173.0 2099
DATE DATE__OBS DATE-OBS TELESCOP INSTRUME WAVELNTH CAMERA BUNIT ORIGIN CONTENT QUALITY QUALLEV1 HISTORY COMMENT BLD_VERS HCAMID SOURCE TOTVALS DATAVALS MISSVALS SATVALS DATAMIN2 DATAMAX2 DATAMED2 DATAMEA2 DATARMS2 DATASKE2 DATAKUR2 DATAMIN DATAMAX DATAMEDN DATAMEAN DATARMS DATASKEW DATAKURT CTYPE1 CTYPE2 CRPIX1 CRPIX2 CRVAL1 CRVAL2 CDELT1 CDELT2 CUNIT1 CUNIT2 CROTA2 CRDER1 CRDER2 CSYSER1 CSYSER2 WCSNAME DSUN_OBS DSUN_REF RSUN_REF CRLN_OBS CRLT_OBS CAR_ROT OBS_VR OBS_VW OBS_VN RSUN_OBS T_OBS T_REC T_REC_epoch T_REC_step T_REC_unit CADENCE DATASIGN HFLID HCFTID QLOOK CAL_FSN LUTQUERY TSEL TFRONT TINTNUM SINTNUM DISTCOEF ROTCOEF ODICOEFF OROCOEFF POLCALM CODEVER0 CODEVER1 CODEVER2 CODEVER3 CALVER64
-------------------- ----------------------- ----------------------- -------- ---------- -------- ------ ----- ------------ ----------- ------- -------- --------------------------------------------------------------------------------------------------------------------------------------------------------------- ----------------------------------------------------------------------------------------------------------------- -------- ------ ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- -------- -------- -------- ------- ------------ ----------- -------- --------- --------- --------- ----------- ----------- ----------- -------- --------- --------- ---------- ----------- -------- -------- ----------- ----------- ------ ------ -------- -------- ------ ------ ---------- ------ ------ ------- ------- ------------------------- --------------- ------------ --------- ---------- -------- ------- ----------- ------------ ----------- ---------- ----------------------- ----------------------- ----------------------- ---------- ---------- ------- -------- ----- ------ ----- ------- --------------------------------------- --------- --------- ------- ------- ----------------------------------------------------------- ----------------------------------------------------------- -------- -------- ------- ---------------------------------------------------------------- ------------------------------------------------------------- ------------------------------------------------------------- --------------------------------------------------- --------
2012-09-05T07:57:40Z 2010-08-01T03:39:41.00Z 2010-08-01T03:39:41.00Z SDO/HMI HMI_FRONT2 6173.0 2 Gauss SDO/JSOC-SDP MAGNETOGRAM -- -- Polynomial Coefficients used for Doppler velocity correction: 1.333019e+02 5.532295e-04 -1.414157e-06 1.826364e-10 \nCROTA2 corrected by adding -0.0702 degrees De-rotation: ON; Un-distortion: ON; Re-centering: ON; Re-sizing: OFF; RSUNerr=0.5; correction for cosmic-ray hits -511 3 hmi.lev1[:#6348058,#6348046,#6348010,#6348082,#6348094,#6348166,#6348060,#6348048,#6348012,#6348084,#6348096,#6348168,#6348062,#6348050,#6348014,#6347966,#6348086,#6348098,#6348064,#6348052,#6348016,#6347968,#6348088,#6348100,#6348066,#6348054,#6348018,#6347970,#6348090,#6348102,#6348068,#6348032,#6348020,#6347996,#6348104,#6348116,#6348070,#6348034,#6348022,#6347998,#6348106,#6348118,#6348072,#6348036,#6348024,#6348000,#6348108,#6348120,#6348074,#6348038,#6348026,#6348002,#6348110,#6348122,#6348076,#6348040,#6348028,#6348004,#6348112,#6348124,#6348078,#6348042,#6348030,#6348006,#6348114,#6348126,#6348080,#6348056,#6348044,#6348008,#6348092,#6348164] 11637782 11637782 0 10235 -4470.989746 4470.989746 -0.05855 -0.278637 62.809631 -0.783281 2636.481201 -2267.47168 1035.622192 -0.06031 -0.268451 28.155598 -21.853561 1292.156616 HPLN-TAN HPLT-TAN 2041.515869 2047.844238 0.0 0.0 0.504285 0.504285 arcsec arcsec 179.929855 -- -- -- -- Helioprojective-cartesian 151855544707.36 149597870691 696000000 111.963272 5.789389 2099 1798.364614 31499.633969 2845.311372 945.377441 2010.08.01_03:40:38_TAI 2010.08.01_03:40:30_TAI 1993.01.01_00:00:00_TAI 45.0 secs 45.0 1 1001 11 0 4875802 hmi.lookup[4875802][][3][36][58][0][82] 22.380854 28.071377 6 10 /home/couvidat/cvs/JSOC/proj/lev1.5_hmi/apps/../libs/lev15/ /home/couvidat/cvs/JSOC/proj/lev1.5_hmi/apps/../libs/lev15/ 6 4 1 $Id: HMI_observables.c,v 1.20 2010/11/12 21:38:52 couvidat Exp $ $Id: interpol_code.c,v 1.1 2010/09/16 20:46:48 couvidat Exp $ $Id: interpol_code.c,v 1.1 2010/09/16 20:46:48 couvidat Exp $ $Id: polcal.c,v 1.4 2010/09/23 20:57:27 schou Exp $ --
Total running time of the script: (0 minutes 17.662 seconds)