Using Remote Data Manager

This is an example to show how to use remote data manager to handle remote data in Sunpy.

Remote data manager is used to handle the usage of remote files in the code base with file verification using hashes.

from sunpy.data import manager

Let’s start by defining a function which uses some remote files. Let’s assume the function requires a file, say http://data.sunpy.org/sample-data/predicted-sunspot-radio-flux.txt, which has the SHA256 hash ‘4c85b04a5528aa97eb84a087450eda0421c71833820576330bba148564089b11’.

@manager.require('test_file',
                 ['http://data.sunpy.org/sample-data/predicted-sunspot-radio-flux.txt'],
                 '4c85b04a5528aa97eb84a087450eda0421c71833820576330bba148564089b11')
def test_function():
    pass

To access the downloaded file inside the function, you can use get function manager.get returns a pathlib.Path object.

@manager.require('test_file',
                 ['http://data.sunpy.org/sample-data/predicted-sunspot-radio-flux.txt'],
                 '4c85b04a5528aa97eb84a087450eda0421c71833820576330bba148564089b11')
def test_function():
    return manager.get('test_file')

The first time the function is called, the file will be downloaded. During subsequent calls, no downloading will take place.

test_function()  # The file will be downloaded
test_function()  # No downloading here

Out:

Files Downloaded:   0%|          | 0/1 [00:00<?, ?file/s]

predicted-sunspot-radio-flux.txt:   0%|          | 0.00/4.23k [00:00<?, ?B/s]

                                                                             
Files Downloaded: 100%|##########| 1/1 [00:00<00:00,  4.64file/s]
Files Downloaded: 100%|##########| 1/1 [00:00<00:00,  4.63file/s]

PosixPath('/home/docs/sunpy/data_manager/predicted-sunspot-radio-flux.txt')

In case the user wants to skip the hash check, there is a helper context manager skip_hash_check.

with manager.skip_hash_check():
    test_function()

Out:

Files Downloaded:   0%|          | 0/1 [00:00<?, ?file/s]

predicted-sunspot-radio-flux.txt:   0%|          | 0.00/4.23k [00:00<?, ?B/s]

                                                                             
Files Downloaded: 100%|##########| 1/1 [00:00<00:00,  5.06file/s]
Files Downloaded: 100%|##########| 1/1 [00:00<00:00,  5.04file/s]

If the user knows the function is going to use a file and want to replace it with another version they can do that too.

with manager.override_file('test_file', 'http://data.sunpy.org/sample-data/AIA20110319_105400_0171.fits'):
    test_function()

Out:

Files Downloaded:   0%|          | 0/1 [00:00<?, ?file/s]

AIA20110319_105400_0171.fits:   0%|          | 0.00/4.21M [00:00<?, ?B/s]

AIA20110319_105400_0171.fits:   1%|          | 41.8k/4.21M [00:00<00:12, 338kB/s]

AIA20110319_105400_0171.fits:   2%|2         | 105k/4.21M [00:00<00:11, 370kB/s] 

AIA20110319_105400_0171.fits:  11%|#         | 443k/4.21M [00:00<00:07, 499kB/s]

AIA20110319_105400_0171.fits:  17%|#7        | 730k/4.21M [00:00<00:05, 649kB/s]

AIA20110319_105400_0171.fits:  44%|####4     | 1.87M/4.21M [00:00<00:02, 905kB/s]

AIA20110319_105400_0171.fits:  86%|########6 | 3.64M/4.21M [00:00<00:00, 1.26MB/s]

                                                                                  
Files Downloaded: 100%|##########| 1/1 [00:00<00:00,  1.07file/s]
Files Downloaded: 100%|##########| 1/1 [00:00<00:00,  1.07file/s]

Total running time of the script: ( 0 minutes 6.562 seconds)

Gallery generated by Sphinx-Gallery