Scraper#

class sunpy.net.scraper.Scraper(pattern=None, regex=False, *, format=None, **kwargs)[source]#

Bases: object

A scraper to scrap web data archives based on dates.

Parameters:

pattern (str) – A string containing the url with the date encoded as datetime formats, and any other parameter as kwargs as a string format. This can also be a uri to a local file patterns. Deprecated in favor of format. Default is None.
regex (bool) – Set to True if parts of the pattern uses regexp symbols. This only works for the filename part of the pattern rather than the full url. Be careful that periods . matches any character and therefore it’s better to escape them. If regexp is used, other kwargs are ignored and string replacement is not possible. Default is False. Deprecated in favor of format. Default is False.
format (str) – A string containing the url with the date and other information to be extracted encoded as parse formats, and any other kwargs parameters as a string format, the former represented using double curly-brackets to differentiate from the latter. The accepted parse representations for datetime values are as given in PARSE_TIME_CONVERSIONS. This can also be a uri to a local file patterns. Default is None.
kwargs (dict) – A dictionary containing the values to be replaced in the pattern. Will be ignored if regex is True.

pattern#

The pattern with the parse format.

Type:: str

datetime_pattern#

The parse pattern in the datetime format.

Type:: str

now#

The pattern with the actual date. This is not checking if there is an existent file, but just how the pattern looks with the current time.

Type:: datetime.datetime

Examples

>>> from sunpy.net import Scraper
>>>
>>> pattern = ('http://proba2.oma.be/{instrument}/data/bsd/{{year:4d}}/{{month:2d}}/{{day:2d}}/'
...            '{instrument}_lv1_{{year:4d}}{{month:2d}}{{day:2d}}_{{hour:2d}}{{month:2d}}{{second:2d}}.fits')
>>> swap = Scraper(format=pattern, instrument='swap')
>>>
>>> print(swap.pattern)
http://proba2.oma.be/swap/data/bsd/{year:4d}/{month:2d}/{day:2d}/swap_lv1_{year:4d}{month:2d}{day:2d}_{hour:2d}{month:2d}{second:2d}.fits
>>>
>>> print(swap.datetime_pattern)
http://proba2.oma.be/swap/data/bsd/%Y/%m/%d/swap_lv1_%Y%m%d_%H%m%S.fits
>>>
>>> print(swap.now)  
http://proba2.oma.be/swap/data/bsd/2022/12/21/swap_lv1_20221221_112433.fits

Methods Summary

`filelist`(timerange)	Returns the list of existent files in the archive for the given time range.
`matches`(filepath, date)	Checks if the given filepath is how the file path is expected to look on given date based on the pattern.
`range`(timerange)	Gets the directories for a certain range of time.

Methods Documentation

filelist(timerange)[source]#

Returns the list of existent files in the archive for the given time range.

Parameters:: timerange (TimeRange) – Time interval where to find the directories for a given pattern.
Returns:: filesurls (list of str) – List of all the files found between the time range given.

Examples

>>> from sunpy.net import Scraper
>>> pattern = ('http://proba2.oma.be/{instrument}/data/bsd/{{year:4d}}/{{month:2d}}/{{day:2d}}/'
...            '{instrument}_lv1_{{year:4d}}{{month:2d}}{{day:2d}}_{{hour:2d}}{{minute:2d}}{{second:2d}}.fits')
>>> swap = Scraper(format=pattern, instrument='swap')
>>> from sunpy.time import TimeRange
>>> timerange = TimeRange('2015-01-01T00:08:00','2015-01-01T00:12:00')
>>> print(swap.filelist(timerange))  
['http://proba2.oma.be/swap/data/bsd/2015/01/01/swap_lv1_20150101_000857.fits',
 'http://proba2.oma.be/swap/data/bsd/2015/01/01/swap_lv1_20150101_001027.fits',
 'http://proba2.oma.be/swap/data/bsd/2015/01/01/swap_lv1_20150101_001157.fits']

While writing the pattern, we can also leverage parse capabilities by using the {{}} notation to match parts of the filename that cannot be known beforehand:

>>> from sunpy.net import Scraper
>>> from sunpy.time import TimeRange
>>> pattern = 'http://proba2.oma.be/lyra/data/bsd/{{year:4d}}/{{month:2d}}/{{day:2d}}/{{}}_lev{{Level:1d}}_std.fits'
>>> lyra = Scraper(format=pattern)
>>> timerange = TimeRange('2023-03-06T00:00:00','2023-03-06T00:10:00')
>>> print(swap.filelist(timerange)) 
['http://proba2.oma.be/swap/data/bsd/2023/03/06/swap_lv1_20230306_000128.fits',
'http://proba2.oma.be/swap/data/bsd/2023/03/06/swap_lv1_20230306_000318.fits',
'http://proba2.oma.be/swap/data/bsd/2023/03/06/swap_lv1_20230306_000508.fits',
'http://proba2.oma.be/swap/data/bsd/2023/03/06/swap_lv1_20230306_000658.fits',
'http://proba2.oma.be/swap/data/bsd/2023/03/06/swap_lv1_20230306_000848.fits']

Notes

The search is strict with the time range, so if the archive scraped contains daily files, but the range doesn’t start from the beginning of the day, then the file for that day won’t be selected. The end of the timerange will normally be OK as includes the file on such end time.

matches(filepath, date)[source]#

Checks if the given filepath is how the file path is expected to look on given date based on the pattern.

Parameters:

filepath (str) – File path to check.
date (datetime.datetime or astropy.time.Time) – The date for which to check.

Returns:

bool – True if the given filepath matches with the calculated one for given date, else False.

range(timerange)[source]#

Gets the directories for a certain range of time.

Parameters:: timerange (TimeRange) – Time interval where to find the directories for a given pattern.
Returns:: list of str – All the possible directories valid for the time range given. Notice that these directories may not exist in the archive.