Function to get robots.txt is structured form. Memoised — polite_fetch_rtxt • dopingdata

Function to get robots.txt is structured form. Memoised

Usage

polite_fetch_rtxt(..., user_agent, delay, verbose)

Arguments

...: arguments passed to robotstxt::robotstxt()
user_agent: user agent string
delay: default delay
verbose: logical