check_rtxt.Rd
Function for checking robots.txt file
check_rtxt(url, delay, user_agent, force, verbose)
web address for download
default delay
user agent string
force re-downloading of robots.xtx
logical