Jump to content
To block spammers, this forum has suspended new user registration ×
Comet Forums
To block spammers, this forum has suspended new user registration

how to download offline pages without being stopped by the website Bot?


Sam Rag

Recommended Posts

Hello,

I started just recently to use your software and I guess it is great. I wanted to offline download some pages from this website, that uses a very strong bot.
See examples of the pages
 

https://www.almaany.com/ar/dict/ar-en/DUI/?page=3

https://www.almaany.com/ar/dict/ar-en/agreement/?page=3

https://www.almaany.com/ar/dict/ar-en/laws/?page=3

https://www.almaany.com/ar/dict/ar-en/Emiratisation/?page=3

it started to work fine, and downloaded the pages as should be but after 40 links it stopped and the rest of the pages did not download. I understand we need to respect that website and not to overload it with download requests, but is there a way for example to batch add 2000 links and have an option in your software that says download one page ever 2 or 3 seconds and not consecutively? Or at least does some temporization as if a human being is downloading the pages, and not the software! Is there a slow but sure way to do this? Pardon me if my question sounds naiive, but I am new to your software. I hope someone can help.

Thanks a lot.

Sincerely,

Rag

Link to comment
Share on other sites

Please sign in to comment

You will be able to leave a comment after signing in



Sign In Now
×
×
  • Create New...