What should I be looking for, when wget returns 500 but the same URL works fine in my web browser? I don’t see any access_log entries that seem to be related to the error. DEBUG output created by Wget 1.14 on linux-gnu. <SSL negotiation info stripped out>—request begin— GET /survey/de/tools/clear-caches/password/<some-token> HTTP/1.1 User-Agent: Wget/1.14 (linux-gnu) Accept: */* Host: testing.thesurveylab.net Connection: Keep-Alive—request end— HTTP request sent, awaiting response… —response begin— HTTP/1.0 500 Internal Server Error Date: Wed, [...]

I am currently attempting to transfer over 1 million files from one server to another. Using wget, it seems to be extremely slow, probably because it starts a new transfer after the previous one has been completed. Question: Is there a faster non-blocking (asynchronous) way to do the transfer? I do not have enough space on the first server to compress the files into tar.gz and transferring them over. Thanks! Asked by Nyxynyx Run 1 [...]

I read recently in blogs that by default wget on linux is 750, so only root can execute it. I would like to allow users wget and change it to 755, but I read around the web that it is a big security risk.. Asked by giorgio79 Setting the permissions to 755 is no security risk. The security risk is, if you have software installed that have bugs (for example a blogging software, or other [...]

I am trying to get a site mirroring with wget, and came across a strange behavior. Say I am mirroring an internal site named www.example.com in the following way, all seems to be working fine: wget -mkE http://www.example.com However, as I need to access the server by IP and not domain name, I try with: wget -mkE -D www.example.com –header “Host: www.example.com” http://IPv4_ADDR But in this case, it only downloads the first page and exits. [...]

I’ve tried several different combinations of rules and nothing seems to be working. I know that you can’t use prerouting table for a request coming from the localhost so I used the output table and that just returns errors when using wget. I have wget configured to use a proxy 127.0.0.1:8118. When I use the command below I get my public IP. If I change the command to remove the –no-proxy I get the IP [...]

I’m trying to download a zipball of a git repo: e.g. wget https://github.com/zeromq/jzmq/zipball/master This works fine in a web browser but on unix the file gets a weird name…how do I do this? Asked by DD. When you say weird name, what do you mean? This should however just be a ZIP-file. It just doesn’t have the extension. You can check this with the following command: file master You can unzip it with this command: [...]

I have this command in crontab: wget –quiet –delete-after http://boms.ro/admincp/cron/s/9abf0f42c1e4f55fdb87d8237cdde And when I run it with the –debug argument I get the following response: Caching boms.ro => 188.240.2.30 Created socket 3. Releasing 0x00000000010c97e0 (new refcount 1).—request begin— GET /admincp/cron/s/9abf0f42c1e4f55fdb87d8237cdde HTTP/1.0 User-Agent: Wget/1.12 (linux-gnu) Accept: */* Host: boms.ro Connection: Keep-Alive—request end——response begin— HTTP/1.0 406 Not Acceptable Date: Thu, 12 Apr 2012 18:29:45 GMT Server: LiteSpeed Connection: Keep-Alive Keep-Alive: timeout=5, max=100 Cache-Control: private, no-cache, max-age=0 Pragma: no-cache [...]

I am trying to copy a (.html) file from a Windows XP Professional shared folder onto a server running Ubuntu Linux 10.04 LTS. As it’s a shared folder the usual ‘wget’ doesn’t seem to work. I suspect I’m using the wrong address style completely. I have tried: http://192.168.1.66/SharedFolder/Data.html //192.168.1.66/SharedFolder/Data.html smb://192.168.1.66/SharedFolder/Data.html //192.168.1.66/SharedFolder/Data.html 192.168.1.66/SharedFolder/Data.html I wondered if this is even possible and if it is, could someone give me some pointers? I’ve successfully pinged the Windows box [...]

I am trying to wget a link, this wget works fine on my local machine but it doesn’t do so on the server. i tried to check the response header and i got the following on my local laptop curl -I http://en.cihan.com.tr/rss?user=albawaba\&type=12 HTTP/1.0 200 OK Server: Apache-Coyote/1.1 X-Powered-By: Servlet 2.4; JBoss-4.2.2.GA (build: SVNTag=JBoss_4_2_2_GA date=200710221139)/Tomcat-5.5 Set-Cookie: JSESSIONID=E65E0CC7CF956F45962FD8571F3AB02A; Path=/ Content-Type: application/xml;charset=UTF-8 Content-Length: 80043 Date: Thu, 29 Mar 2012 12:19:26 GMT Set-Cookie: NSC_fo-sv-bs.djibo.dpn.us=ffffffff092a094245525d5f4f58455e445a4a4229a0;expires=Thu, 29-Mar-2012 12:20:51 GMT;path=/ X-Cache: MISS [...]

Can I get the number of downloaded files with wget -r and use that as a variable? I want to write a script that runs multiple wget commands in it (using -q so i can control output) and then at the end, add up the number of downloaded files and echo that to the user. Is this possible? Asked by E Steven To build on Jed’s suggestion and take it the rest of the way, [...]