Monday, October 21, 2013

Check and curl the file list.

So, I had downloaded a bunch of files using curl.
However, the number of files seem to be less than what I had in the list.

Simple program to check if the file exists, else download using curl.

My trouble was that the list had repeated items. Had not checked before running around... Sigh!


clc

fid = fopen('urls.txt');
flist = textscan(fid, '%s');
fclose(fid);

for jj = 1:length(flist{1})
    fn = flist{1}{jj}
   
    fnlocal = fn(28:length(fn));
   
    if exist(fnlocal,'file')>0
        jj
        fnlocal
        pause(0.5)
        % do nothing
       
    else
        %curl
       
        command = ['curl -L -O ' fn];
        system(command)
       
    end
end

No comments: