Internetcard github download

[LordNocturnus 1]

Hello

i am currently trying to download files from github (and yes i know about wget) and i need to know if they exist or not (making a list of all existing files is not feasible, because there is not enough space on the disk for the list).

I tried multiple things:

1. wget and filesystem.size always gave a length of 0 even for existing files

2. copying wget in my own program and setting a flag if there is a chunk works ... but not with every file (all files that exist have the same structure and length) and i need all of the existing ones them

Now i tried to do it myself with internet.request() but i only get 4 files (also the empty ones) then 6 fail and 4 succeed again(see test.txt t = True and f = False). This is because of to many open connections and i do not know how to close them.

The internet.close() from the wiki dose not work for some reason. (attempt to call nil)

How can i detect if the file exists and close the connection after the download?

test.txt:

tttttffffffffttttffffffttttffffffttttffffffttttffffffttttffffffttttffffffttttfffffft

internet.lua:

local c = require("component")
local internet = require("internet")
local f = io.open("test.txt", "wb")

imax = 0
xmax = 75
ymax = 180
zmax = 87
line = ""

link = "https://raw.githubusercontent.com/LordNocturnus/sf-"
folder = "/master/"

pos = 0
i = 0
while i <= imax do
 xb = pos
 while xb <= xmax do
  yb = 0
  while yb <= ymax do
   zb = 0
   while zb <= zmax do
    print(xb,yb,zb)
    file = tostring(xb) .. "-" .. tostring(yb) .. "-" .. tostring(zb) .. ".mb3d"
    status, user = pcall(internet.request ,link .. tostring(i) .. folder .. file)
    print(status)
    print(user)
--    for i,v in pairs(user) do
--     print(i,v)
--    end
    if status then
     line = line .. "t"
    else
     line = line .. "f"
    end
--    internet.close()
--    f = io.open(file, "wb")
--    for chunk in user do
--     print(chunk)
--     f:write(chunk)
--    end
--    f:close()
--    fs.remove("home/" .. file)
    zb = zb+1
   end
   yb = yb+1
  end
  xb = xb+1
 end
 i = i+1
end
f:write(line)
f:close()

 

[Fingercomp 37]

There are a bunch of questions, so let me start from the easiest ones to solve.

There is indeed no such thing as internet.close. You call the close method directly on the response object — and after you no longer need it. You can't read from a socket if you closed it.

-- this code...
local imax = 10

do
  local i = 1

  while i <= imax do
    print(i)
    i = i + 1
  end
end

-- ...is equivalent to the following code:
for i = 1, imax do
  print(i)
end

-- I recommend using for loops as there's less code to write

Now, the main question: how to check if a file exists. The GitHub server returns HTTP 404 status code if a location, the file in our case, does not exist. And, fortunately, the response object provides a method to get the status code: response.response() (from the docstring: function():number, string, table -- Get response code, message and headers).

local c = require("component")
local internet = require("internet")
local f = io.open("test.txt", "wb")

local imax = 0
local xmax = 75
local ymax = 180
local zmax = 87

local link = "https://raw.githubusercontent.com/LordNocturnus/sf-"
local folder = "/master/"

local pos = 0

for i = 0, imax do
  for xb = pos, xmax do
    for yb = 0, ymax do
      for zb = 0, zmax do
        print(xb, yb, zb)
        local file = xb .. "-" .. yb .. "-" .. zb .. ".mb3d"
        local status, response = pcall(internet.request, link .. i .. folder .. file)

        if not status or not response or response.response() >= 400 then
          print("Download of " .. link .. i .. folder .. file .. " failed.")
        else
          local f = io.open(file, "wb")

          for chunk in response do
            f:write(chunk)
          end

          f:close()
          socket:close()
        end
      end
    end
  end
end

 

[LordNocturnus 1]

thanks for the quick answer.

i tried your code but got a compare number with nil in the line and i newer had a 404 error before even with wget it just wrote a empty file

if not status or not response or response.response() >= 400 then

[Fingercomp 37]

The code below tries to make sure that the request finished, and then it reads the response. Unfortunately, I can't test the program in the game for various reasons. Could you try using it?

local c = require("component")
local computer = require("computer")
local internet = require("internet")
local f = io.open("test.txt", "wb")

local imax = 0
local xmax = 75
local ymax = 180
local zmax = 87

local TIMEOUT = 5  -- in seconds

local link = "https://raw.githubusercontent.com/LordNocturnus/sf-"
local folder = "/master/"

local pos = 0

for i = 0, imax do
  for xb = pos, xmax do
    for yb = 0, ymax do
      for zb = 0, zmax do
        print(xb, yb, zb)
        local file = xb .. "-" .. yb .. "-" .. zb .. ".mb3d"
        local url = link .. i .. folder .. file
        local status, response = pcall(internet.request, url)

        if not status or not response then
          print("Download of " .. url .. " failed.")
        else
          local startTime = computer.uptime()

          while true do
            local result, isFinished = pcall(response.finishConnect)
            
            if result and isFinished then
              break
            elseif not result then
              print("Could not connect to " .. url .. ".")
              os.exit()
            elseif computer.uptime() - startTime > TIMEOUT then
              print("Request to " .. url .. " timed out")
              os.exit()
            else
              os.sleep(0.25)
            end
          end

          local f = io.open(file, "wb")

          for chunk in response do
            f:write(chunk)
          end

          f:close()
          socket:close()
        end
      end
    end
  end
end

 

[LordNocturnus 1]

Thanks agian for the fast answer.

it successfully downloads a file and times out if there is no file but i get a indexing global value (nil) for this line:

          socket:close()

[Fingercomp 37]

Oh, right... I forgot about that, sorry. Replace socket:close() with response:close().

[LordNocturnus 1]

now i get a "attempt to call method 'close' (a table value)" error

[Molinko 43]

local c = require("component")
local computer = require("computer")
local internet = require("internet")
local f = io.open("test.txt", "wb")

local imax = 0
local xmax = 75
local ymax = 180
local zmax = 87

local TIMEOUT = 5  -- in seconds

local link = "https://raw.githubusercontent.com/LordNocturnus/sf-"
local folder = "/master/"

local pos = 0

for i = 0, imax do
  for xb = pos, xmax do
    for yb = 0, ymax do
      for zb = 0, zmax do
        print(xb, yb, zb)
        local file = xb .. "-" .. yb .. "-" .. zb .. ".mb3d"
        local url = link .. i .. folder .. file
        local status, connection = pcall(internet.request, url)

        if not status or not connection then
          print("Download of " .. url .. " failed.")
        else
          local startTime = computer.uptime()

          while true do
            local response, isFinished = pcall(connection.finishConnect)
            
            if response and isFinished then
              break
            elseif not response then
              print("Could not connect to " .. url .. ".")
              os.exit()
            elseif computer.uptime() - startTime > TIMEOUT then
              print("Request to " .. url .. " timed out")
              os.exit()
            else
              os.sleep(0.25)
            end
          end

          local f = io.open(file, "wb")

          for chunk in response do
            f:write(chunk)
          end

          f:close()
          connection:close()
        end
      end
    end
  end
end

Fixed some typos and made some things more readable. And a bug...

[LordNocturnus 1]

ok i changed the programm a bit:

local c = require("component")
local computer = require("computer")
local internet = require("internet")
local f = io.open("test.txt", "wb")

local imax = 0
local xmax = 0
local ymax = 41
local zmax = 1

local TIMEOUT = 5  -- in seconds

local link = "https://raw.githubusercontent.com/LordNocturnus/sf-"
local folder = "/master/"

local pos = 0

for i = 0, imax do
  for xb = pos, xmax do
    for yb = 0, ymax do
      for zb = 0, zmax do
        print(xb, yb, zb)
        local file = xb .. "-" .. yb .. "-" .. zb .. ".mb3d"
        local url = link .. i .. folder .. file
        local status, connection = pcall(internet.request, url)

        if not status or not connection then
          print("Download of " .. url .. " failed.")
        else
          local startTime = computer.uptime()

          while true do
            local response, isFinished = pcall(connection.finishConnect)
            
            if response and isFinished then
              flagg = true
              break
            elseif not response then
              print("Could not connect to " .. url .. ".")
              flagg = false
              break
            elseif computer.uptime() - startTime > TIMEOUT then
              print("Request to " .. url .. " timed out")
              flagg = false
              break
            else
              os.sleep(0.25)
            end
          end
          if flagg == true then
            local f = io.open(file, "wb")

            for chunk in response do
              f:write(chunk)
            end
 
            f:close()
            connection:close()
            flagg = false
          end
        end
      end
    end
  end
end

but now i get for file 0 41 0 (first existing file) in line

for chunk in response do

a "attempt to call a nil value" error