Jump to content
  • Sky
  • Blueberry
  • Slate
  • Blackcurrant
  • Watermelon
  • Strawberry
  • Orange
  • Banana
  • Apple
  • Emerald
  • Chocolate
  • Charcoal
  • 0
LordNocturnus

Internetcard github download

Question

Hello

i am currently trying to download files from github (and yes i know about wget) and i need to know if they exist or not (making a list of all existing files is not feasible, because there is not enough space on the disk for the list).

I tried multiple things:

1. wget and filesystem.size always gave a length of 0 even for existing files

2. copying wget in my own program and setting a flag if there is a chunk works ... but not with every file (all files that exist have the same structure and length) and i need all of the existing ones them

Now i tried to do it myself with internet.request() but i only get 4 files (also the empty ones) then 6 fail and 4 succeed again(see test.txt t = True and f = False). This is because of to many open connections and i do not know how to close them.

The internet.close() from the wiki dose not work for some reason. (attempt to call nil)

How can i detect if the file exists and close the connection after the download?

test.txt:

tttttffffffffttttffffffttttffffffttttffffffttttffffffttttffffffttttffffffttttfffffft

internet.lua:

local c = require("component")
local internet = require("internet")
local f = io.open("test.txt", "wb")

imax = 0
xmax = 75
ymax = 180
zmax = 87
line = ""

link = "https://raw.githubusercontent.com/LordNocturnus/sf-"
folder = "/master/"

pos = 0
i = 0
while i <= imax do
 xb = pos
 while xb <= xmax do
  yb = 0
  while yb <= ymax do
   zb = 0
   while zb <= zmax do
    print(xb,yb,zb)
    file = tostring(xb) .. "-" .. tostring(yb) .. "-" .. tostring(zb) .. ".mb3d"
    status, user = pcall(internet.request ,link .. tostring(i) .. folder .. file)
    print(status)
    print(user)
--    for i,v in pairs(user) do
--     print(i,v)
--    end
    if status then
     line = line .. "t"
    else
     line = line .. "f"
    end
--    internet.close()
--    f = io.open(file, "wb")
--    for chunk in user do
--     print(chunk)
--     f:write(chunk)
--    end
--    f:close()
--    fs.remove("home/" .. file)
    zb = zb+1
   end
   yb = yb+1
  end
  xb = xb+1
 end
 i = i+1
end
f:write(line)
f:close()

 

Link to post
Share on other sites

8 answers to this question

Recommended Posts

  • 0

There are a bunch of questions, so let me start from the easiest ones to solve.

There is indeed no such thing as internet.close. You call the close method directly on the response object — and after you no longer need it. You can't read from a socket if you closed it.

-- this code...
local imax = 10

do
  local i = 1

  while i <= imax do
    print(i)
    i = i + 1
  end
end

-- ...is equivalent to the following code:
for i = 1, imax do
  print(i)
end

-- I recommend using for loops as there's less code to write

Now, the main question: how to check if a file exists. The GitHub server returns HTTP 404 status code if a location, the file in our case, does not exist. And, fortunately, the response object provides a method to get the status code: response.response() (from the docstring: function():number, string, table -- Get response code, message and headers).

local c = require("component")
local internet = require("internet")
local f = io.open("test.txt", "wb")

local imax = 0
local xmax = 75
local ymax = 180
local zmax = 87

local link = "https://raw.githubusercontent.com/LordNocturnus/sf-"
local folder = "/master/"

local pos = 0

for i = 0, imax do
  for xb = pos, xmax do
    for yb = 0, ymax do
      for zb = 0, zmax do
        print(xb, yb, zb)
        local file = xb .. "-" .. yb .. "-" .. zb .. ".mb3d"
        local status, response = pcall(internet.request, link .. i .. folder .. file)

        if not status or not response or response.response() >= 400 then
          print("Download of " .. link .. i .. folder .. file .. " failed.")
        else
          local f = io.open(file, "wb")

          for chunk in response do
            f:write(chunk)
          end

          f:close()
          socket:close()
        end
      end
    end
  end
end

 

Link to post
Share on other sites
  • 0

The code below tries to make sure that the request finished, and then it reads the response. Unfortunately, I can't test the program in the game for various reasons. Could you try using it?

local c = require("component")
local computer = require("computer")
local internet = require("internet")
local f = io.open("test.txt", "wb")

local imax = 0
local xmax = 75
local ymax = 180
local zmax = 87

local TIMEOUT = 5  -- in seconds

local link = "https://raw.githubusercontent.com/LordNocturnus/sf-"
local folder = "/master/"

local pos = 0

for i = 0, imax do
  for xb = pos, xmax do
    for yb = 0, ymax do
      for zb = 0, zmax do
        print(xb, yb, zb)
        local file = xb .. "-" .. yb .. "-" .. zb .. ".mb3d"
        local url = link .. i .. folder .. file
        local status, response = pcall(internet.request, url)

        if not status or not response then
          print("Download of " .. url .. " failed.")
        else
          local startTime = computer.uptime()

          while true do
            local result, isFinished = pcall(response.finishConnect)
            
            if result and isFinished then
              break
            elseif not result then
              print("Could not connect to " .. url .. ".")
              os.exit()
            elseif computer.uptime() - startTime > TIMEOUT then
              print("Request to " .. url .. " timed out")
              os.exit()
            else
              os.sleep(0.25)
            end
          end

          local f = io.open(file, "wb")

          for chunk in response do
            f:write(chunk)
          end

          f:close()
          socket:close()
        end
      end
    end
  end
end

 

Link to post
Share on other sites
  • 0
local c = require("component")
local computer = require("computer")
local internet = require("internet")
local f = io.open("test.txt", "wb")

local imax = 0
local xmax = 75
local ymax = 180
local zmax = 87

local TIMEOUT = 5  -- in seconds

local link = "https://raw.githubusercontent.com/LordNocturnus/sf-"
local folder = "/master/"

local pos = 0

for i = 0, imax do
  for xb = pos, xmax do
    for yb = 0, ymax do
      for zb = 0, zmax do
        print(xb, yb, zb)
        local file = xb .. "-" .. yb .. "-" .. zb .. ".mb3d"
        local url = link .. i .. folder .. file
        local status, connection = pcall(internet.request, url)

        if not status or not connection then
          print("Download of " .. url .. " failed.")
        else
          local startTime = computer.uptime()

          while true do
            local response, isFinished = pcall(connection.finishConnect)
            
            if response and isFinished then
              break
            elseif not response then
              print("Could not connect to " .. url .. ".")
              os.exit()
            elseif computer.uptime() - startTime > TIMEOUT then
              print("Request to " .. url .. " timed out")
              os.exit()
            else
              os.sleep(0.25)
            end
          end

          local f = io.open(file, "wb")

          for chunk in response do
            f:write(chunk)
          end

          f:close()
          connection:close()
        end
      end
    end
  end
end

Fixed some typos and made some things more readable. And a bug...

Link to post
Share on other sites
  • 0

ok i changed the programm a bit:

local c = require("component")
local computer = require("computer")
local internet = require("internet")
local f = io.open("test.txt", "wb")

local imax = 0
local xmax = 0
local ymax = 41
local zmax = 1

local TIMEOUT = 5  -- in seconds

local link = "https://raw.githubusercontent.com/LordNocturnus/sf-"
local folder = "/master/"

local pos = 0

for i = 0, imax do
  for xb = pos, xmax do
    for yb = 0, ymax do
      for zb = 0, zmax do
        print(xb, yb, zb)
        local file = xb .. "-" .. yb .. "-" .. zb .. ".mb3d"
        local url = link .. i .. folder .. file
        local status, connection = pcall(internet.request, url)

        if not status or not connection then
          print("Download of " .. url .. " failed.")
        else
          local startTime = computer.uptime()

          while true do
            local response, isFinished = pcall(connection.finishConnect)
            
            if response and isFinished then
              flagg = true
              break
            elseif not response then
              print("Could not connect to " .. url .. ".")
              flagg = false
              break
            elseif computer.uptime() - startTime > TIMEOUT then
              print("Request to " .. url .. " timed out")
              flagg = false
              break
            else
              os.sleep(0.25)
            end
          end
          if flagg == true then
            local f = io.open(file, "wb")

            for chunk in response do
              f:write(chunk)
            end
 
            f:close()
            connection:close()
            flagg = false
          end
        end
      end
    end
  end
end

but now i get for file 0 41 0 (first existing file) in line

for chunk in response do

a "attempt to call a nil value" error

Link to post
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Answer this question...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

Loading...

×
×
  • Create New...

Important Information

By using this site, you agree to our Terms of Use and Privacy Policy.