Saving Binary Data

[ATastyPeanut 0]

Hello OC community,

 

I am hoping I can get help with a project where I'm saving geolyzer data.  The issue is how to make the data files small enough.  My current method is to write the single digit density values 0, 1, 2, 3 into string form so that it's a string of 255 numbers. Then using the debug card's deflate function on it for final writing to the file.  However, that's wasting a bunch of space since its saving the characters as 8bit values when I only need 2bit.

 

The new way I wish to use involves saving binary data.  But not the standard 8bit files but a shorter 2bit data.  As in saving the values 00, 01, 10, 11 to represent air, low density, intermediate density and high density respectively.  But not having them write like 00000000, 00000001, 00000010, 00000011.  Follows is my attempt at the code for saving 8bit values, unfortunately, I couldn't get it to work, though.

local fs = require "filesystem"

--create fake testing data
local testDensity = {}
local j = 0
for i=1,255 do
	testDensity[i] = j
	if j > 2 then
		j = 0
	else
		j = j + 1
	end
end

--opens test binary file
local file = io.open("testBinaryFile", "wb")
for i=1, 255 do
	local bit
--attempt to convert the table's value into a binary value using the escape character \
	if testDensity[i] == 0 then
		bit = "\0"
	elseif testDensity[i] == 1 then
		bit = "\1"
	elseif testDensity[i] == 2 then
		bit = "\2"
	else
		bit = "\3"
	end
	file:write(bit)
end
io.close(file)

The result of this code is an empty file and I am unsure how to fix it to write the binary data, and then how to make that binary data 2bit and not 8bit.

 

Thank you for any assistance provided.  I'm sorry if this post didn't follow the common guidelines, it is my first post on the site. 

 

 

[ATastyPeanut 0]

So I wrote a proof of concept code that does successfully write characters to stand for the series of 4 base4 numbers and also successfully reads the file back.

 

http://pastebin.com/SKAkEM8w

 

Sorry if I was supposed to put the code in a spoilers tag, I couldn't figure out how to.

local fs = require "filesystem"
local string = require "string"

local testDensity = {}
local j = 0
for i=0,255 do
	testDensity[i] = math.floor(math.random()*3)
	
end

local number1 = 0
local number2 = 0
local number3 = 0
local number4 = 0
--convert a passed number to base 4
local function convToFour(num)
	number4 = math.floor(num)%4
	number3 = (math.floor(num/4))%4
	number2 = (math.floor(num/16))%4
	number1 = (math.floor(num/64))%4
end
--convert a passed number to base 10 from base 4, numbers must be in range of 0-3 for all places
local function convToTen(num)
	local bTenNum1 = math.floor(num/1000)
	local bTenNum2 = math.floor((num-(bTenNum1*1000))/100)
	local bTenNum3 = math.floor((num-bTenNum1*1000-bTenNum2*100)/10)
	local bTenNum4 = math.floor(num-bTenNum1*1000-bTenNum2*100-bTenNum3*10)
	local bTenNum11 = bTenNum1*4^3
	local bTenNum22 = bTenNum2*4^2
	local bTenNum33 = bTenNum3*4^1
	local bTenNum44 = bTenNum4*4^0
	return(math.floor(bTenNum11+bTenNum22+bTenNum33+bTenNum44))
end

local str = ""
for i=0,63 do
local compressedNum = 0 
--turns 4 table values into a fake base 4 number that the convToTen can work with
compressedNum = testDensity[i*4] * 1000
compressedNum = compressedNum + testDensity[i*4+1] * 100
compressedNum = compressedNum + testDensity[i*4+2] * 10
compressedNum = compressedNum + testDensity[i*4+3]
--then uses that number to write a character assosiated with it
str = str..string.char(convToTen(compressedNum))
end

print(str)
print(string.len(str))
local file = io.open("testBinaryFile", "w")
file:write(str)
io.close(file)

str = ""
--tests to make sure it can write and read correctly
local file = io.open("testBinaryFile", "r")
str = file:read("*all")
io.close(file)

local recompiledString = ""
for i = 1,64 do

	convToFour(string.byte(string.sub(str,i,i)))
	recompiledString = recompiledString..tostring(number1)
	recompiledString = recompiledString..tostring(number2)
	recompiledString = recompiledString..tostring(number3)
	recompiledString = recompiledString..tostring(number4)
end

print(testDensity[0]..table.concat(testDensity))
print(recompiledString)

[GreaseMonkey 7]

First things first, I'll clean up what you have:

--create fake testing data
local testDensity = {}
local j = 0
for i=1,255 do
	testDensity[i] = j
	if j > 2 then
		j = 0
	else
		j = j + 1
	end
end

--opens test binary file
local file = io.open("testBinaryFile", "wb")
for i=1, 255 do
	local d = testDensity[i]
	file:write(string.char(d))
end
file:close()

I'm not sure if io.close(file) is properly supported - it's a bit cleaner to just use file:close().

Does that code work? I mean you still have one byte per value instead of 2 bits per value, but does it write data to the file now?

In order to pack several values into a single byte, you'd probably want to shift left by 2*n or multiply by 4^n. Shifts are only available in Lua 5.3, but they're a cleaner option for this kind of thing. In order to read the data back you'll want to use a binary-AND (the & operator) and a right shift, and for that you need Lua 5.3, OR the 5.2 bit32 library (it's cleaner to use 5.3 to be blunt).

So basically, you have to read and write 4 2bit values at a time.

[ATastyPeanut 0]

First off thank you for the reply, the string.char() function is much cleaner than what I had before.  Unfortunately, the file is still blank when I open it to see if it saved correctly.  As for the Lua version, I am using 5.3.  

 

Does the write() function only accept strings so that when it gets a non-string value it errors out and doesn't write anything?  I also tried using io.write(d) after setting the io.output(file), but the file is still empty.

 

Would never have guessed writing binary data would be this difficult :/ thanks for the help.  Would have figured I could just write a bunch of text 0's and 1's then call some function to turn it into binary.

[GreaseMonkey 7]

Are you checking the files in-game or outside of it? You should be doing the former and not the latter.

Do you have enough space on your disk to actually do this?

You may have success if you do file:flush() before doing file:close(), but if that's what it takes to make it actually appear on disk, then that is probably a bug.

[ATastyPeanut 0]

Turns out your right, it's indeed a bug with the modpack I'm using.  After trying the same code you suggested earlier but in a different pack it worked exactly as one would expect it to, revealing a bunch of nonsense characters when edited in OC.  Since saving binary data seems to not be an option I'm going to instead be using the string.byte() and string.char() functions.

 

My plan is to write(string.char(#)) where the # is value i get using string.byte() on a value from 0-255.  This decimal value I determine by doing a conversion from base 4 to base 10.  Essentially writing the same binary data. But using the method of writing a string with the  same value instead of the binary itself.  I will post the converter I come up with when done.

 

Thank you for the help you've provided with this issue GreaseMonkey, don't know what I would have done without  ya.