Lua minifier

[nmarshall23 0]

Description:

 I would like to tool that would help prepare code for use on a Microcontroller. Something like a Lua code minifier, and to rewrite my api calls to components, and any other library constants. Like the sides and redstone color api.

 

 

[Molinko 43]

the closest thing to what you want that is available is miniOS or even skex bios. both are smaller lightweight. skex is particularly great for microcontrollers

[mpmxyz 27]

I've just written a lua parser in lua for that purpose.

The first part - to cut the code into 'tokens' like numbers, strings etc. - is lightweight enough to be executed from within OpenComputers.

The second part - to generate a tree representing the structure of the code - currently needs > 50 MiB during initialization.

(It should be possible to move the resource intensive parts out of OpenComputers via a file with a precomputed table.)

Obviously this is only reading the code.

But I already have ideas for the next step:

-remove comments and unnecessary whitespace (only needs the first part)

-rename variables (i.e. "thisIsMyLongVariable = thisIsMyLongFunction" -> "a = b0", can be improved by knowing variable visibility)

-replace "local a=1 local b=2" by "local a,b=1,2" (needs a more detailed analysis of the variables referenced behind the '=')

I'm going to publish the program in the forum when it's ready. (It might take some time though.)

[natedogith1 4]

Why does generating a tree need so much memory? Is it because you're making a tree for a large block of code or something?

[mpmxyz 27]

I'm using a LR(1) parser and I haven't separated building its parsing table and the actual parsing yet. (But that's easy since this table stays the same.)
I implemented it that way because serializing this table was less of a priority while testing. (It only takes 3 seconds to build with a lot of profiling included; fast enough for testing without the need to have a separate 'build step'.)
The parsing table itself consists of around 200 tables with 50k entries. (I'm considering adding code to make it a LALR parser which should cut down on the number of states.)
That's why. (In short: It's still work in progress and I wanted to get it running first. Now would be the time for optimization.)

EDIT: I've just implemented LALR and parsing table serialization.
The table now has 5000 entries and needs only 48 KiB disk space. (90% decrease in size)
Memory measurements using pure lua peaked at 1.5 MiB for a small file and 2.9 MiB for a 50 KiB file. So it's going in the right direction. (I still haven't done any optimization yet.)
There are several things that take their memory:
-a self made regex library to cut code into tokens
-a lot of memoization (memory -> speed tradeoff); I have to check here if I forgot to mark some tables as 'weak'.
-all the strings and tables that are the output of parsing need their space (approx. 20k strings with a few thousand tables for the 50 KiB file)

 

EDIT2: Writing a code compressor using the parsing library was quite easy. Just by removing comments and whitespace and by modifying local variables it's able to achieve a compression ration of about 2:1. (It's not even touching global variables or table indexing!)

I'm going to refine it a bit before publishing. That should be done within the next days.

Edited February 18, 2015 by mpmxyz