More authority over the 'harware'. (OC Developer Request)

[Varscott11 0]

This is more of a request of the Open Computers developers and may be a bit far fetched, but hear me out. At the very beginning when a BIOS is run, it is running in a sandboxed environment. This makes perfect sense as the Lua interpreter will blindly execute instructions on a real CPU on a real computer which can lead to major security risks if not protected properly. This is done by 'machine.lua' but this is where a problem starts to arise. Theoretically there is enough to build a fully functioning operating system, but it is actually quite limited and this is kind of strange. In real operating systems, the kernel has complete undisputed control over the entire computer. It stands to reason that this is where the sandbox security should be implemented and would allow for so many possibilities that would make the mod shine. Here are some things to be considered:

SEMI-PREEMPTIVE MULTITHREADING:
I am aware it is not possible to yield from a Lua hook but you can force a coroutine to terminate. This is demonstrated in machine.lua. This combined with some clever mechanisms can result in an almost hybrid between preemptive and cooperative multithreading system with lightning fast interrupt (event) dispatching.

MEMORY MANAGEMENT:
A computer can only use a relatively small amount of memory by default. This makes perfect sense. I mean if 30 people on a server all have a mega server farm and each in game computer can allocate 1GB, you will run out of memory on the host computer very fast. The problem is that memory management is very limited without collectgarbage. With it you can implement some sort of memory tracing allowing you to terminate a coroutine (via preempt checks for example) if it exceeds allowed memory. This actually makes the system more secure than it currently is because you can stop a malicious program from infinitely allocating memory until the computer runs out crashing it.

VIRTUAL MEMORY SWAP FILE:
With the unrestricted use of the debug library it can be possible to serialize upvalues and globals and write them to a disk when they are guaranteed not to be used in order to save precious memory. 

KERNEL SECURITY:
Having complete control is actually more of a benefit to everyone (granted the kernel code is trusted). The kernel can implement security on it's level that is more relevant to it's interests as well as prevent any other code from exploiting the actual mod or Lua interpreter. After all, code has to be chosen to be run by the kernel and can be sanctioned at will. A simple file system protection layer can ensure a program cannot write or read from the kernel on the disk completely user spacing any program.

Of course I'm not saying remove all security measures from the mod. That's just crazy. However simply adding an option to the config file to allow something like 'disableSandbox' that would grant the entire Lua Standard Library would be a godsend. If anything it would be completely sufficient to wrap all these functions that prevent the kernel from doing anything that can directly harm the host or going any lower level. Plus it is up to the user to truly decide if something is trusted. If the user wants to install a kernel to their host server with complete access, they would have the freedom to do so if they do so understand the risk. The config file is full of these sorts of trust dependent options already. So long as a person cannot access the save and .minecraft directories from a Lua state, that should be mostly it. For those who are happy with what there is they will have no change as the normal OC sandbox should be enabled by default. To top it all off, I have managed to implement all of the above in an unsandboxed Lua environment, so it would be so amazing if it can be ported.

I don't know what goes into developing OC but in my mind it shouldn't take more that skipping the sandbox init if the config says to do so. These are my thoughts and they may just very well be dreams but if there are any OC developers reading this or anyone really, tell me what you think of this idea.

[payonel 21]

I appreciate the interest and enthusiasm. I disagree with a few of the points you've made if you would allow me to explain.

 

16 hours ago, Varscott11 said:

This combined with some clever mechanisms can result in an almost hybrid between preemptive and cooperative multithreading system with lightning fast interrupt (event) dispatching.

"almost" as in it isn't. Killing a thread pre-emptively is the only thing you can do in this case and that is not multithread nor cooperative. You can't resume coroutines from the same Lua state either. It isn't lightening fast, debug hooks are actually extremely costly. Also, we cannot persist debug hooks, thus when a machine is unloaded and reloaded in chunks, any state and special hook magic you might make would be lost.

 

16 hours ago, Varscott11 said:

The problem is that memory management is very limited without collectgarbage

We collect garbage when you've yielded 10 times. Also, lua state will collect garbage dynamically

16 hours ago, Varscott11 said:

stop a malicious program from infinitely allocating memory until the computer runs out crashing it

You can already do that, see pcall. Or, run untrusted code in a sandboxed coroutine. The coroutine will die if they OOM.

16 hours ago, Varscott11 said:

VIRTUAL MEMORY SWAP FILE:

We looked into this, unfortunately you cannot arbitrarily serialize all value types in lua.

 

Kernel Security:

There is nothing MORE secure about unboxing any part of the sandbox. In fact, you could just run the machine.lua code, again, inside minecraft, and run an OS inside that sandbox. Sandbox inception! You wouldn't have debug hook, but hooks do not increase security, we only use them for stopping long running processes. And you don't need to do that, we do it for you.

 

16 hours ago, Varscott11 said:

skipping the sandbox init if the config says to do so

We are not in favor of making it this easy for uninformed server owners to expose their private server environments to users. Users have brought this up before and we (developers) are in strong agreement about this.

 

 

Hopefully you're still reading this far, and not just mad at me for shooting down your sincere request, inspired by your obvious passion. I wanted to encourage you to ignore the constraints of the sandbox, and rather, look at what you really can do with developing a custom kernel. There is a lot more power than you might give credit to the system as it is. I whole heartedly agree that the virtual world in OC can be restricting, and at times extremely frustrating. I sympathize. Look at OpenOS. I have spent thousands of hours of my life developing OpenOS. WIthout a question, the single hardest part has been memory optimization and refinement. I'm a C++ developer (irl); I live in a world where I have complete and precise control over memory management (allocation, monitoring, profiling...). The way Lua allocates memory in chunks for its stack makes knowing the true cost of operations and new code actually very difficult to measure. I built an OC emulator primarily to provide a way for me to profile and measure memory usage. But none of this would have been "easier" outside the sandbox.

 

Edited December 6, 2018 by payonel
type, can -> cannot

[Varscott11 0]

I'm not mad at all and in fact I do actually agree with many of the points you have made. Some things I didn't know for example was the OC collects garbage at 10 instruction intervals. This is HUGE for me to know. Since I don't know a lot about the Minecraft Java environment I didn't know either that debug hooks aren't persisted. The fact that all types can be arbitrarily serialized now that I see that, can pose security problems not just for OC but for a kernel as well. The only thing I am not sure about is the OC killing a long running process for you. Does it kill the coroutine that is long running or just stop the Lua state. I can search for this myself in machine.lua so don't worry about that, but I'm pretty sure it doesn't notify the kernel via a signal so there is no way to know what happened and to do any cleanup if necessary. I will gladly do work arounds if I can to ensure security and stability (My kernel is centered around strong security). I just need to understand how the mod operates to do such work arounds and this gives me an insight. All in all though I respect your opinions and I'm now a bit reinsured that though it is limited in many ways, there are options. I recently started to work in the mod as a way to get away from the super low-level work of developing a kernel irl with C and x86 ASM and it is a quite difficult switch from complete control to less control. Thanks for your review!

[payonel 21]

btw, I made a typo (which I'll correct inline), I meant to say that (in regards to memory swapping) we cannot serialize all data types. So for a kernel to really swap memory, it would only be able to swap specific segments of data ... and could not force arbitrary user code to fully swap. It is sadly outside what Lua can do.

 

We kill the the current coroutine that is long running. So you can catch it yourself, in your kernel, without the kernel crashing. This isn't a detail of machine.lua, but rather, how lua hooks work.

 

And yes, I empathize with the challenge of that switch. Strangely I found have found that I enjoy the two contrasting worlds (lua <--> c++).

[Varscott11 0]

I did think that was strange. I though it meant that you can and loading it back could be dangerous. For the purpose of a virtual memory manager in a game, It really doesn't need to fully swap code segments. That is a bit overkill. After all most memory usage comes from data (usually) such as tables and what not anyway. I wouldn't really say it is not possible to swap code either. You cannot swap the current state of a coroutine but you can swap static chunks via string.dump and load.