Operating Systems under OpenComputers' Lua architecture

[tim4242 1]

Operating Systems

A small bloody giant guide through the weired and awesome world of operating systems.

 

EDIT: This is pretty dead, for explanation see the last post.

 

FIrst off I am not a professional LUA programmer or professional operating system developer, so there will propably be some mistakes in this tutorial (series) and I always welcome suggestions and fixes. I am also not a native english speeker so fixes for my spelling or... unfortunate sentences are also welcome, but now to the intro!



There are a few things you should propably know before starting the endevour for your very own OS and also about this guide:

• This post will be updated frequently so you should check it out once in a while .
• Operating Systems are complicated, so you should propably have some experience with LUA and OpenOS.
• An Operating System is not one singular program, it's a collection of very different programs so you should know about a lot of different things from graphics to networking depending on the comlexity of the system you're planning.
• This will not be a step by step guide because OS' are so complicated so I will cover some aspects only very breefly or skip over them completely, just saying that it has to be done. If someone wants to make a more in depth guide on those aspects (or on any part of this in general) you are free to do so and I will maybe add it to the post it references (you will get credited of cource). If there is overwhelming demand for something I may do a guide myself.
• I am not perfect and there will be errors so if you know how to do something better show me and I will change my post and credit you.
• If you have anything to say about the topic covered in a specific post that isn't mentioned just say it and if it is imporant it will be added (and you'll be credited).
• I will write an example OS while making this guide I will use its code for the implemenation part of the segments. I will release a update to it after every segment is written, everyone can contribute!

This guide will be segmented into a few parts, how many I don't know yet. Each part is segmented into three:

• Overview - What will be covered, this will be one paragraph for the most part.
• Theory - Here the theoretical bases of the content will be discussed, this will be quite long.
• Implementation - This will discuss the content by looking at code, this will be bloody long.

Also a few things about how I will structure my code for the most part:

When I start a new segment in code I will mark it with something like this:

--[[SHORT_NAME]]

When it comes to functions I have a definition comment above them or where ever they are defined, this has a very specific format:

• A short description of what the function is there for.
• The arguments in this format:

  @arg name_of_the_arg (argument_type) : Short_description.

• The return value in this format:

  @return (return_type / other_return_type) Short_description.

On the topic of function, I name my functions in camel case (firstWord) and I start my argument names with an underscore.

Now lets get started.

Part O: The plan

Overview:

We take a look at a way to structure your OS.

This part is here for the people who don't have a clear plan for how to make their personal OS, if you do have a clear plan you can just skip it. But this could still have some value for this group, just to see how precise their plan really is and what should be in such a plan in general.

Theory:

One way to structure your OS, which has been used quite often workes how follows:

The basic functionality of the OS containing things like multi process support, file systems and basic networking are all in one unit often called the kernel.

There are two classic ways to make a kernel:

• Make all system services part of the kernel (centralised). This type is normally faster.
• Make the kernel small with a lot of user services registered into the kernel (decentralized). This type is normally more reliable.

Also you should think about what the OS should provide, common things are:

• File system(s)
• Networking
• GUIs

Now you know the basic structure of your OS you should think about a few things:

• A central idea. (Common things are things like: Useability or Extendability (and making money))
• What type of user is it aimed at.
• What functionality should it posess.
• Should it follow standards like the OOSS (OpenOS Standard) or something *nix like.
• How open it should be.
• How big and complicated it should be.
• How fast you want it to be.

When you got those things figured out, you can start thinking about the details.

Implementation / How I like to do things:

I personally start with the hardest part, at least for me: Naming.

I really struggle when I need to name things so I just named the example OS: "TUTOS", which stands for Tutorial Operating System, pretty straight forward.

Then I think about at whom it should be aimed at, who my target audience is and I decided my target audience are people following this guide so I decided that it should show a lot of ways to make an OS at a time. For that reason I made the decision to make it very open and not follow any standard completely but make a hybrid between *nix style, OOSS style and my own style. It also should be part of a superset of OpenOS to show how to make it easy to migrate and how to make compatibility layers for programs that can run on OpenOS and on TUTOS.

Then I got to the stage to think about what it should do, it should again show the most stuff I can possibly show so I decided to make it very flexible so it can run with a graphical and a text UI, when I get around to making one. There are also a bunch of other things I want to put into this OS but those will be discussed in detail when I make them.

Another thing TUTOS should be able to do is update itself with new software, may it be an actual TUTOS update or new programs or libraries. For that feature it has to have its own package manager (it will have its own guide).

The next point was easy it is for the community so it's going to be open, very open.

As a consequence of the decision to show as much as possible it has to be quite complicated, thats just a fact.

For the next part, I want it to be fast of course, but tahts not a priority.

So that step done, lets get to programming.

Part I: The initialization

Overview:

This will cover how your OS will be started and initialized it will also cover the basic stuff needed to get your system up and running.

Theory:

When a OpenComputers computer boots up and works it first runs whatever code is on the EEPROM.

By default (on the Lua BIOS EEPROM) it runs the file called "init.lua" which is at the root of a file system.

So now you have two ways to start your OS: Either you start using the standard BIOS or you write your own, if the first is the case you can skip the next paragraph.

If you decide to write your own BIOS you will find that there isn't a lot there when the BIOS starts, you just have the standard LUA functions and libraries and of course the "driver" APIs: Computer and Component. EEPROMS also have severe space limitations you'll need to worry about.

If you choose this you'll need to write most of the stuff you need your self, like file loading.

But using this method you have a little more freedom when booting your OS, like changing the startup file or changing what is available when your kernal starts, this could also be used to add more security.

That you know what file gets run when the OS starts you can start writing your OS.

When the OS gets started you still have almost nothing, but depending on the BIOS you may have a bit of basic stuff like very basic file loading.

But in general you have to build up your basic operations like file system operations so you can get the real system started, at this point you should propably also set up system information and more advanced output. But most importantly you have to load your kernel and, in turn the needed libraries. By this I mean the internal module loading system, events and other system services.

After the basic services are loaded you can start thinking about the head of your OS, so the shell or GUI, this will be a quite complicated little set of programs that the user/s will interact with, it will need all the system services loaded before to work. It will also depend on a collection of programs that the user can interact with, common ones are things like "cd" for text based ones or some kind if file explorer for graphical UIs. These programs are one thing taht really seperates the good from the not so good OSs, depending on if they are usefull and plentyfull.

Another thing you'll have to think about is how similar it should be to OpenOS (which is the de facto standard) on two levels, the API level and the use level.

Being similar on the API level means that you expose the same functions even if they are very different in the background, being similar on the use level on the other hand means being controlled similarly (e.g. the command "cd" Changes Directory).

You would think about it to make it easier to migrate for programs and users, but more on this in a later part.

Now to going over specific parts of the initializing phase:

The basic functions:

These should be really basic and fast because they are only used in the very controlled enviroment of the initialization after that they shouldn't be used in favour of more advanced library functions. This could mean that these early functions are not controlled for r/w protection.

For that reason alone these should be deleted after the libararies are loaded or they are not needed anymore.

The system services:

These services provide all the interaction with the OS, they can vary greatly from OS to OS depending on the functionality it has, for example an OS that provides r/w protection could expose a library to find the needed level to read or change a file, or it could expose networking protocols, registered users, shutdown and errors... You get what I mean, there is a lot of stuff there.

These services may be VERY opaque depending on the OS and its use. (I personally rate them from OSX to Linux).

When I finish this part of my example OS I will give more information to this.

Implementation:

(Ladys and gentlemen, open your text editors or IDEs and go to the start line.

Start programming in 3...

2...

1...

CODING!!!)

(Maybe that was a bit too much)

Lets say you want to make a custom EEPROM like I have for my example OS, you want to run from a different file (not "/init.lua" but "/boot/boot.lua") and you also want to supply a function (a loadfile implementation ( _G.loadfile(file_system_address, path_to_file) )) (Some people think that this is bad practise but I'm going to cover it anyway for completeness sake).

One good resource when writing your own EEPROM is looking at already existing ones, like the standard one or any other ones you know. Another resource for finding out what is available to you is the wiki entry about writing OSs. But now to the code.

First I set up my variable/s, for this one it is only one:

local bootPath = "/boot/boot.lua"

As you can propably see it is the path to the boot file.

Next I do some miscellaneous setup:

do  local screen = component.list("screen")()  local gpu = component.list("gpu")()  if gpu and screen then component.invoke(gpu, "bind", screen) endend

First I bind a monitor so I can print error messages.

local eeprom = component.list("eeprom")() --Getting the address of the currently installed EEPROMlocal function getData() return component.invoke(eeprom, "getData") end local function setData(_data) return component.invoke(eeprom, "setData", _data) end

Then I define some helper functions for getting/setting the data (boot address) for the EEPROM.

Now to the juicy stuff: The API definition.

In this example there is only one function in the API: _G.loadfile(file_system_address, path_to_file)

but for your project there can be a lot more.

I have ripped this loadfile implementation from the standard bios, with just a few differences:

--[[  Low level load file implementation.  @arg _addr (string) : The address of the file system to use.  @arg _path (string) : Path to the file to load.  @return (function / nil, string) The loaded chunk or nil and the error.]]_G.loadfile = function(_addr, _path)  local fil, err = component.invoke(_addr, "open", _path) --Open the file  if not fil then return nil, err end --If an error occured: Return nil and the error  local buf = "" --The string buffer to write the contents of the file to  repeat --Read until nothing is left to read    local dat, err = component.invoke(_addr, "read", fil, math.huge) --Read data from file    if not dat and err then return nil, err end --If an error occured: Return nil and the error    buf = buf..(dat or "") --Write the read data or an empty string to the buffer  until not dat  component.invoke(_addr, "close", fil) --Close the file  local chunk, err = load(buf, "=".._path) --Load the chunk from the file  if not chunk then return nil, err end --If an error occured: Return nil and the error  return chunk, nil --Return The loaded chunk (and nil for the error)end

There isn't anything interesting there just some stolen code .

Now to the second juicy part: Loading the boot file and running it.

This, again is mostly ripped from the standard BIOS and all the differences are commented:

local boot, err --Helper variable declarationlocal data = getData() --Calling my helper functionif data and type(data) == "string" and #data > 0 then --A little more safety when detecting if the current address is valid                                                        boot, err = loadfile(data, bootPath) --If it is valid load the file from the given address or set the errorendif not boot then --If there is no valid address in the EEPROM memory  setData("") --Emptying the EEPROM memory  for addr in component.list("filesystem") do --Iterate over all available file systems    boot, err = loadfile(addr, bootPath) --Try to load the file from a file system    if boot then --If it worked      setData(addr) --Set the EEPROM memory to the address of the current file system      break         --And then stop iterating    end  endendif not boot or err then --If there is an error  error("no bootable medium found" .. (err and (": " .. tostring(err)) or ""), 0) --Print itendcomputer.beep(1000, 0.2) --Give a signal that it workedboot() --Run the loaded chunk

The actual code I put on the EEPROM is not commented to keep its size down because EEPROMs have severe size limitations (4kb / 4000 characters).

To make it posible to use the OS without having to have build a OpenOS computer first I also included a standard init.lua file, it can be found here. It's just a slightly modified version of the BIOS.

Sooo, we are finally starting the real OS in boot.lua, first I do a little cleanup because I like a clean start:

--I really don't like this function just laying around, therefore I have to destroy it  _G.boot_invoke = nil --(Literally the first line I wrote)

Then I set up some OS data: Name and version:

--[[OS_DATA SETUP]]  _G.OS_DATA = {}  _G.OS_DATA.NAME = "TUTOS" --Name / identification  _G.OS_DATA.VERSION = "0.0.3-\"KOMMENTARE\"" --Printable version  _G.OS_DATA.FULL_WRITE = _G.OS_DATA.NAME.." ".._G.OS_DATA.VERSION --Full writable name / version string  _G.OS_DATA.VERSION_COMP = 002 --Computer comparable version

Next I set up basic output that doesn't stop the program like error, even if that is an exellent debugging tool before you have real IO set up.

This mostly means setting up a screen and a GPU and then writing a function that wraps it all into a nice package.

The screen is easy, just pick the first with a keyboard:

--Setting up the main screen  local screen = component.list("screen", true)() --One screen if no screen has a keyboard  for addr in component.list("screen", true) do --Iterate over all available screens    if #component.invoke(addr, "getKeyboards") > 0 then --If it has at least one keyboard...      screen = addr --...Set it as main screen      break    end  end

GPU is quite similar, but now you are searching for the best one:

local gpu = component.list("gpu", true)() --Get the first gpu  for addr in component.list("gpu") do    if component.invoke(addr, "maxResolution") > component.invoke(gpu, "maxResolution") then --If the gpu has a bigger resolution...      gpu = addr --...Set it as main gpu    end  end

Now to binding the GPU, just standard code:

local w, h --Convenient variables  if gpu and screen then --If the system has a gpu and a screen    component.invoke(gpu, "bind", screen) --Bind the gpu to the screen    w, h = component.invoke(gpu, "maxResolution") --Getting the maximal resolution of the current gpu    component.invoke(gpu, "setResolution", w, h) --Maximise the resolution    component.invoke(gpu, "setBackground", 0x000000) --Set background color to black    component.invoke(gpu, "setForeground", 0xFFFFFF) --Set foreground (text) color to white    component.invoke(gpu, "fill", 1, 1, w, h, " ") --Clear the screen / gpu buffer  end

The function is a little more complex, I went for a slightly more advanced output, that supports general types of output like errors, but I think the code is pretty self explenatory:

local y = 1 --Variable to keep track of the current offset  --[[    Convenience function to log boot progress.    @arg _msg : The message to display    @arg _type : The type of the message. Can be one of the following                 - "nothing": Prints no prefix and in white                 - "info" (standard): Sets the prefix to "[INFO]" and the text to white                 - "error": Sets the prefix to "[ERROR]" and the text to read                 - "success": Sets the prefix to "[OK]" and the text to green  ]]  function status(_msg, _type)    if gpu and screen then      if _type == "error" then --When the type is error        _msg = "[ERROR] "..tostring(_msg) --Set the prefix to "[ERROR]"        component.invoke(gpu, "setForeground", 0xFF0000) --Set the color to red      elseif _type == "success" then --When the type is sucess        _msg = "[OK] "..tostring(_msg) --Set the prefix to "[OK]"        component.invoke(gpu, "setForeground", 0x00FF00) --Set the color to green      elseif _type == "nothing" then --When the type is nothing        component.invoke(gpu, "setForeground", 0xFFFFFF) --Set the color to white      else --Otherwise ("info" or no value)        _msg = "[INFO] "..tostring(_msg) --Set the prefix to "[INFO]"        component.invoke(gpu, "setForeground", 0xFFFFFF) --Set the color to white      end        component.invoke(gpu, "set", 1, y, tostring(_msg)) --Write the text on screen        component.invoke(gpu, "setForeground", 0xFFFFFF) --Reset text color      if y == h then --If the offset has reached the screen height        component.invoke(gpu, "copy", 1, 2, w, h - 1, 0, -1) --Move the screen content up by one        component.invoke(gpu, "fill", 1, h, w, 1, " ") --Clear last line      else        y = y + 1 --Add one to the offset      end    end  end

Once you have basic output you can start actually loading your kernel, in this case I use a function for early file execution, it comes with logging so I don't have to think about it when actually executing files.

One note worthy thing is that this function makes use of the earlier defined loadfile implementation:

local bootAddr = component.invoke(component.list("eeprom")(), "getData") --Convenience variable containing the address of the boot filesystem  --[[    Low level dofile implementation (executes a file).    Automatically outputs log messages.    @arg _path : The path to the file to execute, has to be on the boot file system.    @return The loaded chunk of nil with the error  ]]  function dofile(_path)    if not component.invoke(bootAddr, "exists", _path) then --If the file doesn't exist      status("File ".._path.." doesn't exist", "error") --Output that the file doesn't exist      return nil, "File ".._path.." not found" --Return nil and the error    end    local data, err = loadfile(bootAddr, _path) --Load the file or get errors in the file    if data == nil then --If an error occured      status(err, "error") --Output the error      return nil, err --Return nil and the error    end    status("Loaded file: ".._path, "success") --Output taht the file was loaded    local stat, err = pcall(data) --Execute the loaded chunk    if not stat then --If an error occured      status(err, "error") --Output the error      return nil, err --Return nil and the error    end    status("Executed file: ".._path, "success") --Output that the execution was successful    return err, nil --Return the chunks return value (and nil for the error)  end

Now that the initial setup is done I have time to give the user some output, just to tell him that this actually works, so I output the name and version of the OS he is running:

--Output the name and version of the OS boxed in  status("+"..string.rep("-", OS_DATA.FULL_WRITE:len()).."+", "nothing")  status("|"..OS_DATA.FULL_WRITE.."|", "nothing")  status("+"..string.rep("-", OS_DATA.FULL_WRITE:len()).."+", "nothing")  status("", "nothing")

I also tell the user what I'm doing periodically, the first one is this one:

status("Loading system services") --Output that the system is now loading the services

After that we can finally start loading services, because this OS very community centric and open I decided to take a bit of a unique approach, I make the service implementations configurable.

To get that to work I load the first part of my kernel, the service loader. But that will be covered in the next chapter so I'm going to skip the service loading for the moment .

So after that is done and I have acess to the services the boot process is almost finished, one last thing is setting up the enviroment.

I do this in a different file, the badly named "env_setup.lua" by this code:

local env_func, err = loadfile(bootAddr, "/boot/kernel/env_setup.lua") --Load env_setup.lua (needs better name)  if not env_func then error("Error loading env_setup.lua: "..tostring(err)) end --If an error occured: Output it  status("Loaded env_setup.lua", "success") --Output that the file was loaded successful  local stat, env_set = pcall(env_func) --Set up the enviroment by executing env_setup.lua (needs better name)  if not stat then error("Error setting up enviroment: "..env_set) end --If an error occured during execution: Output it

Pretty much the only note worthy thing about this is that it doesn't use the dofile function defined earlier, this is because this deletes the native libraries from the global namespace which are used by the previously defined functions.

(The file can be found here).

The enviroment setup is split up into two parts:

Cleaning up:

This removes any native libraries which are replaced with OS services from the global namespace.

Setting up:

Here I set up the functions that are available in _G:

--[[  Global function for reading text files from the file system.  @arg _path (string) : The path to the file.  @return (string / nil, string) The loaded string or nil and the error.]]_G.readfile = function(_path)--[[  Global function to load a chunk from a file in the file system.  @arg _path (string) : The path to load from.  @return (function / nil, string) The loaded chunk or nil and the error.]]_G.loadfile = function(_path)--[[  Global function to run a file in the file system.  @arg _path (string) : Path to the file to execute.  @return (value / nil, string) : The return value of the given file or nil and the error.]]_G.dofile = function(_path)--[[  Global function to load libraries or modules from the file system.  @arg _name (string) : The module / library name / identification.  @return (table / nil, error) The library / module or nil and the error.]]_G.require = function(_name)

Now that the enviroment is clean the last thing to to is start the shell.

(I haven't finished enough of my example OS to really write more here for now, will follow shortly).

Credits: tim4242

Edited June 4, 2016 by tim4242

[tim4242 1]

The guide is currently on hold 'cause I'm really quite ill at the moment.

I know there isn't a lot here yet but I can't do a whole lot about it right now, sorry .

 

I am very much better now an currently in the process of getting TUTOS to a presentable state, so commenting and so on.

I will host it on OpenPrograms on Github. At the moment I'm in the MiscPrograms repo but if there is demand (someone who wants to contribute in one way or another) I will apply for my own repo. (For the newest code see https://github.com/tim4242/MiscPrograms/tree/master/tim4242/TUTOS)

The code is massively wip and commented, for a comment overload look at "/boot/kernel/services.lua".

 

After I finish commenting I will continue this guide.

 

I'll try to get anything done as soon as possible but it takes a while to write an OS completely from scratch, even more so because I decided not to look at OpenOSs source

for anything and not reuse any of their code (except for the BIOS).

Edited January 4, 2016 by tim4242

[jhagrid77 4]

That sucks Tim, I hope you get better, I know it's not much, but it truly helped me understand that there is a lot of work that needs to be done in order to make an OS and that it isn't that simple at all.

[tim4242 1]

Since there doesn't seem to be a lot of interest in this I will stop working on this, at least for a while. I'll get going with some other project, maybe you'll like it more.

[ChillBacon 0]

I think this project is really interesting and I think it's quite sad that more people don't enjoy it as much as I do  I hope that the project will be continued in the future

[tim4242 1]

Well I'm happy some one actually read it, because I can't imagine it to be an interesting read and also I was getting to the limits of what the forum can do which you can see from the formatting, this means it was really frustrating to write the last part of it.

If I ever pick this up again I'll propably create a little website for it, this way I won't be frustrated with the formatting because before this is finished it's going to be much bigger then now, and by then it'll either be THE guide on operating systems in OC or no one will read it.

But enough with the ranting, I'll only continue this if there is reasonable demand for it and right now there isn't.

[tim4242 1]

To see if there is demand I added a poll, if there are ten or more votes in total on the 1 of june I'll create a small webpage and put it up for download, this will of course take a while but it will be on the way, if the counter doesn't reach 10 I'll declare this project dead.

 

So if you want this tutorial continued vote now!

 

EDIT: The poll is over and only 4 people voted so this is dead now on hiatus for a potentially infinite amount of time.

[SashaGelert 0]

I... I am curious as to why someone would put such effort into a custom OS, just to abandon the project because the community as a whole hasn't indulged in it!  It is still a very worthwhile project and would help many others trying to do this for their first time, especially as when one looks over the Wiki, most every page could do with a proper dedicated tutorial.

[tim4242 1]

My reasoning is a simple calculation, if I develop this OS and tutorial I help four people, if I develop another project of mine I help around 100 people, I only have a limited amount of time and if I did both, both would suffer.

But in all honesty I was toying to revive this on "the backbourner" and work on it from time to time when I need a break from 231 and 219a, well we'll see.

I also see the same problems with the wiki as you do, before I decided to do this I was thinking about creating another wiki that allowed a few more advanced things like submitting tutorials and semi-live communication (even if tis would take a much larger community). I actually build the backend from the remains of 197c but I'm a lousy frontend designer so I decided to do this.

Everything depends on what the community wants because right now 231 just trumps everything else I do.