Jump to content
  • Sky
  • Blueberry
  • Slate
  • Blackcurrant
  • Watermelon
  • Strawberry
  • Orange
  • Banana
  • Apple
  • Emerald
  • Chocolate
  • Charcoal
Solra Bizna

Real CPU for OpenComputers. Interested?

Recommended Posts

Edit: The project is go. My Patreon is up, and scurvy is a thing of the past. Check there for status updates.

Edit 2: The project is dead. My problems got worse than I thought possible. Thanks for the support while it lasted; you literally saved me from scurvy. The complete (incomplete) source is here.

This is going to be one heck of a first post. I apologize if it violates the TOS; I wasn't able to get the TOS to open, so I assumed it was a typical one. If I am in violation, just don't approve the post and I'll disappear back into the woodwork.

tl;dr = Want make mod. Give OpenComputers a real CPU. Real languages. Need money. (Not much!)

I'll start at the most interesting part.

Ever since I stumbled on the "Custom Architectures" page on the OpenComputers wiki, I've had the idea of implementing a real CPU architecture module bounding around in my head. Interrupts, memory protection, the works. This would allow running other languages (C, C++, assembly...) on OpenComputers computers.

There were other things I needed to do, but the idea wouldn't leave me alone. I kept coming up with (and finding solutions to) little problems with spare brainpower, until finally I thought something like the following:

 

Such a project would take time away from important things. Not only would you need to write (or find) an emulator for a real ISA, you'd have to write an entire operating system from scratch, which fits a useful computer system in only a few hundreds of kilobytes of RAM. You'd also have to keep it updated whenever OpenComputers or Minecraft updates, provide documentation, give support... You're capable of doing this. You'd love to do this. But the only way you'll have the time to do this right is if you're being paid to do this.

I thought that had shelved the project, but later I remembered Patreon.

So now I'm writing a pitch and posting it here, to gauge the interest level for a project like this. If you find the concept interesting, please take the time to read this whole post. I'm not asking for money with no collateral, no qualifications (see "My background" below), and not even a prototype. If there's interest, if just a few people say they might become a patron to the tune of just a few dollars after they see a prototype, then I can begin to work on one.

The CPU:

The CPU would emulate a real ISA the way a current CPU runs Lua code. The emulator would be meticulously detailed and fully capable. Privilege levels, memory protection / mapping, DMA, interrupts, the works.

Thanks to the recent OpenComputers work on architecture modularity, it would interface with OpenComputers components using the same interfaces that Lua code uses now. Every device would be compatible without needing to be changed. Future devices will be compatible without being written with this in mind. Computers with Lua CPUs will be able to network with computers with this type of CPU, without having to care at all---imagine a high-traffic server running a tight bit of C code to process requests quickly, while its clients are all running simple Lua code.

Importantly: Lua CPUs and other CPUs can coexist in the same world.

The emulator would parcel out CPU cycles as Minecraft time passes, much like current CPUs do (on a coarse basis) with Lua code. It wouldn't be accurate on a cycle-by-cycle basis to any real, hardware CPU, but more complex instructions would take more "cycles"; this would make this emulated CPU roughly as resistant to Denial Of Service as a Lua CPU.

The CPU would still be much slower than a real CPU; slow enough that you could practically see the numbers crunching. On the other hand, that's also part of the appeal of OpenComputers, and this is definitely true of current Lua code (especially IO heavy code). As we all know from experience with this mod (and elsewhere if we're old enough), even a slow computer is usable with good software.

Possible architectures:

* 680x0: Very nice architecture. Reasonably compact. Unfortunately, not targeted by any current version of LLVM (though GCC can still target it).

* PowerPC: Super nice. My personal favorite ISA. Well-supported by LLVM and GCC. Hilariously bloated machine code---4 bytes per instruction, even simple ones. Loads of registers can make emulation slow.

* x86: I hate hate hate the x86 architecture, but I might be persuaded to tolerate it. Poses serious problems to emulation due to things like its ridiculous instruction encoding.

* MIPS: Not very familiar with it, but has wide adoption, supposedly doesn't suck, and GCC and LLVM both target it.

* SPARC: Branch delay slots are weird, but other than that a solid architecture.

* ARM: Not quite as nice as PowerPC, but better code density. Conditional execution bits are neat. Supported by GCC and LLVM. Probably best choice.

The OS:

Such a CPU would, of course, also require an operating system. Implementing the OS directly in Lua isn't an option, as Lua depends on OS services for things like memory management and IO.

The operating system would have an API based on a subset of POSIX. Its implementation is significantly simplified by not needing to implement a filesystem---the "hard disk" and "floppy disk" are components in their own right, implementing file-level access rather than being block devices, which means they'd be accessed the same way as any other component.

The simplest OS would be unprotected and straightforward, like the current OpenOS or Classic MacOS. It wouldn't make use of memory protection or virtual memory of any kind. It would be well suited for little personal computers or single-duty controllers.

Next simplest would be one that uses memory protection to achieve process separation. Virtual memory would also be an option. This would be more complex to implement, and use more emulated resources. It would be a necessity for secure multi-user systems.

Finally, a preemptively multitasked OS, like any modern UNIX, or post-NT Windows. This would possibly be the most complex part of the mod... though, I am sad to admit, it wouldn't be the first time I'd done something of that sort. (I'd definitely need to dust off the my old Tanenbaum tome...)

I envision all three being options in the final version of the mod, with players selecting between them based on their needs and resources. A 4MiB server behemoth (with a 12MiB RAID for swapping?) might run the multitasking OS, with its clients (typically <1MiB of RAM) running a simpler OS more suited for single-user applications. All three OSes would have more or less the same userspace APIs (subsetted as appropriate). A similar situation existed with Windows 98 and Windows NT, or DOS and Novell NetWare.

In addition, writing your own OSes would be possible, just as with the Lua CPUs. In fact, it was while researching for my own NetWare-like OS for the Lua CPU that I stumbled on the wiki article on "Custom Architectures"...

The languages:

* Assembly: Of course... but only the most crazy-hardcore people will actually use this.

* C: Little more than high-level assembly. Will definitely be able to run code compiled outside the game (with clang or GCC). Might make an LLVM-powered component (a "Compiler") that allows compilation of code by computers inside the game, calling an outside compiler (throttled?). Maybe even a C compiler that runs on the computers ingame? (I wrote part of a C99 compiler once, it's actually fairly easy.)

* C++: C...plus plus. Almost everyone using C for real projects has switched to C++, especially now that C++11 made it nice. Supported as for C, with the caveat that C++ is really really complicated (main C++11 standard is over 700 pages long!) so I am definitely never writing my own compiler for it.

* Python: It seems to require too much memory to run on a real CPU with so little memory. I admit, I'm not a fan of Python and I don't know a lot about it. If it can be made to run in the few hundred kilobytes available, nothing will stop it from running on the emulated computers. (Swap can help!)

* Lua: Of course! One of the reasons Lua appeals to me as a language is its tiny memory footprint. With a small penalty to memory usage, emulated CPUs could even run the same Lua programs that Lua CPUs do. (Bonus: There wouldn't need to be a 64-bit pointer hack anymore!)

The advantages:

* Control: This would give far, far more control over the amount of memory and other resources that are used by the computers. This is particularly important now that we have Microcontrollers and Drones; writing a simple EEPROM-resident program would be much easier with C than it currently is with Lua.

* Memory: An emulated CPU would not use any more resources than a Lua one, and C is even more comfortable in <1MiB than Lua is.

* Speed: For some data-heavy tasks, an emulated CPU will handily outperform a Lua one.

* Learning: Many people who get into either ComputerCraft or OpenComputers do it for the fun learning experience. Many people who "migrate" from ComputerCraft to OpenComputers do it because they're ready for something a little more hardcore. This provides a path to upgrade along to learn even more---things that are even more applicable outside Minecraft, as they will benefit from "real" languages and techniques.

The disadvantages:

* Speed: Running Lua natively will always be faster and more memory efficient than running Lua in an emulator. Faster than emulated C code? Depends on the task.

* Learning: C/C++ is not nearly as easy to learn as Lua.

(Yeah, both were also listed in advantages...)

My background:

I've been an invisible OpenComputers and ComputerCraft user for a very long time. I've played Minecraft since it was in Alpha. My interest grew when I found IndustrialCraft, and even more when I found ComputerCraft. OpenComputers, however, was something more than that. It was love at first sight.

I've been fascinated by low-level details of computers ever since I was a child. One of my earliest memories is of formatting floppies on my father's Atari ST. This was before I could read. (I enjoyed the sounds it made while it erased his precious documents...) When I was older, I crippled one of his Macs by editing the System file with ResEdit. One day, he gave me a copy of the 68020 Programmer's Manual, which I read cover to cover (despite not understanding much of it).

Back in high school, I competed with a few other students in RoboWar. It had a simple RPN-based virtual machine, which powered extremely resource-limited robots. You chose the robot's specifications (trade CPU speed for hit points? an energy shield? better weaponry?) and wrote the program for it. It had very little sensor information, and even less CPU time---faster robots had the luxury of a few hundred instructions per second. Advanced robots used interrupts, unrolled loops, jump tables, and other such techniques to destroy their opposition.

My knack for this kind of low-level hackery, combined with my passion for it, and willingness to sacrifice vast amounts of my free time to it, quickly led my robots to dominate the tournaments. When the other students lost interest in updating their robots to compete, I began pitting my prize robot---Swooper---against the winners of the global RoboWar tournaments. (Often, he won. Not often enough to enter one of the global tourneys, though.) When that got boring, I programmed robots with other goals in mind... a bouncing basketball, a robot that used mines to draw pixel art, robots that communicated with one another... Ultimately, the program length limit became too limiting.

Meanwhile, I'd learned some "real" programming languages... HyperTalk, Perl, Java, C... I also became proficient in 680x0 and PowerPC assembly. Armed with the tools and techniques I learned in RoboWar, and copies of the manuals for the CPUs in question, I would often attempt to beat GCC's rapidly-improving optimizer on certain tight inner loops. Eventually, it improved enough to nearly always win, and I began to trust it for most things. We've had a strong relationship ever since. :P

Flash forward ten years after graduation and my toolchest has grown enormous. I've programmed in over 40 different programming languages (some of which I designed), on systems ranging from a single 65C02 with mere kilobytes of RAM to many-core behemoths with 32GiB RAM per node. I developed the technology behind what was to have been the world's first multi-node, fully-connected persistent game world: the ill-fated "Ataraxia Project". It would have run on nodes with over 100 cores, connected via a custom-designed "fishbone" topology. (I'm still upset that it died...we were only a few weeks from alpha when the team fell apart.)

At the slightest hint of an excuse---or even without one---I am distracted by cravings to get involved with the low-level workings of a computer. On a recent project (a control device for a simulation of a rocket, actually), I realized that we could reduce costs significantly by using a 65C02 processor and interfacing the various controls almost directly to its bus. As part of this project, I wrote in C++ a cycle-accurate* 65C02 emulator, including its interrupts, IO, and even several not-frequently-used pins that we ended up making use of.

(*Read-modify write instructions are one cycle too fast, but they haven't yet been used in timing-critical code in any of our ROMs, so it isn't important.)

I laid out the memory space. I built and tested the bus logic---all of it---in a simulator. I put together a list of the necessary components, and devised the maze of connections to connect them to one another. I spent a week eliminating a transistor here, an IC there, getting the component count down... I hooked up dummy versions of our devices to the emulated 65C02's bus. It all came together. The same code that targeted the real device now ran perfectly on the emulator. And it was the most fun I'd had in years.

This is just the tip of the iceberg. This should serve to emphasize that my skills and my passion converge more or less on this spot.

The money-grab:

I know that, by blatantly asking for money, I'm asking a lot more than most Minecraft mod authors; but I intend to give a lot more than your average mod author as well. Don't get me wrong, there are many awesome, complete, well-supported mods out there (OpenComputers/ComputerCraft in particular are really great about this)... however, most smaller mods (including countless satellite mods of those two) are only half-completed or are abandoned when Minecraft updates.

I'm not asking idly, or just so I can fall into money doing something I enjoy. I'm currently unemployed. For complicated reasons, it's not likely that I'll be able to get employed for a long time. I have very little money, so even just a few dollars a month makes a really big difference in my life.

Money means I can spend less time worrying about my next meal and more time working on the mod. That means I'd be committed to keeping it up to date, tracking along with OpenComputers development (and possibly contributing fixes / patches to OC as necessary). Documenting it (and fixing holes in the OC wiki). Supporting it.

Most importantly, money means it can compete with other (paid) projects for what little productive time I have.

Hence the pitch.

If enough people---not even all that many---are willing to pledge just a few bucks a month, the project becomes worthwhile. So if enough people post in this thread, saying that they might patronize me when I have something to show for it, I can dive in, getting everything set up and making a proof-of-concept architecture module. It might not be able to do anything but print "Hello world!", but the code would be open and all the work behind it would be plain for all to see. If any of those people then go on to actually patronize me, then the real work can begin.

What say you?

Link to post
Share on other sites

Very well written and detailed post!

I saw the title of this post and was immediately interested. Taking a look on the possible architectures, I agree with you on that ARM would probably be the best architecture for this situation. The idea of being able to play with low-level CPU stuffs in Minecraft is very exciting in my opinion. I've always had a fascination with low-level programming and CPU architectures. Ever since I started playing with ComputerCraft and OpenComputers, I've always been fantasizing over a lower-level computer Minecraft mod. And what you have proposed, is exactly what I've always dreamed of.

 

Unfortunately, I'm unable to help fund this project due to personal reasons. But if I were able to, I'd gladly fork over a chunk of cash. I hope others have the same desire for a low-level computer mod and will be able to get this project going.

Link to post
Share on other sites

I saw the title of this post and was immediately interested. ... The idea of being able to play with low-level CPU stuffs in Minecraft is very exciting in my opinion. I've always had a fascination with low-level programming and CPU architectures. Ever since I started playing with ComputerCraft and OpenComputers, I've always been fantasizing over a lower-level computer Minecraft mod. And what you have proposed, is exactly what I've always dreamed of.

Thanks for the reply. I'm glad I'm not the only one enthusiastic about the idea!

Even though the jury is still out on whether it's going to be feasible for me to work on this mod, I've done a few hours of research in between (and sometimes instead of) things. I've made a few preliminary decisions:

  • The CPU would be compatible with ARMv7-A, and would boot in big-endian mode to aid interoperability with other computers on the network. (And because I prefer big-endian.)
  • The OS would be named Orca---short for "Open Real Computers on ARM". The variants would be named "Orca Personal", "Orca Professional", and "Orca Server". They would be downwards compatible with each other to a large degree. Professor Tanenbaum would be very annoyed with me (shortly before asking who the heck I am :P) as none of the variants would be microkernel based.
  • Orca would use ELF binaries. Shared library support would be inevitable, but it would be a while before it was in. (OpenOS uses Lua features to achieve much the same benefits.)
  • At first, all development would be done with external tools. Recent versions of clang and the GNU linker can be used to compile both EEPROM images and ELF executables/drivers targeting big-endian ARMv7-A. The best part? No need to build or install a separate cross-compiler toolchain. Ordinary installs of either tool are cross-compile capable. (Sweet!)
Regardless of interest or lack thereof, I can't begin seriously working on this until at least the fifth of February. (And for now, I really need to get back to what I'm supposed to be doing. :))
Link to post
Share on other sites

Love it! Can't offer anything but providing input from the OC side of things, though (i.e. I can contribute intellectually but not financially). You might want to swing by the IRC some time, there are a few people already working (at different stages) on low level architectures in OC. From what I can tell, one of the main hurdles for such a project is that OC's architecture API is relatively high-level, e.g. it assumes maps (the data type) be available for communicating with components. Which is obviously because it was somewhat modeled around the existing Lua archs, and more with other scripting languages in mind. Building a bridge for that would seem to be something useful for any low level arch.

Link to post
Share on other sites

From what I can tell, one of the main hurdles for such a project is that OC's architecture API is relatively high-level, e.g. it assumes maps (the data type) be available for communicating with components.

I had a few ideas for how to deal with that. Encoding the maps as a fairly primitive array of tagged key/value pairs was one. This would place the parsing burden on the software, which would be free to permute it to/from whatever data structure it prefers. I was also planning on making some sort of layer where the software could define a packed representation for specific cases and the packed representation would be used when possible. No matter what, IO would be somewhat... unrealistic. Which is fine by me since real IO can get pretty hairy. :P
Link to post
Share on other sites

I would be willing to pitch in on the financial side.

 

I was thinking along the same lines myself, and concluded I'd want a cpu that gcc/llvm already target. I'd even skip the OS in favor of pure work code. But knowing me I'd never get it done.

 

I figure there's absolutely no way to run a full c++ compiler ON an OC computer (std templates alone would brutally murder it) what's your plan there?

 

- Only accepting pre-compiled code from the client

Leaving the question of how it is uploaded.

 

- Invoking compiler on server

Maybe there's a compiler block, similar to the raid block, it acts like an fs but you also have a compile action available on it...

Bigger issue is the dependency - many servers won't have the compilers installed.

 

- Invoking compiler on client

Maybe said block would transfer to the client and let the client computer compile.

 

- Some combination of the above?

(Fall back on client, then precompiled?)

Link to post
Share on other sites

I would be willing to pitch in on the financial side.

I'll get right on that prototype, then. :D

 

I was thinking along the same lines myself, and concluded I'd want a cpu that gcc/llvm already target. I'd even skip the OS in favor of pure work code. But knowing me I'd never get it done.

Bare-metal code would be a pretty big application of this... 4KiB isn't a lot, but you can do a lot more with 4KiB of machine code than 4KiB of Lua. (That's around 2,000 Thumb instructions!)

 

I figure there's absolutely no way to run a full c++ compiler ON an OC computer (std templates alone would brutally murder it)

Murder doesn't even begin to cover it. :)

 

- Only accepting pre-compiled code from the client

Leaving the question of how it is uploaded.

Something with paste events and an external packer perhaps? Or a TCP-powered outside server and an Internet card? For my own stuff, I have a webserver that my OC computers can download my working set from.

 

- Invoking compiler on server

Maybe there's a compiler block, similar to the raid block, it acts like an fs but you also have a compile action available on it...

Bigger issue is the dependency - many servers won't have the compilers installed.

Something like this was the long-term approach I had in my mind. The headers and such could "live" in an OpenComputers computer, which would have a Compiler Card (maybe 2-3 tiers on the spectrum of assembly/C/C++?). The preprocessor (or part of it) would run in OpenComputers land and send the preprocessed code to the Compiler Card, and there's a linker in there somewhere...

The servers that don't have the software installed would be problematic, but an install of LLVM and the GNU linker, sufficiently complete to compile this stuff, should be easy for me to portablize. (Is that a word?)

 

- Invoking compiler on client

Maybe said block would transfer to the client and let the client computer compile.

- Some combination of the above?

(Fall back on client, then precompiled?)

That would make things a little more complicated than I'd like, as far as the Minecraft end of things.

In the short term, requiring any code to be compiled outside and transferred inside is workable. It's very similar to the way embedded development is normally done. Once all that's up and running I can start thinking about compiler cards and the like. Maybe far down the line Orca will even be compilable and runnable entirely from inside Minecraft...

(Orca itself, along with enough bits to actually be useful, would be available on a craftable floppy just like OpenOS currently is.)

Link to post
Share on other sites

This might be helpful:

http://jpc.sourceforge.net/home_home.html

It's an x86 emulator written in Java, capable of running Linux and playing Doom.  Minecraft is written in Java, so I figured you could borrow code from this open-source emulator.

Thanks. I'll keep that in mind for when I have the basic functionality in place. It claims to be "fast" which tells me it probably employs JIT, which definitely means it will be good to take a look at their approach; I'm not at all sure how to do code generation (and therefore JIT) in pure Java.

Currently, I've got most of a bare skeleton in place... the application-level core registers, a memory subsystem... The ARM Architecture Reference Manual is long (2736 pages), and I don't have a lot of time to spare for this, but nevertheless I'm making progress.

Link to post
Share on other sites

I'm not dead, and neither is the mod. :)

The last couple months have been... eventful and distracting. The uncontested highlight was that I managed to get scurvy (again). But that's a story for another day... Now that things seem to be done happening to me for a while, I can return to this mod.

Turns out, emulating a complete ARMv7-A system is a lot of work. Almost 2,000 lines of untested code so far. I have a complete instruction decoder for the ARM instruction set, and a decent number of emulated instructions. The current obstacle is coprocessor support, which is necessary to support the system control registers, of which there are... too many. Once that's complete, I can actually start testing my code, and get a hasty prototype into the game.

P.S. I've recently had cause to interact with x86 systems on a low level again and... I'm really glad nobody tried to convince me to implement one of those instead. Just... ick. :P

Link to post
Share on other sites

Still going!

I spent an inordinate portion of the last several days hunting down that rarest of beasts---a serious, fundamental flaw in the most widely-used toolchain on the planet, which fundamentally breaks an entire class of program, yet has a practical impact on probably just one person. :P (Edit: Turned out to be a documentation bug at best. Fundamentally breaking an entire class of program without anyone noticing did seem improbable...)

The emulator has executed its first instruction. Now I must merely fill in the gaps. Prototype isn't far off now...

Link to post
Share on other sites

After months of blood and sweat, and thousands of pages of reference manual, the prototype is here.

It's a cobbled-together proof-of-concept, but a "proofy" one nonetheless. An alpha-quality emulator is there, in Minecraft. Download it and give it a try!

Now that I have a prototype, I have set up my Patreon page. Please consider supporting me so I can continue to invest time in this project. Even a few pledges of $3/month would get me to the point where I could carry this project to completion. (At a significantly greater pace than brought it to prototype, since I would be able to better justify the time expenditure...)

Word of caution, the prototype isn't very user friendly. It basically consisted of me filling out the emulator to the point that it could run interesting code, then doing the absolute minimum work possible to "integrate" it with OpenComputers. Future releases of the mod will be much better in this regard.

mcjarm-proto.zip

Link to post
Share on other sites
[16:35:13] [OpenComputers-Computer-4/INFO]: If you are reading this message, you have just run the simple JARM test ROM!
[16:35:17] [OpenComputers-Computer-3/INFO]: ROM size: 1487 bytes
[16:35:22] [OpenComputers-Computer-1/INFO]: RAM size: 1024 KiB
[16:35:22] [OpenComputers-Computer-1/INFO]: Run complete. Terminating.

Yay!

Link to post
Share on other sites
[16:35:13] [OpenComputers-Computer-4/INFO]: If you are reading this message, you have just run the simple JARM test ROM!
[16:35:17] [OpenComputers-Computer-3/INFO]: ROM size: 1487 bytes
[16:35:22] [OpenComputers-Computer-1/INFO]: RAM size: 1024 KiB
[16:35:22] [OpenComputers-Computer-1/INFO]: Run complete. Terminating.
Yay!
Yay! (Except that it doesn't have the "[JARM Prototype]" bits which the readme said should be there. I guess those aren't always there. Oops. >_>)

This gives me an excuse to say that, after dealing so much over the last year with Minecraft's insanity, and after the last few months of fumbling in the dark between bits of untested code, the actual code to interface with OpenComputers was a breath of fresh air. The third or fourth time in a row I had a nontrivial question, and the answer turned out to be "The way that actually makes sense, and gets good performance"... I realized that this was going to be the rule when dealing with OpenComputers. I can't describe how relieving that was.

Link to post
Share on other sites

I love this idea!

heck, if an OS can be made for this to do something with a language as high-level as python, I could then have some real fun with this thing :3

I could probably go as far as hacking MC's GL-interface and adding my own characters to the game without OpenGlasses and all the unneeded management annoyances.

(on a server with everyone watching me develop this stuff)

Link to post
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

Loading...

×
×
  • Create New...

Important Information

By using this site, you agree to our Terms of Use and Privacy Policy.