So, for a few days now (apparently about five days according to GitHub - it seems a lot longer), I've been trying to work on a pretty simple package container solution. The idea being just like a tar container, it is just a group of files and folders, stored in a single file. Currently, I am not working on implementing any sorts of compression, that's much too difficult and I wanna make sure the core system works first. I designed a primitive specification for the package, and created a method in a library to create the package. That I was okay with. Extracting all the information from this package is where my head is starting to spin, haha. Frankly, if there's already an existing solution, I'd really prefer to use that. Programming something like this means fixing countless bugs and writing entire libraries before the product is even stable enough to be used by people in the first place! If something does exist, please do tell me.
If nothing like this exists, I would appreciate some assistance at very minimum brining version one up to at least a functional level. By help, I mean that while developing the unpacking code right now, I'm pretty much running around with my eyes closed. Anything that makes this process less horrific, is really really appreciated. This project was started, because I wanted to start a little package manager project too. I know OOPM exists, but I wanna make one, simply because I can. That's besides the point. While the package format is designed for this use, it should be usable on its own based on my format.
I know the format is ... meh ... at best. I already have a new structural system in mind for version two. However, before I go off redoing this project, I want to at least get version one working! I have pretty much very social media outlet in the App Store, even some I can't fully understand (such as QQ since I'm [procrastinating] learning Mandarin)! Discord, Skype, and all those common ones of course, so I should be easy to contact if need be. I'd really appreciate any leads for existing software you find, or any help you can provide at least getting V1 working.
While I'm talking about it, I'll mention how I plan on redoing the internal structure of the packages. Right now, to read anything you need to scan it from the start and search for flags that say that an attribute has ended, that a section has begun, etc. The main change in my new theoretical design, is that the package descriptor and metadata sections will be partly fixed-length. Meaning, in the beginning of the package will be a fixed-length table. This table will contain the pointers and sizes of various values. Meaning, regardless of the size of the title for the package, the descriptor table will always be exactly the same length. Importantly, it will contain a pointer to the body structure, where the new design is really different.
In the V2 system, there will be an actual hierarchy stored. At the lowest level, the first object a package's tree will have is (a) trunk(s). This can either be a relative trunk (meaning it allows for the package to extract to whatever directory you're in), or an absolute trunk (so it can install library files into /lib). Everything else in the package will be a child of a trunk. Along with that, every object must have a unique ID (unique, in the sense that it's not reused within the same package). To store and recall the data, instead of having every file and folder prefix with a special identifier, attributes terminating in nulls, the content, and more nulls (as V1 does) it will use a table at the beginning of the body section to find data. This table will be variable length, since it changes based on the contents, but it would be a simple linked list. The first portion of each row would be fixed-length, containing things like the file's ID, it's parent ID, it's type (file, folder, trunk), and a pointer to the next object in the list. After that comes the name of the object, but that's last since it's variable length. Meaning, if you load only the file directory table into memory, you can pick and choose which files to extract, analyse the structure, etc. Since the IDs and pointers for the files themselves are stored separately from the data, it's nice and easy to load into memory, it shouldn't be extremely large, and should be easy to work with. One change that could be made, is have two variable length attributes in each file directory row. Now each row also contains the pointers of the children of that object. That would make viewing the hierarchy itself very easy and quick to do, since jumping between nodes would be simple and easy.
I figured I would give a description of this format just in case. I may start working on this before V1 even get's finished. The reason is, this will likely be its own GitHub repository, since this data structure could be used for other things potentially too, like databases.
Again, thanks for any help you can provide, or information you can give. Sorry this message is so long, I'm not the best at writing short concise messages - as you can see, I'm more of a long, detailed, rambling on type of writer. I'll just stop writing now. Thanks again!
Edited by coderboy14 I forgot to share the repository's url
You can post now and register later.
If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.
So, for a few days now (apparently about five days according to GitHub - it seems a lot longer), I've been trying to work on a pretty simple package container solution. The idea being just like a tar container, it is just a group of files and folders, stored in a single file. Currently, I am not working on implementing any sorts of compression, that's much too difficult and I wanna make sure the core system works first. I designed a primitive specification for the package, and created a method in a library to create the package. That I was okay with. Extracting all the information from this package is where my head is starting to spin, haha. Frankly, if there's already an existing solution, I'd really prefer to use that. Programming something like this means fixing countless bugs and writing entire libraries before the product is even stable enough to be used by people in the first place! If something does exist, please do tell me.
If nothing like this exists, I would appreciate some assistance at very minimum brining version one up to at least a functional level. By help, I mean that while developing the unpacking code right now, I'm pretty much running around with my eyes closed. Anything that makes this process less horrific, is really really appreciated. This project was started, because I wanted to start a little package manager project too. I know OOPM exists, but I wanna make one, simply because I can. That's besides the point. While the package format is designed for this use, it should be usable on its own based on my format.
I know the format is ... meh ... at best. I already have a new structural system in mind for version two. However, before I go off redoing this project, I want to at least get version one working! I have pretty much very social media outlet in the App Store, even some I can't fully understand (such as QQ since I'm [procrastinating] learning Mandarin)! Discord, Skype, and all those common ones of course, so I should be easy to contact if need be. I'd really appreciate any leads for existing software you find, or any help you can provide at least getting V1 working.
-- UPDATE --
Wow. I really forgot to add the link to the GitHub repository for this project. The repository is here [https://github.com/OcilDiamond/opkgf].
Tangent about V2
While I'm talking about it, I'll mention how I plan on redoing the internal structure of the packages. Right now, to read anything you need to scan it from the start and search for flags that say that an attribute has ended, that a section has begun, etc. The main change in my new theoretical design, is that the package descriptor and metadata sections will be partly fixed-length. Meaning, in the beginning of the package will be a fixed-length table. This table will contain the pointers and sizes of various values. Meaning, regardless of the size of the title for the package, the descriptor table will always be exactly the same length. Importantly, it will contain a pointer to the body structure, where the new design is really different.
In the V2 system, there will be an actual hierarchy stored. At the lowest level, the first object a package's tree will have is (a) trunk(s). This can either be a relative trunk (meaning it allows for the package to extract to whatever directory you're in), or an absolute trunk (so it can install library files into /lib). Everything else in the package will be a child of a trunk. Along with that, every object must have a unique ID (unique, in the sense that it's not reused within the same package). To store and recall the data, instead of having every file and folder prefix with a special identifier, attributes terminating in nulls, the content, and more nulls (as V1 does) it will use a table at the beginning of the body section to find data. This table will be variable length, since it changes based on the contents, but it would be a simple linked list. The first portion of each row would be fixed-length, containing things like the file's ID, it's parent ID, it's type (file, folder, trunk), and a pointer to the next object in the list. After that comes the name of the object, but that's last since it's variable length. Meaning, if you load only the file directory table into memory, you can pick and choose which files to extract, analyse the structure, etc. Since the IDs and pointers for the files themselves are stored separately from the data, it's nice and easy to load into memory, it shouldn't be extremely large, and should be easy to work with. One change that could be made, is have two variable length attributes in each file directory row. Now each row also contains the pointers of the children of that object. That would make viewing the hierarchy itself very easy and quick to do, since jumping between nodes would be simple and easy.
I figured I would give a description of this format just in case. I may start working on this before V1 even get's finished. The reason is, this will likely be its own GitHub repository, since this data structure could be used for other things potentially too, like databases.
Again, thanks for any help you can provide, or information you can give. Sorry this message is so long, I'm not the best at writing short concise messages - as you can see, I'm more of a long, detailed, rambling on type of writer. I'll just stop writing now. Thanks again!
Edited by coderboy14I forgot to share the repository's url
Link to post
Share on other sites