1. Require looks if the library has been loaded before and if so it just returns the cached version, otherwise it searches in some predefined paths (usually the current directory, /lib and /usr/lib) for the library, reads the content of the file and uses the load() function to parse and execute the lua code, it caches the returned value from the executed lua / library file (so it is faster if the library is required again) and returns it. You can find the OpenOS implementation in /lib/package.lua
2. You don't have to make any library for a custom filesystem. In theory you can do everything with the filesystem component. But many systems implement a object oriented filesystem api and also a way to mount other filesystem component (like /mnt in OpenOS). Again you can look at OpenOS implementation in /lib/filesystem.lua and /boot/90_filesystem.lua.
If you want a custom directory structure you can also do that. OpenOS uses a unix like structure but you can also make your own. If you use Lua BIOS the only requirement is a init.lua in the root directory, but you can also change that by writing a custom BIOS, if you want.
EDIT: You might want to add basic programs to list, copy, rename, delete and edit files and directories and also to go to another directory (like "cd").