Advanced PRG File Format

[TomXP411]

This is my proposal for an advanced PRG file format. I'd like to place it here for discussion. 

The format consists of a 64 byte header and an 8 byte block header.

File header is

16 FF  - the magic value. Regular PRGs have the address first, and this is an illegal address. So the loader knows to handle this as an advanced PRG file.

01 00 00 Format version

01 00 00 File version (you can increment this as you save newer versions of your program)

56 bytes of free space. You can use this for internal revision control, but the primary intent is to store a text description of the file/program in here. 

Block header:

01: 01 - Main Memory, 02 - VERA

02-03: Block length. Two bytes, little Endian

03-04: Unused

05: Bank number (VERA or high memory bank)

06-07: Load Address

07-length+8: data

A more thorough specification is here: https://docs.google.com/document/d/1q4IhDcdZ12B9EqySmHs5tHblFYIzGlzmPuHl3QkiUL0/edit?usp=sharing

 

I wanted to discuss the format and possibly extend the CX16 ROM and write a C packer for Advanced PRG files. 

 

 

[TomXP411]

Some questions have come up in another thread:

Quote

Why not use Executable And Linkable Format?

ELF is a both larger and more complex than we need for the commander. APRG is as small as we can make the container format, while still carrying necessary information, such as load address and block length. Each block header only uses 6 bytes: Type, address, and length. I padded 2 bytes just to make the header 8 bytes long.

Quote

 Is it meant that the loader runs after the program has started?

If we get this into the KERNAL, the loader would be executed by the LOAD command. It would analyze the first two bytes of the file and decide from there whether to treat this as a single block or switch to APRG mode. 

Quote

As block type $10 seems to be something where the player and the loaders works together.

Blocks other than 1, 2, and 3 are not handled directly by the loader. Instead, your program would re-open the data file and read blocks for graphics, sound, game levels, or other data. Think of this as something like a ZIP file, where all of your program's data is combined into a single file. You can use the Directory block (04) to locate your individual chains in the file. 

Again, in an ideal world, the services to read the directory and seek to individual blocks would be handled by the KERNAL. However, I suspect that's too much to ask, so I'm hoping our reference implementation will include pre-made subroutines that programmers can tack on to their program in either assembly or C. 

Quote

As for the PC side (compiler) instead of writing in many languages you could look into golang, you can write one implementation that can run on all platforms. As for the reference loader I guess the ultimate goal is to get it into the ROM - so assembly is perhaps the only way to go?

I intend to write the packer in C. This will cover every platform used for writing for the Commander - including the Commander itself. 

 

[Lorin Millsap]

Not saying these aren’t valid points, but it is unnecessary. Does it have value. For a multi-task operating system sure. But for a system where each program actually doesn’t have to concern itself with other programs it’s benefits are minimal. Why create special file types when your program can handle its own data just fine and can just put all its relevant stuff in the same directory. Accomplishes the same goal with much simpler code.  Also technically parts of GEOS are in ROM and GEOS can do much of what you are describing. We have done some tests and while more work needs to be done in simplified terms we can run GEOS software.   Sent from my iPhone using Tapatalk

[hth313]

I think a load format for banked data is very useful and it should be in the ROM.

That way we can just load a file and have it placed in memory without having to re-invent the wheel by providing custom loaders. Does it prevent anyone from rolling their own? Not at all, you are free to load a plain binary and load data and program chunks from the file system as you see fit. I on my side prefer to have a binary file created by a tool and not need to bother how it gets loaded into memory, even when I need to use the bank system.

I like the ELF format myself as it rather simple to use when the requirements are simple, yet it provides a lot of flexibility and can be quite expressive when needed. Most of the complexity comes when adding debug sections or linkable sections, but that is not what we are doing here. I will describe it a bit below to give an idea what it is and show that it actually have a lot of similarities to the proposed format.

An ELF file has an ELF header that starts with a magic number (actually the string ELF) that identifies it as ELF. You only need to sanity check that it is 32-bit ELF, inspect the processor type and values related to program headers that describe where the program headers are in the file, their size and how many they are.

Each program header describes a chunk of memory to be loaded. The headers appear after each other. A header can describe an area to be cleared, or it can point to actual raw data to be loaded and it can specify a start address.

Thus, in a file you have:

-----------------

ELF Header

---------------

Program header 1

----------------

Program header 2

----------------

...

----------------

Program header n

----------------

Load data

----------------

The file can also contain sections, mainly for debug or vendor specific stuff, but you totally ignore them. Even if sections are in the file, you will automatically ignore them when you use the file offsets to locate the program headers and data.

For a banked system you could use a program header for each bank. The address is 32-bit so you can easily combine a bank number with the lower address.

If you compare this to the proposed format you will see that the structure is very similar. The ELF header is 52 bytes (in 32-bit ELF) compared to 64 in the proposed format. Each program header is 32 bytes in ELF and 8 in the proposed format.

The main difference is that in ELF the program headers are located after each other and contains a field that gives the file offset of the actual data.

A loader can interpret relevant fields and do some simple sanity checking, it does not need to be any more complicated. Here is a rough load algorithm:

1. 
   Open file, read ELF header and check the magic number, sanity check some fields in the ELF header to see that it is 32-bit ELF, for 6502 and that there is at least one program header. Store number of program headers, the size and keep a pointer to the current one, initialize it with the offset of the first program header as given in the ELF header.
   


2. 
   Seek to the next program header, load it and inspect type, read start address (if program type), size, offset in file to actual data.
   


3. 
   Load raw data from file, if the specified area is larger than what is in file, fill the rest with zero.
   


4. 
   Step to next program header, decrement header counter, if more program headers then go to 2.
   


5. 
   Jump to the start address of the program.
   

The main advantage of using an existing format like ELF is that it makes it possible to use other tools for inspecting and altering the executable file. It also avoids re-inventing the wheel again as someone already gave this some thought. We only need to specify how we use the format for the CX16.

[Fnord42]

I'm undecided as to how much we actually need this (meaning: I am totally unable to judge if it is a good use of possibly scarce rom space), but it seems like a good thing to have.

hth313's argument for ELF seems reasonable to me. OTOH, reinventing wheels is undeniably part of the fun in a project like this.

I'm curious to hear more opinions about this.

[Lorin Millsap]

You can already load banked data.


Sent from my iPhone using Tapatalk

[desertfish]

10 hours ago, hth313 said:

Seek to the next program header

How does one do this though? As far as I am aware there is no concept of seeking in the Commander X16 DOS routines. 

So it will mean you'll have to read all required headers into memory and then manipulate them from there as you cannot seek back in the file to read more data from the beginning.

2 hours ago, Lorin Millsap said:

You can already load banked data.

What do you mean exactly?  There are no ROM routines to do this automatically for you without having to roll your own code

[kliepatsch]

I agree that ROM space could be scarce and therefore an advanced loader might not make it. But I find it still interesting, because if it is made available as a third-party tool it might still be useful (but less so than if it is already in ROM, admittedly)

13 minutes ago, desertfish said:

So it will mean you'll have to read all required headers into memory and then manipulate them from there as you cannot seek back in the file to read more data from the beginning.

That's what I am worried about, too. If I understand correctly, in ELF you would have to make it possible to save information from an unknown number of block headers, whereas in TomXP's suggestion you just need to know about the block you are currently reading. I also like the idea to use an illegal address as the format identifier. To me, his suggestion seems well suited for the X16.

I think, if the goal is to get the feature into ROM, we better hurry up ?

[Lorin Millsap]

Yeah. This isn’t going to happen. It’s not needed, for the most part it’s not useful, and we already have routines for loading data into the banks and VRAM.


Sent from my iPhone using Tapatalk

[desertfish]

It does seems useful to me but it's more important to (hopefully) get some small bugs ironed out in the rom code that exists now (there are a few pull requests on github already that I think are pretty easy and important to include...)