Binary data for the X16

[rje]

My space game, ever in slow-time development, uses binary data for ship data, character data, map data, and game state data.  I load these into high RAM and use them as needed, pulling them into main RAM long enough for a display or whatever.

Perl is really useful for creating binary files -- more useful than C, in that these sorts of tools are scripts, not worthy of dedicated development time with a proper language (like C).

Here's the easiest way to create a one-record binary file.

use strict;

EVERY Perl script should start with that line there.  That enforces some discipline in both the developer and the Perl interpreter.

Now let's declare our data and shove it into Perl variables.  This gives the data some tweakability, decoupled from the binary file's structure... although it also gives us an idea as to what lengths we want our data to fit into.

my $header = "TEMPLE OF APSHAI GAME STATE";   # fits into 31 characters
my $name   = "MATT HEFFERNAN";                # fits into 15 characters
my $gx     = 502;                             # global X, looks like 2 bytes
my $gy     = 500;                             # global Y, looks like 2 bytes
my $sx     = 3;                               # local X, 1 byte
my $sy     = 10;                              # local Y, 1 byte
my $life   = 10;                              # 1 byte

So we have seven pieces of data up there.  Now let's write them.

open my $fp, '>', 'GAMESTATE.BIN';    # fopen for writing ('>')

print $fp pack 'AC', 'X', 16;   # two byte throwaway 'header'
print $fp pack 'A31x', $header;      
print $fp pack 'A15x', $name;
print $fp pack 'vv', $gx, $gy;
print $fp pack 'CC', $sx, $sy;
print $fp pack 'C', $life;
print $fp pack 'x16';

close $fp;

 



The 'pack' command puts our data into a binary format.  We specify that exact format with the string that follows each pack... that string is a template telling the command how to pack in the data.

 

For example,

pack 'A5', 'HELLO';

tells Perl to pack in 5 ASCII characters from the source string 'HELLO'.

Add a zero value with 'x'.  For example, 

pack 'A15x', $name;

tells Perl to pack $name into 15 ASCII characters, then add a zero byte to the end of that.  Or this line, which simply adds 16 zero bytes to the file:

pack 'x16';

'v' encodes an integer in little-endian format.  'CC' encodes two unsigned char values.  And so on.

The trickiest example in my code is

pack 'AC', 'X', 16;

which first encodes a single ASCII character ('A'), an unsigned char value (16).  This is the throwaway two bytes that the X16 will ignore when you load the file to a specific address.  Since I load the file to a specific address, these two bytes are completely irrelevant.  Just on a whim, I put "X16" into them.

The resulting binary file is exactly 73 bytes long, and looks like this:

00000000  58 10 54 45 4d 50 4c 45  20 4f 46 20 41 50 53 48  |X.TEMPLE OF APSH|
00000010  41 49 20 47 41 4d 45 20  53 54 41 54 45 20 20 20  |AI GAME STATE   
00000020  20 00 4d 41 54 54 20 48  45 46 46 45 52 4d 41 4e  | .MATT HEFFERMAN|
00000030  20 00 f6 01 f4 01 03 0a  0a 00 00 00 00 00 00 00  | ...............|
00000040  00 00 00 00 00 00 00 00  00                       |.........|

[Edmond D]

While Perl is useful for these types of tasks, perhaps now the tools for creating graphics/sound binaries have developed enough on the X16? I've phrased it as a question since I'm not there yet in coding for the X16 platform, nor have I investigated yet.



The other thought is that it becomes another tool one has to learn to do this type of work, something that requires a little background knowledge and investment of time. While many may already have this skillset, consider the users who just don't have the experience or the bandwidth to take on another thing. Right now my feeling is the X16 platform requires one to know or learn too many things from too many places that really aren't in coding up an application, rather just tasks to get at the business end of coding. I think one of the successes of the VIC-20 was that with the user's guide and the programmers guide one had all one needed to get going with no side tasks that required building software tools. 



That being said,  I'm no Mozart or Picasso. A sound editor or sprite composer in my hands would only allow me to easily create something bad like a smashed bug on a blue Triscuit. ? But I'd like to have the ready built tools rather than learn the bare minimum on another complex technology that I wouldn't have to know/use otherwise. 



PS - I have learnt and used Perl in the past, and might use it in the future someday, but that effort came with some $ involved, and the X16 is a hobby platform.

 

[kliepatsch]

I think everyone can use whatever they know for this kind of task. Albeit admittedly some tools allow for more comfort than a Hex editor. I am using Python for generating binary data. It would look like this

struct_1 = [12, 13, 14]

 

struct_2 = [0, 80, 0, 90, 0, 100]

 

data = struct_1 + struct_2 + struct_1 # appending data is as easy as this.

 

data_bytes = bytes(data)

 

f = open("DATA.BIN","wb") # "wb" stands for write binary

 

f.write(data_bytes)

 

f.close()

 

 

 

[BruceMcF]

On 9/25/2020 at 10:09 AM, rje said:

My space game, ever in slow-time development, uses binary data for ship data, character data, map data, and game state data.  I load these into high RAM and use them as needed, pulling them into main RAM long enough for a display or whatever.



Perl is really useful for creating binary files -- more useful than C, in that these sorts of tools are scripts, not worthy of dedicated development time with a proper language (like C).

Thanks for that ... as a long time AWK user, Perl is surely an easier learning curve for me than Python.

[rje]

I note that his Python code looks familiar:  if I was writing raw bytes, that's what the Perl code would look like.

Note that he has to first break down larger numbers, and encode strings into bytes.  He might have to do some custom coding if endianness is a problem.

 

... I want AWK on the X16.

 

 

 

[Edmond D]

As much as open flexible general purpose tools are useful, I think for the X16 world that specific purpose tools would serve the users in the community better. 



In the examples given above, each binary file needs a source file to create it. While source code version control and storage have come a long way, it's still maintenance work. Each resource is going to need its own generating script. Then there is a need to have a tool that can present the data in the file for human inspection. 

For specific resources, such as game state data, that's going to be tied to the game itself and most likely requires the approach of general purpose tools.

But for a sprite resource a well designed sprite editor would eliminate a per-sprite development tool by serving to work on any sprite file. It also eliminates the need for a separate tool to look at the resource files. While it would be a stand alone specific purpose application, if it was an open community project then improvements could come from those who have the time, motivation, and the skill set. Everyone would benefit. 

I see the x16 as a "fun/simple" development platform, but for the serious/commercial work being done on a modern computer that overcomes most of the limitations.

Which comes to the question - should X16 development be on a modern platform, or limited to tools on the X16 itself?

[BruceMcF]

On 9/29/2021 at 1:34 PM, rje said:

... I want AWK on the X16.

AWK on the X16 is the thing most likely to make me learn the ins and outs of CC65. Programming in C is too much like working for a living for me to want to do it on a regular basis, but to get an AWK implemented, I'd make an exception.

However, it might be just System 7 AWK, leaving development toward a One True Awk compatible interpreter to a smarter and/or more ambitious person than I.

I realized that since the System 7 sources are available, the YACC and {that other one which escapes me for a minute} could be compiled and run on their own AWK sources to generate an all-K&R-C set of sources for the System 7 AWK. Then it is a matter of porting to the heavily modified Small C that CC65 uses and to build in replacements for the Unix functions the System 7 assumed ... I imagine a menu based system for loading and saving the AWK script and input and output into their own strings of HighRAM segments could make for a fairly nimble system.

[rje]

On 9/29/2021 at 2:35 AM, kliepatsch said:

 

struct_1 = [12, 13, 14]

 

struct_2 = [0, 80, 0, 90, 0, 100]

 

data = struct_1 + struct_2 + struct_1 # appending data is as easy as this.

 

data_bytes = bytes(data)

 

f = open("DATA.BIN","wb") # "wb" stands for write binary

 

f.write(data_bytes)

 

f.close()



 

Hey, that doesn't write bytes to the output channel, it creates a text file containing the Python expression "[12, 13, 14, 0, 80, 0, 90, 0, 10, 0, 12, 13, 14]". The binary file should have 12 bytes in it, since there are 12 byte values in the concatted array. Maybe I'm doing something wrong?

 

And, by the way, the Perl version looks very similar, and also creates a text file, although just the digits and not the Python expression.

@struct_1 = ( 12, 13, 14 );

@struct_2 = ( 0, 80, 0, 90, 0, 100 );

@data = (@struct_1, @struct_2, @struct_1);

open my $fh, ">:raw", "DATA.BIN";

print $fh @data;

close $fh;

To write actual bytes, I still need pack():

@struct_1 = ( 12, 13, 14 );

@struct_2 = ( 0, 80, 0, 90, 0, 100 );

@data = (@struct_1, @struct_2, @struct_1);

open my $fh, ">:raw", "DATA.BIN";

print $fh pack "C*", @data;

close $fh;



 

[rje]

On 9/29/2021 at 1:40 PM, BruceMcF said:

AWK on the X16 is the thing most likely to make me learn the ins and outs of CC65. Programming in C is too much like working for a living for me to want to do it on a regular basis, but to get an AWK implemented, I'd make an exception.

My online friend Alex had written a very very very low C heap manager, for the purpose of implementing a CSH variant on the C64.  I've got his code, but I didn't use it.  I'm having enough trouble writing a compiler down to interpretable bytecode.  

I attach them here for your amusement.
heap.c
heap.h
testcsh.c
testcsh.notes

[rje]

As for AWK, I search for "practical examples" instead of tutorials that show me how to do uninteresting things.

    <CONDITION> [<ACTION>] <SETTINGS> <SOURCE FILE NAME>

^ This looks like the generic template for AWK statements.  Note I'm using [brackets] because PETSCII doesn't have {braces}.  No worries.

First, let's posit that this particular AWK is only for one-liners. No reason to think so, except for simplicity maybe.

All the settings/field separators etc are fine.

 

 

    1 MY_FILE

Cats the file to the "terminal" (does it pause at the bottom of the screen?)

    'NR>1 && NR < 5' MY_FILE

Cats lines 2 through 4.  Note that the initial condition, if complex, is in single quotes.

    NF MY_FILE

Prints lines that have non-space characters on them.  I.E. ignores lines which only have whitespace.

    1  RS=''  file

Ignores lines that only have the newline character.  Note that the Record Separator assignment is outside of the action, so isn't in brackets.

   '{ print $1, $3}' FS=, OFS=, file

Prints columns 1 and 3 of every line.  Note the input and output field separators are both the comma.

   'NF { print $1, $3 }' FS=, OFS=, file

If the line has fields, then print the columns, otherwise ignore the line.

 

 

* * *

Now.

Can we have multi-line scripts, complete with BEGIN and END options?

I think variables would be no problem, so we could have variable assignment in the action blurbs.

Associative arrays are, I think, just out of reach, without some shoehorning.  I'd defer that until the rest of it works beautifully.

The most interesting parts of regular expressions will probably never work here.  Too big.