File Transfer over Serial-Port (using base64 encoding, like a MIME!)

[Xiphod]

I made a version of this awhile back for the earlier serial card, but guess I never put in WIP while waiting for the V1.1 official cards to get released.

I couldn't find my C# solution used to send files to the X16 from the PC side. But I've recreated the general premise of it using some Python. The receiver side runs native on the X16 and uses the TexElec serial card.


Basically this project combines a few things:
- the BASIC command line argument parser (optional and you can modify the BASL to use your own preferred defaults, just search "default" in the code)
- the decode64 project (base64 encode/decoding)
- and the serial connection.

One problem in sending binary data across serial is: how do you send a zero? Or, how do you send a long sequence of zero's? Since on the serial line, zero is generally used as a "no-data-on-the-line" indicator. You could switch to something like 0x55, but... any 255-byte is valid binary (like in 6502 code or an image data file), including 0xFF.

There are various work around to this. But one option is to proxy all the binary data into some other encoding - and so for that, I use the "base64" encoding that converts the binary into a plain-text that could get passed through essentially any system. It's not the most efficient way - because the encoding actually makes the data-payload larger. But, lots of system have base64 encoders (and I have a base64 encoder on the X16 also- maybe someday we can e-mail attachments from the X16).

This is all a "proof of concept" using BASLOAD, and at 8MHz BASIC parsing is still fairly slow (we're talking like 30 seconds to send 3KB, then having to do the actual decode64 after that). But now with all the buffer-handling sorted out, this is probably a pretty easy port to prog8, nxBASIC, cc65, etc. This BASIC version will need to be throttled to about 2400 baud. And, you also have to experiment with the "per character" delay (it's in Hz on the Python script, so highest is faster, and a value around 60-100 worked ok for me).


The general flow is this:
- buffers received content into BANK 1
- once that bank becomes full (8K), it writes the content to a TEMP file (XFERn.TMP); this was intended to eventually supported transfering very large files (>2MB)
- once all the expected bytes are received, then it starts applying the decode64 to each of the files. I'm still cleaning this part up, but it buffers each of the XFERn.TMP files into a BANK (2-n). At the moment this ends up limiting the filesize, but I have changes in mind to need only 2 banks total (curr and next block) that should get back to the "any size file" goal.

Except as mentioned, small files are already painfully slow, this solution is yet suitable for even 100K-size files let alone megabytes.


In addition to being "base64" encoded, the serial-exchange here has a little comma-separated header: the filename, expected bytes, and then the base64 encoding.

That's why some utility is needed on the PC side, to do the encoding, prepare this header, then ship it over the connected serial line as a suitable rate. Send it too fast, and the serial line will drop characters eventually. One issue is the BSAVE of a full 8K block is a big hiccup on performance - so the UART FIFO might not keep up. I tried a smaller size buffer just to make the BSAVE go faster, but still couldn't reliably get >9600 baud transfers.

There isn't much for error checking. Since I am doing a block at a time, I've thought about adding CRC checks, and then some re-try protocol for failed blocks. And if characters do get dropped, it messes up the accounting towards the total expected bytes. There is an override where you can tap spacebar (on the X16 side) if you think the data-stream has gotten confused (so it just proceeds as if the buffer has become full - you might still end up with a valid transfer if all the noise happened to come after the base64 pad signal).


You can transfer multiple files (one after another). After the first transfer, it preps to received another one. So you could script up multiple transfers. Not real support for sub-directories yet, can only transfer into the CWD.

I've found it useful only for small files, like <1K.


example:
BASLOAD"GETFILES.BASL.TXT" (or LOAD"GETFILES.PRG")
RUN:REM 1200 3 H

That is the BAUDRATE(8n1), 3 is the IOx selection on the serial card (3-7), then High or Low (or 0 for Low, 1 for High, either way). If you just type RUN it will pick a default (I'd like to improve this to try to auto-detect the first serial card and just default to that).


On the PC side, the Python script is used like this:
python prepare.py COM5 1200 80

The serial COMn port, the baud rate, and then the between-characters delay time. For me, delays of 60-120 did work, but higher values did increase chance of dropped characters.

Maybe try the following as a guideline:

1200 120
2400 100
4800 80
9600 40
(yes, it's a little silly - increasing the per-character delay is decreasing the effective baud rate; but it's necessary because even if you buffer up all that data, eventually you have to spend CPU to process it; not so bad in the old days when we had small files anyway)

If you don't have a serial port, I use cheap USB-serial adapters (have to for the telescopes).

[Xiphod]

The dots "." at the beginning are just the program waiting for the serial line to settle with no-content for a few seconds. In testing I found serials that would always flood me at startup, giving the program a confused filename prefix.

On the PC side, I used ZOC. And this needs a null-modem cable (with tx/rx wires "twisted") to communicate between a PC and the X16. There is an elaborate way that the WiFi side with ZiModem could be used - in ZiModem you'd have to set a receive port, then on the PC sender just have to know the network-card (ESP32) assigned IP and that port number. Maybe some day I'll test that out again - but for now the focus was on wired serial connection.

[Xiphod]

EDIT: The GETFILESN.BASL.TXT was updated from the original. Fixed HANGUP code.

Here is a version that works across the "network" (WiFi) side of the card. Yes, finally got first wireless transfer over to the X16 SD card (using the TexElec network-card and standard X16 ROMs and equipment)!

Some pre-steps to get this going: Use BASTERM/ROMTERM first to initialize the Zimodem configuration using a few things...
(connect to the Fxx0 port of your network/serial card; dial over to the last baud rate it was set at)
- ATB2400 (set the desired baud rate, I tested at 2400; the interpreted nature of BASIC just makes it slow for this, but it's PoC)
- ATE0 (turn echo off, halves the xfer rate and interferes with the exchange parsing)
- AT&W (write the settings so you don't have to remember to apply these again after power reset)

To get "network" comm working over this card, the only real change is in the SERIAL.INIT having to add the following (which is done in the GETFILESN sample done below; N for Network-version, notice the port numbers are 9Fx0 set instead of the 9Fx8 ones):

Code: Select all

SERIAL.INIT:
...same...
...then add:  
  X = $02 : REM SET RTS BIT 1    0000 0010
  POKE REG.WRITE.MCR, X
  X = $20 : REM SET DSR BIT 5    0010 0000
  POKE REG.WRITE.MSR, X

This is still painfully slow to transfer files (had to use 2400 baud and like 30ms+ in between each transmit characters). And it still seemed prone to dropping characters with larger samples files. But you can connect to the X16 network card with any telnet (ZOC has this, and a feature to "send text file" during the connection -- essentially "speed typing" ASCII content for you, but if you "type too fast" relative to the baud rate, it can drop characters)


NOTE: This works by assuming a little connection protocol implemented in the Zimodem firmware on the ESP32. You'll use something like telnet (or I used a full featured terminal called ZOC) to make the connection and then send the Plain Text file. It uses ATA6400 to specify to listen in on port 6400 for new connections, then "ATA" to answer when a client is connected.