r39 patched Kernal to fix LOAD into HiRAM functionality

[ZeroByte]

The page thing is more optimized than that for block reads. It only checks twice per block (once in LOAD after the block transfer and once in DOS prior to doing the copy.

The DOS routine now sets a ZP variable = Y index where bank swap happens. It then does a loop only having to add cpy ZP per byte.

if the data ptr is < 9f00 or > BFFF then it uses the original loop that doesn’t waste time checking for page boundaries

LOAD calls block read repeatedly and updates its own pointer after each call, so that pointer must get wrapped down but this check only happens roughly 4 times per KB.

it would be easy enough to early exit if initial call was >= C000 and return out of memory.error, and maybe the same for page 9F. Personally I prefer the don’t do this or it crashes method. ?

[ZeroByte]

On 12/3/2021 at 7:44 PM, BruceMcF said:

If that is built into the binary page adjustment, then you need the binary page adjustment to always take place on a true binary page boundary ...

Re-reading your post, I noticed this bit.

Actually, the destination pointer during the block copy routine is NOT page-aligned at all. And by extension, neither is the shell loop in the LOAD routine itself. LOAD could probably force the issue by requesting a block read of N bytes to become page aligned, but that would be a lot of logic to run for each iteration of the loop, and it would be fighting against the fat32 driver's attempts to keep itself page-aligned, and a full 256-byte block transfer would almost never happen. Since LOAD is high level and DOS is low level, it's probably best for DOS to get to control the flow.

In debugging, I found that the very first block copy of a LOAD usually transfers $FE bytes and not $100 bytes. (PRG header removed from first transfer). Thus, when i was loading multi-bank data starting at A000, I found that the first write was FE bytes, then $100 for a few more writes. So the first bank boundary crossing was actually a 256-byte write starting at BFFE. Interestingly, this means that it's probably quite common for most of the bytes copied to incur the +1 clock cycle "penalty" for page crossing in (ZP),Y addressing mode.

I'm not sure whether the source buffer is page-aligned, but I'm fairly certain that it must be.

For anyone interested, the reason the 2 skipped PRG header bytes causes this is that the DOS keeps a kind of cache in bank0. It loads from SD one cluster at a time - I didn't go further down the rabbit hole to see how that operation happens and whether it scraps the buffer if you switch between two open files or anything like that. I just know that LOAD calls it, it returns all or part of the current page of buffer. When LOAD sees that it has not received all of the data, it calls "macptr" again which gets directed into fat32_read, which empties the next page of buffer. If at any point in this loop, the buffer is empty, it refills it with one cluster of data (512 bytes) until that's drained, etc.

P.s. - if you told me 2 years ago that I'd end up this deep in a file I/O routine, I'd've probably laughed at you

[BruceMcF]

On 12/3/2021 at 11:08 PM, ZeroByte said:

... it would be easy enough to early exit if initial call was >= C000 and return out of memory.error, and maybe the same for page 9F. Personally I prefer the don’t do this or it crashes method. ? ...

 

On 12/4/2021 at 1:31 AM, ZeroByte said:

Re-reading your post, I noticed this bit.



Actually, the destination pointer during the block copy routine is NOT page-aligned at all. And by extension, neither is the shell loop in the LOAD routine itself. LOAD could probably force the issue by requesting a block read of N bytes to become page aligned, but that would be a lot of logic to run for each iteration of the loop, and it would be fighting against the fat32 driver's attempts to keep itself page-aligned, and a full 256-byte block transfer would almost never happen. Since LOAD is high level and DOS is low level, it's probably best for DOS to get to control the flow. ...

Yes, the second is a pretty strong argument for the "don't do this" approach.

You wouldn't want to go a page at a time and fail out when the load is within 256 bytes of $9F00, since you want to be able to load up to EXACTLY $9EFF. So the game is not worth the candle. The place to do boundary checks is higher up than this routine ... in the code that is using this routine to load something into RAM.

[ZeroByte]

I think honk the optimal way to stop load at 9EFF would be to add one more check to the block copy routine: It does a series of checks to limit transfer size to the smallest of: full page, bytes in buffer, file bytes remaining, bytes requested by caller.

Add one more check: if page is 9E then limit to $100-ptr_low.

This would make LOAD start the next loop at 9F00 where it can error out.

I still hold the position that this extra range checking functionality adds too much size and performance impact to DOS and the Kernal.

 

[desertfish]

From the (limited) time I spent with this patched kernal it seems to work fine for me!  So yeah go ahead with the pull-request on github.  This is quite an important one, as it will probably fix a whole lot of data-load issues when running from sd-card ? 

(also I want to integrate it with my custom v39 rom build, because a few other patches are in there as well)

edit: found an issue when loading to banked addresses when LSB=$02 , thinks work ok if LSB=$00

[BruceMcF]

On 12/4/2021 at 10:46 AM, ZeroByte said:

... I still hold the position that this extra range checking functionality adds too much size and performance impact to DOS and the Kernal.

I reckon that ACTUALLY Kernel way to do it would be to offer a Kernel call that does a sanity check on the operands of a load, returning carry clear if the load stays within a "zone" (Low RAM, I/O Page, High RAM window), carry set if it WOULD cross a boundary.

And let the programmer decide whether to sanity test or not.

[Scott Robison]

On 12/4/2021 at 3:53 PM, BruceMcF said:

I reckon that ACTUALLY Kernel way to do it would be to offer a Kernel call that does a sanity check on the operands of a load, returning carry clear if the load stays within a "zone" (Low RAM, I/O Page, High RAM window), carry set if it WOULD cross a boundary.



And let the programmer decide whether to sanity test or not.

Unfortunately the kernal doesn't know how much it's loading until it has loaded. As least in as much as the kernal works with serial IEC devices.

[BruceMcF]

On 12/4/2021 at 8:28 PM, Scott Robison said:

Unfortunately the kernal doesn't know how much it's loading until it has loaded. As least in as much as the kernal works with serial IEC devices.

Oh, wait ... yeah, the block size count is in the directory system in the 1541, not in the Kernel. Finding it for the SD filesystem might not be too challenging, but on the IEC bus, it's entirely arbitrary how the connected bus records file sizes.

[Scott Robison]

On 12/4/2021 at 6:34 PM, BruceMcF said:

Oh, wait ... yeah, the block size count is in the directory system in the 1541, not in the Kernel. Finding it for the SD filesystem might not be too challenging, but on the IEC bus, it's entirely arbitrary how the connected bus records file sizes.

Exactly. And the internal DOS is just an abstracted interface to fit with the kernal API (I think).

[Edmond D]

On 12/3/2021 at 3:37 PM, kelli217 said:

There's a limit to how sophisticated these things can be, given speed and size constraints of retro or retro-like systems.

There's limit to anything on every platform retro or not. The balancing of protections and limitations is an art in my opinion. I'm risk adverse and want to be able to choose a safe path first over convenience. That's me ?



The retro factor I think shouldn't be a limitation on a safe approach. I've never crashed my VIC 20, but I spent a career helping people address crashes on PCs. Part may be the complexity of the technology, but some of it I feel is mindset that detaches actions from consequences. The technology has progressed over the years, the mindset should evolve too. 



The people in the X16 community seems to pride itself on providing sound technology and ingenious solutions. While no solution is perfect, coming up with a workable solution that protects the user in most cases is desirable. I see that Zerobyte has put some protection into the code, which I appreciate.