Internet interface

[msx]

Hello there, as discussed in this post, i'd like to address the issue of how to connect the x16 to the internet. It is something probably everyone wants to do so maybe we could prepare some ground (i'm already looking forward to BBS, simple multiplayer games, etc :)).

So, i think the general idea is to have some external device which will handle the network stack, and relays data to the X16 via uart, spi or any other channel. It would be great if said hardware could be made inexpensively with off the shelf parts and minimum extra, like an ESP32 or some other network-enabled arduino.

Now, one thing that i consider very important is to have a common, abstract interface to do simple network access, so that every hardware developer can build their api against it and offer programmers an uniform, vendor indifferent way to access the internet. Let's not relive the errors of the past era where every program must include code to deal with all possible hardwares ? Remember when you had to choose which soudn card you had and the game developer had to code for all of them?

So, if anyone has some ideas, we could perhaps come to a consensus.

My first idea is to have a specific device that handle tcp sockets, i'll paste my previous post:

Quote

So i was wondering if it would be possible to create a "tcp socket" device, where the filename is an IP address. It would need to be managed by the microcontroller, but it shouldn't be hard at all. It would work like this:

OPEN 1, 20, 0, "142.250.184.100:80"

OPEN 2, 20, 0, "api.server.com:555"

The microcontroller will open the socket connection and associate it with the file number. Subsequent read or write will be routed to the socket, and errors can be mapped to the usual FILE_NOT_FOUND, DEVICE NOT PRESENT etc.

The only problem is that filenames are 16 character long AFAIK, so they can hold any ip address (15 chars top) but not the port. Using the secondary address as port is a problem becouse i think it's at most 128. Now an address+port is 6 bytes so it CAN be fitted in 16 char, for example using hex (C345A100:00C1), but it's pretty unelegant. It would require a conversion routine which is annoying. Perhaps someone has some suggestion (if the whole proposal makes any sense :P). 

There's a problem with the size of filenames that's a bit unfortunate.

Another idea could be to reserve some space for a jump table that is to be populated by a "driver" the user can load before the program, if it makes any sense? The API can be as simple as an open/read/write/close set.

I think TCP sockets will cover most of our needs, but beside it, UDP might also be interesting and could be similarly developed. Higher level protocols, i'm not sure. Perhaps an HTTP interafce could be implemented but i think it will be pushing things too far. (A simple HTTP protocol can be directly written on the x16 side, in case.)

A flag could be reserved to create a SSL connection instead of a regular one (obviously the cryptography will be on the microcontroller).

[BruceMcF]

One simple approach is to use the secondary address as a generic index to a protocol string, and send the protocol string along with the secondary address it is associated with on the command channel. Supposing the device is device #14:

OPEN15,14,15

PRINT#15,"SA2:P80"

CLOSE15

 

[lamb-duh]

I think tcp is overkill for most applications. I might be wrong, but I suspect most of what people will want to do is talk directly between two x16s, or between an x16 and a server that expects to be talking to an x16 (not to, for example, an imap mail server or an http server). What if instead of tcp, we had some kind of virtual phone exchange? (which would itself run on top of either tcp or udp)

I haven't thought through all the details, but here's a sketch of how it might work: 

- the x16 is connected to a "modem" over whatever kind of serial is available

- at poweron, the modem connects to the virtual telephone exchange over tcp or udp (or over whatever kind of network the modem wants to). the modem contains some kind of secret data that authenticates it with the exchange server as having a certain identity (a "telephone number"--but it could just as easily be a username)

- the x16 will send a command, asking the modem to "dial a number". the modem relays this to the exchange server, which will cause the modem belonging to that phone number to "ring" if it's connected. software on that end can accept or reject the call.

- if the call is accepted, then the server will proxy packets between the two clients

this set up requires a centralized exchange server to act as a proxy, but it would allow anyone to write machine-to-machine multiplayer pretty easily. the proxy server can be protected from abuse by limiting throughput to a few kilobytes per second, and a single active connection per person.

[Scott Robison]

6 hours ago, lamb-duh said:

I think tcp is overkill for most applications. I might be wrong, but I suspect most of what people will want to do is talk directly between two x16s, or between an x16 and a server that expects to be talking to an x16 (not to, for example, an imap mail server or an http server). What if instead of tcp, we had some kind of virtual phone exchange? (which would itself run on top of either tcp or udp)



{snipped}

I think TCP and / or UDP accessibility via the X16 is fine, or via the smart add on card / interface, or via a proxy server. Whatever people want to do.

Given that the C= 64 and C= 128 can handle TCP, I don't think it would be an issue for the X16 to handle it if people were interested.

My biggest concern with a virtual exchange type of setup is the fact that it is one more piece that has to be working for two machines to communicate. If it should disappear at any point, no one can connect to anything. Which is fine if you want to use it for some sort of MMO (where the first M could be massively or moderately) but some of us like the good old BBS style direct connection (over telnet in this case). And it could work for people who wanted to scratch an itch to do IMAP or POP3 or SMTP or HTTP or ... whatever their imagination demands!

[TomXP411]

The most common way people will likely hook to the Internet is probably via a serial device hooked to the User port, with some sort of Hayes AT emulation. Of course, that will only work for one connection at a time, with either raw TCP or Telnet mode communication. 

For interfacing with multiple endpoints at the same time, or for UDP,  I'd suggest looking at the Espressif AT command set. https://docs.espressif.com/projects/esp-at/en/latest/AT_Command_Set/TCP-IP_AT_Commands.html

 

[lamb-duh]

10 hours ago, Scott Robison said:

My biggest concern with a virtual exchange type of setup is the fact that it is one more piece that has to be working for two machines to communicate. If it should disappear at any point, no one can connect to anything. Which is fine if you want to use it for some sort of MMO (where the first M could be massively or moderately) but some of us like the good old BBS style direct connection (over telnet in this case). And it could work for people who wanted to scratch an itch to do IMAP or POP3 or SMTP or HTTP or ... whatever their imagination demands!

This is always going to be a problem with tcp though, unnless you're directly connecting one x16 to another over tcp, which will have its problems since most of them will be behind at least one nat router. The exchange would be running very simple software (which would probably be either public domain or MIT if i wrote it), which could be run by anyone themselves anywhere. tcp is just really bad for making direct connections between two clients because of changing addresses and various nat routers that try to stop incoming connections. It does has a centralized server, but you really need a centralized server to do peer to peer tcp.

I don't think there's anything wrong with writing IMAP, POP, SMTP or HTTP clients/servers for the commander x16 if people want to communicate with the internet. I just think that most of what people will want to do is use the internet to communicate directly with another x16.

[Scott Robison]

6 hours ago, lamb-duh said:

This is always going to be a problem with tcp though, unnless you're directly connecting one x16 to another over tcp, which will have its problems since most of them will be behind at least one nat router. The exchange would be running very simple software (which would probably be either public domain or MIT if i wrote it), which could be run by anyone themselves anywhere. tcp is just really bad for making direct connections between two clients because of changing addresses and various nat routers that try to stop incoming connections. It does has a centralized server, but you really need a centralized server to do peer to peer tcp.



I don't think there's anything wrong with writing IMAP, POP, SMTP or HTTP clients/servers for the commander x16 if people want to communicate with the internet. I just think that most of what people will want to do is use the internet to communicate directly with another x16.

TCP could be the protocol used to access a "proxy" (game server) ... IP works just fine making outbound connections. As you allude, listening for connections behind a NAT is trickier, but not impossible. https://stackoverflow.com/questions/1511562/how-do-i-make-a-tcp-server-work-behind-a-router-nat-without-any-redirection-co ...

[ZeroByte]

UDP and TCP are the same as far as making NAT pinholes is concerned. The one and only thing that makes UDP "easier" to do this with is that both ends can start sending packets immediately, which will simultaneously open a dynamic NAT pinhole on both ends, whereas TCP must do the handshake thing, so the "listening" side can't send anything until it's received a SYN packet first.

Even so, the UDP version of this depends on a few things being "just so." - i.e. this whole "full cone NAT" stuff that you see mentioned in XBox Live, etc. Namely, the NAT box should use the same source port as the packet used. This isn't always possible, and many boxes only map the local port for the specific remote host:port.

In general, as a network engineer, I always recommend customers who want inbound connections to use a static IP on the inside host, and either map the specific port(s) or a range of ports to that inside IP.

We could always just allow IPv6 and then it's not even a problem anymore. ?

 

 

[Scott Robison]

3 hours ago, ZeroByte said:

UDP and TCP are the same as far as making NAT pinholes is concerned. The one and only thing that makes UDP "easier" to do this with is that both ends can start sending packets immediately, which will simultaneously open a dynamic NAT pinhole on both ends, whereas TCP must do the handshake thing, so the "listening" side can't send anything until it's received a SYN packet first.



Even so, the UDP version of this depends on a few things being "just so." - i.e. this whole "full cone NAT" stuff that you see mentioned in XBox Live, etc. Namely, the NAT box should use the same source port as the packet used. This isn't always possible, and many boxes only map the local port for the specific remote host:port.



In general, as a network engineer, I always recommend customers who want inbound connections to use a static IP on the inside host, and either map the specific port(s) or a range of ports to that inside IP.



We could always just allow IPv6 and then it's not even a problem anymore. ?

Exactly, I think the target audience for X16 is tech savvy enough to configure their home routers to allow inbound connections. I have a dynamic yet mostly static IP address that I use to access stuff at home.

More to the point, lest I be misunderstood. I'm not necessarily suggesting that the X16 itself needs to support IP protocols. A smart add on card could do that. I was just trying to suggest that there is no reason why TCP (or other IP related protocols) couldn't be used between the two remote systems, whether or not a proxy was involved.

[ZeroByte]

TCP is ready-made to be completely handled by some device and presented to the X16 as if it were some kind of file stream.

Personally, I'd like to see an expansion card that has several data ports like the VERA ports, but something like 8 or 16 of them that you can ask the card to map onto any active network socket. Then you can just read from them as long as there's data available, and write to them for as much as you need. The card can handle all of the packetization, windowing, fragment reassembly, etc. UDP would actually be more challenging to use as an application because the onus of these tasks is placed on the application itself for everything except per-packet checksum validation. TCP will just look like an input stream as if it were from a serial connection. That's the whole reason it was made. Going to UDP just because "NAT is hard" will end up surprising you at just how much work you may need to end up doing on the back end that would have been taken care of for you "automatically" by TCP.