• Solved issue with SyncTerm after up/download on my BBS

    From AKAcastor@21:1/162 to All on Tue Apr 16 21:59:44 2024
    On the BBS here (Another Millennium - another.tel port 23) we had experienced an unusual bug causing sessions with SyncTerm to become unstable after a file transfer. Some menus and prompts would sometimes not display, seemingly erratic behaviour. I never noticed this on other BBSes, and only when using SyncTerm to connect to my BBS - no other client I used exhibited the problem, so it seemed kind of mysterious.

    The problem was identified using the logging feature of SyncTerm and solved
    by patching the telnet softmodem in DOSBox-X.

    Some background:

    On telnet connections, there are IAC messages (Interpret As Command) that are used to configure the data link between server and client. These IAC messages did not exist for dialup modem use - they are specifically part of the telnet protocol - so Maximus BBS does not manage IAC messages.

    This BBS is using telnet emulation via the softmodem built into DOSBox-X.
    The softmodem interface allows Maximus and other DOS software to communicate via telnet as though it was a dialup modem. The telnet protocol is invisible
    to the DOS software. DOSBox-X interprets IAC messages and responds accordingly.

    One of the functions of IAC messages is to configure a link to pass ASCII (7-bit) data or binary (8-bit) data. Especially when transferring files it is important to have a binary link to avoid file corruption. On this BBS many other areas also use 8-bit data - if you have IBM-PC characters (CP437)
    enabled, that's 8-bit.

    Back to the problem:

    In the SyncTerm session log, when a file transfer completed I found this:

    Xfer Notice: Finishing Session (Sending ZFIN)
    Telnet Info TX: DON'T Binary Transmission
    Telnet Info TX: WON'T Binary Transmission
    Telnet Info RX: WON'T Binary Transmission

    After a file transfer, SyncTerm is sending an IAC DON'T Binary along with an

    IAC WON'T Binary and has received from the DOSBox-X running the BBS an IAC WON'T Binary message. That is, SyncTerm is requesting the BBS to disable binary (8-bit) mode and telling the BBS it is no longer using binary mode,
    and the BBS (DOSBox-X) is complying and also disabling binary mode.

    With binary mode disabled, the BBS interface becomes erratic in SyncTerm.

    Solution:

    The telnet IAC handler in DOSBox-X softmodem.cpp was modified to deny
    requests to disable binary mode, and in case a client disables binary mode a request to enter binary mode is sent.

    The SyncTerm session long now looks like this:

    Xfer Notice: Finishing Session (Sending ZFIN)
    Telnet Info TX: DON'T Binary Transmission
    Telnet Info TX: WON'T Binary Transmission
    Telnet Info RX: WILL Binary Transmission
    Telnet Info TX: DO Binary Transmission
    Telnet Info RX: DO Binary Transmission
    Telnet Info TX: WILL Binary Transmission
    Telnet Info RX: WILL Binary Transmission

    After this, the session remains in binary mode and the BBS continues to operate smoothly.


    Chris/akacastor


    --- Maximus 3.01
    * Origin: Another Millennium - Canada - another.tel (21:1/162)
  • From Digital Man@21:1/183 to AKAcastor on Wed Apr 17 11:57:46 2024
    Re: Solved issue with SyncTerm after up/download on my BBS
    By: AKAcastor to All on Tue Apr 16 2024 09:59 pm

    Solution:

    The telnet IAC handler in DOSBox-X softmodem.cpp was modified to deny requests to disable binary mode, and in case a client disables binary mode a request to enter binary mode is sent.

    This sounds like a hacky-work-around and not a 'fix'.

    In binary transmit mode, when the user hits the ENTER key in SyncTERM, it will send just ASCII CR to the telnet server. This is not the proper/normal sequence for the Telnet "NVT" (Network Virtual Terminal) 'enter' key sequence which is CR/LF when not in binary transmit mode.

    From https://datatracker.ietf.org/doc/html/rfc854:
    Note that "CR LF" or "CR NUL" is required in both directions
    (in the default ASCII mode), to preserve the symmetry of the
    NVT model.

    If ASCII (non-binary-tx) mode is negotiated succesfully when SyncTERM first connects to telnet server, then it should be free to return to ASCII mode later.

    It just sounds like there's a bug in the DOSBOX-X terminal server with tracking the remote and/or local "BINARY TX" state and that needs to be fixed rather than just blindly rejecting requests to exit binary transmission mode.

    Maybe you would have had better luck with DOSBOX-staging instead of of DOSBOX-X?
    --
    digital man (rob)

    Synchronet "Real Fact" #105:
    Synchronet YouTube channel: https://www.youtube.com/c/Synchronet
    Norco, CA WX: 74.1øF, 34.0% humidity, 6 mph W wind, 0.00 inches rain/24hrs
    --- SBBSecho 3.20-Linux
    * Origin: Vertrauen - [vert/cvs/bbs].synchro.net (21:1/183)
  • From AKAcastor@21:1/162 to Digital Man on Wed Apr 17 12:55:56 2024
    Solution:

    The telnet IAC handler in DOSBox-X softmodem.cpp was modified to deny requests to disable binary mode, and in case a
    client disables binary mode a
    request to enter binary mode is sent.

    This sounds like a hacky-work-around and not a 'fix'.

    I don't disagree with that, but then also I consider running modem-based DOS software inside DOSBox with telnet emulation a bit of a hacky work around in the first place, so it doesn't really offend my sensibilities. ;)

    If ASCII (non-binary-tx) mode is negotiated succesfully
    when SyncTERM first connects to telnet server, then it
    should be free to return to ASCII mode later.

    ASCII mode was never negotiated successfully, the session was in binary mode from the beginning. SyncTerm is incorrectly 'returning' to ASCII mode after file transfers.

    I don't think I see a pressing need for my system to support ASCII mode telnet as I expect any client should support binary mode. (maybe there's some edge cases in niche implementations or a benefit of ASCII mode that I am missing?)

    It just sounds like there's a bug in the DOSBOX-X
    terminal server with tracking the remote and/or local
    "BINARY TX" state and that needs to be fixed rather
    than just blindly rejecting requests to exit binary
    transmission mode.

    Maybe you would have had better luck with DOSBOX-
    staging instead of of DOSBOX-X?

    I agree there is likely a bug hiding in the telnet emulation, I may dig deeper for it yet. In the meantime, convincing SyncTerm to stop switching to ASCII modes removes the issue from my system.

    Telnet emulation in DOSBox-Staging is broken also. I don't know if it has the same behaviour with ASCII vs binary mode but it has other telnet protocol bugs, like sending binary data from DOSBox does not escape the IAC bytes. (due to this binary transfers work when DOSBox is receiving, but not when sending)

    I haven't ruled out DOSBox-Staging in the long term, but DOSBox-X has been working well for me and I have already been patching it to suit my needs (mostly a bit of cleanup/hackjobs in softmodem.cpp). Eventually I hope to get the softmodem code cleaned up enough to submit to the github project.


    Chris/akacastor


    --- Maximus 3.01
    * Origin: Another Millennium - Canada - another.tel (21:1/162)
  • From Digital Man@21:1/183 to AKAcastor on Wed Apr 17 13:39:55 2024
    Re: Solved issue with SyncTerm after up/download on my BBS
    By: AKAcastor to Digital Man on Wed Apr 17 2024 12:55 pm

    If ASCII (non-binary-tx) mode is negotiated succesfully
    when SyncTERM first connects to telnet server, then it
    should be free to return to ASCII mode later.

    ASCII mode was never negotiated successfully, the session was in binary mode from the beginning.

    ASCII mode is the default, so if there's no mention of "BINARY TX" in either direction during initial conneciton, then ASCII mode was successfully negotiated. As I recall, SyncTERM doesn't initate any explicit Telnet commands regarding BINARY TX upon initial connection, so unless the Telnet server does (e.g. sends "DONT BINARY TX"), then SyncTERM is correct in assuming ASCII mode.

    SyncTerm is incorrectly 'returning' to ASCII mode after
    file transfers.

    SyncTERM *attempts* to return to ASCII mode, but the server can definitely NAK the messages (as you've discovered).

    If the Telnet server explicitly requested the client to be in BINARY TX mode during initial connection, then I suppose it could be considered "incorrect" for SyncTERM to want to *return* to a mode it wasn't already in. But I'm not clear that's the case since I didn't see a log of telnet commands sent and received from the start of the connection.

    I don't think I see a pressing need for my system to support ASCII mode telnet as I expect any client should support binary mode. (maybe there's some edge cases in niche implementations or a benefit of ASCII mode that I am missing?)

    I guess you'll find out. :-)

    It just sounds like there's a bug in the DOSBOX-X
    terminal server with tracking the remote and/or local
    "BINARY TX" state and that needs to be fixed rather
    than just blindly rejecting requests to exit binary
    transmission mode.

    Maybe you would have had better luck with DOSBOX-
    staging instead of of DOSBOX-X?

    I agree there is likely a bug hiding in the telnet emulation, I may dig deeper for it yet. In the meantime, convincing SyncTerm to stop switching to ASCII modes removes the issue from my system.

    Telnet emulation in DOSBox-Staging is broken also. I don't know if it has the same behaviour with ASCII vs binary mode but it has other telnet protocol bugs, like sending binary data from DOSBox does not escape the IAC bytes. (due to this binary transfers work when DOSBox is receiving, but not when sending)

    I haven't ruled out DOSBox-Staging in the long term, but DOSBox-X has been working well for me and I have already been patching it to suit my needs (mostly a bit of cleanup/hackjobs in softmodem.cpp). Eventually I hope to get the softmodem code cleaned up enough to submit to the github project.

    k, cool. I might play with connecting to your BBS and see what I find wrt its telnet server and comment further.... or not. :-)
    --
    digital man (rob)

    Synchronet/BBS Terminology Definition #90:
    XON = Transmit On (ASCII 17, Ctrl-Q)
    Norco, CA WX: 79.5øF, 29.0% humidity, 6 mph W wind, 0.00 inches rain/24hrs
    --- SBBSecho 3.20-Linux
    * Origin: Vertrauen - [vert/cvs/bbs].synchro.net (21:1/183)
  • From AKAcastor@21:1/162 to Digital Man on Wed Apr 17 14:03:42 2024
    ASCII mode is the default, so if there's no mention of
    "BINARY TX" in either direction during initial
    conneciton, then ASCII mode was successfully
    negotiated. As I recall, SyncTERM doesn't initate any
    explicit Telnet commands regarding BINARY TX upon
    initial connection, so unless the Telnet server does
    (e.g. sends "DONT BINARY TX"), then SyncTERM is correct
    in assuming ASCII mode.

    The server is negotiating binary mode, as well as character mode, on connection. I imagine every BBS is at least negotiating character mode. The handful that I checked when I was initially setting my system up were negotiating character mode and binary on initial connection.

    SyncTerm is incorrectly 'returning' to ASCII mode after
    file transfers.

    SyncTERM *attempts* to return to ASCII mode, but the
    server can definitely NAK the messages (as you've discovered).

    Sure, but that's kind of hacky. hehe ;)

    If the Telnet server explicitly requested the client to
    be in BINARY TX mode during initial connection, then I
    suppose it could be considered "incorrect" for SyncTERM
    to want to *return* to a mode it wasn't already in. But
    I'm not clear that's the case since I didn't see a log
    of telnet commands sent and received from the start of
    the connection.

    Yes, that's what happened - telnet server negotiated binary during initial connection.

    I connected to Roon's BBS to compare the behaviour (also running Maximus, but in OS/2 not DOSBox) and in the SyncTerm log I see similar behaviour by SyncTerm. On connection, the BBS negotiates binary mode, after a file transfer SyncTerm requests ASCII mode, and in this case the server complies. If a second file transfer is attempted, SyncTerm will request a switch to binary mode and then after the transfer back to ASCII mode again.

    I don't think I see a pressing need for my system to support ASCII mode telnet as I expect any client should support binary (maybe there's
    some edge cases in niche implementations or a benefit of ASCII mode that I am missing?)

    I guess you'll find out. :-)

    Sounds exciting, doesn't it! haha

    Given that my BBS (and others I connect to, for example those using Vmodem in OS/2) works well in binary mode from first connection until file transfer, I don't see a reason for supporting ASCII mode between file transfers.

    I mean there is one reason to support it - for the sake of completely supporting the telnet protocol. But I feel like in this case that's only a matter of principle.

    k, cool. I might play with connecting to your BBS and
    see what I find wrt its telnet server and comment further.... or not. :-)

    I would appreciate any insight! You have a lot more experience with the protocol and what different clients/servers may expect in the real world than probably anybody here.


    Chris/akacastor

    --- Maximus 3.01
    * Origin: Another Millennium - Canada - another.tel (21:1/162)
  • From Roon@21:4/148 to AKAcastor on Sat Apr 20 23:01:11 2024
    Hello AKAcastor,

    Answering a msg of <16 Apr 24>, from you to All:

    wow. glad you found it. buuuut... it's not (only?) a dosbox problem,
    my maximus/2 behaves the same with synchterm... :)

    Regards,
    --
    dp

    telnet://bbs.roonsbbs.hu:1212 <<=-

    ... Uptime: 1d 10h 41m 40s
    --- GoldED/2 1.1.4.7+EMX
    * Origin: Roon's BBS - Budapest, HUNGARY (21:4/148)
  • From Digital Man@21:1/183 to AKAcastor on Tue Apr 23 00:29:32 2024
    Re: Solved issue with SyncTerm after up/download on my BBS
    By: AKAcastor to Digital Man on Wed Apr 17 2024 02:03 pm

    k, cool. I might play with connecting to your BBS and
    see what I find wrt its telnet server and comment further.... or not. :-)

    I would appreciate any insight! You have a lot more experience with the protocol and what different clients/servers may expect in the real world than probably anybody here.

    I looked at the SyncTERM source and determined that though it attempted to track and restore the binary mode of the Telnet connection, it didn't take into account the server initiated commands for setting BINARY TX mode (in either direction). I committed a fix for that. If you have a chance and want to try it out, it's in Git and now and should be in nightly builds (e.g. for Windows, MacOS) within 24 hours. Let me know if you do try it and what you discover. I expect it'll resolve this issue.
    https://gitlab.synchro.net/main/sbbs/-/commit/2bae9a96e637c06d8f48c8c6a3e74835 --
    digital man (rob)

    Rush quote #45:
    The treasure of a life is a measure of love and respect
    Norco, CA WX: 55.4øF, 95.0% humidity, 3 mph WNW wind, 0.00 inches rain/24hrs --- SBBSecho 3.20-Linux
    * Origin: Vertrauen - [vert/cvs/bbs].synchro.net (21:1/183)
  • From Chris Gerlinsky@21:4/148 to Roon on Sun Apr 21 01:59:42 2024
    Answering a msg of <16 Apr 24>, from you to All:

    wow. glad you found it. buuuut... it's not (only?) a dosbox problem,
    my maximus/2 behaves the same with synchterm... :)

    I tested on your board before I had posted and didn't notice trouble after a file transfer, but then I didn't test very well! The symptoms can be a bit funny can't they.

    I am not sure why Maximus doesn't like to operate within an ASCII telnet connection as I would expect the telnet interface (Vmodem or DOSBox-X in our cases) to translate anything that needs to be translated. I wonder if other BBS software has the same behaviour.

    If there's a way to send a telnet negotiation command through the Vmodem connection, you can include after file transfers (or in the top of the file menu?) the telnet IAC commands to switch to binary mode and that may solve the problem for you.


    Chris/akacastor

    --- Blue Wave/DOS v2.30
    * Origin: Roon's BBS - Budapest, HUNGARY (21:4/148)
  • From Roon@21:4/148 to Chris Gerlinsky on Tue Apr 23 20:50:54 2024
    Hello Chris,

    21 Apr 24 01:59, you wrote to me:

    Answering a msg of <16 Apr 24>, from you to All:

    wow. glad you found it. buuuut... it's not (only?) a dosbox
    problem, my maximus/2 behaves the same with synchterm... :)

    I tested on your board before I had posted and didn't notice trouble
    after a file transfer, but then I didn't test very well! The symptoms
    can be a bit funny can't they.

    I am not sure why Maximus doesn't like to operate within an ASCII
    telnet connection as I would expect the telnet interface (Vmodem or DOSBox-X in our cases) to translate anything that needs to be
    translated. I wonder if other BBS software has the same behaviour.

    If there's a way to send a telnet negotiation command through the
    Vmodem connection, you can include after file transfers (or in the top
    of the file menu?) the telnet IAC commands to switch to binary mode
    and that may solve the problem for you.

    did you try other terminal emulators or just synchterm?

    Regards,
    --
    dp

    telnet://bbs.roonsbbs.hu:1212 <<=-

    ... Uptime: 0d 12h 53m 55s
    --- GoldED/2 1.1.4.7+EMX
    * Origin: Roon's BBS - Budapest, HUNGARY (21:4/148)
  • From Roon@21:1/162 to Chris Gerlinsky on Tue Apr 23 12:49:20 2024
    side note, blue wave worked on my bbs :)

    --- Blue Wave/DOS v2.30
    * Origin: Roon's BBS - Budapest, HUNGARY (21:4/148)

    wooo! :)

    --
    dp


    --- Maximus 3.01
    * Origin: Another Millennium - Canada - another.tel (21:1/162)
  • From AKAcastor@21:1/162 to Digital Man on Wed Apr 24 17:49:20 2024
    I looked at the SyncTERM source and determined that though it attempted to track and restore the binary mode of the Telnet
    connection, it didn't take into account the server
    initiated commands for setting BINARY TX mode (in
    either direction). I committed a fix for that. If you
    have a chance and want to try it out, it's in Git and
    now and should be in nightly builds (e.g. for Windows,
    MacOS) within 24 hours. Let me know if you do try it
    and what you discover. I expect it'll resolve this
    issue.

    Thanks for that. I haven't had a chance to try it out yet but will take a look.

    I did spend a bit more time on the DOSBox-X softmodem telnet implementation, I found that my hacky approach to forcing binary mode was in some cases (Netrunner) causing an IAC command loop. So I cleaned up the tracking of telnet state and now don't send IAC WILL in response to an IAC DO if it didn't cause a state change.

    I haven't properly implemented ASCII mode yet - I have some more questions. In ASCII mode I understand a raw CR is sent as CR/NULL (or CR/LF in case of a newline). So any CR should be translated to CR/NULL and then from CR/NULL back to CR at the other end (or however that end chooses to interpret it), correct?

    Other than CR (and sequences starting with the IAC character 0xFF) are there any other characters or sequences that either need to be escaped or can't be passed via ASCII mode telnet? I'm not sure if I'm still missing other things.


    Chris/akacastor

    --- Maximus 3.01
    * Origin: Another Millennium - Canada - another.tel (21:1/162)
  • From AKAcastor@21:1/162 to Roon on Wed Apr 24 18:03:24 2024
    If there's a way to send a telnet negotiation command through the
    Vmodem connection, you can include after file transfers (or in the top
    of the file menu?) the telnet IAC commands to switch to binary mode
    and that may solve the problem for you.

    did you try other terminal emulators or just synchterm?

    I have only noticed the sequence of file transfer -> switch to ASCII mode tripping up noncompliant telnet servers with Syncterm, but I haven't been methodically testing various terminals or anything.

    Usually when I dial your BBS I use Telix inside DOSBox-X and that's working great.

    Interesting that you've found the problem happening on your system, that suggests that Vmodem has an issue with its telnet implementation, I think?

    Chris/akacastor

    --- Maximus 3.01
    * Origin: Another Millennium - Canada - another.tel (21:1/162)
  • From AKAcastor@21:1/162 to Roon on Wed Apr 24 18:09:06 2024
    side note, blue wave worked on my bbs :)

    --- Blue Wave/DOS v2.30
    * Origin: Roon's BBS - Budapest, HUNGARY (21:4/148)

    Excellent!! Y2K-compliant Blue Wave door for Maximus in OS/2. I should properly test my DOS setup and see if it's OK too. :)


    Chris/akacastor


    --- Maximus 3.01
    * Origin: Another Millennium - Canada - another.tel (21:1/162)
  • From Digital Man@21:1/183 to AKAcastor on Wed Apr 24 18:35:00 2024
    Re: Solved issue with SyncTerm after up/download on my BBS
    By: AKAcastor to Digital Man on Wed Apr 24 2024 05:49 pm

    Thanks for that. I haven't had a chance to try it out yet but will take a look.

    I did spend a bit more time on the DOSBox-X softmodem telnet implementation, I found that my hacky approach to forcing binary mode was in some cases (Netrunner) causing an IAC command loop. So I cleaned up the tracking of telnet state and now don't send IAC WILL in response to an IAC DO if it didn't cause a state change.

    Yup, that's how Telnet is: they used the same byte value for a request and an acknowledgement. Dumb design.

    I haven't properly implemented ASCII mode yet - I have some more questions. In ASCII mode I understand a raw CR is sent as CR/NULL (or CR/LF in case of a newline). So any CR should be translated to CR/NULL and then from CR/NULL back to CR at the other end (or however that end chooses to interpret it), correct?

    I've never really found a use case for translating ENTER to CR/NUL. But if the Telnet server receives CR/NUL *or* CR/LF (while in ASCII mode), it should treat that as a bare CR.


    Other than CR (and sequences starting with the IAC character 0xFF) are there any other characters or sequences that either need to be escaped or can't be passed via ASCII mode telnet? I'm not sure if I'm still missing other things.

    That's it.
    --
    digital man (rob)

    Sling Blade quote #9:
    Doyle Hargraves: Morris here is a modern-day poet, kinda like in olden times. Norco, CA WX: 61.8øF, 67.0% humidity, 3 mph WNW wind, 0.00 inches rain/24hrs --- SBBSecho 3.20-Linux
    * Origin: Vertrauen - [vert/cvs/bbs].synchro.net (21:1/183)
  • From AKAcastor@21:1/162 to Digital Man on Wed Apr 24 18:46:06 2024
    I found that my hacky approach to forcing binary mode was in some cases (Netrunner) causing an IAC command loop. So I cleaned up the tracking of telnet state and now don't send IAC WILL in response to an IAC DO if it didn't cause a state change.

    Yup, that's how Telnet is: they used the same byte
    value for a request and an acknowledgement. Dumb design.

    I fully agree!

    Other than CR (and sequences starting with the IAC character 0xFF) are
    there any other characters or sequences that either need to be escaped or can't be passed via ASCII mode telnet? I'm not sure if I'm still missing other things.

    That's it.

    If both ends properly translate CR to CR/NULL (and back), doesn't that negate the need for binary mode? (or is both ends properly translating it hoping for too much?)


    Chris/akacastor

    --- Maximus 3.01
    * Origin: Another Millennium - Canada - another.tel (21:1/162)
  • From Digital Man@21:1/183 to AKAcastor on Thu Apr 25 15:01:23 2024
    Re: Solved issue with SyncTerm after up/download on my BBS
    By: AKAcastor to Digital Man on Wed Apr 24 2024 06:46 pm

    If both ends properly translate CR to CR/NULL (and back), doesn't that negate the need for binary mode? (or is both ends properly translating it hoping for too much?)

    Technically, binary TX mode is supposed to be required (according to the original Telnet RFCs) for 8-bit character transmission. In modern times (and clarified in RFC1123), 8-bit characters are sent/received in Telnet ASCII mode without issue. So yes, I agree, if both sides correctly encoded/decoded CR<->CRLF, theoretically, binary TX mode should not be needed.
    --
    digital man (rob)

    This Is Spinal Tap quote #10:
    Dozens of people spontaneously combust each year... just not widely reported. Norco, CA WX: 57.4øF, 68.0% humidity, 2 mph WSW wind, 0.00 inches rain/24hrs --- SBBSecho 3.20-Linux
    * Origin: Vertrauen - [vert/cvs/bbs].synchro.net (21:1/183)