QBX: Progress Update

In the course of Discord updates about QBX, it came to my attention that it had been about a month and a half since I'd made any update in that forum. Development had definitely not been stagnant, so I decided to review changes since the previous update there -- about 6 weeks prior. Here's what happened with QBX over the course of the past 6 weeks:

• Over 450 commits.
• The completion of INT 21h emulation, along with a comprehensive test suite that verifies the interrupt interface layer on top of the DOS kernel.
• Maturation of the DOS kernel's file I/O layer, including, to some degree, the Short File Names support for non-Windows platforms.
• File I/O support in the QuickBASIC parser & interpreter.
• Lots of IDE bugs fixed. Lots still remain, I'm sure (as well as gaps).
• Improved number formatting & parsing.
• Conversion of numbers to/from strings (CVI, MKL$, etc.), including to/from MBF binary representation for floats (CVSMBF, etc.).
• Help file parsing & display (I don't distribute the HLP files, but if you give it the files from the QB71 distribution, it'll display them.)
• On that note, contextual help for menus and implemented dialogs, and F1 to jump to the topic for the word under the cursor.
• Partial support for having files loaded that aren't part of the project source (such as .TXT files).
• Event registration & dispatch.
• Proper handling of error handlers and event handlers in cross-module situations.
• New core dialogs, such as the F2 "SUBs" dialog.
• One small but key detail in line parsing/formatting: End-of-line comments stay at the column they're at unless the reformatted code bumps them to the right.
• Support for "pinning" variables to memory locations (VARSEG/VARPTR, SSEG/SADD) so that buffers can be passed to interrupts, etc.
• BLOAD/BSAVE support.
• COMMON blocks for sharing variables between modules in a multi-module program.
• Proper segregation of namespaces between modules, including allowing SUBs and FUNCTIONs to be declared with differing but semantically-identical TYPEs.
• VARPTR$() for "X" PLAY and DRAW commands.
• VGA ROM font at segment F000.
• Starting SUBs and FUNCTIONs by typing the opening line.
• STATIC variables & SUBs.
• Identifier canonicalization.

So, you know. A few things. :-)

QBX: Help Files

This makes me really happy 🙂

https://youtu.be/GBkzloes26U

QuickBASIC comes with help files that are encoded in a proprietary binary format (with two layers of compression, no less!). Fortunately, someone out there put in the work reverse-engineering the format and wrote a document accurately describing how to decode and interpret the bytes. The document does an excellent job describing what could otherwise be a nightmare of complex binary formats.

The QuickHelp format employes two layers of compression. First, the text is compressed using a combination of tokenization and run-length encoding.

The tokenization involves identifying words or phrases that are repeated and adding them to a table, after which every instance of them in the text can be replaced with a reference to that table. Run-length encoding is much simpler: If you see the same byte multiple times in a row, just encode how many times it was repeated. A horizontal rule of 78 horizontal line characters can then just be "repeat this character 78 times" rather than literally 78 characters directly.

Then, after the tokenization pass, the resulting byte stream is compressed using Huffman compression. Huffman compression is based on a simple idea: Instead of using a rigid scheme of 8 bits for every byte, use fewer bits for bytes that show up more commonly, at the expense of rare bytes which then take longer than 8 bits. As long as there is a noticeable bias to some byte values, it can quite effectively compress things. As with many compression algorithms, Huffman compression requires you to treat a file that is really a stream of bytes as a stream of bits, automatically transitioning from one byte to the next as needed.

To get back the original data, you have to apply these steps in reverse, first the Huffman compression and then the Keyword compression. If you get even one detail wrong, everything after that point will almost certainly be indecipherable noise. Fortunately, though, the documentation I found was precise and accurate enough that it was relatively straightforward to write the code and it works a treat. 😃

Once the encoding in the file is sorted out, you then have the semantic meaning of the data to worry about. In a QuickHelp database, as it's called, there's a list of Topics, and each Topic can be linked to by one or more Context Strings. Within the text of a Topic, each line stores its text as a series of spans, each with its own formatting, and then a series of links, each of which specifies a start & end character on the row and the Context String or Topic Index (index into the list of topics) to which to link.

The lookup of help topics for keywords is pretty simple. The context strings simply are the keyword. But, help topics providing contextual help for menus and dialogs are a bit less obvious. Reverse-engineering the mappings required some trial and error with the actual QuickBASIC, checking which help pages with what text appeared from each dialog. QBX only has a small subset of the dialogs anyway, but it was important that the help context strings be mapped correctly.

In this video, the program you're seeing is my QBX project, but the help data is coming from the actual BAS7QCK.HLP file from a QuickBASIC 7.1 installation.

QBX: File I/O! More generally, DOS INT 21h!

All has been quiet on the QBX-tern front, as one might say, for a few weeks now. The reason is that I've been coding my butt off creating an INT 21h DOS API interrupt emulation layer and a test suite that exercises every single function. I finished that a few days ago, and now QBX implements file I/O on top of that:

Since QBX implements file I/O using the emulation layer, it should be possible for a program that mix-and-matches QuickBASIC's file access abstraction with DOS interrupt calls to work.