BRB, TT and SR App Refresh

This week we finished a major refresh of our main scripture apps. This blog goes over what we have done, explaining why we've made various changes. Weekly updates resume today.

Our Tagging System

We have a tagged file format that we use for markup in all of our Bible texts. It is a simple set of 2 character tags that are inside angled brackets just like html. By limiting tags to 2 characters they do not overwhelm the English prose. These are similar to html tags but they are much more dense and tuned to the problems of marking up scripture.

We originally saw this general approach in one of the early open source Bible texts we found on the internet years ago. We liked it because it was much more dense than html markup. It allowed easy editing of the English without getting in the way. It was easy to parse and convert into other formats.

The original use of these tags was for the address system, mostly chapter and verse marks, and for basic formatting, in particular paragraph breaks. For tags that span text the opening and closing forms are marked by capitalization, a strange and useful convention.

Over the years we added several more tag series. Initially we added markup for keywords, then more tags for marking speaker voice, then quote links then editor filters.

We have several more new types of markup planned. The need to get those tags running is why we needed to rework our tooling that converts those tags into html. This is what we have been working on for the past several weeks.

HTML Conversion

Tag systems in running prose text do not naturally nest. The 4 main tag families, keywords, voice, quotes and filters, can turn on and off at any given place in the running text, without any concern about what is going on with the other markup. This is the nature of running text. It is in part based on the history of the text itself.

HTML has a similar set of non-nesting tags, 'i' for italics, 'b' for bold, 'u' for underline and 's' for strike through. These tags can be used at any point in the flow of text. They change the style of the text that follows, but they combine with any currently open tags.

Most other html tags are strictly nested, including the common 'div' and 'span' tags. They do not combine across the flow of text, but based on their hierarchy in the document.

The non-nesting tags in html were the obvious original choices for mapping the non-nesting tags in our scripture markup. We have used them since we first started doing this work years ago.

This old choice has 2 fundamental problems. The first is that the default formatting in HTML for these tags rarely matches the formatting implied by the tags we use in the text. So the style system must turn off default html formatting. This adds complexity to style sheets. It also does not look good if there is no style sheet support, say in email.

The other problem is more complex. When quoting bits of scripture, say in a quote box, the current set of earlier formatting must be discovered and then properly started and stopped at those quote box boundaries.

The code for this has always been a mess. Adding new tags has always meant trouble with changes to fragile code. There had to be a better way to think about the problem.

Intermediate Forms

In the past 3 years I have done a bunch of work on merging Aramaic texts into Paleo. That work caused me to need an intermediate form that could be reworked several times in memory before the output of the merge is created.

Use of an intermediate form made much of this other code much easier to write and reason about. So as I have been thinking about the problem of our regular English markup I decided it needed to be based on an internal intermediate form too. In other words a 2 stage conversion process rather than 1 stage as has been previously done in our English texts.

This 2 stage conversion allows all manner of hard conversion problems to become trivial to solve. It does take more memory, but computers these days have lots of memory, not a problem.

The use of an intermediate form also means that all active formatting for any given bit of text can be merged together before html generation. So there is no longer any reason to use the non-nesting html tags like i, b, u, and s. Everything now just runs with properly nested html span tags. The style system is conceptually much easier because there are no default styles that must be turned off. So there is now hope to eventually see scripture quotes in email.

Typographical Changes

The most dramatic change you will see in our scripture apps starting this week is much better typography.

This is because there is essentially perfect styling control over every single bit of the generated inline scripture text. Ryan has been busy reworking the details of this as fine grain style control finally arrived.

The most dramatic difference you will see is that nothing in any given line of markup causes line spacing to change. So the superscripts of verse numbers and the filter medallions are no longer causing line height jitter.

There is now essentially perfect control over the flow of letters and spaces in the HTML. This lets the formatting engine in the browser always do the right thing in terms of line breaks and spacing. One example to note is how filter medallions at the start of new lines are now always correctly lined up with the left edge of paragraphs.

Ryan has slightly expanded the line height to follow recommended settings for text in browsers. This will make the text easier for some people to read. The range of possible font sizes is still 7, but the differences between each size step is now smaller. This should make the more extreme font sizes more useful to more users.

Other Changes

Once the fundamental style system was being reworked, we have reworked styles across the apps themselves. These are subtle, but you will notice them in the apps as you mess around.

Previously, the scripture apps have had help text spread around in various places. Help text was there for new users but cluttered the experience of regular users. We have added a series of circled information markers, (i), that can bring up help when needed. This cleans up the overall look and feel.

Themes

For several years we have had both light and dark lighting options. Those options have always been called 'lighting' in the top right menu. They were always under direct user control.

This update changes the name to 'Theme' and it now has 3 different choices. By default all of our apps follow the theme of the underlying device. This choice is called 'device.' If someone wants to override this, and force either a light or dark theme, then this is possible to do in the Theme menu choice.

Most devices report their theme to the app. They do this in real time. Some devices switch to dark theme at sunset, and then back to light theme at sunrise. They notify apps so they can follow along. We now support and respond to these notifications when the 'device' theme is selected.

A few places still do not support device themes. The Brave browser introduced a regression in the past couple months where it no longer reports device theme to the app. It only reports its own internal theme. That is a bug outside of our control. In any case, most of the time, device theme as the new default just works.

Lots Changes

The lots popup is a popular feature of our scripture apps. Many of us use it all the time.

We have made a few changes with this release. Let me explain.

First we moved the directions behind an (i) medallion. This to clean up that popup for regular users.

Second, we fixed a bug related to opening up the scripture apps to a random page by down-swiping the front cover. This was there originally and stopped working at some point. It is now working again.

Third, we have removed the 'via wind' choices in the lots popup itself.

This was added after reading the biography of William Branham. In that biography we learned of how he had been camping and how the wind opened his Bible to important passages.

This of course cannot work in a software based version of the Bible. So the source of the wind was shifted to the clock. Technically this was an interesting solution, and we have had it as an option for several years.

But...

This is not a feature that Branham himself would have used if he had a copy of one of our scripture apps on his phone. We believe he would have simply used the regular lots calculator using a regular number.

We have also seen the wind choice used in ways that are counter to its intent. Asking Joshua in prayer for a number is the starting point of asking him other questions in prayer too.

The obvious point is that no number may come at all. This either means Joshua does not want to speak or someone does not want to listen. Because Joshua can speak loudly, we consider no number to be a case where there is no lot. This is a feature of the normal number buttons.

Alternatively, the wind buttons could always be pressed, Joshua could be forced to apparently speak. This is a big fundamental problem with the wind style lots buttons. So they are now out.

Fanning The Bible

Fanning the Bible, and opening it to a page, is still an interesting traditional way to find a random location. We still support that use case. Use the front cover and swipe down for that scenario. (Or click the front cover on non-touch devices.)

In this use case the target is an entire chapter in the BRB or story in the TT. The reader must then pick out what applies, if anything. This is a more traditional way to test scripture knowledge or to find a daily devotional. The clock is used to generate the eventual page. Using the clock for this is a reasonable software match to quickly fanning pages in regular paper Bibles.

Search Changes

We have rewritten the search functions in both the BRB and TT apps. This is an important function, and as I commented in the last blog, it was old code.

It is now fast enough to index all words. In previous code a list of about 10 words were not included in the index to limit the size. Now all words are in the index, no exceptions.

'The' is the most common word, used in the BRB over 24,000 times. 'And' is the second most common word, used in the BRB over 21,000 times. It was faster to include them in the index than to leave them out, which was curious. Searches which would result in over 10,000 results are thus now possible.

But, in testing, these lug down the browser's ability to display the result. So, searches with over 10,000 results are prevented.

These high-use words can be combined with other words which is still an interesting use case. Searches like 'in the house' will find all verses where all 3 words are found.

The word 'master' occurs over 8600 times in the BRB. This word is the highest occurrence word that has important meaning to someone doing searches. So the 10,000 limit on search results is not a serious constraint. Searching on 'master' still pushes most browsers.

'The' is still the most common word in the TT app. It occurs over 2400 times in the TT. This is the upper limit on the largest possible length of the search results. As this can be handled by modern devices there is no search result limit error possible on the TT app.

The search code is now mostly shared between both apps. They differ some because of the different address systems of the 2 apps. So the search code will be easier code to maintain in the future.

The new search code has been updated to work better in many scenarios. Searches like 'mother-in-law' now work correctly, even including the hyphenation. Searching 'mother' alone will also find the word 'mother' as used in 'mother-in-law.' Numbers can now be searched with or without embedded commas. Though if commas are used they must be correctly placed.

The TT app now has better headings in the search results, calling out book and story headings in the search results. This is a nice new touch.

More Planned

We have more work planned in our family of scripture apps. I am expecting to be working with Ryan on this for another week or so. Ryan can now start working on new things, which I will be adding code to support.

Shop Work This Week

The last bits of optional hardware for the new Voron printer was finished this week. The printer now has chamber lighting and a webcam for remote monitoring of the printer.

The software configuration is also mostly finished, including the exhaust fan and LED status control on the print head.

Several more test prints have been run, including the first big piece used to tune the printer. The tuning work still needs to be finished. I should be returning next week to my normal routine of code work in the mornings and shop work in the afternoons.

More Later,

Phil