My favorites | Sign in
Project Home Downloads Wiki Issues Source
New issue   Search
  Advanced search   Search tips   Subscriptions
Issue 147: Syntax improvements and support for math environments
3 people starred this issue and may be notified of changes. Back to list
Status:  Done
Owner:  ----
Closed:  May 2012

Sign in to add a comment
Project Member Reported by, Nov 26, 2010
What steps will reproduce the problem?

I am too lazy to go over issues other people reported, or requests for enhancements that are related to the fragility of current syntax and its implementation. Main problems: 
 - often not possible to nest environments
 - often not possible to escape certain characters that are used as tags
 - hard to predict which environment will "win" if many tags are present on a line
 - quite easy to get environments "crossed" (and generate broken HTML) 
 - difficult to extend the syntax without introducing many more problems of the above types
 - not really possible to use the known Javascript solutions for including mathematics in wiki files without a minimal syntactic support from vimwiki

To make the syntax and behaviour predictable, easy to maintain and extend, the syntax should rely on `syntax region` instead on `syntax match` for most environments: this is the only practical way to avoid most of the unpleasant problems mentioned above.

What version of the product are you using? On what operating system?
version 1.1.1
let g:vimwiki_camel_case = 0

Please provide any additional information below.
I have tried to address some of the problems by changing the files

syntax/vimwiki.vim, syntax/vimwiki_default.vim

Headers, text styles and code/pre regions are defined, as well as $math$ and \[display math\], so that it is possible to nest the regions where nesting seems sensible. No assumption on what is inside the math environments is made (except that \ and $ are escaped by prepending \).

I have more or less not touched links, lists and tables, since I assume a lot of vimwiki code deals with these and no one would like to see that broken. But they clearly need the same treatment if vimwiki is to become more robust and flexible.

In testing it on my wiki files, this new highlighting does not seem to break anything (but I do not have many tables, and do not use CamelCase auto-links). With a complicated markup, these new syntax files are a huge improvement. With a rather chaotic markup, vim does not seem to always work perfectly (or perhaps I do not know how to fine-tune the region's definition), but it is a world of difference already compared to the current (ver. 1.1.1) situation.

Attaching four files:
# vimwiki01.vim             (this can replace  syntax/vimwiki.vim)
# vimwiki_default01.vim     (this can replace  syntax/vimwiki_default.vim)
#            (a small test file)
# QuickTest.png             (two views of the test file in vim)
One view shows the full entered text (conceallevel=0), the other view shows what conceallevel=2 looks like when a few more `conceal cchar=..` syntax matches are loaded (these are not included in vimwiki01.vim, as this is by far not the most important thing vimwiki needs right now).    

Just to be clear: these files only change what it looks like in Vim. Vimwiki will use its own "syntax" when doing :Vimwiki2HTML, so you will *not* get what you see unless you use a very simple markup only.
(that is also why the attached syntax files are so large: I had left most of the original regular expression stuff there so that Vimwiki can still function).

Is there any willingness to make some pretty significant changes to the Vimwiki code? In the end, making Vimwiki's "parsing" behave like Vim's region parsing would make it incredibly simple to support different syntaxes and extending the current one(s) to support other people's needs.
It would not surprize me if it actually led to a noticeable reduction in the code size (I mean for a similar feature set).

By the way, supporting math is a pretty important issue, it is nowadays pretty much a standard feature for wikis. There is no comparison between a math environment and "features" such as underlined, deleted or whatever text. Dealing with the actual formulas is outsourced to some local Javascript or a server script (that is, other developers), we just need to be able to mark up something as math. Our aim should be MathJax, it is apparently quite flexible, so people using math lightly may get away with a very simple setup.
15.6 KB   View   Download
8.0 KB   View   Download
2.1 KB   View   Download
153 KB   View   Download
Nov 26, 2010
Project Member #1 habamax

I really appreciate what you have done so far. I will check the code (not sure when - I do have quite a heavy load of stuff to do at my current job) and try to give a feedback.

And yes, I am not against any significant changes to vimwiki's code. 

BTW, I'd like :Vimwiki2HTML to be consistent with vimwiki's default syntax as much as possible.

Status: Accepted
Nov 26, 2010
Project Member #2 habamax
(No comment was entered for this change.)
Labels: -Type-Defect Type-Enhancement
Nov 26, 2010
Project Member #3 habamax
Would you like to have committer role?

It would be nice if there was a branch with all of your changes we could merge when it is time.
Nov 26, 2010
Project Member #4
The 'code' I added is really simple, but very hard to read due to the fact all the regexes are in a different file (I'd estimate that if written as a normal syntax file, it would be perhaps 50 lines, plus about the same or a little less for the things I have not done yet: lists etc.). I just tried to follow the idea that a perhaps in the future the same syntax file could work with slightly different regexes as well.

But I would like to take that much further, and make a translator to HTML that would work without using not much other information but the syntax file and a little extra information, this would make changes and some feature additions very simple (for instance, adding math support would be 10 lines of extra code and then just to write up documentation for users of vimwiki how to configure suitable Javascript). Here is the essence):
 - for the translation to HTML (or other formats in the future), vimwiki code should focus on correctly recognizing the main block environments; all the inline ones (that are presently limited to be on one line of the wiki source) could be handled using vim's `synstack()` and `synID()` functions (instead of a whole lot of matching that seems to have lots of hardcoded regexes, and not just the ones from the syntax file, and that attempts to replicate what vim's syntax matching can do based on the syntax file); perhaps one should try to make it match all environments, block or inline. So what saying: any explicit matching should be done just as an optimization, to skip over a text that cannot introduce a new (or terminate currently active environment) tag, in order to reduce the amount of calls to synstack and synID.
 - acually, it would not be the first time something of this sort has been done: see the vimscript `syntax/2html` <!--(maybe `matchit.vim`)-->: from about the line 820 (before this line, it mostly deals with folds and colors); the main loop is about 100 lines of code, a lot of which deals with diff and virtual columns and other options and settings Vimwiki2HTML does not care about
 - according to the official documentation, `:TOhtml` is "very slow", because it (among other things) walks over all characters one-by-one and calls `synID` on each of them to see if it has changed; in a vimwiki, inside most environments, one can safely skip over all "nonspecial" characters... well, if it wasn't for such silly things as `VimwikiTodo` or URLs or automatic CamelCase links (these are probably the only real problem, the previous ones are caught because of :, not to mentions the current `<blockquote>` mechanism that does not seem to have any vim syntax marking at all), which would cut down on the number of (presumably expensive) `synID` calls.
- running on a test file, it turns out Vimwiki2HTML could be 50 or 80% slower than TOhtml, when I simply run synID for each character of the file, we get a tiny fraction of translator's times (perhaps 10%), so it seems that this should be really the way to go. If this can be made to work, and I thing it can, adding a support for another (not too different) wiki syntax would be a matter of minutes (plus hours of testing). Of course, tables or lists that have a lot of special handling while writing are another story.

Nov 26, 2010
Project Member #5
About the commiter role: right now I cannot see how much time I will have to play with this. If I manage to knock up a short, more-or-less working HTML translator, I will let you know.

But making a branch that could later be "merged": I do not see that as too promising, because I am just trying to make a more solid foundation for the project, I will almost certainly never have time to learn the workings of the current code (4000 lines?) and trying to work with that: it would be much easier for someone who already knows the code to start moving it in the more promising direction, which is what I am proposing. Foundation would be hard to "merge" to an existing project, the project has to be moved onto the foundation.

I will let you know if I have a working germ of a translator, I am pretty sure that it is possible, but I want it much shorter and easier to than over 1000 lines of code the syntax/2html or Vimwiki2HTML have. I am just a little worried about some of the current syntax rules that are not very well thought out: these design issues could result in a very large penalty in terms of code complexity, so if there is going to be no willingness at all for adjusting the worst issues with the current syntax, I may run out of time before it is in a useable form for general use. With a "rational" wiki syntax, I'd estimate that 150 lines of vim code could easily generate the HTML tag tree, but I do not know how much it takes to correctly escape all the characters that need it... I am afraid there may be a lot of cases that require special handling (since HTML is not quite something that would pass as a rational design either).

I am attaching the QuickTest.png that is much better cropped, I had no idea the original one I posted was double the size. Should I delete the old one? It seems I could, but cannot change the attachment in the original posting.
129 KB   View   Download
Jan 16, 2011
Project Member #6 habamax
 Issue 170  has been merged into this issue.
May 4, 2012
Project Member #7 habamax
We'll reopen it if needed with issues specified.
Status: Done
Sign in to add a comment

Powered by Google Project Hosting