BTW, OC code seems to be running complex reformatting code before writing AND 
after reading this value. They never simply assume it to be plain-text. :-/

All the above was checked against OCPL code only, I don't know about OCDE 
branch yet.

OCPL stores HTML-lized comments in the database (it doesn't really 
matter if 'text_html' field is set to FALSE). OKAPI must save it in HTML 
either way. However, escaping plain-text doesn't work. If we put 
"<b>" in, it still gets converted to "" before display! NONE of 
this process is documented. There doesn't seem to be ANY way to force 
OCPL to treat a string as either plain-text nor HTML. It's always 
something in between! In other words, we cannot guarantee how it gets 
displayed in the end. Since text_html=0 doesn't add <br/>s on its own, 
we can only add proper <br/>s and hope it's okay. 

This issue was closed by revision r388.

Actually, it is NOT fixed. It is now "more consistently broken".

I have verified this with OC.de and came to the conclusion that the log storage 
format is odd, but it is not broken.

There are two types of log formats:

a) plain text logs: text_html=0, and the text is stored in a "HTML display" 
form, i.e. with HTML entities encoded and line feeds replaced by <br />. The 
application will either display it this way, or decode this to its original 
form if necesseray, e.g. when the log is to be edited or another encoding is 
needed.

b) HTML logs: text_html=1, and the text is stored as HTML code as provided by 
the user.

The current OKAPI implementation correctly creates the form (a). In the code 
comments, you write:

# If we put "<b>" in, it still gets converted to "" before display!

This is not true for OC.de - the log texts are passed to the web browser 
unencoded*, and the browser will display it as plain text "". I can hardly 
believe that OC.pl decodes this to "" before output ...

* Actually, OC.de code does two modifications when displaying text_html=0 logs: 
it replaces ASCII smileys by images, i.e. ":-)" by a <img 
src=".../smiley-smile.gif" />, and it inserts <a href...> tags for hyperlinks.

# (it doesn't really matter if 'text_html' field is set to FALSE)

text_html MUST be set to 0 for plain text logs - otherwise they would be 
presented to the user in their HTML-encoded form when re-editing the log!


Conclusion: The services/logs/submit documentation is not correct at least 
regarding OC.de and should be updated.

Enabling HTML logs by OKAPI would be nice, too. :-) Of course, implementation 
would be complex: It would need a HTML purifier, like in the OC or GC code, 
which strips "bad tags" from log texts and maybe closes open tags (the latter 
one still is missing at OC.de, but GC does ist).

> the log texts are passed to the web browser unencoded

sorry, this should read: "the log texts are passed to the web browser 
unDEncoded"

Unfortunatelly, I don't have time to dig into it again. If you want, you may 
fix the docs, but I think you should assume that comment 3 is true for OCPL (I 
am not 100% sure, but usually I don't make mistakes when I check for such 
things).

This issue was updated by revision r604.

The new comment_format option is broken: HTML comments must be saved with 
text_html=1, not text_html=0!

If you set text_html=1 for comment_format=HTML, everything is fine. I have 
tested this on OC.de and verified in OC.pl source code - there is no decoding 
when displaying cache_logs.text, just smiley replacement and inserting 
hyperlinks.

> If you set text_html=1 for comment_format=HTML, everything is fine. 

Rethought: It is NOT fine, because HTML data must not be inserted as it is into 
the database. It must be purified first to ensure that it does not contain 
malicious tags, javascript code etc. There is no purifing afterwards before 
display!

The comment_format option in this form is not useful: with text_html=0 it is 
broken, and with text_html=1 it would be dangerous. It should be discarded and 
not be reimplemented before OKAPI includes an HTML cleaner.

I don't follow. This parameter is ignored, and the documentation states it is 
ignored. What exactly do you want me to do?

This parameter encourages developers to submit HTML log comments, which does 
not work.

> What exactly do you want me to do?

Remove the comment_format parameter and document that OKAPI currently does not 
support HTML logging.

Just to clarify:

1. I insert a new log entry: http://i.imgur.com/sGFcSqT.png
2. In the database it has text_html=0: http://i.imgur.com/Qe8FfeJ.png
3. It is still displayed in HTML: http://i.imgur.com/jGUMUiD.png

I don't see a way to fix it in OKAPI.

> Remove the comment_format parameter and document that OKAPI
> currently does not support HTML logging.

But it kind of does, and I can't turn it off. As I said, many developers 
started to use this "hidden" functionality already. I would like them to 
indicate that they are doing it on purpose, that's why I added this parameter.

Perhaps, it should work differently on OCDE, but I think it is the best way for 
OCPL.

Shit, then I must have overlooked some decoder in the OC.pl code. :-(

Sry, first screenshot was wrong, I had submitted the log entry as Text and 
later faked it because I had saved it to some wrong directory and did not find 
it when uploading here. ;-) This is was I did as step 1:

Here it is, in OC.pl code

viecache.php:
  $rs = sql("SELECT ... `cache_logs`.`text` `text` ... FROM `cache_logs` ...");
  $record = sql_fetch_array($rs);
  $tmplog_text = $record['text'];
  $tmplog_text = str_replace($smileytext, $smileyimage, $tmplog_text);
  $tmplog_text = help_addHyperlinkToURL($tmplog_text);
  $tmplog_text = tidy_html_description($tmplog_text);

common.inc.php:
  function tidy_html_description:
    $tidy =  tidy_parse_string(html_entity_decode($text, ENT_NOQUOTES, "UTF-8"), ...);

This is not done in OC.de code.

This issue was updated by revision r610.

To sum up: Currently, OKAPI submits HTML logs for OCPL, and PLAINTEXT logs for 
OCDE. The 'submit' docs are still unclear on that.

Alternative:

Could you implement HTML for OCDE? If you could, then I would also try to 
implement PLAINTEXT for OCPL. This way, comment_format would become valid.

I could implement it if you allow me using this code from within OKAPI:

https://github.com/OpencachingDeutschland/oc-server3/tree/master/lib/htmlpurifie
r-4.2.0

This library is stable, though we have lots issues on improving HTML 
purification in OC.de todo list, either because it throws away tags or 
attributes which need not to be thrown away, or because it does not close open 
tags. This might be solved by

- updating or customizing htmlpurifier-4.2.0, or
- looking for a replacement which does it better, or
- writing a own one,

but that is not on my todo list for 2013.

Okay. Since this library would be needed for OCDE only, you could include it 
directly from OCDE code (without committing it to OKAPI). BTW, OCPL's purifying 
is crappy too.

This issue was updated by revision r637.

Fixed in r642.

Take a look at it. I modified OCDE part slightly, but I tested on OCPL only.

This issue was updated by revision r644.

Documentation should make clear what 'auto' is. On OC.de it must NOT be used on 
plaintext comments, because the HTMLPurifier might strip things from the text 
which look like (unwanted) html tags.

They are many existing apps which do not know about comment_format's existence. 
Some of them submit plaintext, some of them submit html. We need to add the 
auto option in order for these apps to work properly with OCDE (and to be 
backward compatible with OCPL too).

> On OC.de it must NOT be used on plaintext comments

It is not used for plaintext comments, just the "auto" ones.

BTW, with this configuration (auto paragraphs), HTMLPurifier works fine for 99% 
of usual plaintext entries: http://i.imgur.com/T0897Cc.png

I added a link to this issue, so that future developers may know exactly what's 
going on. Fixed in r646.

Thanks for the documentation update and thumbs-up. :-)

> HTMLPurifier works fine for 99% of usual plaintext entries:

Ok, but that won't help if the HTML entities are not encoded after purifing. 
Auto MUST NOT be used on plain text on any OC node except the current/old OC.nl 
installation, i.e. all current applications which rely on 'comment' field to be 
plain text format (except on OC.nl) are broken should be fixed.

I don't understand. Give me an example - when does something break?

Adding nl2br() for 'auto' option (r650) resolves the main problem, i.e. 
eaten-up line breaks.

Here is an example for the remaining problem with auto + plaintext. Surely it 
won't happen often that someone enters plain text which looks like HTML tags. 
Someone hit the wrong key - 'a' is directly besides of '<':

Log text:
"We went <a 10 km but far > 5 km to find the cache."

OKAPI submit request: 
http://local.opencaching.de/oc-server/server-3.0/htdocs/okapi/services/logs/subm
it?cache_code=OC10004&logtype=Comment&comment=We+went+%3Ca+10+km+but+far+%3E5+km
+to+find+the+cache.

Result:

This issue was updated by revision r651.