[PATCH] assert_tag erroneously warns on valid HTML4

A valid HTML4 document

The attachment was validated (independently) using the www.w3c.org html validator.

I've started seeing these all over the place with Typo also, even on XmlBuilder-generated files which pass multiple rounds of validation before being handed to assert_tag.

Unfortunately, html-scanner is not a complete HTML parsing solution. It only works with well-formed XHTML. I have added documentation to assert_tag to make these precondition explicit:

      # <strong>Please note</strong: #assert_tag and #assert_no_tag only work
      # with well-formed XHTML. They recognize a few tags as implicitly self-closing
      # (like br and hr and such) but will not work correctly with tags
      # that allow optional closing tags (p, li, td). <em>You must explicitly
      # close all of your tags to use these assertions.</em>

If someone were to submit a patch that extended html-scanner to handle optional closing tags, that'd be great.

THAT IS NOT THE PROBLEM.

This has *NOTHING* to do with well-formed [X]HTML--with Typo, we're seeing this with the output from rxml files.

The scanner is broken somehow. It's griping about problems that simply don't exist in the original XML--we don't have mis-nested tags at all, yet we're still seeing these assert_tag warnings.

I'll see what I can do to build up a simple test case and post it here later today or tomorrow.

Scott, Jamis mass-closed a bunch of tickets related to assert_tag and XHTML compliance, as you may have noticed. Do proceed to look into the trouble with html-scanner, but please keep these forums civil. Thanks!

Any progress on this defect? Is it related to the partial-versus-exact match issue?

Is this also a problem with assert_select or can this slowly be forgotten as assert_tag fades into the background?

Replying to jarkko:

Is this also a problem with assert_select or can this slowly be forgotten as assert_tag fades into the background?

Does assert_select erroneously warn on valid HTML4?

assert_select uses, underneath, the same HTML parsing library as assert_tag, and so has the same restrictions on input. You must use only valid XHTML, which means you must not use implicit closing tags. Tags that, by definition, have no closing tag (like link, br, hr, and so forth) are correctly parsed, but tags that (according to HTML4) have optional closing tags (like li, p, and such) must be explicitly closed.

Finally, note that XML documents that include tags that, according to HTML, must not have explicit closing tags (e.g. link, br, etc) will not be parsed correctly, by default. You ought to take a look at http://weblog.jamisbuck.org/2007/1/4/assert_xml_select to see how to use assert_select (and assert_tag) safely with XML documents. (It's a feature, not a bug.)

Regarding the optional closing tags, in my opinion, it's just good practice to close all of your tags explicitly. However, if someone were to submit a patch that was fully backwards compatible and supported all optional closing tags, it would certainly be considered.

Lastly, if anyone can demonstrate a case where a valid XHTML document is emitting warnings incorrectly, please do report it. Until then, I'm closing this ticket (again).

Replying to scott@sigkill.org:

THAT IS NOT THE PROBLEM. This has *NOTHING* to do with well-formed [X]HTML--with Typo, we're seeing this with the output from rxml files. The scanner is broken somehow. It's griping about problems that simply don't exist in the original XML--we don't have mis-nested tags at all, yet we're still seeing these assert_tag warnings. I'll see what I can do to build up a simple test case and post it here later today or tomorrow.

Repopening, since the reason this bug was reopened has apparently not been fixed or commented upon.

"we're seeing this with the output from rxml files...It's griping about problems that simply don't exist in the original XML"

My understanding was this was reopened because someone was getting warnings when scanning XML documents, which I addressed in my last comment.

I won't close this right away, though...if nothing else has been said about XHTML documents causing warnings, I'll close again in a few days.

Make HTML::Document#initialize default xml argument to true if text has <?xml instruction

Jamis, I added a patch that makes HTML::Document#initialize default the xml argument to whether the text has the "<?xml ...>" instruction or not. That way the parser will still be backwards compatible with HTML 4 (which is not valid XML) and in addition work with XML in general.

Notice that when we say in the documentation that the parser only works with strict XHTML, that is not quite true, since we are self-closing certain tags (in the method HTML::Node#childless?) in order for the parser to work with HTML 4, which is not valid XML and doesn't require the trailing slash to close elements. This backwards compatibility makes sense but should be reflected in the documentation.

I realize the parser was intended to parse HTML and not be a generic XML parser. However, many Rails developers will expect to be able to use assert_select on their XML feeds and other XML documents (I'm using it for VoiceXML) and so are likely to continue to run into warning messages and/or broken behaviour in the future.

I was hoping that with my patch (see uploaded html_document.patch) we are covering all the bases and no workarounds or monkey patches will be needed.

Just to clarify, as far as I know, the parser currently works correctly with HTML and XHTML, and so in effect, does what it's advertised to do. The intention of my patch is to make the parser retain HTML and XHTML support and in addition work transparently with XML in general. There are quite a few developers working with XML output so I think this would be valuable.

Make HTML::Document#initialize default xml argument to true if text has <?xml instruction at the begging