htmlparser-user Mailing List for HTML Parser

Hi

I have been using htmlparser 1.5 for a short while now and am getting 
pretty flustrated. :-) Not because it doesn't seem to be a good utility 
and expose a rich API, but because sufficient documentation to get 
started with key issues is severly lacking. However, it seems of much of 
my flustration is due to the API's flexability - the ability to do the 
same things using different techniques. However, for someone gettings 
started using the parser, comparing and picking these techniques is 
difficult and I implore the developers and users who have experience 
using the API to share their knowledge in relavant forums, add more 
demos and write a JavaWorld (or other) basic "getting started" article 
on basic usage. This would also lead to more exposure to other java 
developers unaware of the htmlparser project.

Anyway, as the title says I need to transform some html for presentation 
   to use in another web site. To do this I need to remove some tags 
(and children), alter som tags, remove some attributes, alter some 
attributes etc.

Up to now I have been following a basic simple procedure using a 
combination of registering tag handlers and using filters to achieve my 
goal. From what I have seen there seems to be many ways of retrieving 
tag content, but only one way to change tag content while parsing (by 
overriding doSemantivAction() in a tag subclass):

Ok, my code looks something like this:

// register tag handlers for tags I know I want to handle

PrototypicalNodeFactory factory = new PrototypicalNodeFactory();

factory.registerTag(new LinkTag() {
   public void doSemanticAction() throws ParserException {
     // do something with the Link tag or attributes
   }
});

factory.registerTag(new BodyTag() {
   public void doSemanticAction() throws ParserException {
     // do something with the body tag or attributes
   }
});

//factory.registerTag(new general tag that processes all tags
//                    even those that have specific handlers
//                    see text below

Parser parser = new Parser(url);
parser.setNodeFactory(factory);

// this step seems a bit ridiculous. why doesn't parser have a
// public NodeList getNodeList() method? Or am I missing something?

NodeList list = new NodeList();

for (NodeIterator iter = parser.elements(); iter.hasMoreNodes(); ) {
   list.add(iter.nextNode());
}

// get the body tag only. can be part of a complete html document
// or just a html fragment (containing a body tag)

NodeList body = list.extractAllNodesThatMatch(
   new NodeClassFilter(BodyTag.class), true);

// remove unwanted tags and their children

body.keepAllNodesThatMatch(new NotFilter(
      new NodeClassFilter(ScriptTag.class)), true);
body.keepAllNodesThatMatch(new NotFilter(
      new NodeClassFilter(MetaTag.class)), true);

// print the result for debugging

for (int i = 0; i < l2.size (); i++) {
   System.out.print(l2.elementAt(i).toHtml());
}

Does this look correct? It seems to get the result I want...

However, what I can't quite figure out is how to register a general tag 
listener that will take care of removing an attribute for all tags (or a 
subset of them), but also have the ability to register the more specific 
tag listeners at the same time. Is this possible? I tried to create a 
subclass of TagNode and override the doSemanticAction() method, but it 
did't work and was never called.

Am I on the right track? Any ideas?

Regards
Lee Francis Wilhelmsen

-- 
Programs should be written for people to read, and only
incidentally for machines to execute.
-- Structure and Interpretation of Computer Programs, MIT Press

I think you will probably just need to subclass CompositeTag and 
register it with the PrototypicalNodeFactory.
See numerous examples in the org.htmlparser.tags package regarding 
subclassing, and see org.htmlparser.parserapplications.SiteCapturer for 
an example of registration.


Bart Smith wrote:

> Hello,
> I need to create my own custom tags to search for.
> On the wiki there is an article "Writing your own scanners" but it 
> says do not use this one.
>  
> I have to create a custom tag without attirbutes. it will just be some 
> data between <mytag> </mytag>
>  
> Thanks
> Randy
>
>
> <http://us.rd.yahoo.com/my/navbar/sethp/*http://www.yahoo.com/r/hs%20%0A> 

Hello,
I need to create my own custom tags to search for.
On the wiki there is an article "Writing your own scanners" but it says do not use this one.
 
I have to create a custom tag without attirbutes. it will just be some data between <mytag> </mytag>
 
Thanks
Randy

		
---------------------------------
Do you Yahoo!?
 Make Yahoo! your home page   

hi,everybody!
  I I need a kind of XPath to describe or mark
patterns of the interesting elements.But I am not sure
htmlparser can support Xpath.If not,How can I solve
this problem? any information is appreciated!


   

_________________________________________________________
Do You Yahoo!?
注册世界一流品质的雅虎免费电邮
http://cn.rd.yahoo.com/mail_cn/tag/1g/*http://cn.mail.yahoo.com/

You might be adding nodes (tags) to the node currently being visited by 
the node visitor, so it is recursing and applying your changes many, 
many times to the same node.

Qingyi Gu wrote:

>Hi,
>
>I am using "NodeVisitor" to modify the links in the
>HTML page.  It works pretty well, but I got the
>following errors for certain page.   Anybody has any
>idea how to fix it.  I appreciate your help.
>
>**************************************************************
>java.lang.StackOverflowError
>	at
>org.htmlparser.lexer.InputStreamSource.getCharacters(InputStreamSource.java:588)
>	at org.htmlparser.lexer.Page.getText(Page.java:954)
>	at
>org.htmlparser.lexer.PageAttribute.getRawValue(PageAttribute.java:383)
>	at
>org.htmlparser.Attribute.toString(Attribute.java:706)
>	at
>org.htmlparser.nodes.TagNode.toHtml(TagNode.java:699)
>	at
>org.htmlparser.tags.CompositeTag.toHtml(CompositeTag.java:156)
>	at
>org.htmlparser.tags.CompositeTag.putEndTagInto(CompositeTag.java:151)
>	at
>org.htmlparser.tags.CompositeTag.toHtml(CompositeTag.java:161)
>....
>************************************************************
>
>Thanks,
>Jenny
>
>
>		
>__________________________________ 
>Do you Yahoo!? 
>Yahoo! Small Business - Try our new resources site!
>http://smallbusiness.yahoo.com/resources/ 
>
>
>-------------------------------------------------------
>SF email is sponsored by - The IT Product Guide
>Read honest & candid reviews on hundreds of IT Products from real users.
>Discover which products truly live up to the hype. Start reading now.
>http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click
>_______________________________________________
>Htmlparser-user mailing list
>Htm...@li...
>https://lists.sourceforge.net/lists/listinfo/htmlparser-user
>
>  
>

I believe this is the same situation as bug #976448 Parent TableTag has 
incomplete HTML, or bug #923146 tag nesting rule too strict for forms.
The tag closing mechanism is somewhat intolerant of badly formed html.
It's doing one option, and that is to close the table off and start another.

You could add your own <table>, <td> and <tr> custom tags that have 
their own scanner. Your scanner could be more tolerant.
See custom tag examples.

e3oginos X wrote:

> hi
> i have a runned into a situation where a site's html had contained two 
> closing </td> (for example </td> </td>) instead of one.  This still 
> displayed fine in a normal browser, however when running my program 
> that is using the htmlparser, it automaticly replaced the second </td> 
> as a </table> and totally messed up the output.
> how can i stop this from happening? how can i can provide custom 
> methods in the way of handling badly coded html situations such as this?
>
> thanks
>

Can you post the error message you are getting regarding "can't resolve".

umasurya sykam wrote:

> I am a user of Htmlparser1_4 with j2sdk1.4.2_06.
>  
> Everything is working fine.
>  
> But there r 2 problems that r very important to get solved.
>  
> 1.It is telling any Visitor even i regester it is giving can't resolve 
> symbol(error)
>  
>  
> 2.If at all any nested tables in HTML page it is unable to find out 
> the text using method
> getExtractedText().
>  
>  
> So, some one plz give me solution.
>  
> Mainly Visitors r not working inmy s/w except that everything is fine
>  
> First of all i am verythankful to u for providing this kind of great 
> software
>  
> Plz reply me as soon as possible
>  
> Regards Umasuryasykam
>
>  
>
> __________________________________________________
> Do You Yahoo!?
> Tired of spam? Yahoo! Mail has the best spam protection around
> http://mail.yahoo.com
>

I am a user of Htmlparser1_4 with j2sdk1.4.2_06.
 
Everything is working fine.
 
But there r 2 problems that r very important to get solved.
 
1.It is telling any Visitor even i regester it is giving can't resolve symbol(error)
 
 
2.If at all any nested tables in HTML page it is unable to find out the text using method
getExtractedText().
 
 
So, some one plz give me solution.
 
Mainly Visitors r not working inmy s/w except that everything is fine
 
First of all i am verythankful to u for providing this kind of great software
 
Plz reply me as soon as possible
 
Regards Umasuryasykam

 

__________________________________________________
Do You Yahoo!?
Tired of spam?  Yahoo! Mail has the best spam protection around 
http://mail.yahoo.com 

hi
i have a runned into a situation where a site's html had contained two 
closing </td> (for example </td> </td>) instead of one.  This still 
displayed fine in a normal browser, however when running my program that is 
using the htmlparser, it automaticly replaced the second </td> as a </table> 
and totally messed up the output.
how can i stop this from happening? how can i can provide custom methods in 
the way of handling badly coded html situations such as this?

thanks

_________________________________________________________________
Express yourself instantly with MSN Messenger! Download today it's FREE! 
http://messenger.msn.click-url.com/go/onm00200471ave/direct/01/

Hi,

I am using "NodeVisitor" to modify the links in the
HTML page.  It works pretty well, but I got the
following errors for certain page.   Anybody has any
idea how to fix it.  I appreciate your help.

**************************************************************
java.lang.StackOverflowError
	at
org.htmlparser.lexer.InputStreamSource.getCharacters(InputStreamSource.java:588)
	at org.htmlparser.lexer.Page.getText(Page.java:954)
	at
org.htmlparser.lexer.PageAttribute.getRawValue(PageAttribute.java:383)
	at
org.htmlparser.Attribute.toString(Attribute.java:706)
	at
org.htmlparser.nodes.TagNode.toHtml(TagNode.java:699)
	at
org.htmlparser.tags.CompositeTag.toHtml(CompositeTag.java:156)
	at
org.htmlparser.tags.CompositeTag.putEndTagInto(CompositeTag.java:151)
	at
org.htmlparser.tags.CompositeTag.toHtml(CompositeTag.java:161)
....
************************************************************

Thanks,
Jenny


		
__________________________________ 
Do you Yahoo!? 
Yahoo! Small Business - Try our new resources site!
http://smallbusiness.yahoo.com/resources/ 

It's possible to use the Lexer, which just gives a flat, linear sequence 
of nodes,
but the parser will parse all of a composite tag (and place the child 
nodes in a node list) before it hands it back to you.
I guess you could tailor the nodes returned, so only some are composite 
and process the rest linearly.


Rob Shields wrote:

> Hi Derrick,
>
> So is it not possible to use HTMLParser as a SAX-like parser? I recall 
> I used HTMLParser in a very SAX-like way about 3-4 years ago.
>
> Rob
>
> From: Derrick Oswald <Der...@Ro...>
> Reply-To: htm...@li...
> To: htm...@li...
> Subject: Re: [Htmlparser-user] performance benchmarks
>
> Sorry, no. There were some old benchmarks that indicated htmlparser is 
> 40% to 600% faster than JTidy.
> In any case, the current SAX implementation of  HTML Parser is based 
> on the DOM model under the hood.
>
>
>
>
> -------------------------------------------------------
> SF email is sponsored by - The IT Product Guide
> Read honest & candid reviews on hundreds of IT Products from real users.
> Discover which products truly live up to the hype. Start reading now.
> http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click
> _______________________________________________
> Htmlparser-user mailing list
> Htm...@li...
> https://lists.sourceforge.net/lists/listinfo/htmlparser-user
>

Hi Derrick,

So is it not possible to use HTMLParser as a SAX-like parser? I recall I 
used HTMLParser in a very SAX-like way about 3-4 years ago.

Rob

From: Derrick Oswald <Der...@Ro...>
Reply-To: htm...@li...
To: htm...@li...
Subject: Re: [Htmlparser-user] performance benchmarks

Sorry, no. There were some old benchmarks that indicated htmlparser is 40% 
to 600% faster than JTidy.
In any case, the current SAX implementation of  HTML Parser is based on the 
DOM model under the hood.

Sorry, no. There were some old benchmarks that indicated htmlparser is 
40% to 600% faster than JTidy.
In any case, the current SAX implementation of  HTML Parser is based on 
the DOM model under the hood.

Rob Shields wrote:

> Hi
>
> I'm working on a product that uses a javaCC based HTML parser and I 
> have suggested that a SAX-like parser may be faster for our use. Are 
> there any performance benchmarks available for HTMLParser?
>
> Thanks
> Rob
>
>

Hi

I'm working on a product that uses a javaCC based HTML parser and I have 
suggested that a SAX-like parser may be faster for our use. Are there any 
performance benchmarks available for HTMLParser?

Thanks
Rob

The only ones that have 'children' are derived from CompositeTag.
There aren't any tag classes for the font, strong and h1 through h5 tags 
you mention so they are parsed as generic tags which are not composite.
You can make them composite by adding them to the set to be recognized, 
by defining classes for them:

public class FontTag extends CompositeTag
{
    private static final String[] mIds = new String[] {"FONT"};
    public String[] getIds () { return (mIds); }
}

and adding them to the list of tags to be recognized:
    PrototypicalNodeFactory factory = new PrototypicalNodeFactory ();
    factory.registerTag (new FontTag ());
    parser.setNodeFactory (factory);

e3oginos X wrote:

> hi
>
> i am using the html parser in a project that recognizes patterns 
> within a page, however when using the getParent() method on a Node 
> recursively to get its "ancestors" it skips <font> <strong> <h1-h5> 
> tags. It seems to only get parents if they are <body> <html> <table> 
> <td> <tr>
> How can i get it to get these tags?
>
> thanks
>

hi

i am using the html parser in a project that recognizes patterns within a 
page, however when using the getParent() method on a Node recursively to get 
its "ancestors" it skips <font> <strong> <h1-h5> tags. It seems to only get 
parents if they are <body> <html> <table> <td> <tr>
How can i get it to get these tags?

thanks

_________________________________________________________________
Express yourself instantly with MSN Messenger! Download today it's FREE! 
http://messenger.msn.click-url.com/go/onm00200471ave/direct/01/