htmlparser-user Mailing List for HTML Parser

Pietro wrote:
> But if i inform something like: 
> "showsearchbysubjectpage.do?categoryid=automoveis&localeufid=sp&localecityid=sao+paulo&localemarketid=5&guideid=1class="
> what is returnded for me is:
> Can i obtain the parameters of an link without having to write my own 
> method with regex to do it ?

Hi,

attributes are property of the tag, so you can get
HREF attribute of A tag with htmlparser. I am not
aware of class, that would split its value into
request parameters .. IMHO you'll on your own,
but try to seek jakarta commons, there might
be a solution.

Regards

Leos

<p>Hi,</p><p>    I'm trying to get the parameters of a given URI, for this i made the
method bellow.</p><p>But if i inform something like:
&quot;showsearchbysubjectpage.do?categoryid=automoveis&amp;localeufid=sp&amp;localecityid=sao+paulo&amp;localemarketid=5&amp;guideid=1class=&quot;<br
/>what is returnded for me is:</p><p>Nome ==&gt; A =&gt; null<br />Nome ==&gt; HREF =&gt;
showsearchbysubjectpage.do?categoryid=automoveis&amp;localeufid=sp&amp;localecityid=sao+paulo&amp;localemarketid=5&amp;guideid=1class=<br
/>Nome ==&gt; null =&gt; </p><p>But i was  expecting for something like:<br
/>categoryid/automoveis<br />localeufid/sp<br />and so on...</p><p>Can i obtain the
parameters of an link without having to write my own method with regex to do it
?</p><p>Regards.</p><p /><p /><p>public Map getParameters(final String uri) {final HashMap
parameters = new HashMap();new LinkTag();for(Iterator i = v.iterator();i.hasNext();)
{return parameters;</p><p /><p><br />LinkTag link = </p><p>link.setLink(uri);</p><p><br
/>Vector v = link.getAttributesEx();</p><p /><p>Attribute attr
=(Attribute)i.next();</p><p>parameters.put(attr.getName(), attr.getValue());</p><p>}</p><p
/><p>}<br /><br /></p>

Dizzy,

Just a sanity check, if you haven't registered this tag it won't be 
used, see http://htmlparser.sourceforge.net/wiki/index.php/CustomTagLinks

but basically:

        Parser parser = new Parser ("http://urlIWantToParse.com");
        PrototypicalNodeFactory factory = new PrototypicalNodeFactory ();
        factory.registerTag (new PreTag ());
        parser.setNodeFactory (factory);


You shouldn't need the scanner if you subclass CompositeTag, since it 
comes with a default CompositeTagScanner,
unless of course you're doing something fancy....

Derrick


Dizzy Reed wrote:

> Hello,
>
> I need to manipulate every tag scanned by my parser, but I want 
> preformatted text to be left alone; after searching a bit (in the code 
> and in the archive), it appeared that I had to write both a PreTag 
> class and a PreScanner class, respectively inspired by StyleTag and 
> StyleScanner. So I pasted those latest two files into my working 
> directory, and all I did was approximately replacing "stlye" with 
> "pre" where it was needed. So basically, it comes down to changing 
> names and replacing:
>
> private static final String[] mIds = new String[] {"STYLE"};
> private static final String[] mEndTagEnders = new String[] {"STYLE"};
>
> with:
>
> private static final String[] mIds = new String[] {"PRE", "CODE", "TT"};
> private static final String[] mEndTagEnders = new String[] {"PRE", 
> "CODE", "TT"};
>
>
>
> This is the beginning of the "handleNode" of my inherited _parser_, to 
> see if this worked:
>
>  public void handleNode(Node node) {
>    NodeList children = node.getChildren();
>
>    if(node instanceof PreTag)
>       System.out.println("PreTag works !!!"); // debug
>
> /* ... some more code below ... */
>   }
>
> Though the detection works, for instance, with a DoctypeTag, this 
> doesn't. I tested it on htmlparser's website's main.html page, which 
> contains some preformatted text, but it is not detected. Can anyone 
> tell me what I've done wrong?
>
> Have a nice day all,
> Anthony.
>

Hello,

I need to manipulate every tag scanned by my parser, but I want preformatted 
text to be left alone; after searching a bit (in the code and in the 
archive), it appeared that I had to write both a PreTag class and a 
PreScanner class, respectively inspired by StyleTag and StyleScanner. So I 
pasted those latest two files into my working directory, and all I did was 
approximately replacing "stlye" with "pre" where it was needed. So 
basically, it comes down to changing names and replacing:

private static final String[] mIds = new String[] {"STYLE"};
private static final String[] mEndTagEnders = new String[] {"STYLE"};

with:

private static final String[] mIds = new String[] {"PRE", "CODE", "TT"};
private static final String[] mEndTagEnders = new String[] {"PRE", "CODE", 
"TT"};



This is the beginning of the "handleNode" of my inherited _parser_, to see 
if this worked:

  public void handleNode(Node node) {
    NodeList children = node.getChildren();

    if(node instanceof PreTag)
       System.out.println("PreTag works !!!"); // debug

/* ... some more code below ... */
   }

Though the detection works, for instance, with a DoctypeTag, this doesn't. I 
tested it on htmlparser's website's main.html page, which contains some 
preformatted text, but it is not detected. Can anyone tell me what I've done 
wrong?

Have a nice day all,
Anthony.

_________________________________________________________________
Essai gratuit: Appareil photo numérique 
http://essai.fr.msn.be/product/product.html?nr=13

Marcio,

I think reading the JSP page directly is not going to tell you much
assuming the JSP tags/scriptlets are generating the useful content.  At
best you could check the JSP page for XML well-formedness using something
like Tidy.

Using something like HtmlUnit or HttpUnit or JWebUnit you can actually
query the page from the web server causing the JSP code to execute.  Then
you are not looking at JSP but actual HTML.


--Marty

--- "Pietrosoft Informatica Ltda." <pie...@ma...> wrote:

---------------------------------

Hi Martin, still don't tried to do anything. I Really don't know what is
better, parsethe raw JSP or the generated html. I think that parse the raw
JSP is the more  easyoption, no? My only work in this case would be find
the jsp in your phisical path andparse the file. But is possible parse the
generated HTML ? Seems to be a more ellegantsolution. 

Regards,

Marcio.


---- Mensagem Original ----
De: "Martin Wegner" 
Para:htm...@li...
Enviado: Ter, Maio 18, 2004 12:39 am
Assunto: Re: [Htmlparser-user] Parsing JSP with Struts...

Are youtrying to parse the raw JSP page or the HTML file generated by 
Struts? 

--- Pietro 
wrote: 
---------------------------------
Hi, 
    I already used htmlParser a couple of years ago. In that occasion i
waslooking for just a simple parser to extract all the String from statics

html files. AndHtmlParser worked perfect. But know i have a new situation,

i gotto extract strings byhtml tags, that's ok. The problem is that now 
the html filesare dinamics, generated byJSP, and this JSPs are returned 
by Struts actions (.do).
    Somebody here have somesolution for this case? i confess that i don't 
knowfor where to start, any help orguideline will be very very welcome. 
Regards, 
Pietro. 

   
    
-------------------------------------------------------This SF.Net email 
issponsored by: SourceForge.net BroadbandSign-up now for SourceForge 
Broadband and getthe fastest6.0/768 connection for only $19.95/mo for the 
first 3 
months!http://ads.osdn.com/?ad_id=2562&alloc_id=6184&op=click_______________________________________________Htmlparser-user
mailing 
lis...@li...https://lists.sourceforge.net/lists/listinfo/htmlparser-user

------------------------------------------------------- 
This SF.Netemail is sponsored by: SourceForge.net Broadband 
Sign-up now for SourceForgeBroadband and get the fastest 
6.0/768 connection for only $19.95/mo for the first 3months! 
http://ads.osdn.com/?ad_id=2562&alloc_id=6184&op=click 
_______________________________________________ 
Htmlparser-user mailing list 
Htm...@li... 
https://lists.sourceforge.net/lists/listinfo/htmlparser-user 



-------------------------------------------------------This SF.Net email
is sponsored by: SourceForge.net BroadbandSign-up now for SourceForge
Broadband and get the fastest6.0/768 connection for only $19.95/mo for the
first 3
months!http://ads.osdn.com/?ad_id=2562&alloc_id=6184&op=click_______________________________________________Htmlparser-user
mailing
lis...@li...https://lists.sourceforge.net/lists/listinfo/htmlparser-user

<p>Hi Martin, still don't tried to do anything. I Really don't know what is better, parse
the raw JSP or the generated html. I think that parse the raw JSP is the more  easy
option, no? My only work in this case would be find the jsp in your phisical path and
parse the file. But is possible parse the generated HTML ? Seems to be a more ellegant
solution. </p><p>Regards,</p><p>Marcio.</p><br />---- Mensagem Original ----<br
/><b>De:</b> &quot;Martin Wegner&quot; <mar...@ya... /><br /><b>Para:</b>
htm...@li...<br /><b>Enviado:</b> Ter, Maio 18, 2004 12:39 am<br
/><b>Assunto:</b> Re: [Htmlparser-user] Parsing JSP with Struts...<br /><br />Are you
trying to parse the raw JSP page or the HTML file generated by <br />Struts? <br /><br
/>--- Pietro <pie...@ma... />wrote: <br />---------------------------------
<br />Hi, <br />    I already used htmlParser a couple of years ago. In that occasion i
<br />waslooking for just a simple parser to extract all the String from statics <br
/>html files. AndHtmlParser worked perfect. But know i have a new situation, <br />i got
to extract strings byhtml tags, that's ok. The problem is that now <br />the html files
are dinamics, generated byJSP, and this JSPs are returned <br />by Struts actions (.do).
<br />    Somebody here have somesolution for this case? i confess that i don't <br />know
for where to start, any help orguideline will be very very welcome. <br />Regards, <br
/>Pietro. <br /><br />   <br />    <br
/>-------------------------------------------------------This SF.Net email <br />is
sponsored by: SourceForge.net BroadbandSign-up now for SourceForge <br />Broadband and get
the fastest6.0/768 connection for only $19.95/mo for the <br />first 3 <br
/>months!http://ads.osdn.com/?ad_id=2562&amp;alloc_id=6184&amp;op=click_______________________________________________Htmlparser-user
<br />mailing <br
/>lis...@li...https://lists.sourceforge.net/lists/listinfo/htmlparser-user
<br /><br />------------------------------------------------------- <br />This SF.Net
email is sponsored by: SourceForge.net Broadband <br />Sign-up now for SourceForge
Broadband and get the fastest <br />6.0/768 connection for only $19.95/mo for the first 3
months! <br />http://ads.osdn.com/?ad_id=2562&amp;alloc_id=6184&amp;op=click <br
/>_______________________________________________ <br />Htmlparser-user mailing list <br
/>Htm...@li... <br
/>https://lists.sourceforge.net/lists/listinfo/htmlparser-user <br /><br /><br />

<p>Hi Martin, still don't tried to do anything. I Really don't know what is better, parse
the raw JSP or the generated html. I think that parse the raw JSP is the more  easy
option, no? My only work in this case would be find the jsp in your phisical path and
parse the file. But is possible parse the generated HTML ? Seems to be a more ellegant
solution. </p><p>Regards,</p><p>Marcio.</p><p><br />---- Mensagem Original ----<br
/><b>De:</b> &quot;Martin Wegner&quot; <mar...@ya... /><br /><b>Para:</b>
htm...@li...<br /><b>Enviado:</b> Ter, Maio 18, 2004 12:39 am<br
/><b>Assunto:</b> Re: [Htmlparser-user] Parsing JSP with Struts...<br /><br />Are you
trying to parse the raw JSP page or the HTML file generated by <br />Struts? <br /><br
/>--- Pietro <pie...@ma... />wrote: <br />---------------------------------
<br />Hi, <br />    I already used htmlParser a couple of years ago. In that occasion i
<br />waslooking for just a simple parser to extract all the String from statics <br
/>html files. AndHtmlParser worked perfect. But know i have a new situation, <br />i got
to extract strings byhtml tags, that's ok. The problem is that now <br />the html files
are dinamics, generated byJSP, and this JSPs are returned <br />by Struts actions (.do).
<br />    Somebody here have somesolution for this case? i confess that i don't <br />know
for where to start, any help orguideline will be very very welcome. <br />Regards, <br
/>Pietro. <br /><br />   <br />    <br
/>-------------------------------------------------------This SF.Net email <br />is
sponsored by: SourceForge.net BroadbandSign-up now for SourceForge <br />Broadband and get
the fastest6.0/768 connection for only $19.95/mo for the <br />first 3 <br
/>months!http://ads.osdn.com/?ad_id=2562&amp;alloc_id=6184&amp;op=click_______________________________________________Htmlparser-user
<br />mailing <br
/>lis...@li...https://lists.sourceforge.net/lists/listinfo/htmlparser-user
<br /><br />------------------------------------------------------- <br />This SF.Net
email is sponsored by: SourceForge.net Broadband <br />Sign-up now for SourceForge
Broadband and get the fastest <br />6.0/768 connection for only $19.95/mo for the first 3
months! <br />http://ads.osdn.com/?ad_id=2562&amp;alloc_id=6184&amp;op=click <br
/>_______________________________________________ <br />Htmlparser-user mailing list <br
/>Htm...@li... <br
/>https://lists.sourceforge.net/lists/listinfo/htmlparser-user <br /><br /><br /></p>

Are you trying to parse the raw JSP page or the HTML file generated by
Struts?


--- Pietro <pie...@ma...> wrote:

---------------------------------

Hi,

    I already used htmlParser a couple of years ago. In that occasion i
waslooking for just a simple parser to extract all the String from statics
html files. AndHtmlParser worked perfect. But know i have a new situation,
i got to extract strings byhtml tags, that's ok. The problem is that now
the html files are dinamics, generated byJSP, and this JSPs are returned
by Struts actions (.do).

    Somebody here have somesolution for this case? i confess that i don't
know for where to start, any help orguideline will be very very welcome.

Regards,

Pietro.



  

    

-------------------------------------------------------This SF.Net email
is sponsored by: SourceForge.net BroadbandSign-up now for SourceForge
Broadband and get the fastest6.0/768 connection for only $19.95/mo for the
first 3
months!http://ads.osdn.com/?ad_id=2562&alloc_id=6184&op=click_______________________________________________Htmlparser-user
mailing
lis...@li...https://lists.sourceforge.net/lists/listinfo/htmlparser-user

<p>Hi,</p><p>    I already used htmlParser a couple of years ago. In that occasion i was
looking for just a simple parser to extract all the String from statics html files. And
HtmlParser worked perfect. But know i have a new situation, i got to extract strings by
html tags, that's ok. The problem is that now the html files are dinamics, generated by
JSP, and this JSPs are returned by Struts actions (.do).</p><p>    Somebody here have some
solution for this case? i confess that i don't know for where to start, any help or
guideline will be very very welcome.</p><p>Regards,</p><p>Pietro.</p><p /><p /><p>  
</p><p>    <br /></p>

Gerret,

Yes, <p> tags are not recognized as special, just Tags, and aren't 
derived from CompositeTag so they won't have children.
This is because <p> tags aren't necessarily closed with a </p>, and 
aren't block level tags, from the HTML spec: http://www.w3.org/TR/html4/

To try making them special on a well formed document that uses </p> 
tags, you would need to add a ptag class derived from CompositeTag to 
the PrototypicalNodeFactory, similar to the custom tag example 
illustrated here: 
http://htmlparser.sourceforge.net/wiki/index.php/CustomTagLinks
In general, the custom ptag class would need to be carefully crafted to 
finish scannng when encountering </p> or another <p> tag and many others 
by correctly implementing getEnders() and getEndTagEnders(). I'm not 
sure how much success you would have. It might work for your specific 
HTML file, but not against all the various HTML that's out there in the 
wild.

Derrick

Gerret Apelt wrote:

> Hi --
>
> I am having a problem with HTMLParser that is illustrated in the code
> snippet below.
> My goal is to extract the text content of all "<P>" elements in an
> HTML document.
> It appears that my NodeFilter does not find text nodes that are children
> of P-Tag nodes.
> However, if I instead try to find the text children of some other tag
> nodes (e.g., the text node children of "table" tag nodes),
> this appears to work fine.
> Is there a difference between P and TABLE tags in terms of 
> HTMLParser's model?
>
> Any help much appreciated.
>
> cheers,
> Gerret
>
> // Parser parser = 
> Parser.createParser("<HTML><BODY><TABLE>HiThere</TABLE></BODY></HTML>");
>   Parser parser = 
> Parser.createParser("<HTML><BODY><P>HiThere</P></BODY></HTML>");
>
> NodeList list = new NodeList ();
> NodeFilter filter = new AndFilter(new NodeClassFilter(StringNode.class),
>                       new HasParentFilter(new TagNameFilter("P")));
> for (NodeIterator e = parser.elements (); e.hasMoreNodes (); )
>  e.nextNode().collectInto(list, filter);
>
> System.out.println(list.size());
>

Hi --

I am having a problem with HTMLParser that is illustrated in the code
snippet below.
My goal is to extract the text content of all "<P>" elements in an
HTML document.
It appears that my NodeFilter does not find text nodes that are children
of P-Tag nodes.
However, if I instead try to find the text children of some other tag
nodes (e.g., the text node children of "table" tag nodes),
this appears to work fine.
Is there a difference between P and TABLE tags in terms of HTMLParser's model?

Any help much appreciated.

cheers,
Gerret

// Parser parser = Parser.createParser("<HTML><BODY><TABLE>HiThere</TABLE></BODY></HTML>");
   Parser parser = Parser.createParser("<HTML><BODY><P>HiThere</P></BODY></HTML>");

NodeList list = new NodeList ();
NodeFilter filter = new AndFilter(new NodeClassFilter(StringNode.class),
			          new HasParentFilter(new TagNameFilter("P")));
for (NodeIterator e = parser.elements (); e.hasMoreNodes (); )
  e.nextNode().collectInto(list, filter);

System.out.println(list.size());