htmlparser-user Mailing List for HTML Parser

Hello community,
I am writing a website parser with htmlparser and I think it's a great 
library.
My problem is, the website I'm parsing shows me a captcha after a 
certain number of crawls.
As a workaround I wrote a redial routine to reconnect my router and get 
a new ip.
That is working quite well, but my problem is, that my jvm seems to 
cache DNS.
I read this post http://forum.vis.ethz.ch/showthread.php?t=13457 and 
applied everything which is supposed there,
but still I can't continue parsing after a reconnect and I get a 
ConnectionTimeoutException from htmlparser.
It seems, that there might still be some kind of cache.
Could anybody tell me, how I can get the new instance of Parser to 
connect after a reconnect.

Thank you.
Hans.

Hi,

Since Tuesday 20 July 2010, you have been invited by 1 of your contacts to join Netlog, 
the social community for over 49 million young people.

[---- Invitation from geeraza ---- ]
34 yrs - male - Baden-Wurttemberg (Germany)
Connect with geeraza:
http://en.netlog.com/go/mailurl/-bT05ODQwODIyNjImbD0xJmdtPTE2JnU9JTJGZ28lMkZyZWdpc3RlciUyRmlkJTNEMjEwNTI2NTY0NSUyNnIlM0R0MQ__
	

On Netlog you can:

- Create your own web page
- Extend your social network
- Publish your music playlists- Share pictures and videos- Post blogs
- And much more ... ....

http://en.netlog.com/go/mailurl/-bT05ODQwODIyNjImbD0xJmdtPTE2JnU9JTJGZ28lMkZyZWdpc3RlciUyRmlkJTNEMjEwNTI2NTY0NSUyNnIlM0R0OTg0MDgyMjYyMQ__

----------------------------------------------------------------
Don't want to receive invitations from your friends anymore?
http://en.netlog.com/go/mailurl/-bT05ODQwODIyNjImbD0zJmdtPTE2JnU9JTJGZ28lMkZub21haWxzJTJGaW52aXRlJTJGZW1haWwlM0QtYUhSdGJIQmhjbk5sY2kxMWMyVnlRR3hwYzNSekxuTnZkWEpqWldadmNtZGxMbTVsZEFfXyUyNmNvZGUlM0QxMTc1MjQxNSUyNmlkJTNEMjEwNTI2NTY0NQ__



----------------------------------------------------------------
Netlog NV/SA. E. Braunplein 18. B-9000 Gent. Belgium BE0859635972. abu...@ne...

Hey,

I have created a Netlog profile with my pictures, videos, blogs and events and I want to add you as a friend so you can see it. You first need to register on Netlog! When you log in, you can create your own profile.

Take a look:
http://en.netlog.com/go/mailurl/-bT05Njg2NzYwNDMmbD0xJmdtPTM3JnU9JTJGZ28lMkZyZWdpc3RlciUyRmlkJTNEMjEwNTI2NTY0NSUyNmklM0R0OTE_

Cheers,
geeraza

----------------------------------------------------------------
Don't want to receive invitations from your friends anymore?
http://en.netlog.com/go/mailurl/-bT05Njg2NzYwNDMmbD0yJmdtPTM3JnU9JTJGZ28lMkZub21haWxzJTJGaW52aXRlJTJGZW1haWwlM0QtYUhSdGJIQmhjbk5sY2kxMWMyVnlRR3hwYzNSekxuTnZkWEpqWldadmNtZGxMbTVsZEFfXyUyNmNvZGUlM0QxMTc1MjQxNSUyNmlkJTNEMjEwNTI2NTY0NSUyNmklM0R0OTI_

Did you set STRICT false:

http://htmlparser.sourceforge.net/javadoc/org/htmlparser/scanners/ScriptScanner.html



On Wed, Jul 7, 2010 at 9:48 PM, Niket Arora <nik...@ex...>wrote:

>  I m parsing a page
> http://www.healthline.com/search?q1=how+to+improve+prostate+blood+levels using
> htmlparser api and I m getting content inside a script tag in some other tag
> and reason for this is html tags are present in a string inside javascript
> tags and are not escaped …. so htmlparser api is closing on those tags.
>
>
>
>
>
>
> ================================================================================================================================================================================================
>
>
>
> <div id="myHealthlineHeader">
>
>         <script>
>
>               if(isLoggedIn()) {
>
>                 document.write("<a href=\"/action/LogOutServlet\">Sign
> Off</a> | <a rel=\"nofollow\" href=\"/myhealthline/account_overview.jsp\">My
> Healthline</a> | Welcome, <strong>" + getNickname() + "</strong>");
>
>                 document.getElementById("myHealthlineHeader").className =
> "hl_state_top_signed_in";
>
>               } else {
>
>
>
>                 document.write("<div
> style=\"float:right;text-align:right;padding:0 5px 0
> 0;\">&nbsp;&nbsp;&nbsp;|&nbsp;&nbsp;&nbsp;<a class=\"underlineless\"
> rel=\"nofollow\"
> href=\"/yourfeedback.jsp?url=%2Fsearch%3Fq1%3Dhow%2Bto%2Bimprove%2Bprostate%2Bblood%2Blevels\">Feedback</a></div>");
>
>                 document.write("<div style=\"float:right\"><a
> class=\"underlineless\" rel=\"nofollow\" href=\"/signin.jsp\">Sign
> in</a>&nbsp;&nbsp;&nbsp;|&nbsp;&nbsp;&nbsp;<a class=\"underlineless\"
> rel=\"nofollow\" href=\"/registration.jsp\">Join Now</a>&nbsp;</div>")
>
>                 document.getElementById("myHealthlineHeader").className =
> "hl_state_top";
>
>               }
>
>         </script>
>
> </div>
>
>
>
>
> ================================================================================================================================================================================================
>
>
>
> Is there anyway to fix this issue?
>
>
>
> Regards
>
> Niket
>
>
> ------------------------------------------------------------------------------
> This SF.net email is sponsored by Sprint
> What will you do first with EVO, the first 4G phone?
> Visit sprint.com/first -- http://p.sf.net/sfu/sprint-com-first
> _______________________________________________
> Htmlparser-user mailing list
> Htm...@li...
> https://lists.sourceforge.net/lists/listinfo/htmlparser-user
>
>

I m parsing a page http://www.healthline.com/search?q1=how+to+improve+prostate+blood+levels using htmlparser api and I m getting content inside a script tag in some other tag and reason for this is html tags are present in a string inside javascript tags and are not escaped .... so htmlparser api is closing on those tags.


================================================================================================================================================================================================

<div id="myHealthlineHeader">
        <script>
              if(isLoggedIn()) {
                document.write("<a href=\"/action/LogOutServlet\">Sign Off</a> | <a rel=\"nofollow\" href=\"/myhealthline/account_overview.jsp\">My Healthline</a> | Welcome, <strong>" + getNickname() + "</strong>");
                document.getElementById("myHealthlineHeader").className = "hl_state_top_signed_in";
              } else {

                document.write("<div style=\"float:right;text-align:right;padding:0 5px 0 0;\">&nbsp;&nbsp;&nbsp;|&nbsp;&nbsp;&nbsp;<a class=\"underlineless\" rel=\"nofollow\" href=\"/yourfeedback.jsp?url=%2Fsearch%3Fq1%3Dhow%2Bto%2Bimprove%2Bprostate%2Bblood%2Blevels\">Feedback</a></div>");
                document.write("<div style=\"float:right\"><a class=\"underlineless\" rel=\"nofollow\" href=\"/signin.jsp\">Sign in</a>&nbsp;&nbsp;&nbsp;|&nbsp;&nbsp;&nbsp;<a class=\"underlineless\" rel=\"nofollow\" href=\"/registration.jsp\">Join Now</a>&nbsp;</div>")
                document.getElementById("myHealthlineHeader").className = "hl_state_top";
              }
        </script>
</div>

================================================================================================================================================================================================

Is there anyway to fix this issue?

Regards
Niket

Hi,

I need to read arbitrary HTML (HTML 4 transitional, XHTML 1.0 strict, ...) extract the body as a fragment and output it again as another (XHTML standard).

Reading the file is simple enough:

		Parser p = new Parser(resource);
		NodeFilter f = new NodeClassFilter(BodyTag.class);
		NodeList listOfBodies = p.extractAllNodesThatMatch(f);
		Node firstBody = listOfBodies.elementAt(0);
		NodeList bodyChildren = firstBody.getChildren();
		System.out.println(bodyChildren.toHtml());

From this hpw can I output either valid HTML 4.0 code or valid XHTML 1.0 code?

Best regards
Oliver