HTML::CruftText - Remove unuseful text from HTML
Version 0.01
SYNOPSIS
Removes junk from HTML page text.
This module uses a regular expression based approach to remove cruft
from HTML. I.e. content/text that is very unlikely to be useful or
interesting.
use HTML::CruftText;
open (my $MYINPUTFILE, '<input.html' );
my @lines = <$MYINPUTFILE>;
my $de_crufted_lines = HTML::CruftText::clearCruftText( \@lines);
...
DESCRIPTION
This module was developed for the Media Cloud project
(http://mediacloud.org) as the first step in differentiating article
text from ads, navigation, and other boilerplate text. Its approach is
very conservative and almost never removes legitimate article text.
However, it still leaves in a lot of cruft so many users will want to
do additional processing.
Typically, the clearCruftText method is called with an array reference
containing the lines of an HTML file. Each line is then altered so that
the cruft text is removed. After completion some lines will be entirely
blank, while others will have certain text removed. In a few rare
cases, additional HTML tags are added. The result is NOT GUARANTEED to
be valid, balanced HTML though some HTML is retained because it is
extremely useful for further processing. Thus some users will want to
run an HTML stripper over the results.
The following tactics are used to remove cruft text:
* Nonbody text --anything outside of the <body></body> tags -- is
removed
* Text within the following tags is removed: <script>, <style>,
<frame>, <applet>, and <textarea>
* clickprint markers -- many web sites have clickprint annotation
comments that explicitly mark whether text should be included.
* Removal of HTML tags in comments: we remove any HTML tags within <!--
--> comments but keep other comment text. This makes the result easier
to process with regular expressions.
* Close tags that span multiple lines within an single open tag. For
example, we would change:
FOO<a
href="bar.com>BAZ
to:
FOO<a >
<a href="bar.com>BAZ
this makes the output easier to process with regular expressions.
SUBROUTINES/METHODS
clearCruftText( $lines )
This is the main method for this module. Removes cruft text from $lines
and returns the result. Generally $lines will be a reference to an
array of lines from an HTML file. However, this method can also be
called with a string, in which case, the string will be split into
multiple lines and an array reference of decrufted html lines is
returned.
has_clickprint ( $lines )
Returns true if the HTML in $lines has clickprint annotation comment
tags. Returns false otherwise.
LICENSE AND COPYRIGHT
Copyright 2012 Berkman Center for Internet & Society at Harvard
University.
This program is released under the following license: Affero General
Public License