Recently, we have updated our Website Link Checker’s engine to a much stronger one. This post describes the features of the new engine. We believe that few broken link checkers, if any, can compete with our latest engine. Before this update, our Website Line Checker tool was able to check only two basic types of web page links:
<meta> content urlvalues with
When we are talking about web site links, we usually do think of
<a> href values. However, from a webmaster’s point of view, it is important that all URLs in the code of the web are correct. For example, it is important to know whether all website’s images (
<img> src) do exist. And there are many other ways to link to resources from web pages. Think about Java applets, Flash objects, new HTML5 media tags, etc. The more different types of linking your broken link checker recognizes the better. On the other hand, without losing any real functionality, it might be possible to omit several obsolete and poorly documented tags, which are usually unsupported by modern web browsers. Considering this, we have decided to implement a link checker engine that would fully cover HTML4 and HTML5 standards and CSS3.
Internally, the new Website Link Checker’s engine distinguishes between three groups of links.
The first group contains all simple links specifications, i.e. a link is fully specified by a value of a single attribute of a single tag. The group is defined by the following list of tags and attributes:
The second group contains links within CSS code. We analyze CSS using a regular expression, which looks for URLs. We search for a CSS code within three types of locations:
styleattribute of arbitrary tag,
- and any file with declared content type “text/css”.
The last group contains links that require specialized code for their extraction. Our engine handles the following kinds of links within this group:
<base>tag – This tag changes the meaning of all relative links within a web page.
<!--[if ...]>– The conditional comment tag is Internet Explorer specific tag that contains HTML code that is interpreted by Internet Explorer and thus unlike other comments, it should be analyzed for links.
<meta> content urlvalues with
<meta> http-equiv="refresh"– The link is a part of the content attribute’s value, which is a list of values separated using semicolon.
<param> valuevalues with
<object> code, classid, codebase, data, and archivevalues – Messy and complicated tag, which attributes are poorly defined and handled differently by each web browser and thus various hacks are commonly implemented in order to achieve the required functionality in all major browsers. This is why its analysis is similarly messed up.
<applet> code, codebase, and archivevalues – Almost the same horror as with the <object> tag.
<form> actionvalues – It is important to care about the value of
<form> methodattribute because requesting the form target page with GET method instead of POST method, or vice versa, can lead to incorrect assessment of the link validity.
<input> formactionvalues – Similarly to the previous case, the correct HTTP method must be used for verification. The method can be found in the
Link Position in Dynamic Pages
Website Link Checker is able to find, check and report all mentioned types of links. It can tell you the exact position of each link within the code of an analyzed web page. This feature is quite tricky. Web sites can contain hundreds or even thousands of web pages and it would be impractical to store all the pages locally, since the reported link checker’s result is usually used once, immediately after its job is finished. When a user clicks on the HTML icon to see positions of the links within the code of the selected web page, Website Link Checker loads a new copy of that page, and that page only, from the target server again. However, with dynamic web pages, it may happen that the newly downloaded content is different from the analyzed one. This is why Website Link Checker performs a heuristic search for the links it found during the link checking process and it tries to find the links within the new content. Thus, unlike many simpler link checkers, it is able to correctly display positions of the links even within dynamic web pages.