We've trained our model using detected changes from corporate sites and some earlier political sites. Each change for our model was human-rated in terms of relevance/importance, and we also feed in other descriptive attributes about each change, such as DOM location, immediate parent tag, and several other attributes.
What sort of noise are you running in to? (Curious)
In an old project we actually did something a bit more based on visual changes. Basically detected visual diffs and got coordinates of offsets to the change. We then found the smallest html container that encompassed the diff and highlighted it.
Using visual diff you can fuzz things a bit to handle artifact and small movements.
A good lib we have some miles on https://github.com/mapbox/pixelmatch
And this write up on niffy is pretty good
Comparing the pure DOM reveals almost constant change due to various inserted Javascript from Google/Facebook/etc modifying the DOM. Looking at text/images also result in a fair amount of expected noise, but it's mostly in the form of interstitial marketing banners, fundraising targets, etc.
We already have various options to "filter" out certain DOM areas. (As an example, remove all footers, headers, or any other CSS selector.) These work really well... but they require a fair bit of setup.
Thanks for the pixelmatch GH repo link... I have it starred so must have taken a look in the past. We need to evolve our image diff, so we may end up using this!
From that, we feed the results into "diff" Lambda functions to compute image, text, HTML, and network diffs. We treat the text diff as the primary diff type, and so only if we have a text diff do we do the other comparison types.
From these diffs, we then feed the text + some DOM info into ML. There, we use added text, deleted text, and, for each, the shortest unique CSS selector, the immediate parent tag, and some other items that I may be missing (possibly some approximation of where in the main text the change appears... e.g., top 10%, top 20% IIRC).
Hopefully this answers your question?
It's not clear where you're using ML from the site - URLs seem chronological, not ranked on "relevance/importance" and I can't see any relevance/importance indicators.
I'm curious to hear some more detail on how you're encoding your visual diff model?
BTW I ran into 2 issues: 1. You can't zoom out on your image diff slider 2. I got this error when I returned to the site after closing it: {"crossDomain" : true, "method" : "GET", "url" : "https : //api.fluxguard.com/publi c/site/7f646558-f754-447a -b627-9b5202c8a1f2/page?l imit=10&publicAccount=cam paignmonitor"} Please contact us if this is error is happening frequently for you.
We aren't surfacing the per page scores at the moment... most of the ML work was done specifically for this content piece, so we haven't adjusted our core presentation to include it (including the flag grading system which we use to build the training model), other than simply listing stand-out examples at the top of the site.
For our customers, we likely will need to build industry/use-specific models. (Amazon is already sort of doing this by providing pharm-specific text classifiers.) Use cases are so disparate at the moment that it's hard to build a general model for everyone.
As for the cross-domain errors, grrr! Thanks, we will look into it... hard to troubleshoot those. Our API stack consists of CloudFormation -> API Gateway -> API Gateway Caching -> Lambda -> DynamoDB... (edit AKA who knows where those errors are! haha)
(And more detail on the problem you are having w/ image diff slider would be appreciated... feel free to email us directly from email at bottom of site. Not sure I understand this issue you're having.)
(edit feel free to email me w/ error details at peter (at) deepdiveduck . com ... the cross-domain scripting errors are a constant thorn in our sides so we'd like to get any other info you have on 'em)
Our bad for not including the scores for each page... our thoughts w/ this content piece were it would be attractive to journalists who may not have an appreciation for underlying scores.
