Author here - Appraise is a new opensource tool for visual test automation. It’s like FitNesse, but for visuals, and working from Markdown that shows up nicely in GitHub repositories.
Appraise can help you:
- Automate acceptance/regression tests for visual look and feel in a visual language, rather than xUnit style code
- Review and approve changes to web pages, visual layouts and browser components quickly through visual inspection
- Publish easily maintainable/verifiable developer docs with visual examples to Github easily (markdown) or as a static site (html)
- Start Spec by Example/BDD from a sketch (hand-drawn, wireframe, or from a graphic tool), easily compare actual outcomes, then just approve the final result to make a regression test
In the near future, Appraise will also help you:
- Speed up visual exploratory testing by making it easier to rebuild visual components from example data/configuration
- Run visual tests quickly/in parallel using AWS Lambda
This is another tool we built for MindMup, and we used it for the last few months internally, to assist with visual checks and test automation. It helped us move away from tedious layout tests that were very difficult to maintain for small changes, but that would be visually easy to approve or reject. Appraise makes visual tests easy to compose and maintain, by letting people describe what they want something to look like (with a drawing, photo, screenshot or a wire-frame), instead of describing how to test it (eg 10 pixels left, this css color). Here it is in action, preventing me from introducing a stupid layout bug: https://gojko.net/assets/appraise-screenshot.png
Under the hood, Appraise uses headless Chrome to take screenshots, so it can inspect anything that Chrome can render.
The initial release of Appraise is alpha-quality. It covers our key usage scenarios, so we think it is ready for some nice community feedback. It’s useful to us, let’s make it useful to you together.
We’ve designed it to be extensible and pluggable, so other ways of executing fixtures, taking screenshots or processing results should be easy to bolt on to it. We’re releasing it under the MIT license, and the code is on GitHub.
Nice! I built a very similar prototype a few years ago when I was working on a mobile app. My test suite would upload screenshots, and I would have a gallery where I could see all the screenshots that had changed. It was incredibly useful, and I wanted to turn it into a side-project or startup. But I was always worried that CircleCI or TravisCI would add the feature and make my service obsolete. I think I was wrong to worry about that though. Percy launched a little while ago and seems to be doing really well: https://percy.io
My new startup has a very large number of competitors, and I've learned the lesson that competitors are generally a good thing.
Appraise looks much more polished and ready to use. I'll definitely be trying it out for my current project. I had a bad UI bug recently where I forgot about scrollbars on Windows (I used offsetWidth instead of scrollWidth.) I would like to take a lot of screenshots on different browsers and platforms, to make sure I don't have any regressions. I'm going to look into using Appraise for that.
I've actually started building a side-project (imagetest.io, the landing page is still pretty bare-bones) in the visual regression testing space, because I didn't couldn't find percy.io or Applitools (the two main services right now) after quite some searching. Once I discovered the others, I was already too into the project to abandon it, and I also think that there is quite some room for improvement in the space.
If you are interested in testing it out, you can reach out to me at the e-mail in my profile. I'm especially interested in making it work for use-cases that a lot of developers have to solve, but are somewhat underdocumented right now, like testing on Windows/IE10 (I have the exact same requirements at a current freelance client).
1. Are there plans for a multi-browser (especially mobile web) test runner?
2. Are there plans to detect change provenance (eg. if I change my heading height on a page, all the content below it will shift down. The change I really want to be notified about is that the header's height changed, not that everything was pushed down because of it)?
3. There are a lot of similar tools (eg. back in the day I used https://github.com/facebookarchive/huxley) - where does Appraise succeed where those tools failed, or what does Appraise do better?
appraise is designed to be used like fitnesse (so example+fixture=> result) instead of record/replay tools that end up being fragile. it makes the examples easy to understand, splitting what is being tested and how it will be tested clearly so both of those aspects are easy to maintain.
it's also designed to work well with github markdown, so your tests become executable API/component specs that are easy to read and also easy to verify.
I'm working on an AWS Lambda runner now, which will also make it quick and cheap to run UI tests in parallel, so it will provide reasonably fast feedback for large test suites.
>This is diffing PNGs? That sounds like a recipe for disaster.
Not necessarily, if you use it for what it was intended -- visual layout tests that are easy for a human to approve/reject when there is a difference. It doesn't know if the change is right or wrong, just helps humans be more effective by showing the difference clearly and aiding in comparisons.
Would be good to have prebuilt rules for eg material design guidelines.
Would it be even possible to test the material design rules across and entire page without having to define what individual components across the entire website look like?
not currently, but that's definitely an interesting idea.
Appraise is designed to make it easy for a human to approve/reject current state or changes, with a smart visual diff and a good way to show actual and expected pictures side-by-side or as an overlay, but it currently doesn't have any smart logic beyond that. Theoretically it could do something smarter as a comparison.
As it stands now, you could run it once to capture the current state, and get someone to approve it, then it becomes a regression test pack and the tool would alert to any changes with a smart diff.
haven't tested it myself, but I assume headless chrome can render PDFs, so the answer should be yes. Try it and if it doesn't work, submit an issue on github and we'll make it work together. You'll probably need to use a fixture producing a local file, check out https://github.com/AppraiseQA/appraise/blob/master/examples/...
I have not used niffy so this is based on their github description, so might be wrong
1) niffy seems to be designed to use UI interactions to lead you to the result, appraise works like fitnesse, separating what is being tested from how it's being tested, so you could go several levels deeper than UI interactions and eg test how your models and views produce outcomes, without the http serving and web pages in between. this makes tests faster and easier to maintain. appraise is more for visual unit testing.
2) niffy seems to be optimised for developers, appraise is intended for cross-functional collaboration between developers, testers and designers. with appraise, the test is a markdown page that renders nicely in github and shows expected results and the conditions under which they should happen.
Oh good I look forward to deleting mountains of these kinds of tests in a couple of years when the shine comes off the apple and people realise the effort of separating the flake from the real failures isn't worth it.
It's always possible I've misread your comment, but if not: please don't dismiss other people's work in an assholish way on HN. It breaks both the site guidelines and the extra rules for Show HNs:
Substantive critique is welcome, of course, but in this case that would mean talking about problems with visual testing and making it clear you're not picking on this particular implementation.
Could you stop with the incredibly arrogant attitude? You wouldn't act like this in real life -- at least I hope not. You would be shunned by everyone around you.
Appraise can help you:
- Automate acceptance/regression tests for visual look and feel in a visual language, rather than xUnit style code
- Review and approve changes to web pages, visual layouts and browser components quickly through visual inspection
- Publish easily maintainable/verifiable developer docs with visual examples to Github easily (markdown) or as a static site (html)
- Start Spec by Example/BDD from a sketch (hand-drawn, wireframe, or from a graphic tool), easily compare actual outcomes, then just approve the final result to make a regression test
In the near future, Appraise will also help you:
- Speed up visual exploratory testing by making it easier to rebuild visual components from example data/configuration
- Run visual tests quickly/in parallel using AWS Lambda
This is another tool we built for MindMup, and we used it for the last few months internally, to assist with visual checks and test automation. It helped us move away from tedious layout tests that were very difficult to maintain for small changes, but that would be visually easy to approve or reject. Appraise makes visual tests easy to compose and maintain, by letting people describe what they want something to look like (with a drawing, photo, screenshot or a wire-frame), instead of describing how to test it (eg 10 pixels left, this css color). Here it is in action, preventing me from introducing a stupid layout bug: https://gojko.net/assets/appraise-screenshot.png
Under the hood, Appraise uses headless Chrome to take screenshots, so it can inspect anything that Chrome can render.
The initial release of Appraise is alpha-quality. It covers our key usage scenarios, so we think it is ready for some nice community feedback. It’s useful to us, let’s make it useful to you together.
We’ve designed it to be extensible and pluggable, so other ways of executing fixtures, taking screenshots or processing results should be easy to bolt on to it. We’re releasing it under the MIT license, and the code is on GitHub.
My new startup has a very large number of competitors, and I've learned the lesson that competitors are generally a good thing.
Appraise looks much more polished and ready to use. I'll definitely be trying it out for my current project. I had a bad UI bug recently where I forgot about scrollbars on Windows (I used offsetWidth instead of scrollWidth.) I would like to take a lot of screenshots on different browsers and platforms, to make sure I don't have any regressions. I'm going to look into using Appraise for that.
If you are interested in testing it out, you can reach out to me at the e-mail in my profile. I'm especially interested in making it work for use-cases that a lot of developers have to solve, but are somewhat underdocumented right now, like testing on Windows/IE10 (I have the exact same requirements at a current freelance client).
1. Are there plans for a multi-browser (especially mobile web) test runner?
2. Are there plans to detect change provenance (eg. if I change my heading height on a page, all the content below it will shift down. The change I really want to be notified about is that the header's height changed, not that everything was pushed down because of it)?
3. There are a lot of similar tools (eg. back in the day I used https://github.com/facebookarchive/huxley) - where does Appraise succeed where those tools failed, or what does Appraise do better?
the screenshot engine is designed to be pluggable, so theoretically this should be easy to do with some kind of mobile-test-farm runner.
> 2. Are there plans to detect change provenance...
You can set the relevant clip region to just the header, so it will only notify you if the header changed and ignore the rest. check out https://github.com/AppraiseQA/appraise/blob/master/examples/...
>3. There are a lot of similar tools...
appraise is designed to be used like fitnesse (so example+fixture=> result) instead of record/replay tools that end up being fragile. it makes the examples easy to understand, splitting what is being tested and how it will be tested clearly so both of those aspects are easy to maintain.
it's also designed to work well with github markdown, so your tests become executable API/component specs that are easy to read and also easy to verify.
I'm working on an AWS Lambda runner now, which will also make it quick and cheap to run UI tests in parallel, so it will provide reasonably fast feedback for large test suites.
Here is what this looks like in the much denigrated Java Enterprise world:
https://eclipsesource.com/blogs/tutorials/rcp-testing-tool-r...
Not necessarily, if you use it for what it was intended -- visual layout tests that are easy for a human to approve/reject when there is a difference. It doesn't know if the change is right or wrong, just helps humans be more effective by showing the difference clearly and aiding in comparisons.
Would it be even possible to test the material design rules across and entire page without having to define what individual components across the entire website look like?
Appraise is designed to make it easy for a human to approve/reject current state or changes, with a smart visual diff and a good way to show actual and expected pictures side-by-side or as an overlay, but it currently doesn't have any smart logic beyond that. Theoretically it could do something smarter as a comparison.
As it stands now, you could run it once to capture the current state, and get someone to approve it, then it becomes a regression test pack and the tool would alert to any changes with a smart diff.
1) niffy seems to be designed to use UI interactions to lead you to the result, appraise works like fitnesse, separating what is being tested from how it's being tested, so you could go several levels deeper than UI interactions and eg test how your models and views produce outcomes, without the http serving and web pages in between. this makes tests faster and easier to maintain. appraise is more for visual unit testing.
2) niffy seems to be optimised for developers, appraise is intended for cross-functional collaboration between developers, testers and designers. with appraise, the test is a markdown page that renders nicely in github and shows expected results and the conditions under which they should happen.
https://news.ycombinator.com/newsguidelines.html
https://news.ycombinator.com/showhn.html
Substantive critique is welcome, of course, but in this case that would mean talking about problems with visual testing and making it clear you're not picking on this particular implementation.