Adobe's David Coletta on Buzzword's Automated Testing Framework
This past Tuesday the Boston Flex User Group held its second meeting. David Coletta from Adobe's Buzzword team spoke about Automated Testing Framework. This was a deeper dive into the topic he presented at Flex Camp Boston 2007. I counted about 50 people at the meeting. It was a very interactive session with a bunch of good questions coming from the audience. Below are my rough notes from the presentation.
David's has a long background in development but hasn't programmed in Flash. Sees work done in Flash and wonders how that was possible, gets same reaction for work he's done in Flex.
Rough Buzzword time line: Jan 2006, started coding; Oct 2006, First public demo; Fall 2007, opened up for real use.
He gave a quick demo of Buzzword. On startup that an issues with new unsupported browsers. They have support for latest Safari but it isn't released yet. Big advantage of online word processor is like Google Docs you can get at it from anywhere. Better than Google Docs because you aren't confined to email only like editing capabilities.
Document browser with support for sorting by last changed, last viewed, role on document, date, number of pages.
Google can't do pagination. Showing picture import, resize, and text wrapping around it. Layout is all done on the fly. Has a nice ruler that no one knows about.
Three models (in the MVC sense used):
document (persistent user content, hierarchical, moving text is just a tree operation), layout (transient position of content, updated after every keystroke), and display (flash/flex objects on the display list, mostly tracking what is part of the flash player).
Primary loop is: hit a character, inserts into document, fires events, triggers layout, visuals, and redraw.
Document internals: Containment (document) versus Inheritance (run).
Buzzword has a built-in inspector. View and set properties manually. Exposes raw data structure.
Typically when using a tree component you can hook it up to XML. With ITreeDataDescriptor you can plug directly into your own custom data structures. Your application should have debugging tools built into it. Flex's easy to use out of the box components without much styling are a good enough interface to create these tools quickly.
Question about why should someone trust Buzzword for documents?: Basically just need to "trust us". SWFs are not over SSL but all documents are. Adobe practices good data security. Need to ask "how bad would a security breach be for Adobe?" and understand they don't want that to happen.
Created a way to test the real application through a system called "LiveTest". Needed to test correctness of document model after every command. Not a replacement for QTP as it is focused on state of internal models. Internally everything is done via command pattern.
To test some feature it records document model after changes to create known good state. Windows/IE used as platform using ActiveX control to write out state into files. These can later be rerun to compare that the same results are produced each time.
Question about how much effort it took to build the test harness? Not sure, maybe 10% of total time.
Comment made about inability to type in full screen mode. It's a security concern which is why its disallowed.
Commands run during a live test know how to output an ActionScript version of themselves that can be pasted into a .as file to create the test file. Could send to server and compile and send back a SWF, but wanted a light-weight solution. Pieces of test are broken into anonymous function, one for each part. Done to work around issue that if ActionScript runs for too long Flash player freezes. Cooperative multiple tasking. Separate functions can run and return control back to UI.
Question about generating ActionScript code versus interrupting XML? Going through API on top of Buzzword to make testing possible. Built this way to ensure you have a good API. Being bale to set breakpoints in code versus XML is very handy.
New code needs to be referenced so they have a file that pulls in all of the tests so they are compiled in.
Manual training of the test. Took XML snapshots of document and layout and wrote them to disk. XML documents have representations of model objects. Has green thumbs up or red thumbs down. Good discipline about running all test before doing a check in.
Test success based on final XML matching checked in XML. On test failure outputs error files and do text compare of expected and actual. Almost no test is visually analyzed, mostly look at raw XML.
Question about sharing the tests? Yes, every developer has all tests, checked into source control.
Question about if you started over today would you change anything? Wouldn't use IE to write files, probably use AIR instead. Might consider an XML representation for capturing test steps. Do use FlexUnit for lower level API testing.
Question about why not QTP? Why doesn't Adobe provide more testing frameworks? Testing tool capturing at a much lower level than QTP. Could refactor testing tool to be more generic.
Question about how something gets retrained? New kind of break, causes document model to change. May cause new XML to be written out and tests will fail. Now need to generate new XML.
Question about how much of the code is covered by the tests? Look to FlexCover.
Question about handling documents from older versions? Can upgrade documents when opened. Have tests that exercise that process. May have fringe cases that cause problems but hasn't been an issue, yet.
Document organizer testing is harder, need to get into correct state, initialize test user with known set of documents.
Browser tests are only running in IE. Haven't really looked at memory profile under multiple browsers.