May 28, 2008

Adobe's David Coletta on Buzzword's Automated Testing Framework

This past Tuesday the Boston Flex User Group held its second meeting. David Coletta from Adobe's Buzzword team spoke about Automated Testing Framework. This was a deeper dive into the topic he presented at Flex Camp Boston 2007. I counted about 50 people at the meeting. It was a very interactive session with a bunch of good questions coming from the audience. Below are my rough notes from the presentation.

David's has a long background in development but hasn't programmed in Flash. Sees work done in Flash and wonders how that was possible, gets same reaction for work he's done in Flex.

Rough Buzzword time line: Jan 2006, started coding; Oct 2006, First public demo; Fall 2007, opened up for real use.

He gave a quick demo of Buzzword. On startup that an issues with new unsupported browsers. They have support for latest Safari but it isn't released yet. Big advantage of online word processor is like Google Docs you can get at it from anywhere. Better than Google Docs because you aren't confined to email only like editing capabilities.

Document browser with support for sorting by last changed, last viewed, role on document, date, number of pages.
Google can't do pagination. Showing picture import, resize, and text wrapping around it. Layout is all done on the fly. Has a nice ruler that no one knows about.

Three models (in the MVC sense used):
document (persistent user content, hierarchical, moving text is just a tree operation), layout (transient position of content, updated after every keystroke), and display (flash/flex objects on the display list, mostly tracking what is part of the flash player).

Primary loop is: hit a character, inserts into document, fires events, triggers layout, visuals, and redraw.

Document internals: Containment (document) versus Inheritance (run).

Buzzword has a built-in inspector. View and set properties manually. Exposes raw data structure.
Typically when using a tree component you can hook it up to XML. With ITreeDataDescriptor you can plug directly into your own custom data structures. Your application should have debugging tools built into it. Flex's easy to use out of the box components without much styling are a good enough interface to create these tools quickly.

Question about why should someone trust Buzzword for documents?: Basically just need to "trust us". SWFs are not over SSL but all documents are. Adobe practices good data security. Need to ask "how bad would a security breach be for Adobe?" and understand they don't want that to happen.

Created a way to test the real application through a system called "LiveTest". Needed to test correctness of document model after every command. Not a replacement for QTP as it is focused on state of internal models. Internally everything is done via command pattern.

To test some feature it records document model after changes to create known good state. Windows/IE used as platform using ActiveX control to write out state into files. These can later be rerun to compare that the same results are produced each time.

Question about how much effort it took to build the test harness? Not sure, maybe 10% of total time.

Comment made about inability to type in full screen mode. It's a security concern which is why its disallowed.

Commands run during a live test know how to output an ActionScript version of themselves that can be pasted into a .as file to create the test file. Could send to server and compile and send back a SWF, but wanted a light-weight solution. Pieces of test are broken into anonymous function, one for each part. Done to work around issue that if ActionScript runs for too long Flash player freezes. Cooperative multiple tasking. Separate functions can run and return control back to UI.

Question about generating ActionScript code versus interrupting XML? Going through API on top of Buzzword to make testing possible. Built this way to ensure you have a good API. Being bale to set breakpoints in code versus XML is very handy.

New code needs to be referenced so they have a file that pulls in all of the tests so they are compiled in.

Manual training of the test. Took XML snapshots of document and layout and wrote them to disk. XML documents have representations of model objects. Has green thumbs up or red thumbs down. Good discipline about running all test before doing a check in.

Test success based on final XML matching checked in XML. On test failure outputs error files and do text compare of expected and actual. Almost no test is visually analyzed, mostly look at raw XML.

Question about sharing the tests? Yes, every developer has all tests, checked into source control.

Question about if you started over today would you change anything? Wouldn't use IE to write files, probably use AIR instead. Might consider an XML representation for capturing test steps. Do use FlexUnit for lower level API testing.

Question about why not QTP? Why doesn't Adobe provide more testing frameworks? Testing tool capturing at a much lower level than QTP. Could refactor testing tool to be more generic.

Question about how something gets retrained? New kind of break, causes document model to change. May cause new XML to be written out and tests will fail. Now need to generate new XML.

Question about how much of the code is covered by the tests? Look to FlexCover.

Question about handling documents from older versions? Can upgrade documents when opened. Have tests that exercise that process. May have fringe cases that cause problems but hasn't been an issue, yet.

Document organizer testing is harder, need to get into correct state, initialize test user with known set of documents.

Browser tests are only running in IE. Haven't really looked at memory profile under multiple browsers.

Tags: buzzword flex testing

December 31, 2007

Flex Camp Boston 2007: Building Buzzword

Rough draft of notes

David Coletta is talking about Building Buzzword.

Jan 2006, started coding in an attic. Oct 2006, public demo at MAX and funding from Adobe. Nov 2006, more people and real offices. May-Aug 2007, monthly preview releases. Fall 2007, open for real use.

Show documents by alphabetically, author, role, last viewed, last modified, # of pages. Custom scroll bar with numbers in it. Able to click on scrollbar to jump to location. Leave comments in document. Also supports tables and advanced formatting capability.

Text on page. Three main pieces: document (content being saved to server), layout (transient position of the content, relay out after every character), and display (Flex/Flash objects on display list). Flow: document to layout. Tries to optimize what changed and handle as few things as possible.

Document internals: Body with one Section has multiple Paragraphs each with multiple Runs. Run has AnchoRun and FontyRun as subclasses. FontyRun has BreakRun, FieldRun, and TextRun and subclasses.

Use custom Tree ITreeDataDescriptor to provide custom tree data providers that can manage any custom data type. Not just XML.

Screen metric - print metrics = micro-spacing to get consistent layout between screen and printing. Very expensive to calculate but vital to getting consistent layout between the two.

Cache of display elements so that only certain objects are shown at once. Buzzword controls what is rendered on screen since the number of text fields for a document would overwhelm the Flash player.

Buzzword back end is .NET and uses low level HTTPService for back end communication. Backend does comment notification, document locking, and other raw CMS features. Login over SSL. Parking HTTP request on the server which then responds when it has data to send to the client.

Testing was vital primarily used FlexUnit. Spent a lot of time developing custom test framework called LiveTest. Focused on document command validity not focused on UI (like QTP). Able to record, debug, train, and playback a test. Generated ActionScript code can be copied back into the platform. Wrote custom framework instead of something like QTP required more constant feeding of unit tests.

Custom skinning scroll bar in Flex 2 was difficult. Hard to get skin to draw text (much improved in Flex 3 with UIComponent skinning model).

Has modular loading. Caches editors, styles, fonts, but if not there is displaying a progress loading bar. Pro: Improves build times, background loading, and good design through information hiding. Con: Can increase overall download, slow overall build, and increased complexity. Debugging across modules is hard, need runtime switching of loading debug versus production. Centralized control of what gets loaded when based on startup mode to prioritize based on startup mode.

Merge user dictionary on a collaborative document into a document dictionary to provide consistent spell checking. Look at for spell checking libraries.

Rich Text Copying is not supported by Flash. Hidden IFRAME or DIV render content from HTML control to allow clipboard content to be rich content. Push focus back to browser so that it picks up copy/paste events.

Off the shelf Flex isn't used much (more for debugging components), leveraging low level components (UITextField). 90% custom AS versus 10% MXML.

David's blog:

Tags: buzzword flex flexcampboston2007