XK72 | Java and Adobe Flash Software

In December last year Rich Riendeau reported excessive packet fragmentation from the throttling implementation in Charles v2.6.1. While the simulation of the bandwidth was accurate, it didn’t necessarily reflect the structure of the packets as they would actually exist on the network in a bandwidth limited situation.

Packets are usually be sent out onto the network in fragments equal to the MTU, however Charles tried to share the throttled bandwidth between multiple connections by allowing data through in chunks of approximately 1/10 of the available bandwidth. Therefore depending upon your throttling configuration, and the number of concurrent connections, Charles could produce more packet fragments than would be expected. So as of Charles v2.6.2 the throttling system has been tweaked to use MTU-size packets.

As a consequence of this there is an MTU setting as part of the throttling configuration. On a broadband connection (and on any Ethernet network) the MTU is often 1500 bytes, however on a modem or PPP connection the MTU is 576 bytes and packets must be fragmented to fit. Charles now has the capability to simulate the MTU differences on different types of connections and thus effect the appropriate realistic fragmentation.

Looking to the future: a feature that has been requested a few times is random throttling to simulate variable Internet conditions. I’ve spoken to several people about it and I think that its time may be near! If you’ve got any thoughts about it I’d be pleased to hear them.

Charles previously stored all of the request and response information in memory, so if you had Charles running for a few hours (or browsed a lot of images) it would quite often run out of its 64 MB of heap and stop recording until you had emptied the session. I have Charles running all day every day and I’d have to clear the session several times a day: frustrating.

Furthermore, in an effort to reduce the memory usage, Charles would only record the first 100 KB (or so, this was configurable) of binary files such as images, videos and executables. This helped to not use up all the free memory so quickly, however it also meant that you’d sometimes miss the end of something you were interested in. For example, with the introduction of AMF parsing (a binary format) I had to extend the default 1 MB cut off for binary files to avoid truncating large AMF messages.

But wait! In version 2.6.2 both problems have been solved. Charles now saves large requests and responses to temporary files on the hard drive, and there is just one configurable recording limit (defaulting to 10 MB per request/response) to prevent excessive storage. Recording still uses memory, and you’ll see the memory indicator increase in Charles the more you record, however it should be significantly slower to fill up completely.

On the 1st of February (one day before I was off to Kiwi Foo Camp) Charles v2.6.2 was released and can be downloaded from the usual place. I’m going to blog separately about the major new features in an effort to look more blog-prolific.

Here are the highlights:

• No more recording limits. Large responses are now saved to temporary files, reducing memory usage.
• MTU support in the throttle settings
• AMF3 / Flex 2 bug fixes

Thanks to bug reports from Patrick Mineault from AMFPHP, I have fixed some more AMF3 parsing bugs in Charles (currently in beta). With that comes an updated release of the AMF3 reference deserializer. This update includes object parsing improvements (externalisable and dynamic types now properly identified), mixed arrays (string keys) and reference bug fixes.

Thanks to everyone who has reported bugs or otherwise done work on uncovering AMF3.

Download the source code here: AMF3Deserializer.zip

A quick maintenance release was made to Charles today. This release includes improved behaviour for the SOAP viewer and AMF viewer (specifically for AMF3 / Flex applications), and a regression bug fix in the AMF3 parser.

The regression bug was related to parsing of externalizable ActionScript 3 classes; after the improvements to the AMF3 parser in v2.6 the implementation for externalizable classes wasn’t reinstituted. Thank you to Mike and Evert for drawing my attention to the problem.

Charles v2.6 has been released today. You can download it from the usual place. Thank you to everyone that helped during the beta period, especially Matthew Buchanan who was the inspiration for the visual improvements and finder of the icon set.

The list of changes:

• Major UI overhaul, including the use of Quaqua and
  JGoodies libraries to improve the native
  look and feel on Mac OS X and Windows respectively.
• Internal windows removed and replaced by a tab navigation if multiple sessions are open.
• Sequence view now uses a table when in Wide layout, to show a quick summary of important attributes.
• XML tree display. View XML documents in a tree format as well as the existing pretty-printed text format.
• XML pretty-printed text format improved to not load DTDs.
• JSON and JSON-RPC support.
• SOAP support.
• AMF3 parsing improvements.
• AMF parser now displays during the response download and shows download progress.
• Fixed nil dereference in Mac OS X proxy configuration in some configurations.
• Rewrite Tool body rules now support non-type mime types and improved debugging output.

Thanks for your feedback on the previous beta versions. I’ve added a few things to the latest beta, and I’m hoping this is the last (or at least second to last) before release.

Download the latest beta here

The recent additions are:

SOAP support. SOAP messages were always viewable as pretty-printed XML, or now as an XML tree; finally Charles presents a higher level view of SOAP messages in the same RPC format as for AMF (request and response on the same screen). I’d appreciate some feedback from developers who do SOAP development!

JSON-RPC support. Extending the JSON support already in the 2.6 betas with support for JSON-RPC. This provides an enhanced view (request and response on the same screen) for people using JSON-RPC.

Info column in sequence view. The info column gives a brief preview of what is in the request/response. For images it shows the dimensions; for SWFs it shows the dimensions, version and fps; for AMF/Flash Remoting and JSON-RPC it shows the method names. This lets you find the requests you’re looking for faster.

The info column will be expanding in the next release of Charles to support more types. Also I think a similar facility can be added to the structure view when it’s in wide mode.

Please let me know if you discover any problems or if you have any suggestions.

The past few weeks I’ve been working on improving the look, feel and usability of Charles. I’ve made a beta version available for download, please feel free to try it out and send me feedback. http://xk72.com/charles/beta.php

There are new icons for the toolbar and the tree. They come from http://www.icondrawer.com/. I’ve also integrated the excellent Swing GUI fixing libraries; Quaqua for Mac OS X and JGoodies for Windows, which help tidy up a few of the Swing look and feel oddities. This also explains why the download file size has ballooned!

The main interface has also changed; there are no longer internal frames with title bars, instead the current session is part of the main window, with tabs appearing when you have more than one session open. I believe that most users work with a single open session so this reclaims several pixels of screen real estate for you.

The sequence view now defaults to a vertical layout and shows a table, enabling you to see more details about each request/response without clicking on it. You can change this behaviour in the Preferences.

The request and response panels now default to combining headers and body viewers into the one set of tabs, rather than in two separate sets of tabs. This reduces the amount of chrome on the screen and is especially useful when you’re in vertical layout mode and vertical space is at a premium! Again you can change this behaviour in the Preferences.

There’s also JSON support and XML viewing as a tree as well as the existing pretty-printed text view.

There you go. As always I appreciate your feedback, and welcome suggestions.

cheers,
Karl

One month later and Charles v2.5 is tested and ready for release. There’s a massive list of changes, which should cover almost everything that has changed.

Please download and try out the new version. Please email me if you have any problems!

It’s been a few months since Charles 2.4.2 was released, and I’m now happy to report that I’m close to releasing version 2.5. This release will include a number of major and minor improvements. Here is a taster:

Improved viewing of cookies, images, multi-part form posts, AMF / Flash Remoting (this is a really neat improvement if you use AMF) and FLV files. Select multiple nodes in the tree to view summary statistics of those requests, or to save them all to disc. Keyboard shortcuts for switching between views. Regex filtering and row limiting on the sequence view. Automatic flushing of old recorded information to stop the session filling up. Host and path matching for recording ignore and tool configuration now with full wildcard support; so you can turn on No Caching on *.swf for example.

That’s only the interesting bits from a big list. I’m currently testing the changes and hope to have it release quality in a week or so.