mark roseman

A Busy Developer's Guide to Tcl/Tk 8.5

Current Status of this Document: This document is current as of the production 8.5.0 release. Last update December 21, 2007.

This document is intended to quickly introduce "mainstream" developers who have been using earlier versions of Tcl/Tk to the most important new features in Tcl/Tk 8.5 that they need to know about. This isn't a manager's high-level guide, but covers things that affect your code on a daily basis. It's also not something that would interest the type of people who follow daily commits to the Tcl/Tk CVS repositories.

I'm not going to cover every features (and there are lots!), and many things that aren't covered or are covered in less detail are very important to some people; my idea of "mainstream" developer is purely subjective. If you want more comprehensive information, it's available elsewhere. A good starting point is the Changes in Tcl/Tk 8.5 page on the Tcler's Wiki.

Tcl 8.5:
What Every Developer Needs to Know

Along with the usual bug fixes and incremental improvements, Tcl 8.5 removes a lot of "warts" in previous versions, and adopting its features will make your programs better. These aren't all necessarily big things, but many are so common that they're definitely worth knowing about.

Dictionaries: like arrays, but pass-by-value

Dictionaries are similar to arrays (i.e. hash tables), in that they allow you to store many values together, each indexed by a key. Unlike arrays, they are values, meaning a dictionary can be efficiently passed as a value (not just a name) to a procedure. They are the first new data type the Tcl core has gained in a long time.

Before 8.5: To pass arrays around, you were forced to either use the "upvar" wart:# accept the name of an array
proc processarray {arrayName} {
     upvar $arrayName a; # you just scared off the Tcl newbies...
     # use $a(name), $a(age), etc...
array set data {name Bob age 20 country USA}
processarray data  ; # eww, have to pass the name

Before 8.5: Or, you could use "array get" and "array set" to pass by value, but with tons of copying, which is excruciatingly slow:# accept the flattened form of the array
proc processarray {arraydata} {
     array set a $arraydata; # copy, copy...
     # use $a(name), $a(age), etc...
array set data {name Bob age 20 country USA}
processarray [array get data]; # copy, copy, copy...

With 8.5: While the string representation of dictionaries look like the "array get" form (i.e. list of "key val key val..."), internally they are very efficient.# accept a dictionary
proc processdict {dict} {
    # ... use [dict get $dict name], [dict get $dict age], etc...
set data [dict create name Bob age 20 country USA]
processdict $data; # hey, actual pass by value!

The expected range of commands to manipulate dictionaries are bundled in the "dict" command. The commands shown here are a useful subset.dict create ?key value...?
dict get dictValue ?key...?
dict set dictValue key value
dict exists dictValue key ?key...?
dict unset dictValue key ?key...?
dict replace dictValue ?key value...?
dict keys dictValue ?glob?
dict values dictValue ?glob?
dict for {keyVar valueVar} dictValue body
dict size dictVal

Dictionaries have a lot more features, and include support for things like nested dictionaries, modifying individual values in a dictionary in-place (append, incr, lappend) and more. See TIP #111 or the "dict" manual page for more information.

New {*} syntax: avoid abuse of eval

It is all too common in Tcl and Tk programs that you have a list of several things, that need to be passed as separate parameters to a command. The existing mechanism is ugly, inefficient, error-prone and in fact can even be dangerous. It is a frequent source of problems in even expert Tcl programmers' code.

Before 8.5: To break up a list into pieces to pass it to a command, you'd use "eval"set kids [winfo children .] ; #returns a list
eval destroy $kids; # destroy needs each window as a separate arg

Before 8.5: The actual correct and always safe way to do this is this ugly wart.set kids [winfo children .]
eval [linsert 0 $kids destroy]; # this looks backwards...

With 8.5: This can be written more clearly and safely using the new {*} syntax, which breaks up the list in place and passes it as separate arguments to the command.set kids [winfo children .]
destroy {*}$kids

This is the first new piece of syntax in Tcl in many years (the parser now has 12 rules, not 11). And yes, it's itself an ugly wart, that was the subject of long consideration. However, the problem it solves is so pervasive and damaging that it is unfortunately a much needed change. It's very much worth checking the "eval" calls in your existing code to see if they can be replaced by {*}, and certainly think twice before using eval in the future to explode arguments.

Note that earlier alphas of 8.5 used "{expand}". These should be changed to use "{*}".

List membership with "in" and "ni": enough with lsearch

This is a small one, but removes one of the most common warts in Tcl programs, checking whether something is a member of a list.

Before 8.5: While lsearch does the job, it's pretty verbose and indirect, making it harder to read. Plus I can never remember the order of arguments to lsearch.if {[lsearch -exact $list $item]!=-1} {
   # item is in the list...

With 8.5: There are new expression operators which let you check if an item is in a list ("in"), or if its not in the list ("ni").if {$item in $list} {

New expr operators: min, max, and **

With 8.5: Again these are small but common things, but expr finally gains "min" and "max" operators, and can now use "**" for exponentiation, which can be more readable than "pow()".expr {min(1,10,20)}
expr {max(2,5)}
expr {2.0*x**3 - 1.2*$x**2 + 3.0*$x + 4.0}

Lassign: assign elements of list into variables

Breaking up a list and storing each element in a separate variable is not hard to do, but it's a very common operation. For example, commands in Tcl can only return a single value, so if they have to return more than one value they pack them all into a list. The caller then has to unpack them.

Before 8.5: The readable way to do this has always been using multiple "lindex" calls, but for the lover of obscurity, it was also possible with "foreach".set coords [getBoundingBox $thing]
set x0 [lindex $coords 0]
set y0 [lindex $coords 1]
set x1 [lindex $coords 2]
set y1 [lindex $coords 3]

foreach {x0 y0 x1 y1} [getBoundingBox $thing] {}; # ewww....

With 8.5: The "lassign" command, originally from TclX, now makes this easy and clear.lassign [getBoundingBox $thing] x0 y0 x1 y1

Incr auto-initialization: simplify counters

Before 8.5: Not hard, but doing counters in Tcl always involved the same dance.if {![info exists counter]} {
   set counter 0
incr counter

With 8.5: Now if a variable doesn't exist, incr will auto-initialize it with a value of 0.incr counter

New "chan" command: deprecates several f* I/O commands

With 8.5: A new "chan" command gathers together a number of operations that work on channels into a single command, to curb some of the global namespace usage for I/O. This deprecates several existing commands, so plan to move to the new versions of those soon.chan blocked    ; # fblocked (DEPRECATED)
chan close      ; # close
chan configure  ; # fconfigure (DEPRECATED)
chan copy       ; # fcopy (DEPRECATED)
chan eof        ; # eof
chan event      ; # fileevent (DEPRECATED)
chan flush      ; # flush
chan gets       ; # gets
chan names      ; # file channels (DEPRECATED)
chan puts       ; # puts
chan read       ; # read
chan seek       ; # seek
chan tell       ; # tell
chan truncate   ; # ftruncate

Tcl Modules: a new, more effective way of packaging extensions

Tcl Modules are an important new mechanism to construct, deliver and deploy Tcl packages. You'll start seeing more and more packages available as Tcl modules, and in particular are the format used by Teapot, the new but long-awaited package repository that Activestate is providing. Yes, after all these years, a central place to locate and download Tcl packages.

Tcl Modules complement, but do not replace the existing package mechanism Because they are simpler, they offer several advantages. Traditional packages consist of several files, with a "pkgIndex.tcl" script pulling the pieces together, identifying the package and version number, etc. Tcl Modules consist of a single, source-able file (with extension ".tm") whose package and version number must be encoded in the filename.

As a result, Tcl Modules are simpler to store (a single file), and much more efficient to search and load (the search paths are more restricted, and only the module's filename needs to be examined).

For more on Tcl Modules, see TIP #189.

Tcl 8.5: Other Things You Should Know

Less common or pervasive than the previous changes, the improvements in this section will still be of use in a good number of situations, or, like the new clock command, have undergone such significant change that it warrants being aware of.

Arbitrary-precision integers. Tcl now transparently supports arbitrarily-sized integers (bignums). Rather than generating overflows that became a source of bugs, Tcl now promotes integer to bignums when needed. Of course, this is particularly good if you know you're going to be dealing with large integers, and are (or otherwise would be) using another library to do so. A new "string is wideinteger" command lets you check if a string will fit into a bignum.

New clock command. The "clock" command has been almost entirely rewritten, to deal with a wide range of corner cases, weird input formats, localization issues, platform-specific bugs, bizarre time zones, and more. The new version is almost entirely compatible, unless you were depending on existing bugs. There is also new finer-grained "milliseconds" and "microseconds" options. Note that because the new clock implementation is mostly Tcl, it is somewhat slower than before. As well, Tcl now includes a very sizable collection of timezone data files, which may present issues for packaging and deployment.

New and improved list operations. A new "lrepeat" command makes constructing large lists easy. Lsearch gets a new "-index" option for searching in nested lists (like lsort). Finally, lsearch, lsort, and switch get new "-nocase" options to do case-insensitive comparisons, which should remove a few "string tolower" commands from your programs.

Encodings for Tcl scripts. Both the "source" command and the "tclsh" and "wish" applications now allow specifying a script's encoding via a "-encoding" flag, as opposed to assuming the system encoding.

New "apply" command for anonymous (lambda) procedures. This will be a welcome addition for people who enjoy functional programming. But it's also a way to create commands on the fly that you can pass around and call, without needing to create new entries in the command table. If you often create new commands with dummy names, and then pass around the names, this may be a better solution.

Files, pipes and exec. Both the "open |" and "exec" commands get a new "2>@1" syntax, which allows you to merge and capture both stdout and stderr. Open also gets a new "b" access mode for opening binary files, which automatically does the equivalent of a "fconfigure -translation binary" for you. Finally, "file attributes" and "file copy" add support for several Mac OS X specific features.

Namespace improvements. For developers who make extensive uses of namespaces in creative ways (more common than perhaps it need be), there are new "namespace ensemble", "namespace unknown", and "namespace upvar" commands, that may simplify some of your existing namespace manipulations.

Modernized TEA build system. For developers who build Tcl extensions who were scared off by earlier, increasingly complex versions of the TEA package specification and tools, newer versions of TEA have removed many of the most annoying problems, so it's worth revisiting converting your extension to use TEA.

Tk 8.5:
What Every Developer Needs to Know

On the Tk side, far and away the most significant change is the inclusion of the "themed Tk" widgets. If you care at all about look and feel, this is one thing you definitely need to know about.

Themed Tk Widgets

Alongside the classic Tk widgets, 8.5 introduces a new, complementary set of widgets, collectively referred to as "themed Tk" widgets. This is based on work done by Joe English, originally under the name "Tile".

Better platform look and feel. For developers working on mainstream Tk applications, the most important aspect of these themed widgets is that they deliver a look and feel more closely resembling the platform they are running on, as opposed to the often-dated appearance on some platforms (e.g. Linux) that many people have complained about. They also include some new widgets, e.g. combobox, that are common in today's user interfaces, but were only available as extensions in standard Tk.

As the name suggests, underlying the themed widget set is a mechanism allowing you to define new appearances via themes or styles, rather than via widget-by-widget customization or option database changes as is done with Tk. However, for most applications, the theming aspect will be secondary; the main value will be in the better appearance and new widgets.

How to use. The existing Tk widgets still remain, available through commands such as "button", but they also now are accessible via a "tk::" namespace, e.g. "tk::button". The themed widgets exist in a "ttk::" namespace, so there is now a "ttk::button" command.

Though the themed widget commands work similarly to the originals, there are important differences. Themed widgets are not a drop-in replacement. In particular, themed widgets generally provide less options for customizing their appearance than regular Tk widgets (e.g. they will often not have options like "-background"). Such changes, if needed (and they should be needed much less often) must be done by defining new widget styles, using the facilities offered by the themed widget package.

Complement, not replacement. Themed Tk widgets provide a better match for the "typical" uses of a widget, at the expense of the flexibility in many regular Tk widgets to turn themselves into something barely recognizable as the original. Because of this, applications will mostly want to use the themed widgets. However, to achieve particular effects, in special situations, it may make sense to use the regular Tk widget, if the flexibility it offers is all you need. One does not replace the other, and they can be mixed-and-matched as necessary within the same application.

Available ttk:: widgets. Alternatives to standard Tk widgets and containers: button, checkbutton, entry, label, menubutton, radiobutton, scale, scrollbar, frame, labelframe, panedwindow. New widgets not present in standard Tk: combobox, notebook, progressbar, separator, sizegrip. Note that these standard Tk widgets do not have (nor really need) an alternative in themed Tk: canvas, listbox, menu, spinbox, text.

The "widget" demo program that comes with Tk, though not currently fully updated to use Ttk everywhere, has added new demos that incorporate the new themed widgets. This is a good starting point to see how to use the new widgets and what they look like.

These themed widgets are the most significant upgrade to Tk in over ten years (since 8.0, the original port to Mac and Windows), and application developers should very seriously consider planning to migrate. This transition is non-trivial, because of the different approach to widget appearance that themed widgets take. The likely gains however are a better look and feel, removal of many existing hacks needed to make standard Tk widgets look better, and replacement of some extensions used to add widgets like notebooks. Widget builders should also examine the themed widget foundations, which provide a powerful way to construct new widgets. Because the themed widgets are still in rapid development, with new widgets planned, it's worth keeping an eye on future changes here.

Tk 8.5: Other Things You Should Know

Again, a few things that may well improve your programs, some without any changes needed on your part.

Antialiased text under X11, Mac OS X. This allows Tk to fit in much better with modern desktops; the lack of antialiased fonts has been a frequent criticism recently, particularly on Linux. Based on Xft on X11, ATSUI on OS X.

Mac Improvements. The Mac OS X version has received substantial improvements in look and feel of both the core and themed widgets, as well as many underlying optimizations, enhancements and modernizing.

Many text widget enhancements. These include smooth-scrolling (e.g. across tall images), new commands for counting characters, replacing text, handling display lines (not just logical lines), and more. See TIP #155 and TIP #169 for more.

Grid configure with widget name. It is now possible to pass the widget name to grid column/rowconfigure, whereas previously these accepted only a row/column index. This should make your grid layout code much more clear and obvious. It also simplifies things when adding new columns or rows after the fact, because you no longer have to adjust all the index numbers in your grid calls.

Tristate checkbutton and radiobuttons. Added a "-tristatevalue" that can be used to set a state where some but not all members of a collection of items are selected.

Full-Screen toplevels. A new "-fullscreen" option allows top-levels to be created which take up the entire screen, removing some of the pitfalls inherent in existing ad hoc methods of doing this.

Color window icons. A new "wm iconphoto" command allows specifying a color window icon, akin to the black and white icons you could use by "wm iconbitmap".

Enhanced font handling. There are now a wider variety of named fonts that you should use rather than specifying a particular font family, size, etc. to ensure that you get the right font for interface elements on each platform; see TIP #145.

Photo images catch memory errors. Photo images will now generate appropriate Tcl errors if enough memory is not available.

User-Data for virtual events. Virtual events can now specify a user-defined, event specific data value, useful for e.g. passing around an object or other additional information the event refers to. The "event generate" command now takes a "-data" option, while a "%d" in the event binding captures the data.

New panedwindow options. A new "-stretch" option allows more control over how different panes share the available space, while "-hide" is now available to control visibility.