Moving some of the comments from the discussion into an answer, with re-wording and re-formatting..
Basically, what it comes down to is that unless you have a super extreme case, they don't really need to be "garbage collected". If you never fetch them, then it doesn't matter if they're there or not.
See, transients are stored in the options table by default. In a base install, the options table will have maybe 100 entries in it. Each transient adds two more entries, but even if you have thousands, they don't affect the site speed, since they're not autoloaded.
On startup, WordPress loads the options into memory, but it only loads options that have their autoload flag turned on. Transients don't get this, and so don't get loaded into memory. Only transients that get actually used later will incur a cost.
From the database's perspective, the options table has indexes on both the option Id and the option name. Transients are always loaded based on the name (key), and so the lookups for them are always simple selects on a single unique key value. Thus the lookup is O(log(n)) and is super fast. With a Big-O of log(n), you'd have to get into the millions and millions of rows before it became noticable. Frankly, the overhead in the setup and teardown of the query, along with the actual data transfer, is way longer. The query itself runs in essentially zero-time by comparison. So simply having extra unused rows doesn't affect anything but using extra disk space.
Indexing in databases is one of those deep-read kind of ideas that doesn't make sense to people who haven't actually understood what's going on behind the scenes. Databases are designed for fast data retrieval, from the ground up, and can handle this sort of thing without issues. This is a pretty good read: http://en.wikipedia.org/wiki/Index_(database)
Now, cleanup in the most obvious way (calling SQL DELETE on them) doesn't actually delete them from the database. It just removes them from the index and marks the row as "deleted". Again, this is just how databases work. To actually clear up the disk space, you have to then continue on and do an OPTIMIZE TABLE afterwards, and this is not a fast operation. It takes time. Probably more time than it's worth. It's probably not enough to give you a savings in CPU time, in total.
If you have some case that is causing a continual insertion of new transients that are not being used, then you need to find the underlying problem instead. What is inserting these transients? Are they using a changing or mutating key? If so, then the plugin or code causing this should be fixed to, basically, not do that. That will be more helpful, because it's likely that the code that isn't creating them properly also isn't retrieving them, and thus doing more work than it has to do.
On the other hand, there may be a case where transients are being created for something like every post. This may indeed be perfectly acceptable. I do this myself in SFC, to store incoming comments from Facebook. Each post has a potential transient associated with it, which means two extra rows per post. If you have 10k posts, you'll have 20k rows in the options table (eventually). This isn't bad or slow, because again, there's very little difference between 100 rows and 20,000 rows as far as databases really care. It's all indexed. It's fast as heck. Sub-sub-milliseconds.
When you start getting into millions of rows, then I'd be worried. When the options table size increases above hundreds of megabytes, then I'd be concerned enough to take a closer look. But generally speaking, this isn't an issue except for extreme cases. It's certainly not an issue for anything smaller than something like a large news site, with hundreds of thousands of posts. And for any site large enough for it to be a problem, you should be using an external object cache of some sort, and in that case, the transients get automagically stored there instead of in the database.