Friday, March 17, 2017

Teaching a Machine to Regurgitate

Artificial intelligence (AI) is a topic that is dominating the news. Often, these stories are cast in a fearful light (e.g. your smart refrigerator will one day roll into your bedroom and kill your family). The truth is that while AI as both a topic of study and a commercialization opportunity is noticeably stronger than it has been in years, there's very little to fear from its ascendancy, at least for now.

To gain intelligence, something must be able to "learn." In order for something to learn, it has to be "taught." This process of teaching a machine to learn something is machine learning. As a discipline in computer science, machine learning has been around for decades, but putting it into practice has been difficult -- until now. Advances in computing power, storage, and data have pushed the field forward in significant ways; consequently, this has fueled its resurgence.

As we teach machines to learn, we as humans also learn how to teach them better. Indeed, years of research and experimentation in the field have led to huge growth in the machine learning ethos, and these techniques can be used to bring a semblance of intelligence to our computers. One of those machine learning methods is the neural network. The idea behind neural networks is to use vast computing resources to mimic the structure of the brain, with cells modeled after neurons that are attached to other neurons, firing as various thresholds are reached. These patterns can be trained to react to certain inputs (stimuli). Much research has been, and continues to be, poured into this area.

One particular type of neural network is a recurrent neural network, or RNN for short. RNNs are interesting in that they form internal, cyclical connections that can be used to mimic dynamic behavior of sorts. Think of it like a bunch of interconnected roads on a map that turn into and out of each other, allowing traffic to flow more than just one direction. This flow of information allows RNNs to learn differently than the traditional feed-forward networks.

Neural networks can be trained to do a number of tasks, including recognizing and regurgitating patterns, from images, to sound and music, to text. It's the latter that Andrej Karpathy explores in this blog on the effectiveness of RNNs. To demonstrate his point, he trains an RNN on text from numerous plays written by Shakespeare. This creates a model that is then used to generate text that reads a lot like Shakespeare, although it's not necessarily sensical.

So what does this all have to do with the CoCo?

As you read Karpathy's blog, it should occur to you that any collection of text could be used to feed into and train an RNN like the one he discusses. Why stop with Shakespeare? Any sufficiently large body of text is representative of a certain style of writing. Terminology, sentence structure, the use of punctuation... all of this can be "learned" and used to generate new data in the same style and vein. The Malted Media CoCo List is itself a large body of text -- it happens to be composed of many authors, each with their own word usage and writing style. However, it has a common structure: everything is in the context of an email, containing headers, subject, body, and signature. There's sentence structure, quoting other messages, and even repeatable terminology (CoCo, GIME, OS-9, etc.)

With this in mind, I decided to apply Karpathy's RNN work to training on the entire set of CoCo List raw text data from December 2003 to September 2016. Gathering the data was a matter of pulling down the monthly archives available on the maltedmedia.com site, then concatenating all of that data together into one large file comprised of 7,985,340 lines of text, 45,272,983 words, or 288,690,945 characters (bytes).

Training on a quad-core Intel Core i7 at 3.4GHz running Ubuntu Linux took exactly 14,654 minutes and 58 seconds (let's be generous and say 14,655 minutes). That's 244.25 hours, or 10.177 days to train an RNN with all of that data. The resulting model file came out to 2GB in size.

Now you can see the model regurgitate somewhat meaningful CoCo List messages by clicking here.

Here's some sample output:

>To: ColorComputer at yahoogroups.com, "James M color at yahoo.com) says
my threat Aaron files and I was as well except reads platfor
allowed by a way to make one.
http://www.boy.com?}_So---Mug
12V quite a %dolleddlls prompted, so that it wouldn't include it and
going for change ports, but what
the lowercase could be group to
allow the earth. I already exist with "owner" had the asked just application on 4 years. Also why
a Dragon 2560 for Gun & e-mak with the disk reference.

Run is replaceing inteal for PC cartridge domain needed
'descriptor box pully Pin Cloud-9 USB drives on my
seems to be the authors are hard come.

The online?


I really happen to have to see it is directed by cpu data registration with Drive
computer??? Anyway, any
rest of the weag gback ever. :

Just polited to "bus and lc and 12 bit format). Been the interruct and discussion out. This)...

Message: 7
11:09 Nox6993 #DSK=500+21O auction
+723 Electronic Oh
Topit controlments in 279-30-1004

-Aaron

-----
Cheers, Glenith at optode.com ?

----------

-------------------------------

> I have currently "need from disk image rares). in specifics plus: *** COCO Daying of
> multi-Page Disks' need on prefer or the "but Magazine GIME" and
> other
> > available?
> NitrOS-9's made command code.
>
> Marcution from acsaging to actually learn more of user removable, but keep this memory or a cool.hn Luis I say and 300, 3. I love one
> but this is not writting that thread but based in that Cloud-9... the 1.4N talk about?
>
> Cocolist platform I have made for tweak and
> work because of
they are not zero and it always looking or have the
> Disto OS-9 disks screen, too, do a disk image you? Is a personal MESS ERROR if there is a end you?
>
> What should possibher.
>
> The
work if not up Wildort Crs9/new prompt possing the faster people work in chance to afford
> and I reavenPil config as not going to a development internal language and
At a glance, it looks like a message on a mailing list. There's a semblance of a header at the top, then there's a body of text, what looks like a signature, and the quoting of another message with the '>' character indicating the previous reply in the thread.

The sentences may be somewhat awkward to read, but for the most part, there are sensible words there.  You can see references to recognizable words in the CoCo lexicon: OS-9, Cloud-9, Disto, etc.

What's interesting is just how well the RNN mimics a CoCo List message. It's not perfect, but it's quite close, and it provides just enough of a mixture at times to make for a bit of humor.

It's a simple, but interesting example of the power and utility of RNNs. I suggest reading Andrej Karpathy's previously referenced blog post for the details and behind this. And don't forget to refresh your browser to get another dose of random CoCo List goodness.

Friday, July 31, 2015

"Made In The U.S.A" - Why American CoCo 3s Are Quality-Challenged

[Note: Back in 2000, I wrote a white paper on the differences between CoCo 3s made in Korea (from the 1986-1987 era) versus the ones manufactured later at Tandy's Fort Worth factory (from 1988 onwards). My conclusion then was that the CoCo 3s made in America were somewhat inferior in a number of respects, and I continue to stand by that. I thought I would revisit the topic and move the contents of the paper into this blog especially in light of some additional information I've come across recently. Most of the information here, including the photos, are from the original 2000 white paper.]

"Made In The U.S.A" - Why American CoCo 3s Are Quality-Challenged

It is well known that over the course of the Color Computer's production, Tandy manufactured Color Computers both overseas and in the United States. One of the more interesting comparisons can be made between the American and Korean models of the Tandy Color Computer 3.

The Korean model of the Color Computer 3 was manufactured in Korea between 1986 and 1987. Sometime in 1988 (see this link for info on Tandy's move), the switch was made to a factory in the United States. Although both models are virtually the same and functionally identical, there are some interesting differences, minor as they are, which may yield clues as to Tandy's decision to move production from overseas to home.

For this post, I'll compare two CoCo 3s:

  • An American made 128K Tandy Color Computer 3 SN#1299880
  • A Korean made 128K Tandy Color Computer 3 SN#1144677

Exterior Differences

Look carefully at a Korean and American CoCo 3 side by side; at first glance, it's not immediately noticeable that they are different. In fact, without the original packaging and box, they look identical. However, just examining the box will give away a CoCo 3's country of origin.

Packaging

Korean CoCo 3's were shipped to Radio Shack stores in a predominately white box with red lettering and graphics. The picture on the front cover of the box shows the outline of a CoCo 3, and the sides of the box give a summary of features.

The later American CoCo 3 was packaged in a more attractive, sturdier white box with blue and red lettering and graphics. Featured on the front top of the box are two children sitting in front of a CoCo 3, while the sides depict graphics and lettering oriented towards younger computer users.

This more "kid friendly" packaging fit well with Tandy's promotion of the CoCo 3 as a game machine towards the end of its production run in 1988-1989.
The later American CoCo 3 box on the left compared to the original Korean CoCo 3 box on the right.

Outside Case

As mentioned earlier, the Korean and American CoCo 3 look virtually identical from the outside. However, there are a few minor clues which can tell the two apart without even opening them up.

The Nameplate - The word "TANDY" on the American CoCo 3's nameplate is slightly larger than it is on the Korean CoCo 3.

The Korean CoCo 3's badge has a smaller "TANDY" logo and no space between the "128" and "K".
This American CoCo 3 badge has a larger, fatter "TANDY" logo and a space between "128" and "K". Note that the red/green/blue color bars have shifted on the plate, indicating a bit of carelessness on the part of Tandy QC. 









The Power Button - On a Korean CoCo 3, the power button on the back protrudes out further in the "OFF" position. On an American CoCo 3, the power button is shorter.


On this American CoCo 3, the power button does not protrude out as far as it does on the Korean CoCo 3, pointing to a different parts supplier. Note the button is in the OFF position.

Here on this Korean CoCo 3, the power button (also in the OFF position) extends out of the case noticeably compared to its American counterpart.
The Bottom - The bottom of the CoCo 3 case reveals different warranty and identification stickers. Also, the grey rubber feet on the American CoCo 3 are glued onto the bottom of the case, whereas the Korean CoCo 3 uses interlocking, black rubber feet that are larger in size.

The American CoCo 3 bottom has a square beige, opaque identity/serial number sticker, as well as warranty seal sticker. Also note the round grey rubber feet (and one missing at the bottom right). Unlike the Korean CoCo 3, these feet are glued in place. Over time, the glue gets soft/tacky and the feet move around and even dislodge.
Here, the Korean CoCo 3 has a rectangular, transparent identity/serial number sticker, as well as several additional stickers and a smaller warranty seal. The plastic feet are also pinned against the case for durability and adhesion.

The Power Cord - The power cord is noticeably “cheaper” on the American CoCo 3. The blades are not as strong and thick as its Korean counterpart, and the shape of the plug is less fluid. The strain relief is not as long and the wire itself is not as robust.
The power cord of the American CoCo 3 on the left vs. the Korean CoCo 3 on the right. The blades are thinner on the American CoCo 3, and the plug body and wire are noticeably thinner and less "rubbery" than its Korean counterpart.

Interior Differences

Upon opening up both CoCo 3's, the differences are more striking internally.

The Motherboard

Upon opening up both CoCo 3s, the differences are more striking internally. On both models of the CoCo 3, the actual motherboards have identical identification numbers, and appear to be exactly the same.  The components, however, are slightly varied.


Capacitors and other Components - While the values of electrical components such as resistors and capacitors on both CoCo 3s appear to be the same, there are some noticable visual differences. The American CoCo 3 has smaller "blob" capacitors which are yellow in color, while the Korean CoCo 3 has slightly larger blue capacitors in the same locations. Likewise, certain ceramic disc capacitors on the American CoCo 3 are orange in color, and on the Korean CoCo 3 are green.

RAM - The Korean CoCo 3 has Korean-manufactured 120ns Samsung RAM chips, while the American CoCo 3 has U.S. made Micron Technology 120ns memory. I have seen American CoCo 3s with 150ns Micron Technology RAM chips as well.
This wide shot of the Korean CoCo 3 motherboard shows the assortment of chips. Specifically, the RAM chips are four Korean-made Samsung KM41464 at 120 nanoseconds.
This American CoCo 3 keyboard shows US-made RAM chips from Micron Technology with 120ns access times. Other American CoCo 3s have contained 150ns variants of the same chip (apologies for the blurry photo).


IC Markings - Most integrated circuit chips, particularly the GIME, have different markings and logos, probably due to newer runs of those chips. Even later American CoCo 3s have both GIME and ROM chips made in Mexico.

Other Internal Differences

One glaring difference internally has to do with the area where the keyboard rests on case. The Korean CoCo 3 has two small round black rubber washers on either side of the pegs where the keyboard rests, in addition to a rectangular black rubber "boot" at the bottom of the keyboard and a round rubber insert directly underneath for support. The American CoCo 3 has none of these.

This Korean CoCo 3 has the rubber cap on the post that the keyboard rests on, as well as a GIME chip made in Korea and  an NEC ROM chip from Japan.

Here, the American CoCo 3's rubber cap is missing. In addition, the rubber block (not pictured) that rests at the bottom of the keyboard is also gone. While the GIME chip here is made in Korea, later models have GIME chips made in Japan. The ROM chip is also made in Mexico.

Conclusions

So what conclusions can we draw from these differences?

The CoCo 3 was nearing its end of life as it moved from its home manufacturing base in Korea to the United States. It would appear that the overseas move gave manufacturing an opportunity to source parts state side with potentially new suppliers.

Without a doubt, quality slipped with the transition from Korean-made to American-made CoCo 3s. The Color Computer manufacturing plant manager, Kenji Nishikawa, was notorious for cutting costs, and it really shows on the later American CoCo 3s. The omission of rubber stoppers and washers, along with the replacement of the more durable pinned plastic black feet to the grey rubber ones which were glued on, shows compromises were made, more than likely for cost reasons. Even the fact that the above badge had a misprint reflects carelessness (this is the ONLY CoCo 3 that I've ever seen with such a badge defect, by the way).

Addendum (July 2015)

One interesting note to this, and which prompted my posting of this white paper as a blog post: I recently acquired a CoCo 3 that was most unusual. It was an American CoCo 3 with the opaque beige "PRODUCT OF U.S.A." sticker on the bottom. However, the feet were not the glued-on grey half spheres, but were instead the nicer black round plastic ones. In addition, the electrical plug was the more durable black wire and blades, along with a Korean style TANDY badge. The serial number of this particular CoCo 3 is 1277278, which is less than the 1299880 profiled here.

I suspect this was one of the early CoCo 3s made in the Fort Worth factory after the move, made with parts that were originally stocked for the Korean factory. There are probably other CoCo 3s that are assembled in the same manner, but this is the first one that I have personally run across, making it somewhat of an anomaly compared to its earlier and later brethren.

Sunday, June 7, 2015

Part III: The Recovery

In the last blog post, I describe how the Microware CoCo 3s were taken from Microware by Scott McGee and given (gratis) to Bro. Jeremy at a mid-1990s Chicago CoCoFEST, whereupon they were shown there and at subsequent fests. During that time, they remained in the possession of Brother Jeremy.

The ending to this three part tale is rather short, but necessary to provide closer to the larger story.

Fast forward to 2012. At this point in time Bro. Jeremy has moved to England and no longer attends CoCoFESTs on a regular basis. Curious about the fate of the Microware CoCo 3's, I contacted our community monk to learn exactly where the computers where. As it turned out, he left them at his home monastery in Wisconsin.

We discussed an arrangement for them to be shipped to me in Louisiana, where they now reside.

And now you know.... the REST of the story.

Wednesday, June 3, 2015

Part II: The Monk Takes The Booty

In my previous blog post I explain how a late night of working at Microware led to the my discovering CoCo related equipment in a cubicle. I shared that discovery with a  co-worker, Scott McGee

Since no one at Microware really had expressed any interest in showcasing the CoCo 3s and other Color Computer related hardware, Scott had the raving idea of donating the CoCo 3s discovered at Microware to Brother Jeremy at the upcoming CoCoFEST. I wasn't particularly fond of the scheme, but didn't put up much resistance to it at the time. Looking back, I should have made a bigger wave and kept the CoCos where they were and made more of a point to display them somehow at Microware's headquarters.

While I cannot remember the exact year, I believe it was either 1995 or 1996. The Chicago CoCoFEST was coming up during its usual April/May time frame. James Jones and I would take the drive every year from Des Moines to Chicago to attend. Scott drove and attended separately, as I recall, with the haul of Color Computers destined for the fest.

Brother Jeremy took possession of the CoCos at that ChicagoFEST and for years thereafter, brought them with him to every fest he attended, setting them up at a booth for display. I'll give him credit -- he took care of the systems and made them accessible for the CoCo Community to see.

However, there was a casualty in this -- the rest of the haul. I cannot recall what happened to the boxes of ROM Paks and the DS-69B digitizer, nor do I know what happened to the original Tandy documents that were house along with the rest of the hardware. It reinforces in my mind that the entire collection of stuff should have never been disbanded.





Thursday, May 28, 2015

Part I: The Discovery

In my previous blog post, I shared photos of six unique Color Computers; four of them are CoCo 3s from Microware Systems Corporation of Des Moines, Iowa (Microware has changed hands a couple of times since that time). In this post, I'll give some background on my Microware experience and how these CoCos came to my attention.

Background

I joined Microware in the summer of 1992 as a software engineer. Getting my foot in the door had everything to do with my ownership of a CoCo and familiarity with OS-9, so everyone at the company knew of my Color Computer interests. I worked directly with Mark Hawkins and Todd Earles, two of the "CoCo 3 mugateers" that you see when you hit CTRL-ALT and the reset button on the CoCo 3. I was told first-hand of the lore and war stories behind the development of the Color Computer 3's BASIC and the ports of OS-9 to the Color Computer family. But I never actually saw any evidence of the development of the product. That is, until late one night...

Finding Gold

The Microware building on 1900 NW 114th Street was arranged in somewhat of an L-shaped fashion with much of the work areas filled with cubicles. I worked on the side of the building that would be considered the bottom of the L. Near my workspace were other cubicles of other employees, as well as a few empty ones where miscellaneous equipment would tend to accumulate.

I was working late one evening when I walked by a cubicle next to the "machine room" (a large air-conditioned enclosed area where Unix servers and OS-9 VME crates used for development were housed). This particular cubicle had been empty earlier in the week, but now housed a bunch of boxes of recently moved equipment. Curious, I stepped in and poked around, whereupon I discovered several Color Computer 3s. In another box were a few more Color Computer 3 cases and some documents scattered about, including the specs for the CoCo 3 on a Tandy letterhead. Still in another box were loads and loads of game and application software cartridges from Tandy. Another box held a DS-69B digitizer and some unrecognizable cartridges. There was even a huge GIMIX 6809 computer named "Sybil" which I later discovered held all the OS-9/6809 sources (even the source to the fabled Level 2 Upgrade), but that's another blog post...

Back to the find... I was ecstatic to see all of this CoCo equipment in one place, and it began to dawn on me that this was Microware's stash of Color Computer goodies they used for development some 10-odd years earlier.

The next day at work, I shared the discovery with another CoCo aficionado and Microware employee, Scott McGee. Scott had started working for Microware about a month before I did and also got his foot in the door through his Color Computer experience. He was keen on viewing the CoCo stash, so together we went to the cubicle and looked through the boxes to get an idea of what was there.

I brought up the idea of cleaning up all of the CoCo equipment and putting it in an area of the building that would showcase Microware's legacy work, but that didn't go anywhere. My impression was that no one at Microware really cared to pay homage to the work that was done on the Color Computer. No matter how I viewed these machines and their legacy, for the rest of the company, there was no interest in celebrating the past.

Everything CoCo related in that cubicle stayed in place for a while, and then it happened...

Wednesday, May 27, 2015

Some Serious CoCo History

One of the things on my to-do list has been to showcase some very specific Color Computer history. Those who have read the book "CoCo: The Colorful History of Tandy's Underdog Computer" will recognize the machines in the next photo:

These are six Color Computers of historical significance. The top shelf contains the Deluxe Color Computer that was featured in the previous blog posts. It's a rare bird, as only one other is known to exist. To its right is a mock-up of the Tandy Color Computer 4. It's simply a shell without any electronics inside, save the 3.5" disk unit. Like the Deluxe Color Computer, it is unique and is the only one of its kind.

The following two photos give another view (apologies for the poor lighting conditions).


The second and third shelves contain four Color Computer 3s of special significance: two PAL versions and two NTSC versions. These four CoCo 3s were the property of Microware Systems Corporation and used to by Mark Hawkins, Tim Harris, and Todd Earles to develop OS-9 Level Two and the "super" extensions to Extended Color BASIC. 
The black cartridge in between the two CoCo 3s in this last photo is an interesting artifact that I'll devote a whole blog post to later.

If these four machines could speak, they would tell some interesting stories. While I wasn't at Microware at the time the CoCo 3 was being developed, I came on board later. Now I think it's time to tell the tale about how these CoCos were discovered and where they have been since that time. Stay tuned for the "rest of the story" in a subsequent blog post.

For now, enjoy the photos!

Monday, May 18, 2015

Under the Cover

Now that we've explored the case, it's time to delve into the innards.

Below is a hi-res picture of the motherboard of the Deluxe Color Computer. You'll immediately notice that it has lots of wires -- undoubtedly modifications to adjust its operation after the board run in order to fix issues.



Since there isn't a service manual in existence for this machine, any attempt to understand its design must start with a survey of the integrated circuits that it contains. I'll enumerate them here:

  • U1 - SC77527P
  • U2 - SC77526P
  • U3 - RC4559N
  • U4 - MC6847P (This particular part DOES have lowercase characters -- I verified this)
  • U5 - MC1489P
  • U6 - MC1372P
  • U7 - MC1488P
  • U8 - S6551P
  • U9 - SN74LS273N
  • U10 - RF Modulator
  • U11 - GI AY-3-8913
  • U12 - SN74LS161AN
  • U13 - MC6821P
  • U14 - SN74LS221N
  • U15 - SN74LS32N
  • U16 - SN74LS86ND
  • U17 - SN74LS09N
  • U18 - SN74LS22NS
  • U19 - SN74LS04N
  • U20 - SN74LS260N
  • U21 - MC6822P
  • U22 - F74244PC
  • U23 - SN74LS273N
  • U24 - 27128 (16K ROM, currently DWDOS for CoCo 2 that I installed)
  • U25 - MC68B09EP
  • U26 - 6349-IN
  • U27 - SN74LS74AN
  • U28 - SN74LS161AN
  • U29 - SN74LS161AN
  • U30 - SN74LS783N
  • U31 - SN74LS244N
  • U32 - N82S153N
  • U33 - SN74LS157N
  • U34 - SN74LS14N
  • U35-U42 - TANDY 8040655
  • U43 - SN74LS12N
  • U44 - SN74LS393ND
There is an additional IC near U20 that is glued upside down on the board with wires coming from it. I have not identified that IC up to now.

The power supply is located on the middle left side of the case,  just as it is on the 64K Color Computer. The keyboard connector accepts the ribbon style mylar cable that is common on the CoCo 2 and CoCo 3. And of course, the 40 pin expansion connector is right where it should be.

Recall from the previous post that a 9 pin serial connector is mounted on the back, as is composite audio and video.

There's quite a bit to digest here, and I'll be out of town a few days on business, so there will be a stretch of time before I can make another post. So have fun and let the comments ensue!