In the old days it was easy to differentiate between the two.  Handheld was the ideal explanation of a game device you hold in your hands.  The emphasis was game portability.  The unit came with its own form of display, and controls built in.  Video game consoles generally remained at the home.  In current times gamers can now purchase attachable LCD screens and other accessories to make certain consoles portable.  Does that make them handhelds?  Our first guideline is simple.

If it plays portable "right out of the box", then it is a handheld.

Then what would we consider a device like Vectrex?  It has both display and controls built into the unit.  Doesn't this make it a handheld?  That brings us to our second guideline.

If the device uses a power cord that requires AC power, and cannot run without it (no batteries), then it is a console.

With the constant evolving of the game market, eventually even these guidelines will be blurred.

This by far has been the hardest to answer.  Technically both consoles and computers are near identical.  They both use separate display devices, and use peripherals to input commands.  Both also use some sort of game media.  In the 1980's thru the 1990's computer makers redesigned lines of home computers and made them into consoles (FM Towns Marty, Commodore 64GS, Amstrad GX400, Zemmix, Amiga CD32, etc.).  Console makers created media, or added features, to give consoles "Computer-like" abilities.  So we need more guidelines.

If the device is capable of running game media upon boot up without the need for tweaking, user input (prompt commands, clicking executables, etc), or settings adjustment...then it most likely is a console.

This rule works easily with current computers and consoles, but may not apply well some older computers.

If the device uses a form of game media coded specifically for that device only, then it may be a console.

Again this rule works easily with current computers and consoles, but may not apply well some older computers.  The introduction of the EVO Smart Console is another example of bending this rule since it runs computer software games.

If the primary function is for playing game media "right out of the box"...then it most likely a console.

Although quite simplistic, this guideline when coupled with the other two guidelines narrows things down.  It works quite well with older computers.  The Commodore 64GS for example is a stripped down Commodore 64 computer designed with only gaming in mind.  If a buyer wanted the device for more then gaming, then they would purchase the Commodore computer.

Lastly..If it is a gaming machine that is encased in something smaller then a computer minitower, yet larger then the cases of its gaming media...then it most likely is a console.

The Sony PlayStation 3 really walked the edge of this rule...hahaha.
Who knows what the future will look like...