LanManager encryption is somewhat similar to UNIX password encryption. The server uses a file containing a hashed value of a user's password. This is created by taking the user's plaintext password, capitalising it, and either truncating to 14 bytes or padding to 14 bytes with null bytes. This 14 byte value is used as two 56 bit DES keys to encrypt a 'magic' eight byte value, forming a 16 byte value which is stored by the server and client. Let this value be known as the "hashed password".
Windows NT encryption is a higher quality mechanism, consisting of doing an MD4 hash on a Unicode version of the user's password. This also produces a 16 byte hash value that is non-reversible.
When a client (LanManager, Windows for WorkGroups, Windows 95 or Windows NT) wishes to mount a Samba drive (or use a Samba resource), it first requests a connection and negotiates the protocol that the client and server will use. In the reply to this request the Samba server generates and appends an 8 byte, random value - this is stored in the Samba server after the reply is sent and is known as the "challenge". The challenge is different for every client connection.
The client then uses the hashed password (16 byte values described above), appended with 5 null bytes, as three 56 bit DES keys, each of which is used to encrypt the challenge 8 byte value, forming a 24 byte value known as the "response".
In the SMB call SMBsessionsetupX (when user level security is selected) or the call SMBtconX (when share level security is selected), the 24 byte response is returned by the client to the Samba server. For Windows NT protocol levels the above calculation is done on both hashes of the user's password and both responses are returned in the SMB call, giving two 24 byte values.
The Samba server then reproduces the above calculation, using its own stored value of the 16 byte hashed password (read from the smbpasswd file - described later) and the challenge value that it kept from the negotiate protocol reply. It then checks to see if the 24 byte value it calculates matches the 24 byte value returned to it from the client.
If these values match exactly, then the client knew the correct password (or the 16 byte hashed value - see security note below) and is thus allowed access. If not, then the client did not know the correct password and is denied access.
Note that the Samba server never knows or stores the cleartext of the user's password - just the 16 byte hashed values derived from it. Also note that the cleartext password or 16 byte hashed values are never transmitted over the network - thus increasing security.
from
http://samba.kn.vutbr.cz/samba/docs/man/Samba-Developers-Guide/pwencrypt.html
