Deniability is also known as repudiability or plausible deniability. As far as I can determine, the idea came out of nowhere. It wasn't really an issue anyone worried about and then it was a feature. I will use PGP as the practical example1).
When you use public key cryptography to encrypt a message, there is no way for the recipient to be sure it is from you. Normally you address that problem by cryptographically signing the message. In some situations, you might only want the recipient of your message to have access to the connection between your identity and your message.
Deniability is intended to break or weaken the connection between a cryptographically signed message and your identity. It must still allow the message recipient(s) to be certain that the message is from you, thus preserving the value of the signature. The goal here is to allow an electronically communicated message to be completely “off the record” in the sense the phrase is used in journalism.
In the event that someone can come up with a tie between you and your cryptographic identity then this is what they will face when attempting to link it to your physical identity:
As an example; email evidence used in something like a court case would follow the regular email evidence process. There would be no incentive to add an extra complicated step that might be challenged.
In a personal communications situation the people involved will know each other. If the reporter is part of the group then it would come down to how the others felt about you and them. If it was an outsider then the report would be treated as an unsubstantiated rumour. A technical proof would be ignored as redundant.
Ironically the very existence of a deniability feature draws attention to the existence of the cyptographic signatures used in a particular messaging system and would tend to reinforce the value of such signatures.
Far and away the most common case is where you can trust the one you are messaging with. If you are using a system that can sign messages then you will have the capability and the motivation to encrypt any messages that have the potential to embarrass you. For this case, deniability only has potential value when that encryption fails.
Here we need to make a distinction between the harm from revealing that a particular person said something and what was actually said. After the failure of encryption, usually most of the harm will be due to the facts that are revealed. Most facts can be verified in some other way than by a cryptographic signature.
This could be considered the true “off the record” case. The one you are messaging with might use your message and cryptographic signature in a way that could cause you problems. In this case, most people would be best off:
… rather than having to try to weaken the effect of a cryptographic signature. Our practical example, PGP, works well in this case as it allows both options. Deniability can be seen here as a compromise solution for a problem that need not exist.
It is not really certain that deniability is even completely possible in this case. The cryptographic signature has still proven whatever it can prove. Whatever indication that the system has given your correspondent of your identity can be recorded somehow, even if just in their memory. It's similar to the difference between someone getting your letter and hearing you say something to them in person.
The method of establishing deniability used in practice involves leaking information so that someone can forge a message after the message has been transferred and verified by the recipient(s).
The fundamental problem here is that the mere possibility of forging a message does not in any way prove that a message was forged. Forgeability uses a simple technical approach to create a much more complicated question of credibility.
In practice you would have to at least imply that your message was a forgery created by someone using a complicated and obscure technical process. This has obvious ethical issues and is unlikely to work, either in the court of public opinion or a court of law. Most media is relatively easy to forge but false claims of such forgery are quite rare which suggests that such claims don't tend to work out. Fabricated evidence of a forgery could work, but the risk is much higher.
Consider the case where a corpus of encrypted email is publicly leaked but it is not possible to decrypt that email at that time. Should it then become possible to decrypt the email later then it can be shown that the email existed before it was possible for anyone to forge it. Forgeability is thus negated. An entity that is saving encrypted messages in hope of future decryption only has to generate proof of the existence of those emails at the time of intercept to negate any forgeability. In general, proof that a message existed before it was possible to forge it negates forgeability.
There is a potential problem of implied intent. Forgeability involves a deliberate act of data leakage. Say you get caught skulking around in the dark with burglary tools in an area with recent breakins. Providing solid proof that you always carry such tools would increase the level of suspicion, not decrease it.
In the same way, using a system that is deliberately designed to allow you to say something then deny having said it might also cause extra suspicion. The concept is quite easy to understand, as opposed to the stuff about public key based signatures.
Consider the situation if someone were to actually forge a message from you using the method you have explicitly made possible. When claiming the forgery you would have to work against the same suspicion.
Instead of releasing the information required to forge your signature to everyone who gets access to the unencrypted messages, you can just release that information to your correspondents. This eliminates the potential problems caused be allowing just anyone to forge your signature. It does however greatly reduce the number of people you might suggest had forged that signature. If you try to suggest that your boyfriend Nathan might of forged your signature you will have a bad time when it comes out that the most technical thoughts Nathan ever has relate to the healing power of crystals.
Forgeability as used for deniability would rarely be applicable. The most likely instance where it would be relevant would be in the case of a compromised endpoint. In that situation you have a plausible way to claim a forgery and thus denyabliliy becomes redundant.
Deniablity is one of those things that seems like a good idea in theory. In practice it has very little value. It is unlikely that the feature is worth the extra risk of the complexity it would take to implement it.