ITCS Unit 1
ITCS Unit 1
Claude Shannon, who is often regarded as the father of information theory. His seminal
contributions, particularly his 1948 paper "A Mathematical Theory of Communication", laid the
mathematical foundation for the study of information, communication systems, and data
encoding.
1. Information:
o Shannon defined information in terms of uncertainty. The more uncertain or
unpredictable an event is, the more information it carries when it occurs.
o The amount of information associated with an event is measured in bits (binary
digits), where a bit is the amount of information gained by knowing the outcome
of a binary (yes/no) decision.
2. Entropy:
o Entropy is a measure of the uncertainty or randomness in a system. In Shannon's
theory, it quantifies the average amount of information produced by a stochastic
source of data.
o Mathematically, the entropy H(X)H(X) of a random variable XX with possible
outcomes x1,x2,...,xnx_1, x_2, ..., x_n is defined as:
where:
where:
BB is the bandwidth of the channel,
SS is the signal power,
NN is the noise power,
S/NS/N is the signal-to-noise ratio.
5. Noiseless and Noisy Channels:
o Noiseless Channel: A theoretical communication channel with no interference or
noise. The transmission rate can be as high as the channel's capacity.
o Noisy Channel: A more practical scenario where noise distorts the signal.
Shannon showed that even in the presence of noise, error-free transmission is
possible up to a certain capacity if the transmission rate is below this capacity.
6. Shannon's Source Coding Theorem:
o This theorem addresses data compression and states that the average length of the
codewords needed to represent the symbols from a source can approach the
entropy of the source. Essentially, it's the foundation of lossless data
compression.
o If the entropy of a source is H(X)H(X), the average number of bits needed to
encode each symbol is at least H(X)H(X). Thus, you cannot compress data to
fewer bits than the entropy of the source without losing information.
7. Shannon's Channel Coding Theorem:
o This theorem demonstrates that error-free communication is possible over a noisy
channel if the data is encoded properly. It states that, given a noisy channel with a
certain capacity CC, it is possible to transmit data at a rate less than or equal to
CC without errors, assuming we use the right encoding (error-correcting codes).
o The practical implementation of this idea leads to the development of error-
correcting codes, such as Hamming codes and Turbo codes, which are used in
modern communication systems to detect and correct errors during transmission.
1. Data Compression:
o ZIP files, JPEG images, MP3 audio, and H.264 video all rely on principles
from Shannon's theory to reduce the size of files while maintaining quality.
2. Error Correction:
o Modern communication systems (e.g., cell networks, internet protocols) use
error-correcting codes to ensure data integrity over unreliable networks.
3. Cryptography:
o Shannon's work laid the foundation for modern cryptography, including
symmetric and asymmetric encryption schemes used for secure communication
(e.g., RSA, AES).
4. Machine Learning:
o Information theory concepts, such as entropy and mutual information, are widely
used in machine learning for tasks like feature selection, decision trees, and
clustering algorithms.
5. Networking:
o Concepts of channel capacity and efficiency guide the design of high-speed,
reliable communication protocols (e.g., TCP/IP).
Conclusion:
1. Uncertainty
For example:
If a coin is flipped, the outcome is uncertain because there are two equally likely
outcomes: heads or tails.
If you know that a coin is always heads, there is no uncertainty about the outcome.
2. Entropy
Entropy is a measure of the average uncertainty or the amount of information
contained in a source of data. It quantifies the level of unpredictability or randomness in a
system.
Mathematically, entropy is defined as the expected amount of information produced by a
random variable, with respect to its probability distribution.
Shannon defined entropy H(X)H(X) for a random variable XX with possible outcomes
x1,x2,...,xnx_1, x_2, ..., x_n as:
Where:
The term p(xi)log2p(xi)p(x_i) \log_2 p(x_i) measures the amount of information (in bits)
contained in outcome xix_i weighted by its probability p(xi)p(x_i).
When an event is very likely (i.e., p(xi)p(x_i) is close to 1), the information content is
low. If the event is highly uncertain (i.e., p(xi)p(x_i) is close to 0.5 for binary events), the
information content is high.
Entropy is always measured in bits when using the logarithm base 2.
Key Points:
1. Maximum Entropy: Entropy reaches its maximum when all possible outcomes are
equally likely. For example, if a coin is flipped, the entropy is 1 bit because the outcomes
are equally probable (50% heads, 50% tails).
2. Minimum Entropy: Entropy is zero when one outcome is certain (i.e., when one
outcome has a probability of 1 and the others have a probability of 0).
For a fair coin flip, there are two possible outcomes: heads (H) and tails (T), each with
probability 12\frac{1}{2}.
H(Coin Flip)=−(12log212+12log212)H(\text{Coin Flip}) = - \left( \frac{1}{2} \log_2 \frac{1}{2} + \frac{1}
{2} \log_2 \frac{1}{2} \right) H(Coin Flip)=−(12(−1)+12(−1))=1 bitH(\text{Coin Flip}) = - \left( \frac{1}{2} (-
1) + \frac{1}{2} (-1) \right) = 1 \text{ bit}
This means that on average, 1 bit of information is required to describe the outcome of the coin
flip.
For a loaded dice with outcomes 1 through 6, but with probabilities not equal, let’s assume the
following probabilities for each outcome:
p(1)=0.5p(1) = 0.5,
p(2)=0.1p(2) = 0.1,
p(3)=0.1p(3) = 0.1,
p(4)=0.1p(4) = 0.1,
p(5)=0.1p(5) = 0.1,
p(6)=0.1p(6) = 0.1
H(X)=−(0.5(−1)+5×0.1(−3.32))=0.5+1.66=2.16 bitsH(X) = - \left( 0.5 (-1) + 5 \times 0.1 (-3.32) \right) = 0.5
+ 1.66 = 2.16 \text{ bits}
Since the die is loaded (i.e., not all outcomes are equally likely), the uncertainty (and thus the
entropy) is lower than the entropy of a fair die (which would be 2.58 bits).
If there is a situation where the outcome is always the same, such as always getting a "1" on a
die, then the entropy is 0 because there's no uncertainty:
3. Relation to Information
For example:
If an event has a probability of 1 (certainty), its information content is 0 bits (no new
information).
If an event has a probability of 0.5 (maximum uncertainty), the information content is 1
bit.
4. Applications of Entropy
Data Compression: In lossless data compression (e.g., ZIP, PNG), entropy is used to
determine the minimum number of bits required to represent a data source. The goal is to
encode data using fewer bits without losing any information.
Cryptography: In cryptography, entropy is used to measure the unpredictability of
cryptographic keys. A good cryptographic key should have high entropy, making it hard
to predict or guess.
Machine Learning: Entropy and related concepts like information gain are used in
decision trees to measure how much uncertainty is reduced after splitting data at each
node.
Communication Systems: Shannon's information theory uses entropy to design efficient
communication systems and to measure the maximum amount of information that can be
transmitted over a noisy channel.
Conclusion:
Entropy is a core concept in information theory that quantifies the uncertainty or randomness in a
source of information. By measuring entropy, we can understand how much information is
required to describe a message or event and how efficiently we can transmit or store that
information. It plays a pivotal role in data compression, cryptography, communication systems,
and machine learning, making it one of the foundational pillars of modern computing.
Leakage refers to the unintentional exposure of information, such as secret keys, intermediate
results, or private data, during cryptographic operations or communication. Quantifying leakage
is an important aspect of designing secure cryptographic systems, as it helps understand how
much of a secret or sensitive information could be deduced by an attacker through various forms
of side-channel analysis or other techniques.
Quantifying Leakage
Leakage is often quantified using information-theoretic metrics, which measure the amount of
information that an attacker can gain about a secret key (or other sensitive data) based on the
observed leakage.
1. Information-Theoretic Leakage:
o The leakage of information can be understood in terms of entropy. If the secret
key has high entropy, then it is harder for an attacker to gain information about it.
If entropy is lower (i.e., there is more leakage), the attacker can learn more about
the secret key.
2. Leakage in Terms of Mutual Information:
o Mutual information is often used to quantify leakage. It measures the amount of
information that an attacker can gain about the secret key KK from some
observed leakage LL. Mathematically, the mutual information is:
where:
If I(K;L)I(K; L) is large, the attacker can deduce significant information about KK, and
the system suffers from significant leakage.
3. Leakage-Resilient Cryptography:
o Some cryptographic schemes are designed to be resistant to leakage, where
leakage resilience refers to minimizing or controlling how much information can
be deduced about the secret key through the leakage.
o One common approach is to employ techniques like masking, where operations
on the key are disguised or randomized to prevent sensitive information from
being exposed during computations.
For instance, if an attacker has access to a partitioned or segmented view of the key space (e.g.,
from different sources of leakage), they may be able to infer parts of the key with higher
probability, but if the partitions are well-designed, the information gained might be minimal.
The key size in cryptographic protocols is crucial for ensuring security, and the lower bound on
the key size can be derived based on the following factors:
1. Security Goal: The key size needs to be large enough to protect against brute-force
attacks. The security of an encryption scheme is typically proportional to the key size. In
general, the larger the key size, the more difficult it is to break the cryptographic system
by trying all possible keys.
For example:
These key sizes are often determined based on the desired security level against brute-
force attacks.
The lower bound on key size kk in the presence of leakage can be approximated by the
following relation:
where:
This suggests that the key size must be large enough to accommodate the potential
leakage. If I(K;L)I(K; L) is very large, the key size must be increased proportionally to
maintain security.
4. Cryptographic Constructions:
o In certain cryptographic constructions, such as public-key encryption schemes,
digital signatures, or hash functions, the lower bound on the key size can
depend on the assumed hardness of certain mathematical problems (e.g.,
factoring large numbers, discrete logarithms, or lattice-based problems).
o For example, in RSA, the key size typically needs to be large enough (e.g., 2048
bits or more) to prevent attacks based on the difficulty of factoring large numbers.
For AES (Advanced Encryption Standard), the key size determines the level of security
provided. If an attacker is able to gain some information about the key through side-channel
leakage (e.g., timing attacks, power analysis), the key size must be sufficiently large to ensure
that the attacker cannot recover the key even with partial information.
If AES-128 is used and an attacker learns a portion of the key due to leakage, the remaining bits
of the key must still be sufficiently large enough to prevent successful attacks. In practice, key
sizes of 128, 192, or 256 bits are used based on the level of security needed.
Conclusion
In summary, leakage refers to the unintended disclosure of sensitive information, and it is crucial
to quantify it to ensure cryptographic security. This is often done through information-theoretic
measures such as mutual information and entropy. Additionally, when dealing with leakage,
it's essential to account for partitions of the secret key and to ensure that the key size is large
enough to withstand potential leakage. The lower bound on key size is determined by the level of
leakage tolerated and the security requirements of the cryptographic system.
Secrecy, Authentication, and Secret Sharing in Cryptography
In the context of cryptography, secrecy, authentication, and secret sharing are foundational
concepts that address the confidentiality, integrity, and distribution of information. Here's an
explanation of each:
1. Secrecy
Key Concepts:
Confidentiality: Ensuring that only authorized individuals (or systems) can access
sensitive information. This is achieved using encryption to transform data into an
unreadable form, which can only be decrypted with the appropriate key.
Encryption: The process of converting plaintext into ciphertext using a cryptographic
algorithm and a secret key. Only those who possess the key can decrypt the ciphertext
back into the original plaintext.
Symmetric Encryption: In symmetric encryption, the same key is used for both
encryption and decryption (e.g., AES, DES).
Asymmetric Encryption: In asymmetric encryption, there are two different keys: a
public key for encryption and a private key for decryption (e.g., RSA, ECC).
Examples of Secrecy:
2. Authentication
Authentication ensures that the identity of a person, device, or system is verified before granting
access or permission to perform a specific action. In cryptography, authentication aims to
confirm that the parties involved in communication are who they claim to be, and that the
message has not been tampered with.
Types of Authentication:
1. User Authentication:
o Involves verifying the identity of a user based on credentials, such as passwords,
biometric data, or hardware tokens. User authentication ensures that only
authorized individuals can access sensitive systems or data.
2. Message Authentication:
o Ensures that a message has not been altered during transmission. Message
Authentication Codes (MACs) or digital signatures are commonly used to
verify the integrity and authenticity of messages.
3. Two-Factor Authentication (2FA):
o Combines something the user knows (e.g., password) and something the user has
(e.g., phone with an authentication app) to provide an additional layer of security.
Examples of Authentication:
Digital Certificates: Used in SSL/TLS to authenticate websites (the site proves its
identity to the user).
Digital Signatures in Email: Used to verify the authenticity of email messages, such as
in PGP (Pretty Good Privacy) or S/MIME.
3. Secret Sharing
Secret sharing is a cryptographic technique that divides a secret (e.g., a cryptographic key,
password, or any sensitive data) into multiple parts, called shares, and distributes these parts to
different parties. The secret can only be reconstructed when a sufficient number of shares are
combined, ensuring that no single party can access the entire secret. This provides distributed
trust and fault tolerance.
Key Concepts:
1. Threshold Scheme:
o In a threshold secret sharing scheme, a secret is divided into nn shares, and any
subset of kk shares (where k≤nk \leq n) can be used to reconstruct the original
secret. The number kk is called the threshold.
o If fewer than kk shares are available, the secret cannot be reconstructed, providing
security against compromise of some shares.
2. Shamir's Secret Sharing:
o One of the most popular secret sharing schemes, Shamir's Secret Sharing
algorithm divides a secret into nn shares and requires any kk shares to reconstruct
the secret. It uses polynomial interpolation over finite fields.
o Shamir’s Secret Sharing ensures that knowledge of fewer than kk shares gives
no information about the secret.
3. Linear Secret Sharing:
o In a linear secret sharing scheme, the shares are chosen such that the secret can be
reconstructed from a linear combination of the shares.
4. Verifiable Secret Sharing:
o In verifiable secret sharing, the share holders can verify the validity of their
shares, ensuring that no malicious party has provided incorrect or fake shares.
Suppose Alice has a secret value SS that she wants to share with 5 people, but she requires that at
least 3 people must collaborate to reconstruct the secret. Using Shamir’s scheme:
Each of these concepts plays a crucial role in securing data and systems in the modern
cryptographic landscape, supporting confidentiality, integrity, and reliable trust in distributed
environments.
In modern cryptography, the security of cryptographic systems is analyzed under various models
and definitions. This helps ensure that the systems are resilient to attacks and that their security is
well-understood. Below, we discuss provable security, computational security, and
symmetric ciphers, focusing on how they relate to each other.
1. Provable Security
Provable security refers to the ability to rigorously prove the security of a cryptographic scheme
based on well-defined mathematical models. The aim is to provide strong evidence that a
cryptographic system is secure by reducing the security of the scheme to a well-studied and hard
problem in mathematics, such as factoring large numbers, solving discrete logarithms, or certain
lattice problems.
Key Concepts:
Example:
RSA Encryption: The security of RSA is provably based on the difficulty of factoring
large integers. If an attacker could factor large semiprimes efficiently, they could break
RSA. However, no polynomial-time algorithm exists for factoring large numbers (under
the assumption of P ≠ NP), so RSA is considered secure as long as the key size is large
enough.
Limits:
Computational security refers to a security model where the system is secure against
computationally bounded adversaries. Instead of providing absolute security (which is
impossible in most practical settings), computational security ensures that breaking the system is
infeasible within any reasonable time frame, even with the most powerful computers available.
Key Concepts:
Example:
AES (Advanced Encryption Standard): AES with a key size of 128 bits is considered
computationally secure against current and foreseeable brute-force attacks. While it is
not theoretically unbreakable, an attack requiring 21282^{128} operations is
computationally infeasible with today's technology.
3. Symmetric Cipher
A symmetric cipher is a type of encryption algorithm where the same key is used for both
encryption and decryption. These ciphers are widely used in cryptographic systems because they
are efficient and relatively fast compared to asymmetric ciphers (e.g., RSA).
Key Concepts:
Key Symmetry: In symmetric encryption, both the sender and the receiver must share a
secret key beforehand. This key is used to both encrypt and decrypt messages.
Efficiency: Symmetric ciphers are generally faster and require less computational power
than asymmetric algorithms because the operations involved (like substitution and
permutation) are simpler.
Key Distribution: The main challenge with symmetric ciphers is key distribution—how
to securely share the secret key between the sender and receiver. If an attacker learns the
key, they can easily decrypt the communication.
1. Block Ciphers: These ciphers encrypt fixed-size blocks of plaintext at a time (e.g., 128-
bit blocks).
o AES (Advanced Encryption Standard): A widely used block cipher with key
sizes of 128, 192, or 256 bits. AES operates on 128-bit blocks and provides high
security and efficiency.
o DES (Data Encryption Standard): An older block cipher that uses a 56-bit key
and operates on 64-bit blocks. DES is now considered insecure due to its small
key size.
2. Stream Ciphers: These ciphers encrypt data one bit or byte at a time, often using a
keystream.
o RC4: A stream cipher that was widely used but is now considered insecure due to
vulnerabilities found over time.
Example of Symmetric Encryption:
AES (Advanced Encryption Standard): AES is the most widely used symmetric
encryption algorithm, often used in secure communications (e.g., HTTPS, VPNs). It is
considered computationally secure for the foreseeable future with a sufficiently long
key (e.g., 128-bit key or longer).
Provable Security applies to both symmetric and asymmetric ciphers and aims to
formally prove the security of a system based on the difficulty of certain mathematical
problems. It typically provides a high level of confidence in the security of a
cryptographic scheme.
Computational Security provides a more practical approach, acknowledging that
absolute security is impossible but ensuring that breaking the cryptosystem requires an
infeasible amount of computational resources. This approach is more realistic and often
the basis for the security of real-world systems.
Symmetric Ciphers are a central part of computational security. Since symmetric
ciphers (e.g., AES) are efficient and widely used, they are often the focus of both
computational security and provable security analysis.
Summary
Together, these concepts help cryptographers design and evaluate secure systems that balance
theoretical security guarantees with practical performance and usability concerns.