It is somewhat lengthy at 72 pages, but I can honestly say it is one of the most informative papers I have ever read.  Literally everything you have ever wanted to know about floating point standards is covered.  It covers everything from how the numbers are stored at the binary level all the way through how mathematical operations are performed on them.

Really, it’s worth reading.  Even if you don’t read all of the mathematical proofs at the end, at least glance at the IEEE standard way of storing floating-point numbers and the section on accuracy.  I guarantee it will make the whole topic far easier to grasp.

Link to article: http://docs.sun.com/source/806-3568/ncg_goldberg.html

So, I decided to write an article about it with the hopes that it would benefit others. php|architect was kind enough to support my writing and published the article in the September issue. I’d like to thank Steph Fox for her ongoing help with both the technical and language aspects of the post.  This article would not have been possible without her excellent guidance.

For those of you looking for the classes referenced and explained in the article, they can both be downloaded below. Note that the NotificationHandler class also includes logging functionality which was not covered in the article.

SMS Article Classes (1.51 KB)

I can say after all of these years, I have finally found the best company I’ve ever had the pleasure of dealing with.  ServInt is a small (in number of employees) VPS-only web host who maintains the best business model anyone could ever ask for.  They have been around since 1995 and have a spotless track record.  Their servers are incredibly reliable and I have not had a single outage since I started using them.  Their prices are extremely reasonable starting at only $50/mo which is still an excellent package including 1 GB burstable memory, half a terabyte of transfer, 15 GB of storage, and 4 dedicated IPs.  Both storage and transfer quantities are easily upgradable and are both extremely reasonable in price.  All packages get root access and free backups in case of an outage or user error on the admin’s part.  Cpanel/WHM or Plesk is also included at no charge.

That being said, there are dozens of companies that can provide more for the same price.  What ended up selling me was their customer support.  I e-mailed their billing department before signing up for hosting and was amazed when a real human, with a name and personal phone-number responded to me.  She was extremely well spoken and went above and beyond what was required of her.

The next day when I signed up, I promptly received a turnup e-mail and went along my way setting up various sites on the new VPS.  I almost had a heart attack when a customer service rep called me and asked if he could assist with migration or any software installation.  I was floored.  A proactive phone call from a hosting company?  Amazing.

Customers are given the standard panel for billing, support tickets, and other miscellaneous services but are also given access to a forum.  Not a dead, useless forum - a lively, active forum.  Other customers regularly discuss their experiences and collaborate with ServInt (not just talk one-way) to solve problems and talk about hosting technology.  Not only does customer support and their NOC staff post on these forums, but the CEO himself frequents the forums and regularly contributes to discussions.  Now thats different!

This wasn’t endorsed in any way by ServInt but they are running a truly excellent business and they deserve to be recognized.  They are by far the most personal hosting company I have delt with and I recommend them to anyone looking for a VPS hosting solution.

declare(Ticks=N) Depreciated
First off, they finally depreciated the declare(Ticks…) construct. I never really found a use for this and actually had it crash a web server due to threading issues. Thankfully, PHP is no longer formally supporting this function along with register_tick_function.

Ternary Default
In the last few months I have found myself using the ternary operator extensively to save space. Most of the time it ends up being in the form:

$var .= $other_var ? 'some string' : '';

Thankfully this can now be shortened to simply:

$var .= $other_var ? 'some string' :;

to omit the false case, and to:

$var .= $other_var ?: 'some other string';

to omit the true case.

Namespace
For some reason namespace isn’t currently a reserved word. In v5.3 Alpha 1, it is.

goto Keyword
This one I don’t like. PHP decided to add the dreaded goto/label: construct. I have absolutely no clue why they would do this since it is never needed in modern programming.

Nothing is groundbreaking but most of the changes are a breath of fresh air. My only concern is goto which I hope does not make it to the release branch…

phpWatch is a general purpose service monitor that is able to send notifications of outages via e-mail or text-message (SMS). The purpose of this system is two-fold: it allows administrators to easily check the status of many different services running on any number of servers and also allows developers to interface with the query and notification APIs.

A demonstration of the administrator view is available at here. The AJAX based user-interface makes it simple to view, add, edit, and delete service monitors as well as notifications on a single page. The configuration page allows users to customize and format the notifications that are sent out during a service outage. Note that when a new version of phpWatch is released, the bottom of the page (where it shows the phpWatch version number) will be updated with a notification.

The developer documentation is available here. The API allows other PHP scripts to query the monitored services in real-time, gather statistics about services, and interact with the notification system to send SMS and e-mail alerts.

If you find any bugs or have a suggestion, please feel free to add them to the bug-tracker on Sourceforge or send an e-mail to me at aaron (at) aaron-rosenfeld (dot) com.

Demonstration Page
http://aaron-rosenfeld.com/phpWatch/demo

Developer Documentation
http://aaron-rosenfeld.com/phpWatch/demo/docs

Download Link
phpWatch v1.0.6 Beta - 56kb

What I have realized is the vast majority of individuals who have non-technology jobs consider anyone who “works with computers” to be IT and want to make it clear that:

COMPUTER SCIENCE IS NOT INFORMATION TECHNOLOGY

Note: Before I even start, I want to make it clear that I did not indent this article to favor IT or Computer Science but I am obviously bias towards my major of Computer Science. I apologize if any of the information is not accurate for a specific person - we all know what you major in may not be what you end up doing…

What differentiates the two is Computer Science is just that - a science. We generally take math through at least differential equations, four to six terms of a lab science, and specialize in fields like artificial intelligence, data structures, bioinformatics, and human-computer interaction.

In my eyes, IT is an applied skill set for computers and networks rather than a science. Just as a carpenter works with wood or a plumber with pipes, an IT professional works with computers and networks. To quote Wikipedia:

IT professionals perform a variety of duties that range from installing applications to designing complex computer networks and information databases.

IT majors generally concentrate on aspects of computing such as server technology, database design, database administration, networking, and basic software design. I am by no means trying to make IT sound like a narrow-minded field - in fact, IT students may come out much more prepared for the “real world” than many Computer Science students.

Almost every company that utilizes computers, in any way, has IT. They deal with everything from day-to-day operations of the computer systems and databases to manage major overhauls of company-wide networks.

Computer Science is entirely different. Again to quote Wikipedia:

Computer Science [...] is the study and the science of the theoretical foundations of information and computation and their implementation and application in computer systems.

Computer Scientists in a (very small) nutshell design algorithms and create software. That is the key difference, in my opinion: Computer Science deals, in general, with software development and the heavily theoretical side of computing rather than (for lack of better words) the nuts-and-bolts of the computing world.

The operating system you are reading this from was most likely developed by computer scientists. The browser you are reading this from was most likely developed by computer scientists. The protocol that takes the 1’s and 0’s from half way across the world and converts into something you can read was most likely developed by computer scientists. The software that NASA uses to get the Space Shuttle into orbit and back was most likely developed by computer scientists. You get the point…

Generally speaking, Computer Scientists are employed by companies that need custom software developed or large applications that need constant additions or changes. They are the heart of companies like Microsoft, Sun Microsystems, Google, and all the other big-name software companies out there. IT on the other hand is a much more “professional oriented” field. They are employed by a much broader range companies ranging from the software giants to banks to newspapers to…anything with a computer network.

Computer Science, however, is a much broader field than the general public understands and I could not even begin to go into every facet in this article. What I want to make clear, however, is there is a distinct difference between IT and Computer Science. Both are extremely valuable but their paths only cross on certain topics.

1. Quotes: Understand the difference between double-quotes and single-quotes. I personally do not like the usage of variables in double quotes:

   $var = "my var's value"

   I much prefer

   $var = 'my var\'s value'

   That is a personal preference but it really is extremely important to know when to use different types of quotes and understand escape sequences.

2. Don’t get == confused with ===: The triple equals checks if the operands are identical — that is they have the same type with all of the same properties. Example with string:

   if('1' == 1){...} // evaluates to true
   if('1' === 1){...} // evaluates to false

3. Use print_r liberally: If there is ever confusion when it comes to objects or arrays, print_r will come in extremely handy. Also remember that print_r($var) will print the output as soon as possible (generally at the top of your page) where print_r($var, true) will return it.
4. Make a database wrapper class: When you eventually delve into the mySQL (or any other database) world, there will be a ton of new functions to learn: mysql_connect, mysql_select_db, mysql_query, mysql_num_rows, mysql_fetch_array to name just a few. Make it easy by making a single database class to wrap all of this together. It will save you a ton of time and headache in the end.
5. Avoid the <? shorthand: Always use the full <?php. This can cause problems with PHP installations that do not have the shorthand enabled and also confuse some XML parsers.
6. Never rely on register_globals: Always use $_GET, $_POST, and other pre-defined variables. This guarantees compatibility with other PHP installations.
7. Choosing a PHP Version: Use a PHP version you are comfortable with: PHP4 classes are incredibly simple to use. There is no visibility keywords to worry about and functions can be called after being instantiated or statically. PHP5 introduces the vast majority of the OOP paradigm, including visibility, the static keyword, class constants, method overloading, and class abstraction. If you have never been exposed to this, it can be quite difficult to take in while also learning PHP. That being said, it never hurts to use PHP5+ as it is fully backwards compatible.
8. Modulize: After working with PHP for a couple of days or so, consider modulizing your commonly used functions, classes, etc. This can be as simple as breaking them into files. I found that I reuse the same code (albeit much cleaned up) that I did 3 years ago.
9. Get into OOP as soon as possible: I remember the first year or two I was using PHP, I made maybe six websites for various organizations I was involved with. The code was so horribly sloppy it still makes me sick to this day. Do not get in the habit of mixing your PHP and HTML! I cannot emphasize this enough! Get in the habit of keeping the vast majority of code in included files and simply call function in the HTML itself.
10. Utilize the PHP documentation fully: PHP, in my opinion, has bar-none the best documentation of any scripting language. php.net contains a page dedicated to every function in PHP along with examples and alternatives for each. At the bottom there is usually a wealth of user-submitted comments which provide even more information.

I wrote this article about a year ago for my old site and decided it was worthy of a re-post.

What is the problem?

Given a sequence of amino-acids (sometimes given as DNA bases which are simply converted to codons and then amino-acids), what is the structure of a protein after a time t? In addition to the general protein-protein forces, the solution in which the protein is folding must be considered. Temperature, pH, and electromagnetic properties all affect the final outcome. In general, however, these other forces are held constant so only protein-protein forces need to be calculated.

General Approach

The most intuitive approach is to re-calculate the protein’s shape at every multiple of some interval of time (called Δh in this case) until the the end condition is reached. That is, starting at t=0 and continue incrementing time by Δh until we arrive at t=t_f (when the protein is completely folded).

The next obvious question is, “When is the protein done folding?” It turns out, whenever the protein is in its lowest energy configuration, it will stop adjusting its shape and come to a rest. Although I will omit the full derivation of why this is the case, I will give another example of why this is true.

If a ball is released from rest at a height above the ground (really any height above the centerpoint of the earth but for simplicity I will say above the earth) it naturally falls towards the ground. This is because its (potential) energy is relatively high. As it accelerates towards the ground, it loses this potential energy (gaining kinetic energy). Once it hits the ground (ignoring any rebound), both the kinetic and potential energy equals 0. This is the low energy (rest) state of the ball. It will not move on its own. This is the same for proteins. Once it is in its lowest energy state it will not “want” to change its shape.

Calculation Methods

Method One
The first method of simulating the protein’s folding is to simply determine the force on each atom and update its position from that. The math behind this is relatively simple to understand. All that needs to be done is calculate the force of each atom on every other atom. Using Newton’s basic laws of motion, the force acting on the i^th amino-acid is given by:

Method Two
Although the previous method works, a (somewhat) easier way to do these calculations ignore the forces of each atom and instead uses the potential energy of the entire system. Letting φ_i be the potential energy of the system at the i^th atom (as a function of all n that cause the potential energy), the force on the i^th atom is given by:

As a side note, in case you are not familiar with what a gradient is, a full description of it can be found on Wikipedia but as a brief overview, a gradient is a vector field where each vector points in the direction of the largest increase of whatever is being measured (in this case φ_i). A gradient can be expanded to be the partial derivative in each direction (in this case x, y, and z). Since this needs to be done numerically in the end, the right side the following uses the value of each derivative at the i^th atom:

This again goes back to the energy state discussion. Because the gradient of φ yields the direction of increasing energy, going in the negative direction of the gradient leads towards the lowest energy and an eventual end condition for the protein’s folding.

Energy of molecules

In the previous section φ was used to represent the energy of a system. In this section the actual value of the energy of molecule-molecule interaction will be investigated. From over one hundred years research, a number of different forces between atoms have been determined. Hydrogen-bonds, electrostatic interaction, and Van der Walls attraction are the three most accepted. In addition to these forces, the angle between atoms must be considered along with how much the bonds can stretch, and how much the bonds can rotate along their axis (sometimes called their Dihedral Angle Permittivity). The actual mathematics behind these forces are studied in depth by chemists. Luckily, the potential energy each cause can be reduced to nothing more than algebraic expressions. Note also that all include some sort of coefficient (e.g. angle-bending coefficient, Dihedral Angle Permittivity coefficient, etc.) and will be denoted with a C followed by a subscript. To start, the total potential energy on the atoms in question is given by:

Expanding each term gives:

Numerical Method

The math above looks great on paper but when it comes to implementation, computers don’t natively have “gradient”, “derivative”, or “integral” functions. To solve this, Maclaurin (or Taylor) series expansion is generally used. Again I will not go into detail about what a Taylor or Maclaurin series is, Wikipedia has more information, but as an overview, both series represent a function as the infinite sum of its derivatives at a given point. The official notation for a Maclaurin series is:

Note that for all of the functions in this article, this series actually equals the function itself after an infinite number of repetitions, not approximately equals. With this understanding, it is possible to use an integration algorithm such as the Verlet algorithm to propagate an atom’s position vector from t = t - Δh to time t = t + Δh without needing to know the velocity of the atom. All that would be needed is the current position, time, and acceleration which can be determined from the force. Again, the math of such an algorithm will be omitted as it beyond the scope of this article.

Computational Complexity

Possibly the largest hurdle in protein-folding simulation is the sheer amount of time the calculations discussed take to complete. All of the summations above take a great deal of time. The two most intensive display O(n²) performance. This alone, when considering n is rarely under 100 and usually in the tens-of-thousands, takes a very long time to complete. Nonetheless, these calculations are minimal in comparison to the other option: trying every combination of conformation to find the lowest energy level.

A principle known as the Levinthal Paradox shows just how intensive a process like this is. Because there are three basic conformations for each amino-acid (α-helix, β-pleated sheet, and the random coil), there are exactly 3ⁿ conformations possible in a given protein with n amino-acids and a computational complexity of O(3ⁿ). To put this in perspective, an amino-acid chain with only 100 amino-acids has 3¹⁰⁰ conformations. That is just about 5 * 10⁴⁷ combinations that would have to be searched. Assuming this was run on a home computer (mind you a very, very high performance home computer) which can calculate 1,000 conformations per second, it would take just about 9.8 * 10³¹ YEARS. So, as it also seems intuitively, doing the calculations in previous sections is vastly more efficient than a random search of all combinations.

A Brief Case Study

To bring all of this article together, I decided to run the algorithm above on a small protein. I eventually settled on Myoglobin for a few reasons. Myoglobin is used by the body for a number of different functions such as a type of marker of muscle injury and oxygen transportation, so it is readily available to be studied. Additionally, it only contains 153 amino-acids making the calculations relatively timely. It also has some bonding characteristics which are favorable for reducing calculation time.

Running the calculations took 3 hours, 17 minutes, 34 seconds on a 21-processor distributed computing platform. Some of this time was spent doing additional calculations to generate additional data for graphs that will be discussed next, however.

For purposes of this article, at each conformation, the software calculated the overall potential energy of the system which was then plotted vs. the position in z and x*y locations of a certain atom (the exact atom was chosen because it had the least number of bonds to minimize calculations). This method of plotting had to be used because otherwise the plot would expand into a 4^th dimension; x, y, z, and energy. An animated graph would be perfect for this, but for demonstration purposes truncating the 3 independent variables seemed acceptable. So to allow all of the data to be shown in a single image x and y were multiplied to give us only three variables: x*y, z, and energy:

As the plot shows, at different values of x, y, and z the energy of the system fluctuates. As was stated previously, getting from one conformation to another involves using a gradient to determine which direction will lower the potential energy the most. The following shows this gradient (to be completely correct, it shows the magnitude of the gradient since the gradient itself is a vector valued function). The dark areas show the highest magnitude (which is negative since it is a minimum) and light show the lowest. As a side note, there is an overall gradient towards the very strong black dot (very strong vectors) that is faint and hard to see.

As it should, the highest magnitude gradient is in the location of the global minimum. The software, in a sense, moved from the bottom left of this plot (since this atom can be considered the origin of the system) up, right, left, and down in the direction of the gradient until it hit the minimum when the atom was at its lowest energy state. In the end, as confirmed by Wikipedia, the structure looks like:

What to make of it all

I wrote this article to give others a perspective on how daunting the problem of protein folding is. What I have explained is considered the easiest when it comes to the mathematics. There are alternative approaches using energy levels of atoms, more advanced vector-based equations, and more accurate methods of describing the motion of atoms than has been explored here. But, the basic ideas were covered here. I hope everyone found this article interesting and informative. Any feedback is always appreciated.

Citations

For Article

• J. T. P. DeBrunner and E. Munck (1969). Levinthal’s Paradox. Retrieved January 26, 2008, from Levinthal’s Paradox Web site.
• Protein folding. (2007, December 25). In Wikipedia, The free encyclopedia. Retrieved January 17, 2008.
• Molecular dynamics. (2007, December 29). In Wikipedia, The free encyclopedia. Retrieved January 16, 2008.
• Myers J.K. and Pace C.N. (1996) Hydrogen bonding stabilizes globular proteins, Biophys. J. 71: 2033-2039.
• National Institutes of Health (2001 January 01). NIH Center for Molecular Modeling. Retrieved January 26, 2008, from Center for Molecular Modeling Web site.
• Steven, Schultz (1999 September 27). Math helps explain protein folding. Princeton Weekly Bulletin, 89, Retrieved September 25, 2008.

For Case Study

• Higgins D., Thompson J., Gibson T. Thompson J. D., Higgins D. G., Gibson T. J.(1994). CLUSTAL W: improving the sensitivity of progressive multiple sequence alignment through sequence weighting,position-specific gap penalties and weight matrix choice. Nucleic Acids Res. 22:4673-4680.
• Leckband, D. and Israelachvili, J. (2001) Intermolecular forces in biology. Quart. Rev. Biophys. 34: 105-267.
• Monte Carlo Method. (2008, January 16). In Wikipedia, The free encyclopedia. Retrieved January 20, 2008.
• Myoglobin. (2008, January 11). In Wikipedia, The free encyclopedia. Retrieved January 21, 2008.
• University of Florida (2006 January 31). ACES II. Retrieved January 20, 2008, from ACES II Manual Release 2.0 Web site.
• University of Washington, (2006 May 11). Raster3d. Retrieved January 26, 2008, from Raster3d Web site.

What hit me today was kind of interesting: Most wireless companies provide an SMS gateway to their subscribers (a large list can be found on Wikipedia). This allows SMS messages to be sent via e-mail. For example, if I wanted to send a message to a Verizon customer with the phone number 123-456-7890, I can e-mail it to 1234567890@vtext.com. This is then forwarded, by Verizon, to the cell-phone. That means, it is completely possible to setup a PHP script, using nothing more than the mail() function, to send a text message.

Although that thought really excited me, I then realized I still had to tackle the other side of the problem: How do I send a text message to my server? This turned out to be a bit of a thought-process challenge but the implementation was very easy.

Most web hosts allow customers to setup mail forwarding. What many people do not know is this forwarding is not limited to other e-mail addresses. You can also “forward” (technically you “pipe” it) mail to a script. How this is actually done depends on the web-server. If Cpanel is installed, it is very simple. Apache requires a module to do this.

With all of these things in my head, I setup the e-mail address sms@aaron-rosenfeld.com to pipe to the script |/path/myroot/sms.php (for the non-Linux type, the first character “|” is required. That is why it’s known as the pipe!). The script did nothing besides write to a log file “Message Received!” when it was executed (another side note: make sure the file being piped to has execute privileges and starts with #!/usr/bin/php -q).

I quickly sent a text-message from my phone to sms@aaron-rosenfeld.com and sure enough I had “Message Received!” in my log file! To read the e-mail itself requires a bit more code:

$handle = fopen('php://stdin', 'r');
$content = '';
while (!feof($handle))
	$content .= fread($handle, 1024);
fclose($handle);

I won’t go into detail but this just reads all the data that was piped from the e-mail to the PHP script. At the end, $content will store the entire e-mail including headers. To parse all of this out into a more readable form I used the following function (where $fullText is the $content from the last code-snippet):

$lines = explode("\n", $fullText);
$inMsg = false;
$this->message = '';
for ($i=0;$i<sizeof($lines);$i++)
{
	if (!$inMsg)
	{
		$this->headers['all'][] = $lines[$i];
		if (preg_match("/^Subject: (.*)/", $lines[$i], $matches))
			$this->headers['subject'] = $matches[1];
		if (preg_match("/^From: (.*)/", $lines[$i], $matches))
			$this->headers['from'] = $matches[1];
	}
	else
		$this->message .= $lines[$i] . "\n";

	if (trim($lines[$i]) == '')
		$inMsg = true;
}

This sticks the e-mail message in $this->message and the headers in their own array, $this->headers[]. From here the possibilities are endless: database queries, server status monitors, checking e-mail, etc. I may post the full classes that I ended up using at some point but for now I though this would be adequate.

Update June 23^rd, 2008: I recently gave a small talk on this subject which required a PDF format of this document. This can be found here.

So I recent bought an awesome cryptanalysis book by Christopher Swenson that is bar-none the best computer science related text I have ever read. I would recommend it to everyone who is interested in the topic of Cryptography or Cryptanalysis.

But that isn’t what I wanted to write about here. I wanted to talk about what I have learned from this book and spent the last few weeks discovering on my own. I’ll start with what most web-developers are used to: password storage. Any PHP or ASP programmer is familiar with the common one-way hash functions such as MD5 and SHA. Both have undergone thorough mathematical benchmarks for reversibility and brute-forcing and are trusted by most everyone.

The obvious drawback of these hashes are just that, though. They are hashes; they are one-way. For many this is fine since user login is simply a matter of comparing hashes. However, password retrieval is impossible and is a feature many clients ask for. Likewise, there are plenty of occasions where entire bodies of text need to be encrypted for transmission or storage purposes.

With that as the intro, I’ll start talking about the cryptography itself. Note that I am by no means an authority on this topic. I have taken two courses on cryptographic methods yet I am still far from understanding the ins and outs of such a diverse and rapidly-evolving field.

My notation: I will usually denote what base a number is in either with a subscript on the right side or use 0x as a prefix for hex or 0b for binary. Assume base-10 if there is no indication. All numbers are written with their most-significant value on the left, and if given a number N, N₀ is the least-significant bit (right-most), N₁ is the second-most-least-significant bit (second right-most), etc. I use parallel vertical bars to represent concatenation of two values. That is:

Basics

There are three basic requirements for any strong cryptographic algorithm: concealment, dispersion, and avalanching. Concealment simply means, given some plaintext (input text) character it should appear different in the ciphertext (encrypted form). This should seem obvious and I will append to this definition in our first example below.

Dispersion means that, given some string of input characters, the resultant ciphertext should not preserve the ordering of the input (even if they do meet the concealment requirement).

Many people consider avalanching part of dispersion but I like to keep them separate since they can each occur independent of the other. Avalanching is when a very slight change in input (only 1 bit even) causes a large change in the ciphertext. Take two MD5 hashes below. Only one character has been changed in the input:

Hello world!: 86fb269d190d2c85f6e0468ceca42a20
Hello wordd!: 5447cf2589c0fb0b119cea40f48b9a51

Requirement One: Concealment

I think the best way to understand the first requirement is to do a case study. To make things easy, we will examine one of the most basic encryption method available: the exclusive-or. If you are unfamiliar with what exclusive-or (XOR from here on) does, it takes two bit inputs and returns 1 if one and only one bit is 1. We denote this operation with a circled multiplication sign. It can be more formally described by . Logically, XOR can be applied to many bits by performing the operation bit-by-bit as in this example:

XOR is a very powerful operation because it is reversible. That is, if a bit p is XOR’ed with a bit q, to give a resultant bit y, the bit p can be determined by simply XOR’ing y with q (or q can be found by XOR’ing y and p).

Example: We have a plaintext of 0×61 0×61 0×72 0×6F 0×6E (”aaron” in hex), denoted P and we want to encrypt it with a single bit 0×11, denoted K. The result is:

This gives us the encrypted text 0×70 0×70 0×63 0×7E 0×7F (the string is actually not printable in ASCII but that is irrelevant here). To get back to the plaintext P, this encrypted text can just be XOR’ed with the key again.

For the average person, the encryption above is a pretty good way of mixing things up; going from “aaron” to “70 70 63 7F”.

Not only that, it definitely meets the first requirement we set previously; the text is concealed. But I am now going to append to that definition: Concealment not only means that a given input character (or number) should be different after encryption, but all instances of that character in the ciphertext should not be the same (although they don’t all have to be different either). In other words, the number of times a letter appears in plaintext should not be easy to determine simply by looking at the ciphertext.

Here is why: Lets look at the ciphertext. There is one very obvious repetition in the ciphertext, the first two bytes, this directly correlates to the first two bytes that were being encrypted! To a trained eye, this will indicate the encryption method may be weak, although this can happen by sheer luck with even very strong encryption methods.

The next logical question is “Who cares? There is still no way to get from 0×70 back to 0×61 without the key.” This is somewhat true, but, it provides those trying to break the code (or obtain the key) crucial information to use to do something called frequency analysis.

Frequency Analysis

You may have played some of those cryptograms-puzzles in newspapers; most of them use basic methods of non-digital cryptography that are based on a simple lookup table (’a’ in the ciphertext means ‘t’ in the plaintext, etc.) or other pattern-based method. These are fun but hold little practical value due to a number of cryptanalysis tricks developed over the years. The main one that I want to go over, though is called Frequency Analysis.

Lets say we have a big block of ciphertext like the following (the spaces mean nothing and are simply there to break the text up):

XFUIF QFPQM FPGUI FVOJU FETUB UFTJO PSEFS UPGPS NBNPS FQFSG FDUVO JPOFT UBCMJ TIKVT UJDFJ OTVSF EPNFT UJDUS BORVJ MJUZQ SPWJE FGPSU IFDPN NPOEF GFODF QSPNP UFUIF HFOFS BMXFM GBSFB OETFD VSFUI FCMFT TJOHT PGMJC FSUZU PPVST FMWFT BOEPV SQPTU FSJUZ EPPSE BJOBO EFTUB CMJTI UIJTD POTUJ UVUJP OGPSU IFVOJ UFETU BUFTP GBNFS JDB

To most people this could be solved pretty easily by guessing but we are going to do something a bit more analytic (although guessing is still a big part of it).

The English language is very well documented. Information on letter-frequency is readily as can be seen in this graph (Wikipedia):

As you can see, there are some letters that are used quite a lot (’E’ and ‘T’) and some that are used very rarely (’Z’ and ‘J’). We can use this information to make more educated guesses at our plaintext-to-ciphertext mapping. I won’t list them all but the three highest frequency letters in our ciphertext are:

F: 39 (0.15)
U: 29 (0.11)
P: 25 (0.09)

The number after each letter is the number of occurances and the number in parenthesis is that number divided by the total number of characters (268). So lets start by matching each of these up with their respective letter in the graph. The most frequently occuring ciphertext letter is ‘F’ (by a lot), and the most frequently occuring letter in English is ‘E’ (by a lot) so we will assume ‘F’ in ciphertext maps to ‘E’ in plaintext. Yes this is a guess, but a very logical one. We will do the same for ‘U’ and map it to the second-most-frequently occurring letter in English: ‘T’. Same for ‘P’ but note that, since ‘O’ and ‘A’ are both very close in frequency we should try both.

This process continues until either the cipher is broken or enough of a pattern is understood that the rest can be figured out. From what three letters that we have mapped, ‘F’ to ‘E’, ‘U’ to ‘T’, and ‘P’ to ‘O’ (it could very well have been ‘A’ and I had to try both), it may be clear that our cipher simply makes every plaintext letter the next letter in the alphabet. After doing the substitutions and placing spaces properly we end up with:

WE THE PEOPLE OF THE UNITED STATES IN ORDER TO FORM A MORE PERFECT UNION ESTABLISH JUSTICE INSURE DOMESTIC TRANQUILITY PROVIDE FOR THE COMMON DEFENCE PROMOTE THE GENERAL WELFARE AND SECURE THE BLESSINGS OF LIBERTY TO OURSELVES AND OUR POSTERITY DO ORDAIN AND ESTABLISH THIS CONSTITUTION FOR THE UNITED STATES OF AMERICA

I know I skimmed over this example and made some assumptions but the idea is what is important. In reality, a simple offset cipher like that is very simple to crack. It is harder when letters are randomly assigned or frequencies don’t match up as well. In those cases we can use the same exact technique but use the well-documented frequencies of two and three letter strings or turn to other techniques (such as examining different letters’ Index of Coincidences).

Back to the Case Study

As our detour has shown, a strong cipher cannot preserve the frequencies of letters. In our first example the only reason this occurred, however, was because our key was just 1 character long. Had it been two or more, the first “a” and second “a” would have been XOR’ed with different values thus yielding different ciphertexts. Case and point: keys should not be one character long and should minimize the chance of a given letter encrypting to the same thing too often.

Requirement Two: Dispersion

Until now I have not addressed the second requirement: dispersion. All the letters in the ciphertext, although in a different form, still represent the character in the equivalent position in the plaintext. There are many easy ways of fixing this but many of them are not reversible or do not shuffle enough for real use. I will explain a couple methods of achieving basic levels of dispersion and will then discuss the primary method that most well-known ciphers use.

Going back to the first example, our plaintext was 0×61 0×61 0×72 0×6F 0×6E and ciphertext calculated to 0×70 0×70 0×63 0×7E 0×7F. A simple method of shuffling is to simply rotate bits or bytes in a predefined pattern. For example, we could move every character left one position to get 0×70 0×63 0×7E 0×7F 0×70. We could do this in any pattern we want as long as it is reversible (in reality it can be irreversible but for our basic methods of encryption we need to have an easy way of going back).

Another method is to shuffle the bits in each byte by a certain amount. If you are unfamiliar with what bit-shifting is, it simple moves all bits left or right by a certain amount and we denote theseby << and >> respectively. It is interesting to note that each bit-shift multiplies the number by 2. Here is an example:

Note that when we shift the binary number 1001 once, we just move every bit over to the left and add a 0 to the right side. The equivalent is shown in base-10 on the right. Assume all bit-shifts from here on are done in binary and then converted back to the base in which we are dealing.

This method is very well suited to disguise numbers (and characters by using their ASCII values) because it is seemingly random to the untrained eye. If we did a single left-shift on each character in original ciphertext, instead of 0×70 0×70 0×63 0×7E 0×7F we would have 0xE0 0xE0 0xC6 0xFC 0xFE. One of the other methods that I personally like is shifting each byte its index plus 1. That is, for the least-significant-byte is shifted 1 time, the second least-significant by 2, etc.

Another important note is some people do shifts in a different fashion. Instead of shifting all the bits over by one, they literally rotate the bits. For our example above:

Note that the leftmost bit is not dropped but wraps back around to the right side. This is my preferred way of doing shifts because it guarantees the number of bits will remain unchanged. Note that I put the base-10 equivalent in just for reference. The math behind rotational shifts like this gets a bit messy (although not too complex) and is really unimportant to us since we are simply using it to mask our original bits.

From now on when I talk about bit-shifts I am referring to this rotational method unless I not otherwise.

Bit shifts, byte shifts, and XOR’s can be combined in many different ways to make cracking a cipher much more difficult. This is pretty impressive for a single-round encryption that has nothing but an XOR base to it.

Requirement Three: The Avalanche Effect

This the first place I will address the avalanche effect but we will develop the ideas more fully in the final section. Our encryption algorithms so far have basically just masked our input text and moved the bytes around. But imagine this: lets say we have a 1 round encryption scheme that XOR’s once with a key given by 0xA1 0×12 0×41 and then rotates all the bytes once. Now, imagine we feed it the plaintext 0×01 0×02 0×03:

Now imagine we do all of that the same except we feed it 0×42 instead of 0×41 for input. Our output would be 0×10 0×41 0xA0. Pretty close to the original isn’t it? This can become a fatal flaw in an algorithm. By trying many different combinations of inputs (or in this case, one), we can easily determine what output byte corresponds to what input byte. From that, the key itself can be found by calculating the difference between the input and output bits in that byte.

The way to combat this is to assure bytes are dependent on many other bytes. In other words, if we added another step to the encryption process that XOR’s every byte with every other byte, a change in any single byte will be propagated to all others and cause a much larger difference between two ciphertexts even if their plaintexts are very similar.

Round-based Ciphers:

Everything we have talked about until now is what’s called single-rounded. An input is given to a function that performs some shuffling and distorting of the data and it then gives us a ciphertext all in one go. The next important topic is using more than one round for encryption. Entire books have been dedicated to this topic so I will barely scratch the surface but I will try and cover the general concepts.

A round based cipher essentially just does what we have been discussing a number of times. Each round ciphertext is manipulated by a function (like what we have been talking about) and the output is used as the input for the next round. What is unique about most round based encryption methods is they don’t use the same key every round. Instead, a key schedule containing one key for each round is generated from the base key. Many times, additional changes are made at the beginning and end of encryption based on the base key itself, however.

Until we get to the last section of this article, we must make the round function reversible. Likewise, the key schedule must be generated in such a way that, given the same base key, all sub-keys will be the same.

To decrypt, the key schedule is regenerated and the rounds are run in reverse but with the round function this time un-doing whatever the original round function did.

What the problem is with this whole scheme is generally round functions are very complex. They involve bit-shifting, byte-shifting, XORing, masking bytes with other bytes, and generally moving things around a lot. Making all of this reversible is actually much harder than it sounds and, in many cases, impossible. Here is an example.

Say I want to make a round function F that XORs the current input P_i with the current key K_i and then XORs itself with the original round input shifted by one byte S(P_i) (where S just denotes the shift function). We know from the previous section this will promote an avalanche effect.

This seems to be a pretty good method but it cannot be reversed. The general form for a given round i can be shown as (P₀ is our starting plaintext and assume our key schedule has already been determined and the key for round i is denoted by K_i):

Thus, if we run this cipher for three rounds:

Now, if we try and run that in reverse we have a problem. Any given step requires the previous steps’ output. When we were encrypting we had access to this. Going the other direction we do not. This would work as a one-way hashing method but it fails for two-way encryption.

Avalanche Effect and the Feistel Structure

The way we get around this problem was discovered by Horst Feistel in the 1950’s. His method, known as the Feistel Structure, is by far one of the most important algorithms in modern cryptology and is used by some of the big-names such as DES and Blowfish. The beauty of his algorithm is any round function can be used, the round function does not have to be reversible, and it does a great job maximizing the avalanche effect in a minimal number of rounds.

The way it works is ingenious. The plaintext is split into two equal pieces (halves) denoted L and R. During encryption the right half is sent through the round function along with the round key. That is then XOR’d with the left half. That entire block is then used as the right half of the next round’s input and the left half of the current round’s input is used as the right. It is easier to understand by reading these:

The most important thing to notice is that the second step does not involve anything from the previous step as in our last example. This can be shown by rearranging the expressions above and changing the indices for easier understanding:

This means, to unencrypt we simply iterate from i=n down to i=0 and we will arrive back at our plaintext! This is quite amazing seeing that we can use any round function even if data is discarded since they are still preserved (in a different form) in the other half. This diagram from Wikipedia shows the whole process very well:

The method above is extremely important and is the basis for most all modern cryptographic algorithms.

That about wraps up what I wanted to cover. I think most of the basics were covered and it gives a good foundation for further learning. I plan on writing more in the future as my own skills progress and would love comments.