Two Challenges of Stealthy Hypervisors Detection: Time Cheating and Data Fluctuations

Editor's Notes

• #2: Hello! Today I’ll tell you about a new security system which detects hidden software
• #3: This talk consists of three parts. First, I’ll describe a hypervisor as a “real cyber threat”. Next I’ll focus on the existing hypervisor detection methods and their drawbacks. Finally I’ll present a new hypervisor detection system, which improved time based detection method.
• #4: The idea is that any computer can be compared with a big ship. Because decks of a ship look like computer modes.
• #5: To protect a ship we need to make sure that there is no terrorist (spy or intruder) all over the ship from the hold to the top of the mast. In terms of computer the situation is absolutely the same, we need to make sure that there is no spyware in each computer mode.
• #6: Here you can see the details of different computer modes. Each of these modes can have backdoor. At the last conference in Virginia one year ago, I discussed kernel mode rootkits; it was the upper deck. Now we are going deeper. Today I’ll tell you about the detection of new suspicious software… about hypervisors.
• #7: All right, we all have laptops and {u-Z-e} use them for various purposes. Of course, any laptop could be infected.
• #8: To remove malware we use an antivirus and we expect that this antivirus works properly.
• #9: This is the basic scheme of how computers work. All these software apps are run by the operating system, which is above {E-B-A-V} the hardware.
• #10: But the situation is changing: the vast majority of computers which have been sold in the last 10 years definitely have a CPU — a processor with “hardware virtualization technology {T-E-K-nologi}”. For Intel processors this technology is called VT-X, and for AMD processors it’s called AMD-V. Today we will focus on Intel processors, but all my ideas work well on AMD CPU as well. Hardware virtualization technology provides a mechanism to put an intermediary between the operating system and the hardware.
• #11: This sóftware is called a hypervisor and it runs directly on the hardware. A hypervisor is a program that operates at a more privileged level than the operating system; it is just below the operating system. In a nutshell, the hardware virtualization is the CPU feature. Does any computer system support hardware virtualization?
• #12: On the left you can see the computers without hardware virtualization, and on the right you can see the computers that support hardware virtualization technology. If you want to check whether your computer supports this technology or not go to this website or use tools like {Cee‑Pee‑Uuu‑Z} CPU-Z to receive system information. Now let’s go through some features of a hypervisor. We will focus on 5 of them.
• #13: The first one is that a hypervisor can control all interaction between the operating system and the hardware. A hypervisor has an unrestricted access to any hardware compónents, like hard disks and network cards. Second, it’s impossible to stop or block a hypervisor by using the operating system, because a hypervisor runs at a more privileged level. Third, there is no built-in tool to detect a hypervisor. Current approaches cannót solve the detection problem in an appropriate way. Next is that a hypervisor can avoid its detection using a variety of countermeasures, today we will focus on the time cheating. What feels óminous is that a hypervisor can be installed invisibly both {BO-UTH} for users and anti-viruses.
• #14: By using the first two advantages, we can create an advanced {ADVA-ANSD} and powerful anti-virus, which enhances the level of security, or for example a tool to {í-S-o} ísolate ínternet banking from malware. It sounds practical! Obviously, this is a plus of “hardware virtualization”. Of course, we can use all these advantages, but what will happen if hackers use all of them? Surely, on the dark side, hackers can create the most powerful “backdoor”. Any doubts {DA-U-T-S} that this can happen?
• #15: Before answering this question let’s look at what kind of backdoor a hypervisor can provide. This backdoor can record keystrokes, steal all your data or even block your computers. Also a hypervisor can be installed by a driver or it can sit in the BIOS
• #16: Moreover, the problem with a stealthy hypervisor detection becomes even worse when “several nested hypervisors” sit on your computer. This is an example of two nested hypervisors which are running at the same time. In this case, a legítimate hypervisor allóws {ELAUS} us to run two operating systems at the same time, like Windows and Mac operating systems simultaneously. It sounds great! At the same time, the malicious hypervisor is loaded first and therefore it is located between the trusted hypervisor and the hardware. That is why it can compromise the trusted one and steal all our data.
• #17: This situation is entirely factual. Look at the first two lines. Here you can see two examples of hypervisors, which are loaded by drivers. And also at the last DeepSec conference, the investigation about undócumented suspicious hypervisor was presented. Look at the last line. This ghost {GO-UST} hypervisor uses BIOS to hide its start-up. All these examples prove that an illegal hypervisor is a “real cyber threat” to our computers, … to all our data, … to our business. We have to meet this challenge; we have to detect a backdoor hypervisor. Of course, researchers are trying to solve the hypervisor detection problem.
• #18: Look at this comparison table with well-known hypervisor detection tools. The first column includes the tool titles. The next one holds the implemented detection methods. And the last two columns show if the tools are resilient and portable.
• #19: Look at the first two lines. You can see that only hardware tools are resilient to hypervisor countermeasures, but they are not portable so they are good only for laboratories. That is why further we will not consider the hardware tools.
• #20: In terms of portability {porta-BÍ-LITY}, the software tools are better, but they are unable to detect stealthy hypervisors, … hypervisors which are applying countermeasures. So far, you can see that there is no tool, which is both {BO-UTH} portable and resilient.
• #21: But today I’ll present you a detection tool which is both {BO-UTH} portable and resilient.
• #22: Now let’s focus on the principles of these software detection tools. How do they work and why are they vulnerable?
• #23: There are four hypervisor detection methods. Let’s start with signature-based detection.
• #24: How does it work? After the installation of a hypervisor, its dispatcher or binary code and corresponding data structures are loaded in the memory. Look at the red rectangles. We can reliably detect a hypervisor by finding its memory fragments. But a hypervisor can hide its memory pages and prevent its detection. For example, a hypervisor can use Extended Page Tables for Intel CPU. You can see that this detection method is vulnerable to hypervisor countermeasures.
• #25: Another method is a detection based on a trusted hypervisor. In this case, a trusted hypervisor is loaded first and it’s close to hardware. That is why it can prevent any installation of a suspicious one. It’s a trusted hypervisor.
• #26: Everything looks okay, but what will happen if a suspicious, … a worm hypervisor installs before the trusted one? What will happen if the worm gains the control first? Of course, in this situation, the worm hypervisor can prevent the installation of the trusted one; it can compromise the trusted one. As a result our computers will be unsafe, will be in danger. Unfortunately, this method is vulnerable to typical man in the middle attack.
• #27: The next method is named behavior-based detection. This method relies on the fact that the result of specific instructions execution depends on whether a hypervisor is present or not. For example if a hypervisor is running by some old AMD processor, the execution of instruction VMSAVE with this parameter causes the system to freeze. Or it stops the system. But in the same computer without a hypervisor, such execution does not freeze the system. You see the difference in the behavior. Unfortunately, this method works only for old CPUs and there are no such “freezing” instructions for new hardware. This is a serious drawback for the detection.
• #28: The last method for today is tíme-based detection. Now I’m going to demonstrate an interaction between an operating system and a hypervisor. Without a hypervisor all events occur in the operating system without long delays.
• #29: But if a hypervisor is present, it intercepts this event. As a result the hypervisor code starts to run and it needs some time, … a delay to process this event. Therefore, the duration of this event becomes longer, in comparison with no hypervisor case. We can precisely detect a hypervisor by measuring this time delay.
• #30: Everything looks ok, but a hypervisor can compensate this time overhead, … this delta time by using time cheating function. As a result, this delta time is decreased and becomes the same as in no hypervisor case. We can see that this method is vulnerable to time-cheating.
• #31: So far we can see that all software detection methods are vulnerable to hypervisor countermeasures. For my further analysis, I chose the time-based detection. Why? I focused on the fact that a hypervisor always spends time to process an event and this time needs to be hidden. It’s impossible to process an event without spending time. That was my idea. Let’s focus on the road map of current situation with time-based detection and mark my contribution on it.
• #32: We know that by using the time values or average time values we can precisely detect only a non-hidden hypervisor.
• #33: But if a hypervisor is using time-cheating it’s impossible to detect it using average values, because they are the same. So, a stealthy hypervisor avoids time-based detection. For this situation there are no appropriate time-based detection methods.
• #34: And today, I’ll describe how to improve it. Look at the yellow arrow. This new method works well even if a hypervisor applies time-cheating.
• #35: To detect a hypervisor as “an event” I choose unconditionally intercepted instructions. Let’s focus on them.
• #36: First of all, what instructions do we call unconditionally intercepted? Execution of each of these instructions is always intercepted by any hypervisor. Or this event causes a hypervisor dispatcher to start. An example of such instructions is “Cee Pee Uu I Dee”.
• #37: How to use them for detection? There are three steps: Read a time counter, execute a block of these instructions and read the time counter repeatedly. I calculate Instructions Execution Time — “I-Ee-Tee” or just the duration, … the delta time, … by subtracting the first value from the second one.
• #38: To make it clear I’ll show how this scheme works. For example, I measured ten (10) “Cee-Pee-Uu-I-Dee” instructions with the help of Time Stamp Counter in two situations: without a hypervisor and with it. You can see a ten times difference between “I-Ee-Tee” averages without a hypervisor and with a non‑stealthy hypervisor. In other words, there is no problem to detect a non-stealthy hypervisor.
• #39: However, if a hypervisor uses time cheating these average values are the same, … are absolutely the same. Their values are often not the same but really close to each other and that is why we cannot be sure about whether a hypervisor is present or not. This is the most complicated situation, that hasn’t been taken into account before. What should we do?
• #40: Let’s think. What steps can be used to detect a hypervisor?
• #41: First of all we need a computer without any hypervisor. We will measure the time in the following cases: without a hypervisor and with it. Next we calculate some statistic value and by comparing their values we can find threshold values. It’s similar to the previous situation. But you know that because of time-cheating attack the average values are not applicable. Now we just have to invent the appropriate statistic to replace an average.
• #42: To do this to invent the statistic value let’s move on to the question. What is happening to the computer during time measurements?
• #43: Look at the interaction between the operating system and other modes during the measurements of “I‑Ee‑Tee”. Without a hypervisor, the operating system interacts only with System Management Mode — S‑M‑M, which is the most privileged mode and its interruptions occur randomly. S‑M‑M suspends the work of the operating system only for a short time. If a hypervisor is loaded, such interruptions to S‑M‑M suspend both {BO-UTH} of them, … the operating system and the hypervisor. In a nutshell, the S‑M‑M is the most privileged mode and its interruptions occur randomly.
• #44: Let’s specify the process of modes switching during “I Ee Tee” measurements. Without a hypervisor, instructions are executed in the operating system. This process is indicated by a small blue loop. Randomly generated {jenerated} interruptions to the S‑M‑M are indicated as dashed blue arrows {ARO‑UZ}.   A scheme with a hypervisor is shown below. The hypervisor catches {CAT-CHIZ} the execution of every “Cee Pee Uu I Dee” instruction from the operating system. This execution is indicated by a red oval loop. The S‑M‑M is duplicated {DUPLICAT} for better clarity. Of course, there is only one mode. S‑M‑M interruptions occur either from the operating system or from the hypervisor in a random way.   Now let’s compare these cases and think about “I Ee Tee” peculiarities here.
• #45: First due to the random nature of the interruptions to the S‑M‑M, the “I Ee Tee” is a random variable. Another point is that “I Ee Tee” is randomly distríbuted in several fixed layers. This layered nature is also caused by the interruptions to the System Management Mode.   Let’s focus on the following “I Ee Tee”characteristics: average value and variability indexes {INDISIZ}, such as the number of layers, variance and fourth order moment. After a hypervisor is loaded all these statistics are increased in comparison with a no-hypervisor case.
• #46: We remember that a hypervisor can reduce the average value by time cheating. In this situation, we have to apply only variability indexes as a possible way for detection. Let’s check these three ideas about random nature with layers and variability indexes.
• #47: The experiment was done in the following way.
• #48: First, I use a tiny {TA-INY} hypervisor with minimum functionality. It has basic functions and time cheating. This is the most complicated case for detection. Next, you can see the source code fragment for measuring “I Ee Tee”. The inner loop runs one thousand (1000) times and measures time, after that I suspended this driver for a two‑second delay to make observations {ABZERVATIONZ} independent. The outer loop runs ten (10) times. As a result, I got a matrix with one thousand (1000) by ten (10).
• #49: Here you can see the example of this matrix for no hypervisor case. Such matrices were measured for both {BO-UTH} situations without a hypervisor and with a hidden one. To make it clear let me show fragments of columns with the help of a “scatter plot”.
• #50: Here you can see scatter plots for two mentioned cases. This is the plot for the case with no hypervisor, and here is the plot with one hidden hypervisor. Each point on the plot represents a value from the column. Numbers of experiments are on the x-axis. The y-axis shows the execution {EX-SECUTION} time in CPU ticks or “I Ee Tee” values. Let’s compare the average values of these two plots.
• #51: We can see that due to time cheating the average values of these two cases are the same. The average is about two thousand and seven (2007). As a result, existing time based detection method does not work.
• #52: However, something is different. Let’s focus on the structures of these plots. Without a hypervisor, a set of points consists of few horizontal layers. There are just 4 layers and no jumps. With one hypervisor present, a set of points consists of several layers with a lot of jumps. All these plots in both {BO-UTH} cases have a layered structure.
• #53: Moreover, we can see that after a hypervisor has been loaded, the number of layers has grown three-fold. Also, a variance has increased {INCRI-ST} five-fold. We see that all variability indexes are increased {INCRI-ST} so they can be used for detection. Perfect!
• #54: We can conclude that all my theoretical {THIO-retical} principles have been confirmed by experiments. In particular, “I Ee Tee” is a random variable, whose spread increases after the loading of a hypervisor. Now we are focusing on three types of statistics: number of layers, variance and fourth (4th-) order moment. To detect a hypervisor we need to compare samples in both {BO-UTH} cases without and with a hypervisor. But it isn’t so easy! Let’s look at the reasons.
• #55: My experiments {I-KSPERiments} show that, apart from the layered {LAYYED} nature “I Ee Tee” samples also include partial deviation and other anómalies, like noise which significantly change variability indexes, so that they cease {SEASE} to be useful for detection. In addition, our analysis shows that “I Ee Tee” is not normally distríbuted. Let’s think which of the statistical tests are applicable in our situation.
• #56: First, we cannot use classical statistical or parametric tests, for example, Student’s t-tests, because all of them require normal distribution. Non-parametric tests are possible. But, our experiments show that these tests, for example a Wilcoxon test, are not precise enough for our experimental data.
• #57: To compare these samples I use another non-classical method, which was introduced by Mark Kornfeld in Russian and later by Johann {Yo-HANN} Strellen in English. Using this method the confidence interval is calculated {CAL-KJU-LATED} as the interval between the minimum and maximum values. Here is the corresponding confidence level. Let’s use this method.
• #58: First, I measured “I Ee Tee” for two situations: without a hypervisor and with it. Then I calculated statistic value, for example, variance according to the matrixes columns {CALOMS}. After that, I compared these variation intervals with each other: blue and red intervals.
• #59: As a result, I found out that the variation intervals overlap so much, that I could not get the threshold value to detect a hypervisor.
• #60: What are the reasons for this data fluctuation? We need to understand why the calculated statistics are so different.
• #61: To answer this question I repeatedly made scatter plots and now show you typical anomalies. {magnify up-left} We see that this scatter plot includes a high outlier {OUTLAE} apart from basic layers. Also, the plot includes one jump. {magnify up-right } This is an example of high outliers. {magnify left-down} And this is an example of low frequency values. All these anomalies considerably influence the variance and the other statistics. As a result, I cannot calculate the appropriate {APRO-UPRIAT} threshold values.
• #62: Let’s look at how to reduce the influence of these anomalies. To remove outliers and low frequency values I applied filtration. As a result, all values whose frequency is lower than 2, 5 or 10% are deleted. Here is the original plot with one outlier and jump. The corresponding variance is about half a million.
• #63: After the filtration, one outlier {OUTLI-YA} was deleted, and variance became about one hundred thousand. You see the difference. Let’s move on to the jump.
• #64: To correct each jump I calculated the statistic values, for example variance, before and after the jump. Next, I added these values using their section lengths. In other words, I averaged out the statistic before and after the jump. Finally, the variance is a bit more than a hundred. These measures significantly helped me to stabilize statistics and their variation intervals.
• #65: I decided to test these ideas every day and do more experiments.
• #66: Here you can see variation intervals for both cases: without and with a hypervisor, which were calculated during three days. Day first, second and third.
• #67: The point is that I received different variation intervals on different days. So it seemed impossible to get the appropriate threshold values to detect a hypervisor.
• #68: Now let’s move on to the question – how to overcome this lack of repeatability?
• #69: I tackled this issue by using a two-step way to calculate new statistics. First, I calculated, for example, variance, for each column in the matrix and achieved a set of values Vee‑one (V1), Vee‑two (V2) and so on Vee‑ten (V10), which I considered {CONSIDED} as a “new sample”. Second step was to calculate the statistics of this new sample. Here “Vee-bar” was the average of this new sample. As a result overlapping of new statistics intervals was seriously reduced.   Another idea was to repeat measurements within 10 days. This period was enough to stabilize variation intervals.   These two approaches were sufficient to get appropriate threshold values. It looked like a success.
• #70: But, what can we do if variation intervals keep overlapping? Look at the green flag. If the calculated value is in the overlapping part it is impossible to detect a hypervisor precisely. What should we do in this situation?
• #71: The first idea was to repeat “I Eee Tee” measurements and recalculate statistics. Here you can see the probabilities of errors of the first and second kinds {KA-INDS}. The thing was that because this overlapping part is really small, the probability of repeating this negative situation is quite low. It is highly unlikely that the calculated value occurs in the overlapping part again because of the multiplication rule for independent events. Let’s join all the previous steps to calculate the threshold values.
• #72: The first step was to obtain the data for ten days. I got five matrixes every day for each case: without a hypervisor and with my tiny {TA-INY} hypervisor. All in all I obtained fifty matrixes for each case. Next was to calculate statistics using a two-step way. The final step was to calculate threshold values. The results are in the table.
• #73: This is the table with threshold values. If the calculated number of layers is “5”, so it is smaller than threshold “7” and it means there is no hypervisor. If the calculated number of layers is bigger than “8”, it means that at least one hypervisor is present. The complete table with threshold values is in my paper.
• #74: Let’s sum up all previous ideas to formulate how to detect a stealthy hypervisor.
• #75: First is to guarantee that there is no hypervisor in BIOS by reflashing it. Next is reinstall operating system, which is downloaded from the official sources. The first two steps look like performing a factory data reset in computers. Next is to calculate threshold values from the no hypervisor case. After that install supplementary software and monitor messages about a hypervisor presence.
• #76: You can say, all right Igor, but your head is in the clouds. The first two steps are good only for crazy computer experts. They are not appropriate for everyday life. Yes, you are absolutely right. Let’s keep both {BO-UTH} feet on the ground. How to make it userfriendly?
• #77: Now I have the basis to guarantee no hypervisor presence in the running computer just by checking the structure of the scatter plot. The details are in the paper.
• #78: Here you can see the architecture of this detection system. At the preliminary stage, I use “measure IET” block, tiny hypervisor, block for calculating statistics and getting thresholds. At the operational stage, I calculate statistics and compare their values with thresholds. This detection system is developed using Cee + + and MATLAB, the source code is free & available here.
• #79: Experiments {IkspErimens} proved that this detects any hidden hypervisor, as well as 2 nested ones. Here you can see cases, which were considered. There are hypervisors based on drivers and BIOS-based ones. Detection system works well on both {BO-UTH} CPUs: on Intel and AMD.
• #80: As a conclusion, I’d like to highlight all the challenges which I faced and how I dealt with them. First, a stealthy hypervisor uses time cheating to prevent its detection. I have solved {SOLVD} this by using variability indexes. Second is the data fluctuation and noise values. I solved them by applying filtration, calculating statistics using two-step way and repeating measurements. One more thing, there were no appropriate methods to compare samples. I used the Kornfeld method for solving this task.
• #81: In a nutshell, in this paper I presented a statistical ruler for hypervisors. This is the first software system, which can detect a stealthy hypervisor and calculate several nested ones even under countermeasures. If you are interested I can show how it works, it will take about twenty minutes. Thank you.
• #84: You can see the meaning of the type I and II errors and how to calculate them.
• #87: The confidence intervals method was introduced by Kornfeld in Russian and later by Strellen in English. This is just the comparison of Kornfeld and Strellen ideas. You can see the similar formulas and results: Kornfeld sorts sample values by ascending, that is why X1 is the minimum and Xn is the maximum. Strellen gets straight the maximum and the minimum. Formulas with the confidence intervals are the same.
• #89: Here you can see the reasons of interruption to System Management Mode, there are signals from Thermal Sensor, RTC alarm, using USB devices instead of COM, like COM keyboard and mouse.
• #90: These are the ways how to debug a hypervisor. To debug a hypervisor driver we can use Vmware with kernel debugger WinDbg. We can also apply hardware emulator Bochs ("box") or AMD SimNow and their own debuggers. It is possible to use classical approach with DbgPrint messages and DbgView to collect it. This approach is appropriate more for demonstration not for debugging. Another approach which is used in BluePill is to send debug messages to COM port. The last method is to use “Debug PCI Card” to see, for example POST BIOS messages.