Difference between revisions of "Login VSI VSImax"

From Login VSI Documentation
Jump to: navigation, search
(Timer Metrics)
 
(48 intermediate revisions by the same user not shown)
Line 1: Line 1:
The logic behind Login VSI is different to conventional benchmarks. In general, most system benchmarks are steady state benchmarks. These benchmarks execute one or multiple processes, and the measured execution time is the outcome of the test. Simply put: the faster the execution time or the bigger the throughput, the faster the system is according to the benchmark.  
+
The logic behind Login VSI is different to conventional benchmarks. In general, most conventional benchmarks are steady state benchmarks. These execute one or multiple processes, and the measured execution time is the outcome of the test. Simply put: the faster the execution time or the bigger the throughput, the faster the system is.  
  
Login VSI is different in approach. Login VSI is not primarily designed to be a steady state benchmark (however, if needed, Login VSI can act like one). Login VSI was designed to perform benchmarks for SBC or VDI workloads through system saturation. Login VSI loads the system with simulated user workloads using well known desktop applications like Microsoft Office, Internet Explorer and Adobe PDF reader. By gradually increasing the amount of simulated users, the system will eventually be saturated. Once the system is saturated, the response time of the applications will increase significantly. This latency in application response times gives a clear indication whether the system is (close to being) overloaded. As a result, by nearly overloading a system it is possible to find out what is the true maximum user capacity.
+
Login VSI takes a different approach. Login VSI is not primarily designed to be a steady state benchmark (however, if needed, Login VSI can act like one). Login VSI was designed to perform benchmarks for SBC or VDI workloads through system saturation. Login VSI loads the system with simulated user workloads using well known desktop applications like Microsoft Office, Internet Explorer and Adobe PDF Reader. By gradually increasing the amount of simulated users, the system will eventually be saturated. Once the system is saturated, the response time of the applications will increase significantly. This latency in application response times gives a clear indication whether the system is (close to being) overloaded. As a result, by nearly overloading a system it is possible to find out what is the true maximum user capacity.
  
After a test is performed, the response times can be analyzed to calculate the maximum active session/desktop capacity. Within Login VSI this is calculated as VSImax. When the system is coming closer to its saturation point, response times will rise. When reviewing the average response time it will be clear that the response times escalate at the saturation point.
+
After a test is performed, the response times can be analyzed to calculate the maximum active session/desktop capacity. Within Login VSI, this is calculated as VSImax. When the system is getting closer to its saturation point, response times will rise. When reviewing the average response time it will be clear that the response times escalate at the saturation point.
  
This VSImax is the “Virtual Session Index (VSI)”. With Virtual Desktop Infrastructure (VDI) and Terminal Services (RDS) workloads this is valid and useful information. This index simplifies comparisons and makes it possible to understand the true impact of configuration changes on hypervisor host or guest level.
+
This VSImax is the “Virtual Session Index (VSI)”. With Virtual Desktop Infrastructure (VDI) and Terminal Services (RDS) workloads this is valuable information. This index simplifies comparisons and makes it possible to understand the true impact of configuration changes on a hypervisor host or at guest level.
  
'''Recommended read:''' [https://www.loginvsi.com/blog/481-calculating-maximum-virtual-desktop-capacity-vsimax-explained Calculating Maximum Virtual Desktop Capacity – VSImax Explained]
+
'''Recommended:''' [https://www.loginvsi.com/blog/481-calculating-maximum-virtual-desktop-capacity-vsimax-explained Calculating Maximum Virtual Desktop Capacity – VSImax Explained]
  
 
==Server side response time measurements==
 
==Server side response time measurements==
  
It is important to understand why specific Login VSI design choices have been made. An important design choice is to execute the workload directly on the target system within the session instead of using remote sessions. The scripts simulating the workloads are performed by an engine that executes workload scripts on every target system, and are initiated at logon within the simulated user’s desktop session context.  
+
It is important to understand why specific Login VSI design choices have been made. An important design choice is to execute the workload directly on the target system within the session instead of using remote sessions. The scripts simulating the workloads are performed by an engine that executes workload scripts on every target system, and are initiated at logon within the simulated user’s desktop session context.
  
An alternative to the Login VSI method would be to generate user actions client side through the remoting protocol. These methods are always specific to a product and vendor dependent. More importantly, some protocols simply do not have a method to script user actions client side.  
+
An alternative to the Login VSI method would be to generate user actions on the client side via a remoting protocol. These methods are always specific to a product and rely on its vendors. More importantly, some protocols simply do not have a method to script user actions on the client side.
  
For Login VSI the choice has been made to execute the scripts completely server side. This is the only practical and platform independent solution, for a benchmark like Login VSI. The relative overhead and footprint of a benchmark engine scripted in AutoIT is small enough (1-5% range) for Login VSI’s purposes.
+
For Login VSI, the choice has been made to execute the scripts on the server side. This is the only practical and platform independent solution for a benchmark, like Login VSI. The relative overhead and footprint of a benchmark engine scripted in AutoIt is small enough (1-5% range) for Login VSI’s purposes.
  
==Calculating VSImax v4.1==
+
==Calculating VSImax v4.1.x==
  
The simulated desktop workload is scripted in a 48-minute loop when a simulated Login VSI user is logged on, performing generic Office worker activities. After the loop is finished it will restart automatically. Within each loop the response times of ten specific operations are measured in a regular interval, five of which are the most important. These measurements happen sixteen times within each loop. The response times of these five important operations are used to determine VSImax.
+
The simulated desktop workload is scripted in a 48-minute loop where a simulated Login VSI user is logged on and performing generic office worker activities. After the loop is finished, it will restart automatically. Within each loop the response times of ten specific operations are measured at a regular interval (16 times within each loop), five of which are the most important. The response times of these important operations are used to determine VSImax.
  
The five operations from which the response times are measured are:
+
The five important operations are:
  
'''Notepad File Open (NFO)'''
+
*'''Notepad File Open (NFO)''': Loading and initiating VSINotepad.exe and opening the openfile dialog. This operation is handled by the OS and by the executable itself through execution (this operation seems almost instant from an end-user point of view).
  
Loading and initiating VSINotepad.exe and opening the openfile dialog. This operation is handled by the OS and by the VSINotepad.exe itself through execution. This operation seems almost instant from an end-user’s point of view.
+
*'''Notepad Start Load (NSLD)''': Loading and initiating VSINotepad.exe and opening a file. This operation is also handled by the OS and by the executable itself through execution (this operation seems almost instant from an end-user point of view).
  
'''Notepad Start Load (NSLD)'''
+
*'''Zip High Compression (ZHC)''': This action copies a random file and compresses it (with 7-Zip) with high compression. The compression will very briefly spike both CPU and disk I/O.
  
Loading and initiating VSINotepad.exe and opening a file. This operation is also handled by the OS and by the VSINotepad.exe itself through execution.This operation seems almost instant from an end-user’s point of view.
+
*'''Zip Low Compression (ZLC)''': This action copies a random file and compresses it (with 7-Zip) with low compression. The compression will very briefly spike disk I/O while also creating some load on the CPU as well.
  
'''Zip High Compression (ZHC)'''
+
*'''CPU''': Calculates a large array of random data and spikes the CPU for a short period of time.
  
This action copy's a random file and compresses it (with 7zip) with high compression enabled. The compression will very briefly spike CPU and disk IO.  
+
'''Note''': NFP (Notepad File Print) is no longer used to calculate VSImax due to the amount of printers configured in an environment. The amount of printers could influence this number. If you have 15 printers, it takes much longer to open the dialog compared to when you only have one printer configured.
  
'''Zip Low Compression (ZLC)'''
+
These measured operations affect many different sub-systems, such as CPU (user and kernel), Memory, Disk, the OS (in general), the application itself, print, GDI, etc. These operations are specifically short by nature. When such operations become consistently long, the system becomes saturated because of excessive queuing on any kind of resource. As a result, the average response times will then escalate, which is clearly visible to end-users. If such operations consistently take longer than expected, the user will regard the system as slow and unresponsive.
  
This action copy's a random file and compresses it (with 7zip) with low compression enabled. The compression will very briefly disk IO and creates some load on the CPU aswell.  
+
===VSImax Metrics===
 +
Once the test is finished, VSImax v4.1 will be calculated. Previous VSImax models (Classic and Dynamic) needed Microsoft Word to function. With the new 4.1 timers this is no longer needed, making it more flexible and applicable to a larger amount of scenarios.  
  
'''CPU'''
+
The following actions are part of the calculation and are weighted as follows (US notation):
  
Calculates a large array of random data and spikes the CPU for a short period of time.
 
 
===VSImax Metrics===
 
Once the test is finished, VSImax v4.1 can be calculated. Previous VSImax models (Classic and Dynamic) needed Microsoft Office Word to function. With the new 4.1 timers this is no longer needed, making it more flexible and applicable to a larger amount of scenarios.
 
 
The following actions are part of the VSImax v4.1 calculation and are weighted as follows (US notation):
 
 
*Notepad File Open (NFO): 0.75
 
*Notepad File Open (NFO): 0.75
 
*Notepad Start Load (NSLD): 0.2
 
*Notepad Start Load (NSLD): 0.2
Line 56: Line 51:
  
 
===VSImax Baseline===
 
===VSImax Baseline===
 +
With the introduction of Login VSI 4.1, we also created a new method to calculate the basephase of an environment. With the new workloads (Taskworker, Powerworker, etc.) enabling ''basephase'' for a more reliable baseline has become obsolete. The calculation is explained below.
  
With the introduction of Login VSI 4.1 we also created a new method to calculate the baseline of an environment. With the new workloads (Taskworker, Powerworker, etc.) enabling 'basephase' for a more reliable baseline has become obsolete. The calculation is explained below.  
+
In total, 15 lowest VSI response time samples are taken from the entire test, the lowest 2 samples are removed and the 13 remaining samples are averaged. The result is the Baseline.  
  
The 13 lowest VSI Index Calculation response time samples are taken from the entire test and are averaged. The result is the Baseline. In short:
+
In short:
  
*Sort the VSI Index Calculation values from lowest to highest
+
*Take the lowest 15 samples of the test
*Take the lowest 13 samples of the VSI Index Calculation
+
*Remove the lowest 2 from those 15 samples
*Average the 13 results and the result is the baseline
+
*Average these 13 samples to calculate the Baseline number
  
==Calculating VSImax v4==
+
===VSImax Average===
 +
The VSImax average response time in Login VSI 4.1.x is calculated on the amount of active users that are logged on to the system. This calculation always uses the average of five Login VSI response time samples + 40% of the amount of active sessions.
  
The simulated desktop workload is scripted in a 48 minute loop when a simulated Login VSI user is logged on, performing generic Office worker activities. After the loop is finished it will restart automatically. Within each loop the response times of six specific operations are measured in a regular interval: twelve times in within each loop. The response times of these six operations are used to determine VSImax.
+
For example, if the active sessions = 60, then latest 5 + 24 (40% of 60) = 29 response time measurements are used for the average calculation.
  
The six operations from which the response times are measured are:
+
To remove noise (accidental spikes) from the calculation, the top 5% and bottom 5% of the VSI response time samples are removed from the average calculation, with a minimum of one top and one bottom sample. As a result, with 60 active users, the last 29 VSI response time sample are taken. From those 29 samples, the top two samples are removed and lowest two results are removed (5% of 29 = 1.45, rounded to 2). At 60 users, the average is then calculated over the 27 remaining results.
  
'''Starting “VSI Notepad”'''
+
===VSImax Threshold===
 +
VSImax v4.1.x is reached when the VSIbase + a 1000ms latency threshold is not reached by the average VSI response time result. Depending on the tested system, VSImax response times can grow 2-3x the baseline average. In end-user computing, a 3x increase in response time compared to the baseline is typically regarded as the maximum performance degradation to be considered acceptable.
  
This operation is handled by the OS (loading and initiating <nowiki>VSINotepad.ex</nowiki>e) and by the <nowiki>VSINotepad.ex</nowiki>e itself through execution. This operation seems almost instant from an end-user’s point of view.  
+
In VSImax v4.1.x, this latency threshold is fixed to 1000ms, which allows better and fairer comparisons between two different systems, especially when they have different baseline results. Ultimately, in VSImax v4.1.x, the performance of the system is not decided by the total average response time, but by the latency is has under load. For all systems, this is now 1000ms (weighted).
  
'''Starting the “File Open” dialogue '''
+
The threshold for the total response time = average weighted baseline response time + 1000ms.
  
This operation is handled for a small part by Notepad and a large part by the operating system. The file open dialogue uses generic subsystems and interface components of the OS. The OS provides the contents of this dialogue.  
+
When the system has a weighted baseline response time average of 1500ms, the maximum average response time may not be greater than 2500ms (1500 + 1000). If the average baseline is 3000, the maximum average response time may not be greater than 4000ms (3000 + 1000).
  
'''Starting the “Print” dialogue'''
+
When the threshold is not exceeded by the average VSI response time during the test, VSImax is not hit and the amount of sessions ran successfully. This approach is fundamentally different in comparison to previous VSImax methods, as it was always required to saturate the system beyond VSImax threshold.
  
This operation is handled for a large part by the OS subsystems, as the print dialogue is provided by the OS. This dialogue loads the print-subsystem and the drivers of the selected printer. As a result, this dialogue is also dependent on disk performance.  
+
===Additonal Info===
 +
VSImax v4.1.x is now always reported with the average baseline VSI response time result. For example: The VSImax was 125 with a baseline of 1526ms. This helps considerably in the comparison of systems and gives a more complete understanding of the system. The baseline performance helps to understand the best performance the system can give to an individual user. VSImax indicates what the total user capacity is for the system. These two are not automatically connected or related.
  
'''Compress the document into a zip file with 7-zip command line (2x)'''
+
For example, when a server with a very fast dual core CPU, running at 3.6 GHz, is compared to a 10 core CPU, running at 2.26 GHz, the dual core machine will give an individual user better performance than the 10 core machine. This is indicated by the baseline VSI response time. The lower this score is, the better performance an individual user can expect.
  
This operation is handled by the command line version of 7-zip. The compression will very briefly spike CPU and disk IO. This action is performed twice: once with no compression (IO intensive) and with high compression (CPU intensive)
+
However, the server with the slower 10 core CPU will easily have a larger capacity than the faster dual core system. This is indicated by VSImax v4.1.x, and the higher the VSImax is, the larger overall user capacity can be expected.
  
'''Starting Microsoft Word with a document'''
+
With Login VSI 4.1.x a new VSImax method is introduced: VSImax v4.1. This methodology gives much better insight into system performance and scales to extremely large systems.
  
This operation will measure the responsiveness of the Operating System and the file system. Microsoft Word is started and loaded into memory, also the new document is automatically loaded into Microsoft Word. When the disk IO is extensive or even saturated, this will impact the file open dialogue considerably.  
+
=Login VSI Timer=
 +
The VSI_Timer41 function measures the responsiveness of the system. It looks at the response of the system at multiple points within each workload. Within the Login VSI 4.1 timer, the timer mechanism has been made more reliable by removing erratic timers and adding more comprehensive timers.
  
These measured operations within Login VSI do hit considerably different subsystems such as CPU (user and kernel), Memory, Disk, the OS in general, the application itself, print, GDI, etc. These operations are specifically short by nature. When such operations become consistently long: the system is saturated because of excessive queuing on any kind of resource. As a result, the average response times will then escalate.  This effect is clearly visible to end-users. If such operations consistently consume multiple seconds the user will regard the system as slow and unresponsive.  
+
'''For example:'''
 +
If we were to start an application, we look at what kind of performance we can expect at that moment in time, e.g. how long does it take to start the application or open the document.
  
 +
==Timer Metrics==
  
[[File:Good-chart.png|400px|thumb|center]]
+
VSImax v4.1 shows the amount of sessions that can be active on a system before the system is saturated. This number gives you an indication of the scalability of the environment (higher is better). VSImax v4.1 is determined by the measurements in the below table:
  
<center>'''Figure 1 Sample of a VSI max responsetime graph, representing a normal test'''</center>
+
{| class="wikitable"
 +
|-
 +
|
 +
'''Measurement ID'''
  
 +
|
 +
'''Measurement Action'''
  
 +
|
 +
'''Measurement Action Detailed'''
  
 +
|
 +
'''Measures Related Resource'''
 +
|-
 +
|
 +
'''NSLD'''
  
[[File:Bad-chart.png|400px|thumb|center]]
+
|
 +
Notepad starts and loads a 1500KB document
  
<center>'''Figure 2 Sample of a VSI test responsetime graph where there was a clear performance issue'''</center>
+
|
 +
Notepad starts and loads a random local 1500KB document file that is copied from the content pool
 +
|
 +
CPU and I/O
 +
|-
 +
|
 +
'''NFO'''
  
 +
|
 +
Measure how long it takes to show the file-open dialog in Notepad
 +
|
 +
VSI-Notepad file open [Ctrl+O]
 +
|
 +
CPU, RAM, and I/O
 +
|-
 +
|
 +
'''ZHC*'''
  
 +
|
 +
Create a Zip file with high compression
 +
|
 +
Compress a local random .pst file that is copied from the content pool (5MB)
 +
|
 +
CPU and I/O
 +
|-
 +
|
 +
'''ZLC*'''
  
Once the test is finished, VSImax v4 can be calculated. Previous VSImax models (Classic and Dynamic) could not be calculated when the system was not saturated. In VSImax v4 this is not a requirement anymore. When the system is not saturated, and it could complete the full test without exceeding the average response time latency threshold, VSImax is now considered the maximum of active sessions that were launched.
+
|
 +
Create a Zip file with low compression
 +
|
 +
Compress a local random .pst file that is copied from the content pool (5MB)
 +
|
 +
I/O
 +
|-
 +
|
 +
'''CPU'''
  
The response times are very different per measurement type, for instance Zip with compression can be around 2800 ms, while the Zip action without compression can only take 75ms. These response time of these actions are weighted before they are added to the total. This ensures that each activity has an equal impact on the total response time.
+
|
 
+
Calculates a large array of random data
In comparison to previous VSImax models, this weighting much better represent system performance. All action have very similar weight in the VSImax total, both in VSImax classic and dynamic the opening of word had far greater impact on the total than other activities. The following weighting of the response times are applied:
+
|
 
+
Creates a large array of random data that will be used in the I/O timer
 
+
|
 
+
CPU
The following actions are part of the VSImax v4 calculation and are weighted as follows (US notation):
+
|-
 
+
|}
•    Start VSINotepad with large text file: 0.5
+
'''*'Compression is measured with 7-Zip''
 
 
•    Start File Open Dialogue: 1.25
 
  
•    Start Print dialogue: 4
+
Other VSI 4.1 related measurements helping determine environmental issues (resources).
  
•    Zip PST file without compression: 6
+
'''Note''': The measurements in the table below are not used for the calculation of VSImax v4.1.
 
 
•    Zip PST file with high compression: 0.175
 
 
 
•    Start Word with new document from document pool: 0.15
 
 
 
 
 
This weighting is applied on the baseline and normal Login VSI response times.
 
 
 
===VSImax Baseline===
 
The average VSImax baseline response time (also called VSIbase) in Login VSI 4.0 is calculated from the results recorded in the baseline phase. In total 30 VSI response time samples are takes by 5 baseline sessions. To ensure the VSIbase represents the optimal performance of the system, the highest 15 results are removed from the average calculation. To ensure no fluke low measurements are affecting the results unrealistically, the bottom 2 results are removed from the average VSIbase calculation. Over the remaining 13 VSI response time measurements the average VSImax baseline response time VSIbase is calculated. So in short:
 
 
 
*Take 30 results collected during the Baseline phase
 
*Remove the top 15 results
 
*Remove the lowest 2 results
 
*Average the remaining results
 
 
 
If the baseline fase is not enabled the calculation is as follows:
 
*Sort the VSImax v4 data (found in analyzer) on Session Count
 
*Select the first 200 results of the VSIindex_Calculation
 
*Sort the results lowest to highest
 
*Select the lowest 30 Results
 
*Average the remaining results
 
 
 
====VSImax average====
 
 
 
The VSImax average response time in Login VSI 4.0 is calculated on the amount of active users that are logged on the system.
 
 
 
There are always 5 Login VSI response time samples averaged and 40% of the amount of “active” sessions is added. For example, if the active sessions is 60, then latest 5 + 24 (=40% of 60) = 29 response time measurement are used for the average calculation.
 
 
 
To remove noise (accidental spikes) from the calculation, the top 5% and bottom 5% of the VSI response time samples are removed from the average calculation, with a minimum of 1 top and 1 bottom sample. As a result, with 60 active users, the last 29 VSI response time sample are taken. From those 29 samples the top 2 samples are removed and lowest 2 results are removed (5% of 29 = 1.45, rounded to 1). At 60 users the average is then calculated over the 28 remaining results.
 
 
 
====Reaching VSImax====
 
 
 
VSImax v4 is reached when the VSIbase + a 2600 ms latency threshold is reached by the average VSI response time result. Depending on the tested system, VSImax response time can grow 2 - 3x the baseline average. In end-user computing, a 3x increase in response time in comparison to the baseline is typically regarded as the maximum performance degradation to be considered acceptable.
 
 
 
Note: In VSImax Dynamic the latency threshold was dependent on the baseline measurement at the beginning of the test. 25% of the baseline measurement was added to a latency threshold. While this was not a problem with most tests, in some cases, especially when the systems performance was very close between two different configurations, the slower system might get a higher VSImax score simply because the higher baseline results gave the slower system a higher latency threshold.
 
 
 
In VSImax v4 this latency threshold is fixed to 2600ms, this allows better and fairer comparisons between two different systems, especially when they have different baseline results. Ultimately, in VSImax v4, the performance of the system is not decided by the total average response time, but by the latency is has under load. For all systems, this is now 2600ms (weighted).
 
 
 
The *threshold* for the total response time is: average weighted baseline phase response time + 2600ms.
 
 
 
When the system has a weighted baseline response time average of 1500ms, the maximum average response time may not be greater than 4100ms (1500+2600). If the average baseline is 3000 the maximum average response time may not be greater than 5600ms (3000+2600).
 
 
 
VSImax v4 is determined before the system has exceeded it threshold. For example, when the VSI max average on system has exceeded the VSI threshold at 80 users, the VSImax will be 79.
 
 
 
When the threshold is not exceeded by the average VSI response time during the test, VSImax is now considered the maximum amount of users that was launched. This approach is fundamentally different in comparison to previous VSImax methods, as is it was always required to saturate the system beyond VSImax threshold.
 
 
 
Lastly, VSImax v4 is now always reported with the average baseline VSI response time result. For example: “The VSImax v4 was 125 with a baseline of 1526ms”. This helps considerably in the comparison of systems and gives a more complete understanding of the system. The baseline performance helps to understand the best performance the system can give to an individual user. VSImax indicates what the total user capacity is for the system. These two are not automatically connected and related:
 
 
 
When a server with a very fast dual core CPU, running at 3.6 GHZ, is compared to a 10 core CPU, running at 2,26 GHZ, the dual core machine will give and individual user better performance than the 10 core machine. This is indicated by the baseline VSI response time. The lower this score is, the better performance an individual user can expect.
 
 
 
However, the server with the slower 10 core CPU will easily have a larger capacity than the faster dual core system. This is indicated by VSImax v4, and the higher VSImax is, the larger overall user capacity can be expected.
 
 
 
With Login VSI 3.6 it is was possible to choose between ‘VSImax Classic’ and 'VSImax Dynamic’.  With Login VSI 4.0 a new VSImax method is introduced: VSImax v4. This methodology gives much better insight in system performance and scales to extremely large systems. ‘VSImax Classic’ and 'VSImax Dynamic’ are not suitable for large systems.
 
 
 
===VSImax Classic===
 
 
 
VSImax Classic is based on the previous versions of VSI, and is achieved when the average VSI'' ''response time is higher than a fixed threshold of 4000ms. This method proves to be reliable when no anti-virus or application virtualization is used.
 
 
 
In practice however, tests have shown a substantial increase of application response time when antivirus and/or application virtualization is used. The baseline response time is typically around 1400 - 1800 ms without application virtualization or antivirus. However, when anti-virus or application virtualization is used, the baseline response time varies between 2500 – 3500 ms.
 
 
 
When the baseline response time is already so high the VSImax'' ''Classic'' ''threshold of 4000ms is too easily reached. ‘VSImax Classic’ will report a maximum long before system resources like CPU, RAM or disk are actually saturated.
 
 
 
It was therefore decided to further optimize VSImax calculation.
 
 
 
===VSImax Dynamic===
 
 
 
Similar to ‘VSImax Classic’, VSImax Dynamic is calculated when the response times are consistently above a certain threshold. However, this threshold is now dynamically calculated on the baseline response time of the test. VSImax Dynamic was introduced in VSI 3.0.
 
 
 
For this the average VSImax response times need to consistently higher than a dynamically calculated threshold. To determine this dynamic threshold, first the average baseline response time is calculated. This is done by averaging the baseline response time of the first 15 VSI users on the system.
 
 
 
The formula for the dynamic threshold is:  Avg. Baseline Response Time x 125% + 3000. As a result, when the baseline response time is 1800, the VSImax threshold will now be 1800 x 125% + 3000 = 5250ms.
 
 
 
Especially when application virtualization is used, the baseline response time can wildly vary per vendor and streaming strategy.
 
 
 
===Baseline update patch 4.0.7===
 
 
 
The Login VSI analyzer used to plot an imaginary line at the start of a test which interferes with the baseline when the Basephase is not used. We have noticed the line can be confusing when analyzing the results and to eliminate any confusing we have decided to remove the incorrect calculation. When the Basephase is not used there will be 6% increase in millisecond on the Baseline. Because the baseline will increase this will have an effect on the VSImax. To be precise the VSImax will increase with 1%. 
 
The following screenshot shows the imaginary line from the previous analyzer. 
 
[[File:Imaginary line.png|400px|thumb|center]]
 
 
 
 
 
 
 
 
 
===VSImax 4.x Metrics===
 
 
 
VSImax v4 is determined by the measurements in the below table.
 
  
 
{| class="wikitable"
 
{| class="wikitable"
Line 227: Line 185:
 
|-
 
|-
 
|
 
|
'''WSLD'''
+
'''NFP'''
  
 
|
 
|
Start Microsoft Word and Load a random document
+
Press print open in VSI-Notepad
  
 
|
 
|
Word Start/Load a local random document file from content pool
+
VSI-Notepad print open [Ctrl+P]
 
|
 
|
CPU, RAM and IO
+
CPU
 
|-
 
|-
 
|
 
|
'''NSLD'''
+
'''UMEM'''
  
 
|
 
|
Start VSI-Notepad and Load a document
+
The percentage of memory used by the sessions
 
|
 
|
VSI-Notepad Start/Load a local random text file from content pool
+
The average percentage of memory used by the active sessions
 
|
 
|
CPU and IO
+
RAM
 
|-
 
|-
 
|
 
|
'''WFO'''
+
'''I/O'''
  
 
|
 
|
Press file open in VSI-Notepad
+
Write the CPU random data file to disk
 
|
 
|
VSI-Notepad file open [ctrl+o]
+
The file created by the CPU random data file is written to the local disk
 
|
 
|
CPU, RAM and IO
+
I/O
 +
|}
 +
 
 +
Other VSI related measurements helping determine environmental issues (resources).
 +
 
 +
'''Note''': The measurements in the table below are not used for the calculation of VSImax v4.1.
 +
 
 +
{| class="wikitable"
 
|-
 
|-
 
|
 
|
'''NFP'''
+
'''Measurement ID'''
  
 
|
 
|
Press print open in VSI-Notepad
+
'''Measurement Action'''
 +
 
 
|
 
|
VSI-Notepad print open [ctrl+p]
+
'''Measurement Action Detailed'''
 +
 
 
|
 
|
CPU
+
'''Measures Related Resource'''
 
|-
 
|-
 
|
 
|
'''ZHC*'''
+
'''FCDL'''
  
 
|
 
|
Compress files with high compression
+
File copy document local
 +
 
 
|
 
|
Compress a local random .pst file from content pool (5mb)
+
Copy a local random document from temp directory to home drive
 
|
 
|
CPU
+
I/O
 
|-
 
|-
 
|
 
|
'''ZNC*'''
+
'''FCTL'''
  
 
|
 
|
Compress files with no compression
+
File copy text local
 
|
 
|
Compress a local random .pst file from content pool (5mb)
+
Copy a local random text file from local temp directory to home drive
 
|
 
|
IO
+
I/O
 
|}
 
|}
'''*'compression is measured with 7zip''
 
  
==Other resources==
+
Other VSI related measurements helping determine environmental issues (resources).
 +
 
 +
'''Note''': The measurements in the table below are not used for the calculation of VSImax v4.1.
  
[https://www.loginvsi.com/blog/481-calculating-maximum-virtual-desktop-capacity-vsimax-explained Blog - Calculating Maximum Virtual Desktop Capacity – VSImax Explained]
+
{| class="wikitable"
 +
|-
 +
|
 +
'''Measurement ID'''
 +
 
 +
|
 +
'''Measurement Action'''
 +
 
 +
|
 +
'''Measurement Action Detailed'''
 +
 
 +
|
 +
'''Measures Related Resource'''
 +
|-
 +
|
 +
'''FCTS'''
 +
 
 +
|
 +
File copy text share
 +
|
 +
Copy a text document from VSIshare to temp directory
 +
|
 +
I/O and Network
 +
|}
  
 
[[Category:Login VSI]]
 
[[Category:Login VSI]]

Latest revision as of 14:36, 3 May 2018

The logic behind Login VSI is different to conventional benchmarks. In general, most conventional benchmarks are steady state benchmarks. These execute one or multiple processes, and the measured execution time is the outcome of the test. Simply put: the faster the execution time or the bigger the throughput, the faster the system is.

Login VSI takes a different approach. Login VSI is not primarily designed to be a steady state benchmark (however, if needed, Login VSI can act like one). Login VSI was designed to perform benchmarks for SBC or VDI workloads through system saturation. Login VSI loads the system with simulated user workloads using well known desktop applications like Microsoft Office, Internet Explorer and Adobe PDF Reader. By gradually increasing the amount of simulated users, the system will eventually be saturated. Once the system is saturated, the response time of the applications will increase significantly. This latency in application response times gives a clear indication whether the system is (close to being) overloaded. As a result, by nearly overloading a system it is possible to find out what is the true maximum user capacity.

After a test is performed, the response times can be analyzed to calculate the maximum active session/desktop capacity. Within Login VSI, this is calculated as VSImax. When the system is getting closer to its saturation point, response times will rise. When reviewing the average response time it will be clear that the response times escalate at the saturation point.

This VSImax is the “Virtual Session Index (VSI)”. With Virtual Desktop Infrastructure (VDI) and Terminal Services (RDS) workloads this is valuable information. This index simplifies comparisons and makes it possible to understand the true impact of configuration changes on a hypervisor host or at guest level.

Recommended: Calculating Maximum Virtual Desktop Capacity – VSImax Explained

Server side response time measurements

It is important to understand why specific Login VSI design choices have been made. An important design choice is to execute the workload directly on the target system within the session instead of using remote sessions. The scripts simulating the workloads are performed by an engine that executes workload scripts on every target system, and are initiated at logon within the simulated user’s desktop session context.

An alternative to the Login VSI method would be to generate user actions on the client side via a remoting protocol. These methods are always specific to a product and rely on its vendors. More importantly, some protocols simply do not have a method to script user actions on the client side.

For Login VSI, the choice has been made to execute the scripts on the server side. This is the only practical and platform independent solution for a benchmark, like Login VSI. The relative overhead and footprint of a benchmark engine scripted in AutoIt is small enough (1-5% range) for Login VSI’s purposes.

Calculating VSImax v4.1.x

The simulated desktop workload is scripted in a 48-minute loop where a simulated Login VSI user is logged on and performing generic office worker activities. After the loop is finished, it will restart automatically. Within each loop the response times of ten specific operations are measured at a regular interval (16 times within each loop), five of which are the most important. The response times of these important operations are used to determine VSImax.

The five important operations are:

  • Notepad File Open (NFO): Loading and initiating VSINotepad.exe and opening the openfile dialog. This operation is handled by the OS and by the executable itself through execution (this operation seems almost instant from an end-user point of view).
  • Notepad Start Load (NSLD): Loading and initiating VSINotepad.exe and opening a file. This operation is also handled by the OS and by the executable itself through execution (this operation seems almost instant from an end-user point of view).
  • Zip High Compression (ZHC): This action copies a random file and compresses it (with 7-Zip) with high compression. The compression will very briefly spike both CPU and disk I/O.
  • Zip Low Compression (ZLC): This action copies a random file and compresses it (with 7-Zip) with low compression. The compression will very briefly spike disk I/O while also creating some load on the CPU as well.
  • CPU: Calculates a large array of random data and spikes the CPU for a short period of time.

Note: NFP (Notepad File Print) is no longer used to calculate VSImax due to the amount of printers configured in an environment. The amount of printers could influence this number. If you have 15 printers, it takes much longer to open the dialog compared to when you only have one printer configured.

These measured operations affect many different sub-systems, such as CPU (user and kernel), Memory, Disk, the OS (in general), the application itself, print, GDI, etc. These operations are specifically short by nature. When such operations become consistently long, the system becomes saturated because of excessive queuing on any kind of resource. As a result, the average response times will then escalate, which is clearly visible to end-users. If such operations consistently take longer than expected, the user will regard the system as slow and unresponsive.

VSImax Metrics

Once the test is finished, VSImax v4.1 will be calculated. Previous VSImax models (Classic and Dynamic) needed Microsoft Word to function. With the new 4.1 timers this is no longer needed, making it more flexible and applicable to a larger amount of scenarios.

The following actions are part of the calculation and are weighted as follows (US notation):

  • Notepad File Open (NFO): 0.75
  • Notepad Start Load (NSLD): 0.2
  • Zip High Compression (ZHC): 0.125
  • Zip Low Compression (ZLC): 0.2
  • CPU: 0.75

This weighting is applied on the baseline and normal Login VSI response times.

VSImax Baseline

With the introduction of Login VSI 4.1, we also created a new method to calculate the basephase of an environment. With the new workloads (Taskworker, Powerworker, etc.) enabling basephase for a more reliable baseline has become obsolete. The calculation is explained below.

In total, 15 lowest VSI response time samples are taken from the entire test, the lowest 2 samples are removed and the 13 remaining samples are averaged. The result is the Baseline.

In short:

  • Take the lowest 15 samples of the test
  • Remove the lowest 2 from those 15 samples
  • Average these 13 samples to calculate the Baseline number

VSImax Average

The VSImax average response time in Login VSI 4.1.x is calculated on the amount of active users that are logged on to the system. This calculation always uses the average of five Login VSI response time samples + 40% of the amount of active sessions.

For example, if the active sessions = 60, then latest 5 + 24 (40% of 60) = 29 response time measurements are used for the average calculation.

To remove noise (accidental spikes) from the calculation, the top 5% and bottom 5% of the VSI response time samples are removed from the average calculation, with a minimum of one top and one bottom sample. As a result, with 60 active users, the last 29 VSI response time sample are taken. From those 29 samples, the top two samples are removed and lowest two results are removed (5% of 29 = 1.45, rounded to 2). At 60 users, the average is then calculated over the 27 remaining results.

VSImax Threshold

VSImax v4.1.x is reached when the VSIbase + a 1000ms latency threshold is not reached by the average VSI response time result. Depending on the tested system, VSImax response times can grow 2-3x the baseline average. In end-user computing, a 3x increase in response time compared to the baseline is typically regarded as the maximum performance degradation to be considered acceptable.

In VSImax v4.1.x, this latency threshold is fixed to 1000ms, which allows better and fairer comparisons between two different systems, especially when they have different baseline results. Ultimately, in VSImax v4.1.x, the performance of the system is not decided by the total average response time, but by the latency is has under load. For all systems, this is now 1000ms (weighted).

The threshold for the total response time = average weighted baseline response time + 1000ms.

When the system has a weighted baseline response time average of 1500ms, the maximum average response time may not be greater than 2500ms (1500 + 1000). If the average baseline is 3000, the maximum average response time may not be greater than 4000ms (3000 + 1000).

When the threshold is not exceeded by the average VSI response time during the test, VSImax is not hit and the amount of sessions ran successfully. This approach is fundamentally different in comparison to previous VSImax methods, as it was always required to saturate the system beyond VSImax threshold.

Additonal Info

VSImax v4.1.x is now always reported with the average baseline VSI response time result. For example: The VSImax was 125 with a baseline of 1526ms. This helps considerably in the comparison of systems and gives a more complete understanding of the system. The baseline performance helps to understand the best performance the system can give to an individual user. VSImax indicates what the total user capacity is for the system. These two are not automatically connected or related.

For example, when a server with a very fast dual core CPU, running at 3.6 GHz, is compared to a 10 core CPU, running at 2.26 GHz, the dual core machine will give an individual user better performance than the 10 core machine. This is indicated by the baseline VSI response time. The lower this score is, the better performance an individual user can expect.

However, the server with the slower 10 core CPU will easily have a larger capacity than the faster dual core system. This is indicated by VSImax v4.1.x, and the higher the VSImax is, the larger overall user capacity can be expected.

With Login VSI 4.1.x a new VSImax method is introduced: VSImax v4.1. This methodology gives much better insight into system performance and scales to extremely large systems.

Login VSI Timer

The VSI_Timer41 function measures the responsiveness of the system. It looks at the response of the system at multiple points within each workload. Within the Login VSI 4.1 timer, the timer mechanism has been made more reliable by removing erratic timers and adding more comprehensive timers.

For example: If we were to start an application, we look at what kind of performance we can expect at that moment in time, e.g. how long does it take to start the application or open the document.

Timer Metrics

VSImax v4.1 shows the amount of sessions that can be active on a system before the system is saturated. This number gives you an indication of the scalability of the environment (higher is better). VSImax v4.1 is determined by the measurements in the below table:

Measurement ID

Measurement Action

Measurement Action Detailed

Measures Related Resource

NSLD

Notepad starts and loads a 1500KB document

Notepad starts and loads a random local 1500KB document file that is copied from the content pool

CPU and I/O

NFO

Measure how long it takes to show the file-open dialog in Notepad

VSI-Notepad file open [Ctrl+O]

CPU, RAM, and I/O

ZHC*

Create a Zip file with high compression

Compress a local random .pst file that is copied from the content pool (5MB)

CPU and I/O

ZLC*

Create a Zip file with low compression

Compress a local random .pst file that is copied from the content pool (5MB)

I/O

CPU

Calculates a large array of random data

Creates a large array of random data that will be used in the I/O timer

CPU

'*'Compression is measured with 7-Zip

Other VSI 4.1 related measurements helping determine environmental issues (resources).

Note: The measurements in the table below are not used for the calculation of VSImax v4.1.

Measurement ID

Measurement Action

Measurement Action Detailed

Measures Related Resource

NFP

Press print open in VSI-Notepad

VSI-Notepad print open [Ctrl+P]

CPU

UMEM

The percentage of memory used by the sessions

The average percentage of memory used by the active sessions

RAM

I/O

Write the CPU random data file to disk

The file created by the CPU random data file is written to the local disk

I/O

Other VSI related measurements helping determine environmental issues (resources).

Note: The measurements in the table below are not used for the calculation of VSImax v4.1.

Measurement ID

Measurement Action

Measurement Action Detailed

Measures Related Resource

FCDL

File copy document local

Copy a local random document from temp directory to home drive

I/O

FCTL

File copy text local

Copy a local random text file from local temp directory to home drive

I/O

Other VSI related measurements helping determine environmental issues (resources).

Note: The measurements in the table below are not used for the calculation of VSImax v4.1.

Measurement ID

Measurement Action

Measurement Action Detailed

Measures Related Resource

FCTS

File copy text share

Copy a text document from VSIshare to temp directory

I/O and Network