Difference between revisions of "Login VSI VSImax"

From Login VSI Documentation
Jump to: navigation, search
(Timer Metrics)
 
(22 intermediate revisions by the same user not shown)
Line 1: Line 1:
The logic behind Login VSI is different to conventional benchmarks. In general, most system benchmarks are steady state benchmarks. These benchmarks execute one or multiple processes, and the measured execution time is the outcome of the test. Simply put: the faster the execution time or the bigger the throughput, the faster the system is according to the benchmark.  
+
The logic behind Login VSI is different to conventional benchmarks. In general, most conventional benchmarks are steady state benchmarks. These execute one or multiple processes, and the measured execution time is the outcome of the test. Simply put: the faster the execution time or the bigger the throughput, the faster the system is.  
  
Login VSI is different in approach. Login VSI is not primarily designed to be a steady state benchmark (however, if needed, Login VSI can act like one). Login VSI was designed to perform benchmarks for SBC or VDI workloads through system saturation. Login VSI loads the system with simulated user workloads using well known desktop applications like Microsoft Office, Internet Explorer and Adobe PDF reader. By gradually increasing the amount of simulated users, the system will eventually be saturated. Once the system is saturated, the response time of the applications will increase significantly. This latency in application response times gives a clear indication whether the system is (close to being) overloaded. As a result, by nearly overloading a system it is possible to find out what is the true maximum user capacity.
+
Login VSI takes a different approach. Login VSI is not primarily designed to be a steady state benchmark (however, if needed, Login VSI can act like one). Login VSI was designed to perform benchmarks for SBC or VDI workloads through system saturation. Login VSI loads the system with simulated user workloads using well known desktop applications like Microsoft Office, Internet Explorer and Adobe PDF Reader. By gradually increasing the amount of simulated users, the system will eventually be saturated. Once the system is saturated, the response time of the applications will increase significantly. This latency in application response times gives a clear indication whether the system is (close to being) overloaded. As a result, by nearly overloading a system it is possible to find out what is the true maximum user capacity.
  
After a test is performed, the response times can be analyzed to calculate the maximum active session/desktop capacity. Within Login VSI this is calculated as VSImax. When the system is coming closer to its saturation point, response times will rise. When reviewing the average response time it will be clear that the response times escalate at the saturation point.
+
After a test is performed, the response times can be analyzed to calculate the maximum active session/desktop capacity. Within Login VSI, this is calculated as VSImax. When the system is getting closer to its saturation point, response times will rise. When reviewing the average response time it will be clear that the response times escalate at the saturation point.
  
This VSImax is the “Virtual Session Index (VSI)”. With Virtual Desktop Infrastructure (VDI) and Terminal Services (RDS) workloads this is valid and useful information. This index simplifies comparisons and makes it possible to understand the true impact of configuration changes on hypervisor host or guest level.
+
This VSImax is the “Virtual Session Index (VSI)”. With Virtual Desktop Infrastructure (VDI) and Terminal Services (RDS) workloads this is valuable information. This index simplifies comparisons and makes it possible to understand the true impact of configuration changes on a hypervisor host or at guest level.
  
 
'''Recommended:''' [https://www.loginvsi.com/blog/481-calculating-maximum-virtual-desktop-capacity-vsimax-explained Calculating Maximum Virtual Desktop Capacity – VSImax Explained]
 
'''Recommended:''' [https://www.loginvsi.com/blog/481-calculating-maximum-virtual-desktop-capacity-vsimax-explained Calculating Maximum Virtual Desktop Capacity – VSImax Explained]
Line 11: Line 11:
 
==Server side response time measurements==
 
==Server side response time measurements==
  
It is important to understand why specific Login VSI design choices have been made. An important design choice is to execute the workload directly on the target system within the session instead of using remote sessions. The scripts simulating the workloads are performed by an engine that executes workload scripts on every target system, and are initiated at logon within the simulated user’s desktop session context.  
+
It is important to understand why specific Login VSI design choices have been made. An important design choice is to execute the workload directly on the target system within the session instead of using remote sessions. The scripts simulating the workloads are performed by an engine that executes workload scripts on every target system, and are initiated at logon within the simulated user’s desktop session context.
  
An alternative to the Login VSI method would be to generate user actions on the client side via the remoting protocol. These methods are always specific to a product and vendor dependent. More importantly, some protocols simply do not have a method to script user actions client side.  
+
An alternative to the Login VSI method would be to generate user actions on the client side via a remoting protocol. These methods are always specific to a product and rely on its vendors. More importantly, some protocols simply do not have a method to script user actions on the client side.
  
For Login VSI, the choice has been made to execute the scripts on the server side. This is the only practical and platform independent solution for a benchmark, like Login VSI. The relative overhead and footprint of a benchmark engine scripted in Auto-IT is small enough (1-5% range) for Login VSI’s purposes.
+
For Login VSI, the choice has been made to execute the scripts on the server side. This is the only practical and platform independent solution for a benchmark, like Login VSI. The relative overhead and footprint of a benchmark engine scripted in AutoIt is small enough (1-5% range) for Login VSI’s purposes.
  
 
==Calculating VSImax v4.1.x==
 
==Calculating VSImax v4.1.x==
  
The simulated desktop workload is scripted in a 48-minute loop when a simulated Login VSI user is logged on, performing generic office worker activities. After the loop is finished, it will restart automatically. Within each loop the response times of ten specific operations are measured in a regular interval, five of which are the most important. These measurements happen sixteen times within each loop. The response times of these five important operations are used to determine VSImax.
+
The simulated desktop workload is scripted in a 48-minute loop where a simulated Login VSI user is logged on and performing generic office worker activities. After the loop is finished, it will restart automatically. Within each loop the response times of ten specific operations are measured at a regular interval (16 times within each loop), five of which are the most important. The response times of these important operations are used to determine VSImax.
  
The five operations from which the response times are measured are:
+
The five important operations are:
  
'''Notepad File Open (NFO)'''
+
*'''Notepad File Open (NFO)''': Loading and initiating VSINotepad.exe and opening the openfile dialog. This operation is handled by the OS and by the executable itself through execution (this operation seems almost instant from an end-user point of view).
  
Loading and initiating VSINotepad.exe and opening the openfile dialog. This operation is handled by the OS and by the VSINotepad.exe itself through execution. This operation seems almost instant from an end-user point of view.
+
*'''Notepad Start Load (NSLD)''': Loading and initiating VSINotepad.exe and opening a file. This operation is also handled by the OS and by the executable itself through execution (this operation seems almost instant from an end-user point of view).
  
'''Notepad Start Load (NSLD)'''
+
*'''Zip High Compression (ZHC)''': This action copies a random file and compresses it (with 7-Zip) with high compression. The compression will very briefly spike both CPU and disk I/O.
  
Loading and initiating VSINotepad.exe and opening a file. This operation is also handled by the OS and by the VSINotepad.exe itself through execution. This operation seems almost instant from an end-user point of view.
+
*'''Zip Low Compression (ZLC)''': This action copies a random file and compresses it (with 7-Zip) with low compression. The compression will very briefly spike disk I/O while also creating some load on the CPU as well.
  
'''Zip High Compression (ZHC)'''
+
*'''CPU''': Calculates a large array of random data and spikes the CPU for a short period of time.
  
This action copies a random file and compresses it (with 7-Zip) with high compression enabled. The compression will very briefly spike both CPU and disk I/O.  
+
'''Note''': NFP (Notepad File Print) is no longer used to calculate VSImax due to the amount of printers configured in an environment. The amount of printers could influence this number. If you have 15 printers, it takes much longer to open the dialog compared to when you only have one printer configured.
  
'''Zip Low Compression (ZLC)'''
+
These measured operations affect many different sub-systems, such as CPU (user and kernel), Memory, Disk, the OS (in general), the application itself, print, GDI, etc. These operations are specifically short by nature. When such operations become consistently long, the system becomes saturated because of excessive queuing on any kind of resource. As a result, the average response times will then escalate, which is clearly visible to end-users. If such operations consistently take longer than expected, the user will regard the system as slow and unresponsive.
  
This action copies a random file and compresses it (with 7-Zip) with low compression enabled. The compression will very briefly spike disk I/O while also creating some load on the CPU as well.
+
===VSImax Metrics===
 
+
Once the test is finished, VSImax v4.1 will be calculated. Previous VSImax models (Classic and Dynamic) needed Microsoft Word to function. With the new 4.1 timers this is no longer needed, making it more flexible and applicable to a larger amount of scenarios.  
'''CPU'''
 
 
 
Calculates a large array of random data and spikes the CPU for a short period of time.
 
 
 
'''Note''': NFP (Notepad File Print) is no longer used to calculate VSImax due to the amount of printers configured in an environment. The amount of printers could influence this number. If you have 15 printers it takes much longer to open the dialog compared to when you have 1 printer configured.
 
 
 
These measured operations within Login VSI do hit considerably different subsystems such as CPU (user and kernel), Memory, Disk, the OS in general, the application itself, print, GDI, etc. These operations are specifically short by nature. When such operations become consistently long: the system is saturated because of excessive queuing on any kind of resource. As a result, the average response times will then escalate.  This effect is clearly visible to end-users. If such operations consistently consume multiple seconds the user will regard the system as slow and unresponsive.
 
  
===VSImax Metrics===
+
The following actions are part of the calculation and are weighted as follows (US notation):
Once the test is finished, VSImax v4.1 can be calculated. Previous VSImax models (Classic and Dynamic) needed Microsoft Word to function. With the new 4.1 timers this is no longer needed, making it more flexible and applicable to a larger amount of scenarios.
 
  
The following actions are part of the VSImax v4.1 calculation and are weighted as follows (US notation):
 
 
*Notepad File Open (NFO): 0.75
 
*Notepad File Open (NFO): 0.75
 
*Notepad Start Load (NSLD): 0.2
 
*Notepad Start Load (NSLD): 0.2
Line 60: Line 51:
  
 
===VSImax Baseline===
 
===VSImax Baseline===
With the introduction of Login VSI 4.1 we also created a new method to calculate the basephase of an environment. With the new workloads (Taskworker, Powerworker, etc.) enabling 'basephase' for a more reliable baseline has become obsolete. The calculation is explained below.  
+
With the introduction of Login VSI 4.1, we also created a new method to calculate the basephase of an environment. With the new workloads (Taskworker, Powerworker, etc.) enabling ''basephase'' for a more reliable baseline has become obsolete. The calculation is explained below.
 +
 
 +
In total, 15 lowest VSI response time samples are taken from the entire test, the lowest 2 samples are removed and the 13 remaining samples are averaged. The result is the Baseline.  
  
In total 15 lowest VSI response time samples are taken from the entire test, the lowest 2 samples are removed and the 13 remaining samples are averaged. The result is the Baseline. In short:
+
In short:
  
*Take the lowest 15 samples of the complete test
+
*Take the lowest 15 samples of the test
*From those 15 samples remove the lowest 2
+
*Remove the lowest 2 from those 15 samples
*Average the 13 results that are left is the baseline
+
*Average these 13 samples to calculate the Baseline number
  
 
===VSImax Average===
 
===VSImax Average===
The VSImax average response time in Login VSI 4.1.x is calculated on the amount of active users that are logged on the system.
+
The VSImax average response time in Login VSI 4.1.x is calculated on the amount of active users that are logged on to the system. This calculation always uses the average of five Login VSI response time samples + 40% of the amount of active sessions.  
  
Always a 5 Login VSI response time samples are averaged + 40% of the amount of “active” sessions. For example, if the active sessions is 60, then latest 5 + 24 (=40% of 60) = 31 response time measurement are used for the average calculation.
+
For example, if the active sessions = 60, then latest 5 + 24 (40% of 60) = 29 response time measurements are used for the average calculation.
  
To remove noise (accidental spikes) from the calculation, the top 5% and bottom 5% of the VSI response time samples are removed from the average calculation, with a minimum of 1 top and 1 bottom sample. As a result, with 60 active users, the last 31 VSI response time sample are taken. From those 31 samples the top 2 samples are removed and lowest 2 results are removed (5% of 31 = 1.55, rounded to 2). At 60 users the average is then calculated over the 27 remaining results.
+
To remove noise (accidental spikes) from the calculation, the top 5% and bottom 5% of the VSI response time samples are removed from the average calculation, with a minimum of one top and one bottom sample. As a result, with 60 active users, the last 29 VSI response time sample are taken. From those 29 samples, the top two samples are removed and lowest two results are removed (5% of 29 = 1.45, rounded to 2). At 60 users, the average is then calculated over the 27 remaining results.
  
 
===VSImax Threshold===
 
===VSImax Threshold===
VSImax v4.1.x is reached when the VSIbase + a 1000 ms latency threshold is not reached by the average VSI response time result. Depending on the tested system, VSImax response time can grow 2 - 3x the baseline average. In end-user computing, a 3x increase in response time in comparison to the baseline is typically regarded as the maximum performance degradation to be considered acceptable.
+
VSImax v4.1.x is reached when the VSIbase + a 1000ms latency threshold is not reached by the average VSI response time result. Depending on the tested system, VSImax response times can grow 2-3x the baseline average. In end-user computing, a 3x increase in response time compared to the baseline is typically regarded as the maximum performance degradation to be considered acceptable.
  
In VSImax v4.1.x this latency threshold is fixed to 1000ms, this allows better and fairer comparisons between two different systems, especially when they have different baseline results. Ultimately, in VSImax v4.1.x, the performance of the system is not decided by the total average response time, but by the latency is has under load. For all systems, this is now 1000ms (weighted).
+
In VSImax v4.1.x, this latency threshold is fixed to 1000ms, which allows better and fairer comparisons between two different systems, especially when they have different baseline results. Ultimately, in VSImax v4.1.x, the performance of the system is not decided by the total average response time, but by the latency is has under load. For all systems, this is now 1000ms (weighted).
  
The threshold for the total response time is: average weighted baseline response time + 1000ms.
+
The threshold for the total response time = average weighted baseline response time + 1000ms.
  
When the system has a weighted baseline response time average of 1500ms, the maximum average response time may not be greater than 2500ms (1500+1000). If the average baseline is 3000 the maximum average response time may not be greater than 4000ms (3000+1000).
+
When the system has a weighted baseline response time average of 1500ms, the maximum average response time may not be greater than 2500ms (1500 + 1000). If the average baseline is 3000, the maximum average response time may not be greater than 4000ms (3000 + 1000).
  
 
When the threshold is not exceeded by the average VSI response time during the test, VSImax is not hit and the amount of sessions ran successfully. This approach is fundamentally different in comparison to previous VSImax methods, as it was always required to saturate the system beyond VSImax threshold.
 
When the threshold is not exceeded by the average VSI response time during the test, VSImax is not hit and the amount of sessions ran successfully. This approach is fundamentally different in comparison to previous VSImax methods, as it was always required to saturate the system beyond VSImax threshold.
  
 
===Additonal Info===
 
===Additonal Info===
VSImax v4.1.x is now always reported with the average baseline VSI response time result. For example: “The VSImax v4.1 was 125 with a baseline of 1526ms”. This helps considerably in the comparison of systems and gives a more complete understanding of the system. The baseline performance helps to understand the best performance the system can give to an individual user. VSImax indicates what the total user capacity is for the system. These two are not automatically connected and related:
+
VSImax v4.1.x is now always reported with the average baseline VSI response time result. For example: The VSImax was 125 with a baseline of 1526ms. This helps considerably in the comparison of systems and gives a more complete understanding of the system. The baseline performance helps to understand the best performance the system can give to an individual user. VSImax indicates what the total user capacity is for the system. These two are not automatically connected or related.
  
When a server with a very fast dual core CPU, running at 3.6 GHZ, is compared to a 10 core CPU, running at 2,26 GHZ, the dual core machine will give and individual user better performance than the 10 core machine. This is indicated by the baseline VSI response time. The lower this score is, the better performance an individual user can expect.
+
For example, when a server with a very fast dual core CPU, running at 3.6 GHz, is compared to a 10 core CPU, running at 2.26 GHz, the dual core machine will give an individual user better performance than the 10 core machine. This is indicated by the baseline VSI response time. The lower this score is, the better performance an individual user can expect.
  
However, the server with the slower 10 core CPU will easily have a larger capacity than the faster dual core system. This is indicated by VSImax v4.1.x, and the higher VSImax is, the larger overall user capacity can be expected.
+
However, the server with the slower 10 core CPU will easily have a larger capacity than the faster dual core system. This is indicated by VSImax v4.1.x, and the higher the VSImax is, the larger overall user capacity can be expected.
  
With Login VSI 4.1.x a new VSImax method is introduced: VSImax v4.1. This methodology gives much better insight in system performance and scales to extremely large systems.
+
With Login VSI 4.1.x a new VSImax method is introduced: VSImax v4.1. This methodology gives much better insight into system performance and scales to extremely large systems.
  
 
=Login VSI Timer=
 
=Login VSI Timer=
Line 99: Line 92:
  
 
'''For example:'''
 
'''For example:'''
If we were to start an application we look at what kind of performance we can expect at that moment in time, e.g. how long does it take to start the application or open the document.
+
If we were to start an application, we look at what kind of performance we can expect at that moment in time, e.g. how long does it take to start the application or open the document.
 
 
==Changes from 4.0 to 4.1==
 
 
 
'''Word Start Load Document (WSLD)'''
 
 
 
All previous versions of Login VSI used Microsoft Office Word within their timer mechanism to calculate the desktops performance. As published in Project VRC whitepaper “Phase 6: Microsoft Office Impact on VDI” the influence of Microsoft Office on the test results can be significant. Removing Microsoft Word from the timer improves the consistency of the test results across different environments, Microsoft Word will still be part of the generic workloads.
 
 
 
*Independent from Office, able to run in environments without Microsoft Office
 
*More reliable when using User Environment Managers (UEM)
 
*More reliable when using application visualization solutions
 
*More reliable when using antivirus scanners and office plugins
 
 
 
'''ZNC to ZLC'''
 
 
 
ZNC (Zip No Compression) has been changed to ZLC (Zip Low Compression). The reasons for this are:
 
*ZNC was too fast, ZNC simply added a header and a footer to the file. This made it very fast requiring a large multiplier compensate this making small changes in response time relatively intense
 
*ZNC with a virus scanner will increase the times a lot which meant that the results could not be compared to each other since ZNC and therefore was unreliable
 
*ZLC uses a single random letter as password to prevent easy caching on a file system-file, this hardly generates any CPU overhead
 
*ZLC uses a low compression: this consumes moderate CPU. ZHC is more CPU bottleneck prone then ZLC. ZLC is more IO prone the ZHC
 
*Detailed weighted ZHC and ZLC is multi-threaded so with a 1 vCPU environment (VM) ZHC and ZLC almost doubles in response times in comparison to 2 vCPU's environments
 
 
 
'''NFP'''
 
 
 
NFP (Notepad File Print) is no longer used due to the amount of printers configured in an environment. The amount of printers could influence this number. If you have 15 printers it takes much longer to open the dialog compared to when you have 1 printer configured.
 
 
 
'''ZHC vs. ZLC'''
 
 
 
ZHC is more CPU prone and ZLC is more IO prone. The two graphs will most likely look the same but depending on which is higher and the difference between ZHC and ZLC you can see where the bottleneck of an environment lies, either IO or CPU.
 
 
 
'''CPU'''
 
 
 
CPU is a new introduced VSImax metric. CPU has proven to be very stable and resilient against applications and outside interactions. This metric shows us the performance of the CPU. CPU creates an array of data and is a purely compute process.
 
 
 
'''IO'''
 
 
 
IO has been added to Login VSI but not to the VSImax calculation. This graph will indicate when writes or throughput are a bottleneck. It is added to the analyzer because it can give you a good indication if IO is the bottle neck, it will not give you the exact reason but an indication. It is recommended that external performance data is used to confirm suspicions.
 
 
 
*Writes multiple random files in different sizes to the storage.
 
 
 
'''UMEM'''
 
 
 
UMEM (Used Memory) is also added in 4.1 this shows the amount of available memory (in %) in a session (VM). This is interesting when the session is running out of memory and starts paging. This is recognized by decrease in IO performance. This graph will fluctuate naturally because segment 1 of the workload has little to no applications open and segment 4 has lots of applications open.
 
 
 
'''NSLD'''
 
 
 
NSLD (Notepad Start Load Document) is most of the time very quick and not IO intensive because most files opened by Notepad are cached by the environment or windows itself.
 
  
 
==Timer Metrics==
 
==Timer Metrics==
  
VSImax v4.1 shows the amount of sessions that can be active on a system before the system is saturated. The blue X shows the point where VSImax was reached. This number gives you an indication of the scalability of the environment (higher is better). VSImax v4.1 is determined by the measurements in the below table:
+
VSImax v4.1 shows the amount of sessions that can be active on a system before the system is saturated. This number gives you an indication of the scalability of the environment (higher is better). VSImax v4.1 is determined by the measurements in the below table:
  
 
{| class="wikitable"
 
{| class="wikitable"
Line 169: Line 116:
  
 
|
 
|
Notepad starts and loads a 1500kb document
+
Notepad starts and loads a 1500KB document
  
 
|
 
|
Notepad starts and loads a random local 1500kb document file that is copied from the content pool
+
Notepad starts and loads a random local 1500KB document file that is copied from the content pool
 
|
 
|
CPU and IO
+
CPU and I/O
 
|-
 
|-
 
|
 
|
Line 180: Line 127:
  
 
|
 
|
Measure how long it takes to show the file-open dialog in VSI notepad
+
Measure how long it takes to show the file-open dialog in Notepad
 
|
 
|
VSI-Notepad file open [ctrl+o]
+
VSI-Notepad file open [Ctrl+O]
 
|
 
|
CPU RAM, and IO
+
CPU, RAM, and I/O
 
|-
 
|-
 
|
 
|
Line 190: Line 137:
  
 
|
 
|
Create a zipfile with high compression
+
Create a Zip file with high compression
 
|
 
|
Compress a local random .pst file that is copied from the content pool (5mb)
+
Compress a local random .pst file that is copied from the content pool (5MB)
 
|
 
|
CPU and IO
+
CPU and I/O
 
|-
 
|-
 
|
 
|
Line 200: Line 147:
  
 
|
 
|
Create a zipfile with low compression
+
Create a Zip file with low compression
 
|
 
|
Compress a local random .pst file that is copied from the content pool (5mb)
+
Compress a local random .pst file that is copied from the content pool (5MB)
 
|
 
|
IO
+
I/O
 
|-
 
|-
 
|
 
|
Line 212: Line 159:
 
Calculates a large array of random data
 
Calculates a large array of random data
 
|
 
|
Creates a large array of random data that will be used in the IO timer
+
Creates a large array of random data that will be used in the I/O timer
 
|
 
|
 
CPU
 
CPU
 
|-
 
|-
 
|}
 
|}
'''*'compression is measured with 7zip''
+
'''*'Compression is measured with 7-Zip''
  
 +
Other VSI 4.1 related measurements helping determine environmental issues (resources).
  
Other VSI 4.1 related measurements helping determine environmental issues (resources). Note, *the measurements in the table below are not used for the calculation of VSImax v4.1.
+
'''Note''': The measurements in the table below are not used for the calculation of VSImax v4.1.
  
 
{| class="wikitable"
 
{| class="wikitable"
Line 243: Line 191:
  
 
|
 
|
VSI-Notepad print open [ctrl+p]
+
VSI-Notepad print open [Ctrl+P]
 
|
 
|
 
CPU
 
CPU
Line 258: Line 206:
 
|-
 
|-
 
|
 
|
'''IO'''
+
'''I/O'''
  
 
|
 
|
Line 265: Line 213:
 
The file created by the CPU random data file is written to the local disk
 
The file created by the CPU random data file is written to the local disk
 
|
 
|
IO
+
I/O
 
|}
 
|}
  
Other VSI related measurements helping determine environmental issues (resources). Note, *the measurements in the table below are not used for the calculation of VSImax v4.1.
+
Other VSI related measurements helping determine environmental issues (resources).  
 +
 
 +
'''Note''': The measurements in the table below are not used for the calculation of VSImax v4.1.
  
 
{| class="wikitable"
 
{| class="wikitable"
Line 293: Line 243:
 
Copy a local random document from temp directory to home drive
 
Copy a local random document from temp directory to home drive
 
|
 
|
IO
+
I/O
 
|-
 
|-
 
|
 
|
Line 303: Line 253:
 
Copy a local random text file from local temp directory to home drive
 
Copy a local random text file from local temp directory to home drive
 
|
 
|
IO
+
I/O
 
|}
 
|}
  
 +
Other VSI related measurements helping determine environmental issues (resources).
  
Other VSI related measurements helping determine environmental issues (resources). Note, *the measurements in the table below are not used for the calculation of VSImax v4.1.
+
'''Note''': The measurements in the table below are not used for the calculation of VSImax v4.1.
  
 
{| class="wikitable"
 
{| class="wikitable"
Line 331: Line 282:
 
Copy a text document from VSIshare to temp directory
 
Copy a text document from VSIshare to temp directory
 
|
 
|
IO and Network
+
I/O and Network
 
|}
 
|}
  
 
[[Category:Login VSI]]
 
[[Category:Login VSI]]

Latest revision as of 14:36, 3 May 2018

The logic behind Login VSI is different to conventional benchmarks. In general, most conventional benchmarks are steady state benchmarks. These execute one or multiple processes, and the measured execution time is the outcome of the test. Simply put: the faster the execution time or the bigger the throughput, the faster the system is.

Login VSI takes a different approach. Login VSI is not primarily designed to be a steady state benchmark (however, if needed, Login VSI can act like one). Login VSI was designed to perform benchmarks for SBC or VDI workloads through system saturation. Login VSI loads the system with simulated user workloads using well known desktop applications like Microsoft Office, Internet Explorer and Adobe PDF Reader. By gradually increasing the amount of simulated users, the system will eventually be saturated. Once the system is saturated, the response time of the applications will increase significantly. This latency in application response times gives a clear indication whether the system is (close to being) overloaded. As a result, by nearly overloading a system it is possible to find out what is the true maximum user capacity.

After a test is performed, the response times can be analyzed to calculate the maximum active session/desktop capacity. Within Login VSI, this is calculated as VSImax. When the system is getting closer to its saturation point, response times will rise. When reviewing the average response time it will be clear that the response times escalate at the saturation point.

This VSImax is the “Virtual Session Index (VSI)”. With Virtual Desktop Infrastructure (VDI) and Terminal Services (RDS) workloads this is valuable information. This index simplifies comparisons and makes it possible to understand the true impact of configuration changes on a hypervisor host or at guest level.

Recommended: Calculating Maximum Virtual Desktop Capacity – VSImax Explained

Server side response time measurements

It is important to understand why specific Login VSI design choices have been made. An important design choice is to execute the workload directly on the target system within the session instead of using remote sessions. The scripts simulating the workloads are performed by an engine that executes workload scripts on every target system, and are initiated at logon within the simulated user’s desktop session context.

An alternative to the Login VSI method would be to generate user actions on the client side via a remoting protocol. These methods are always specific to a product and rely on its vendors. More importantly, some protocols simply do not have a method to script user actions on the client side.

For Login VSI, the choice has been made to execute the scripts on the server side. This is the only practical and platform independent solution for a benchmark, like Login VSI. The relative overhead and footprint of a benchmark engine scripted in AutoIt is small enough (1-5% range) for Login VSI’s purposes.

Calculating VSImax v4.1.x

The simulated desktop workload is scripted in a 48-minute loop where a simulated Login VSI user is logged on and performing generic office worker activities. After the loop is finished, it will restart automatically. Within each loop the response times of ten specific operations are measured at a regular interval (16 times within each loop), five of which are the most important. The response times of these important operations are used to determine VSImax.

The five important operations are:

  • Notepad File Open (NFO): Loading and initiating VSINotepad.exe and opening the openfile dialog. This operation is handled by the OS and by the executable itself through execution (this operation seems almost instant from an end-user point of view).
  • Notepad Start Load (NSLD): Loading and initiating VSINotepad.exe and opening a file. This operation is also handled by the OS and by the executable itself through execution (this operation seems almost instant from an end-user point of view).
  • Zip High Compression (ZHC): This action copies a random file and compresses it (with 7-Zip) with high compression. The compression will very briefly spike both CPU and disk I/O.
  • Zip Low Compression (ZLC): This action copies a random file and compresses it (with 7-Zip) with low compression. The compression will very briefly spike disk I/O while also creating some load on the CPU as well.
  • CPU: Calculates a large array of random data and spikes the CPU for a short period of time.

Note: NFP (Notepad File Print) is no longer used to calculate VSImax due to the amount of printers configured in an environment. The amount of printers could influence this number. If you have 15 printers, it takes much longer to open the dialog compared to when you only have one printer configured.

These measured operations affect many different sub-systems, such as CPU (user and kernel), Memory, Disk, the OS (in general), the application itself, print, GDI, etc. These operations are specifically short by nature. When such operations become consistently long, the system becomes saturated because of excessive queuing on any kind of resource. As a result, the average response times will then escalate, which is clearly visible to end-users. If such operations consistently take longer than expected, the user will regard the system as slow and unresponsive.

VSImax Metrics

Once the test is finished, VSImax v4.1 will be calculated. Previous VSImax models (Classic and Dynamic) needed Microsoft Word to function. With the new 4.1 timers this is no longer needed, making it more flexible and applicable to a larger amount of scenarios.

The following actions are part of the calculation and are weighted as follows (US notation):

  • Notepad File Open (NFO): 0.75
  • Notepad Start Load (NSLD): 0.2
  • Zip High Compression (ZHC): 0.125
  • Zip Low Compression (ZLC): 0.2
  • CPU: 0.75

This weighting is applied on the baseline and normal Login VSI response times.

VSImax Baseline

With the introduction of Login VSI 4.1, we also created a new method to calculate the basephase of an environment. With the new workloads (Taskworker, Powerworker, etc.) enabling basephase for a more reliable baseline has become obsolete. The calculation is explained below.

In total, 15 lowest VSI response time samples are taken from the entire test, the lowest 2 samples are removed and the 13 remaining samples are averaged. The result is the Baseline.

In short:

  • Take the lowest 15 samples of the test
  • Remove the lowest 2 from those 15 samples
  • Average these 13 samples to calculate the Baseline number

VSImax Average

The VSImax average response time in Login VSI 4.1.x is calculated on the amount of active users that are logged on to the system. This calculation always uses the average of five Login VSI response time samples + 40% of the amount of active sessions.

For example, if the active sessions = 60, then latest 5 + 24 (40% of 60) = 29 response time measurements are used for the average calculation.

To remove noise (accidental spikes) from the calculation, the top 5% and bottom 5% of the VSI response time samples are removed from the average calculation, with a minimum of one top and one bottom sample. As a result, with 60 active users, the last 29 VSI response time sample are taken. From those 29 samples, the top two samples are removed and lowest two results are removed (5% of 29 = 1.45, rounded to 2). At 60 users, the average is then calculated over the 27 remaining results.

VSImax Threshold

VSImax v4.1.x is reached when the VSIbase + a 1000ms latency threshold is not reached by the average VSI response time result. Depending on the tested system, VSImax response times can grow 2-3x the baseline average. In end-user computing, a 3x increase in response time compared to the baseline is typically regarded as the maximum performance degradation to be considered acceptable.

In VSImax v4.1.x, this latency threshold is fixed to 1000ms, which allows better and fairer comparisons between two different systems, especially when they have different baseline results. Ultimately, in VSImax v4.1.x, the performance of the system is not decided by the total average response time, but by the latency is has under load. For all systems, this is now 1000ms (weighted).

The threshold for the total response time = average weighted baseline response time + 1000ms.

When the system has a weighted baseline response time average of 1500ms, the maximum average response time may not be greater than 2500ms (1500 + 1000). If the average baseline is 3000, the maximum average response time may not be greater than 4000ms (3000 + 1000).

When the threshold is not exceeded by the average VSI response time during the test, VSImax is not hit and the amount of sessions ran successfully. This approach is fundamentally different in comparison to previous VSImax methods, as it was always required to saturate the system beyond VSImax threshold.

Additonal Info

VSImax v4.1.x is now always reported with the average baseline VSI response time result. For example: The VSImax was 125 with a baseline of 1526ms. This helps considerably in the comparison of systems and gives a more complete understanding of the system. The baseline performance helps to understand the best performance the system can give to an individual user. VSImax indicates what the total user capacity is for the system. These two are not automatically connected or related.

For example, when a server with a very fast dual core CPU, running at 3.6 GHz, is compared to a 10 core CPU, running at 2.26 GHz, the dual core machine will give an individual user better performance than the 10 core machine. This is indicated by the baseline VSI response time. The lower this score is, the better performance an individual user can expect.

However, the server with the slower 10 core CPU will easily have a larger capacity than the faster dual core system. This is indicated by VSImax v4.1.x, and the higher the VSImax is, the larger overall user capacity can be expected.

With Login VSI 4.1.x a new VSImax method is introduced: VSImax v4.1. This methodology gives much better insight into system performance and scales to extremely large systems.

Login VSI Timer

The VSI_Timer41 function measures the responsiveness of the system. It looks at the response of the system at multiple points within each workload. Within the Login VSI 4.1 timer, the timer mechanism has been made more reliable by removing erratic timers and adding more comprehensive timers.

For example: If we were to start an application, we look at what kind of performance we can expect at that moment in time, e.g. how long does it take to start the application or open the document.

Timer Metrics

VSImax v4.1 shows the amount of sessions that can be active on a system before the system is saturated. This number gives you an indication of the scalability of the environment (higher is better). VSImax v4.1 is determined by the measurements in the below table:

Measurement ID

Measurement Action

Measurement Action Detailed

Measures Related Resource

NSLD

Notepad starts and loads a 1500KB document

Notepad starts and loads a random local 1500KB document file that is copied from the content pool

CPU and I/O

NFO

Measure how long it takes to show the file-open dialog in Notepad

VSI-Notepad file open [Ctrl+O]

CPU, RAM, and I/O

ZHC*

Create a Zip file with high compression

Compress a local random .pst file that is copied from the content pool (5MB)

CPU and I/O

ZLC*

Create a Zip file with low compression

Compress a local random .pst file that is copied from the content pool (5MB)

I/O

CPU

Calculates a large array of random data

Creates a large array of random data that will be used in the I/O timer

CPU

'*'Compression is measured with 7-Zip

Other VSI 4.1 related measurements helping determine environmental issues (resources).

Note: The measurements in the table below are not used for the calculation of VSImax v4.1.

Measurement ID

Measurement Action

Measurement Action Detailed

Measures Related Resource

NFP

Press print open in VSI-Notepad

VSI-Notepad print open [Ctrl+P]

CPU

UMEM

The percentage of memory used by the sessions

The average percentage of memory used by the active sessions

RAM

I/O

Write the CPU random data file to disk

The file created by the CPU random data file is written to the local disk

I/O

Other VSI related measurements helping determine environmental issues (resources).

Note: The measurements in the table below are not used for the calculation of VSImax v4.1.

Measurement ID

Measurement Action

Measurement Action Detailed

Measures Related Resource

FCDL

File copy document local

Copy a local random document from temp directory to home drive

I/O

FCTL

File copy text local

Copy a local random text file from local temp directory to home drive

I/O

Other VSI related measurements helping determine environmental issues (resources).

Note: The measurements in the table below are not used for the calculation of VSImax v4.1.

Measurement ID

Measurement Action

Measurement Action Detailed

Measures Related Resource

FCTS

File copy text share

Copy a text document from VSIshare to temp directory

I/O and Network