Login VSI VSImax

From Login VSI Documentation
Revision as of 12:08, 24 October 2017 by John (talk | contribs) (Calculating VSImax v4.1.x)

Jump to: navigation, search

The logic behind Login VSI is different to conventional benchmarks. In general, most system benchmarks are steady state benchmarks. These benchmarks execute one or multiple processes, and the measured execution time is the outcome of the test. Simply put: the faster the execution time or the bigger the throughput, the faster the system is according to the benchmark.

Login VSI is different in approach. Login VSI is not primarily designed to be a steady state benchmark (however, if needed, Login VSI can act like one). Login VSI was designed to perform benchmarks for SBC or VDI workloads through system saturation. Login VSI loads the system with simulated user workloads using well known desktop applications like Microsoft Office, Internet Explorer and Adobe PDF reader. By gradually increasing the amount of simulated users, the system will eventually be saturated. Once the system is saturated, the response time of the applications will increase significantly. This latency in application response times gives a clear indication whether the system is (close to being) overloaded. As a result, by nearly overloading a system it is possible to find out what is the true maximum user capacity.

After a test is performed, the response times can be analyzed to calculate the maximum active session/desktop capacity. Within Login VSI this is calculated as VSImax. When the system is coming closer to its saturation point, response times will rise. When reviewing the average response time it will be clear that the response times escalate at the saturation point.

This VSImax is the “Virtual Session Index (VSI)”. With Virtual Desktop Infrastructure (VDI) and Terminal Services (RDS) workloads this is valid and useful information. This index simplifies comparisons and makes it possible to understand the true impact of configuration changes on hypervisor host or guest level.

Recommended: Calculating Maximum Virtual Desktop Capacity – VSImax Explained

Server side response time measurements

It is important to understand why specific Login VSI design choices have been made. An important design choice is to execute the workload directly on the target system within the session instead of using remote sessions. The scripts simulating the workloads are performed by an engine that executes workload scripts on every target system, and are initiated at logon within the simulated user’s desktop session context.

An alternative to the Login VSI method would be to generate user actions on the client side via the remoting protocol. These methods are always specific to a product and vendor dependent. More importantly, some protocols simply do not have a method to script user actions client side.

For Login VSI, the choice has been made to execute the scripts on the server side. This is the only practical and platform independent solution for a benchmark, like Login VSI. The relative overhead and footprint of a benchmark engine scripted in Auto-IT is small enough (1-5% range) for Login VSI’s purposes.

Calculating VSImax v4.1.x

The simulated desktop workload is scripted in a 48-minute loop when a simulated Login VSI user is logged on, performing generic office worker activities. After the loop is finished, it will restart automatically. Within each loop the response times of ten specific operations are measured in a regular interval, five of which are the most important. These measurements happen sixteen times within each loop. The response times of these five important operations are used to determine VSImax.

The five operations from which the response times are measured are:

Notepad File Open (NFO)

Loading and initiating VSINotepad.exe and opening the openfile dialog. This operation is handled by the OS and by the VSINotepad.exe itself through execution. This operation seems almost instant from an end-user point of view.

Notepad Start Load (NSLD)

Loading and initiating VSINotepad.exe and opening a file. This operation is also handled by the OS and by the VSINotepad.exe itself through execution. This operation seems almost instant from an end-user point of view.

Zip High Compression (ZHC)

This action copies a random file and compresses it (with 7-Zip) with high compression enabled. The compression will very briefly spike both CPU and disk I/O.

Zip Low Compression (ZLC)

This action copies a random file and compresses it (with 7-Zip) with low compression enabled. The compression will very briefly spike disk I/O while also creating some load on the CPU as well.

CPU

Calculates a large array of random data and spikes the CPU for a short period of time.

Note: NFP (Notepad File Print) is no longer used to calculate VSImax due to the amount of printers configured in an environment. The amount of printers could influence this number. If you have 15 printers it takes much longer to open the dialog compared to when you have 1 printer configured.

These measured operations within Login VSI do hit considerably different subsystems such as CPU (user and kernel), Memory, Disk, the OS in general, the application itself, print, GDI, etc. These operations are specifically short by nature. When such operations become consistently long: the system is saturated because of excessive queuing on any kind of resource. As a result, the average response times will then escalate. This effect is clearly visible to end-users. If such operations consistently consume multiple seconds the user will regard the system as slow and unresponsive.

VSImax Metrics

Once the test is finished, VSImax v4.1 can be calculated. Previous VSImax models (Classic and Dynamic) needed Microsoft Word to function. With the new 4.1 timers this is no longer needed, making it more flexible and applicable to a larger amount of scenarios.

The following actions are part of the VSImax v4.1 calculation and are weighted as follows (US notation):

  • Notepad File Open (NFO): 0.75
  • Notepad Start Load (NSLD): 0.2
  • Zip High Compression (ZHC): 0.125
  • Zip Low Compression (ZLC): 0.2
  • CPU: 0.75

This weighting is applied on the baseline and normal Login VSI response times.

VSImax Baseline

With the introduction of Login VSI 4.1 we also created a new method to calculate the basephase of an environment. With the new workloads (Taskworker, Powerworker, etc.) enabling 'basephase' for a more reliable baseline has become obsolete. The calculation is explained below.

In total 15 lowest VSI response time samples are taken from the entire test, the lowest 2 samples are removed and the 13 remaining samples are averaged. The result is the Baseline. In short:

  • Take the lowest 15 samples of the complete test
  • From those 15 samples remove the lowest 2
  • Average the 13 results that are left is the baseline

VSImax Average

The VSImax average response time in Login VSI 4.1.x is calculated on the amount of active users that are logged on the system.

Always a 5 Login VSI response time samples are averaged + 40% of the amount of “active” sessions. For example, if the active sessions is 60, then latest 5 + 24 (=40% of 60) = 31 response time measurement are used for the average calculation.

To remove noise (accidental spikes) from the calculation, the top 5% and bottom 5% of the VSI response time samples are removed from the average calculation, with a minimum of 1 top and 1 bottom sample. As a result, with 60 active users, the last 31 VSI response time sample are taken. From those 31 samples the top 2 samples are removed and lowest 2 results are removed (5% of 31 = 1.55, rounded to 2). At 60 users the average is then calculated over the 27 remaining results.

VSImax Threshold

VSImax v4.1.x is reached when the VSIbase + a 1000 ms latency threshold is not reached by the average VSI response time result. Depending on the tested system, VSImax response time can grow 2 - 3x the baseline average. In end-user computing, a 3x increase in response time in comparison to the baseline is typically regarded as the maximum performance degradation to be considered acceptable.

In VSImax v4.1.x this latency threshold is fixed to 1000ms, this allows better and fairer comparisons between two different systems, especially when they have different baseline results. Ultimately, in VSImax v4.1.x, the performance of the system is not decided by the total average response time, but by the latency is has under load. For all systems, this is now 1000ms (weighted).

The threshold for the total response time is: average weighted baseline response time + 1000ms.

When the system has a weighted baseline response time average of 1500ms, the maximum average response time may not be greater than 2500ms (1500+1000). If the average baseline is 3000 the maximum average response time may not be greater than 4000ms (3000+1000).

When the threshold is not exceeded by the average VSI response time during the test, VSImax is not hit and the amount of sessions ran successfully. This approach is fundamentally different in comparison to previous VSImax methods, as it was always required to saturate the system beyond VSImax threshold.

Additonal Info

VSImax v4.1.x is now always reported with the average baseline VSI response time result. For example: “The VSImax v4.1 was 125 with a baseline of 1526ms”. This helps considerably in the comparison of systems and gives a more complete understanding of the system. The baseline performance helps to understand the best performance the system can give to an individual user. VSImax indicates what the total user capacity is for the system. These two are not automatically connected and related:

When a server with a very fast dual core CPU, running at 3.6 GHZ, is compared to a 10 core CPU, running at 2,26 GHZ, the dual core machine will give and individual user better performance than the 10 core machine. This is indicated by the baseline VSI response time. The lower this score is, the better performance an individual user can expect.

However, the server with the slower 10 core CPU will easily have a larger capacity than the faster dual core system. This is indicated by VSImax v4.1.x, and the higher VSImax is, the larger overall user capacity can be expected.

With Login VSI 4.1.x a new VSImax method is introduced: VSImax v4.1. This methodology gives much better insight in system performance and scales to extremely large systems.

VSImax Calculation

The VSImax calculation has been changed from a percentage to time based. Login VSI looks back 90 seconds (when available else look forward) for an x amount of measuring points. Simply put instead of looking forward we look to the past.

Login VSI Timer

The VSI_Timer41 function measures the responsiveness of the system. It looks at the response of the system at multiple points within each workload. Within the Login VSI 4.1 timer, the timer mechanism has been made more reliable by removing erratic timers and adding more comprehensive timers.

For example: If we were to start an application we look at what kind of performance we can expect at that moment in time, e.g. how long does it take to start the application or open the document.

Changes from 4.0 to 4.1

Word Start Load Document (WSLD)

All previous versions of Login VSI used Microsoft Office Word within their timer mechanism to calculate the desktops performance. As published in Project VRC whitepaper “Phase 6: Microsoft Office Impact on VDI” the influence of Microsoft Office on the test results can be significant. Removing Microsoft Word from the timer improves the consistency of the test results across different environments, Microsoft Word will still be part of the generic workloads.

  • Independent from Office, able to run in environments without Microsoft Office
  • More reliable when using User Environment Managers (UEM)
  • More reliable when using application visualization solutions
  • More reliable when using antivirus scanners and office plugins

ZNC to ZLC

ZNC (Zip No Compression) has been changed to ZLC (Zip Low Compression). The reasons for this are:

  • ZNC was too fast, ZNC simply added a header and a footer to the file. This made it very fast requiring a large multiplier compensate this making small changes in response time relatively intense
  • ZNC with a virus scanner will increase the times a lot which meant that the results could not be compared to each other since ZNC and therefore was unreliable
  • ZLC uses a single random letter as password to prevent easy caching on a file system-file, this hardly generates any CPU overhead
  • ZLC uses a low compression: this consumes moderate CPU. ZHC is more CPU bottleneck prone then ZLC. ZLC is more IO prone the ZHC
  • Detailed weighted ZHC and ZLC is multi-threaded so with a 1 vCPU environment (VM) ZHC and ZLC almost doubles in response times in comparison to 2 vCPU's environments

NFP

NFP (Notepad File Print) is no longer used due to the amount of printers configured in an environment. The amount of printers could influence this number. If you have 15 printers it takes much longer to open the dialog compared to when you have 1 printer configured.

ZHC vs. ZLC

ZHC is more CPU prone and ZLC is more IO prone. The two graphs will most likely look the same but depending on which is higher and the difference between ZHC and ZLC you can see where the bottleneck of an environment lies, either IO or CPU.

CPU

CPU is a new introduced VSImax metric. CPU has proven to be very stable and resilient against applications and outside interactions. This metric shows us the performance of the CPU. CPU creates an array of data and is a purely compute process.

IO

IO has been added to Login VSI but not to the VSImax calculation. This graph will indicate when writes or throughput are a bottleneck. It is added to the analyzer because it can give you a good indication if IO is the bottle neck, it will not give you the exact reason but an indication. It is recommended that external performance data is used to confirm suspicions.

  • Writes multiple random files in different sizes to the storage.

UMEM

UMEM (Used Memory) is also added in 4.1 this shows the amount of available memory (in %) in a session (VM). This is interesting when the session is running out of memory and starts paging. This is recognized by decrease in IO performance. This graph will fluctuate naturally because segment 1 of the workload has little to no applications open and segment 4 has lots of applications open.

NSLD

NSLD (Notepad Start Load Document) is most of the time very quick and not IO intensive because most files opened by Notepad are cached by the environment or windows itself.

Timer Metrics

VSImax v4.1 shows the amount of sessions that can be active on a system before the system is saturated. The blue X shows the point where VSImax was reached. This number gives you an indication of the scalability of the environment (higher is better). VSImax v4.1 is determined by the measurements in the below table:

Measurement ID

Measurement Action

Measurement Action Detailed

Measures Related Resource

NSLD

Notepad starts and loads a 1500kb document

Notepad starts and loads a random local 1500kb document file that is copied from the content pool

CPU and IO

NFO

Measure how long it takes to show the file-open dialog in VSI notepad

VSI-Notepad file open [ctrl+o]

CPU RAM, and IO

ZHC*

Create a zipfile with high compression

Compress a local random .pst file that is copied from the content pool (5mb)

CPU and IO

ZLC*

Create a zipfile with low compression

Compress a local random .pst file that is copied from the content pool (5mb)

IO

CPU

Calculates a large array of random data

Creates a large array of random data that will be used in the IO timer

CPU

'*'compression is measured with 7zip


Other VSI 4.1 related measurements helping determine environmental issues (resources). Note, *the measurements in the table below are not used for the calculation of VSImax v4.1.

Measurement ID

Measurement Action

Measurement Action Detailed

Measures Related Resource

NFP

Press print open in VSI-Notepad

VSI-Notepad print open [ctrl+p]

CPU

UMEM

The percentage of memory used by the sessions

The average percentage of memory used by the active sessions

RAM

IO

Write the CPU random data file to disk

The file created by the CPU random data file is written to the local disk

IO

Other VSI related measurements helping determine environmental issues (resources). Note, *the measurements in the table below are not used for the calculation of VSImax v4.1.

Measurement ID

Measurement Action

Measurement Action Detailed

Measures Related Resource

FCDL

File copy document local

Copy a local random document from temp directory to home drive

IO

FCTL

File copy text local

Copy a local random text file from local temp directory to home drive

IO


Other VSI related measurements helping determine environmental issues (resources). Note, *the measurements in the table below are not used for the calculation of VSImax v4.1.

Measurement ID

Measurement Action

Measurement Action Detailed

Measures Related Resource

FCTS

File copy text share

Copy a text document from VSIshare to temp directory

IO and Network

Other resources

Blog - Calculating Maximum Virtual Desktop Capacity – VSImax Explained