Login VSI Analyzing Results

From Login VSI Documentation
Revision as of 16:53, 27 March 2015 by Omar (talk | contribs) (Analyzer preferences – Custom Multipliers v4.1)

Jump to: navigation, search

The Login VSI Analyzer will process the data collected during the VSI workload. The Analyzer will calculate if the target environment has reached it saturation point and if so, at how many concurrent session. This point is called VSImax.

Analyzing Results Video

VSImax 4.1.x Description

The philosophy behind Login VSI is different to conventional benchmarks. In general, most system benchmarks are steady state benchmarks. These benchmarks execute one or multiple processes, and the measured execution time is the outcome of the test. Simply put: the faster the execution time or the bigger the throughput, the faster the system is according to the benchmark.

Login VSI is different in approach. Login VSI is not primarily designed to be a steady state benchmark (however, if needed, Login VSI can act like one). Login VSI was designed to perform benchmarks for SBC or VDI workloads through system saturation. Login VSI loads the system with simulated user workloads using well known desktop applications like Microsoft Office, Internet Explorer and Adobe PDF reader. By gradually increasing the amount of simulated users, the system will eventually be saturated. Once the system is saturated, the response time of the applications will increase significantly. This latency in application response times show a clear indication whether the system is (close to being) overloaded. As a result, by nearly overloading a system it is possible to find out what its true maximum user capacity is.

After a test is performed, the response times can be analyzed to calculate the maximum active session/desktop capacity. Within Login VSI this is calculated as VSImax. When the system is coming closer to its saturation point, response times will rise. When reviewing the average response time it will be clear the response times escalate at saturation point.

This VSImax is the “Virtual Session Index (VSI)”. With Virtual Desktop Infrastructure (VDI) and Terminal Services (RDS) workloads this is valid and useful information. This index simplifies comparisons and makes it possible to understand the true impact of configuration changes on hypervisor host or guest level.

Server side response time measurements

It is important to understand why specific Login VSI design choices have been made. An important design choice is to execute the workload directly on the target system within the session instead of using remote sessions. The scripts simulating the workloads are performed by an engine that executes workload scripts on every target system, and are initiated at logon within the simulated user’s desktop session context.

An alternative to the Login VSI method would be to generate user actions client side through the remoting protocol. These methods are always specific to a product and vendor dependent. More importantly, some protocols simply do not have a method to script user actions client side.

For Login VSI the choice has been made to execute the scripts completely server side. This is the only practical and platform independent solution, for a benchmark like Login VSI.

Calculating VSImax v4.1.x

The simulated desktop workload is scripted in a 48 minute loop when a simulated Login VSI user is logged on, performing generic Office worker activities. After the loop is finished it will restart automatically. Within each loop the response times of sixteen specific operations are measured in a regular interval: sixteen times in within each loop. The response times of these five operations are used to determine VSImax.

The five operations from which the response times are measured are:

Notepad File Open (NFO)

Loading and initiating VSINotepad.exe and opening the openfile dialog. This operation is handled by the OS and by the VSINotepad.exe itself through execution. This operation seems almost instant from an end-user’s point of view.

Notepad Start Load (NSLD)

Loading and initiating VSINotepad.exe and opening a file. This operation is also handled by the OS and by the VSINotepad.exe itself through execution.This operation seems almost instant from an end-user’s point of view.

Zip High Compression (ZHC)

This action copy's a random file and compresses it (with 7zip) with high compression enabled. The compression will very briefly spike CPU and disk IO.

Zip Low Compression (ZLC)

This action copy's a random file and compresses it (with 7zip) with low compression enabled. The compression will very briefly disk IO and creates some load on the CPU aswell.

CPU

Calculates a large array of random data and spikes the CPU for a short period of time.


These measured operations within Login VSI do hit considerably different subsystems such as CPU (user and kernel), Memory, Disk, the OS in general, the application itself, print, GDI, etc. These operations are specifically short by nature. When such operations become consistently long: the system is saturated because of excessive queuing on any kind of resource. As a result, the average response times will then escalate. This effect is clearly visible to end-users. If such operations consistently consume multiple seconds the user will regard the system as slow and unresponsive.


Good-chart.png
Figure 1 Sample of a VSI max responsetime graph, representing a normal test



Bad-chart.png
Figure 2 Sample of a VSI test responsetime graph where there was a clear performance issue


Once the test is finished, VSImax can be calculated. When the system is not saturated, and it could complete the full test without exceeding the average response time latency threshold, VSImax is not reached and the amount of sessions ran successfully.

The response times are very different per measurement type, for instance Zip with compression can be around 2800 ms, while the Zip action without compression can only take 75ms. These response time of these actions are weighted before they are added to the total. This ensures that each activity has an equal impact on the total response time.

In comparison to previous VSImax models, this weighting much better represent system performance. All actions have very similar weight in the VSImax total. The following weighting of the response times are applied:

The following actions are part of the VSImax v4.1 calculation and are weighted as follows (US notation):

  • Notepad File Open (NFO): 0.75
  • Notepad Start Load (NSLD): 0.2
  • Zip High Compression (ZHC): 0.125
  • Zip Low Compression (ZLC): 0.2
  • CPU: 0.75

This weighting is applied on the baseline and normal Login VSI response times.

With the introduction of Login VSI 4.1 we also created a new method to calculate the basephase of an environment. With the new workloads (Taskworker, Powerworker, etc.) enabling 'basephase' for a more reliable baseline has become obsolete. The calculation is explained below. In total 15 lowest VSI response time samples are taken from the entire test, the lowest 2 samples are removed and the 13 remaining samples are averaged. The result is the Baseline. In short:

  • Take the lowest 15 samples of the complete test
  • From those 15 samples remove the lowest 2
  • Average the 13 results that are left is the baseline

The VSImax average response time in Login VSI 4.1.x is calculated on the amount of active users that are logged on the system.

Always a 5 Login VSI response time samples are averaged + 40% of the amount of “active” sessions. For example, if the active sessions is 60, then latest 5 + 24 (=40% of 60) = 31 response time measurement are used for the average calculation.

To remove noise (accidental spikes) from the calculation, the top 5% and bottom 5% of the VSI response time samples are removed from the average calculation, with a minimum of 1 top and 1 bottom sample. As a result, with 60 active users, the last 31 VSI response time sample are taken. From those 31 samples the top 2 samples are removed and lowest 2 results are removed (5% of 31 = 1.55, rounded to 2). At 60 users the average is then calculated over the 27 remaining results.

VSImax v4.1.x is reached when the VSIbase + a 1000 ms latency threshold is not reached by the average VSI response time result. Depending on the tested system, VSImax response time can grow 2 - 3x the baseline average. In end-user computing, a 3x increase in response time in comparison to the baseline is typically regarded as the maximum performance degradation to be considered acceptable.

In VSImax v4.1.x this latency threshold is fixed to 1000ms, this allows better and fairer comparisons between two different systems, especially when they have different baseline results. Ultimately, in VSImax v4.1.x, the performance of the system is not decided by the total average response time, but by the latency is has under load. For all systems, this is now 1000ms (weighted).

The threshold for the total response time is: average weighted baseline response time + 1000ms.

When the system has a weighted baseline response time average of 1500ms, the maximum average response time may not be greater than 2500ms (1500+1000). If the average baseline is 3000 the maximum average response time may not be greater than 4000ms (3000+1000).

When the threshold is not exceeded by the average VSI response time during the test, VSImax is not hit and the amount of sessions ran successfully. This approach is fundamentally different in comparison to previous VSImax methods, as it was always required to saturate the system beyond VSImax threshold.

Lastly, VSImax v4.1.x is now always reported with the average baseline VSI response time result. For example: “The VSImax v4.1 was 125 with a baseline of 1526ms”. This helps considerably in the comparison of systems and gives a more complete understanding of the system. The baseline performance helps to understand the best performance the system can give to an individual user. VSImax indicates what the total user capacity is for the system. These two are not automatically connected and related:

When a server with a very fast dual core CPU, running at 3.6 GHZ, is compared to a 10 core CPU, running at 2,26 GHZ, the dual core machine will give and individual user better performance than the 10 core machine. This is indicated by the baseline VSI response time. The lower this score is, the better performance an individual user can expect.

However, the server with the slower 10 core CPU will easily have a larger capacity than the faster dual core system. This is indicated by VSImax v4.1.x, and the higher VSImax is, the larger overall user capacity can be expected.

With Login VSI 4.1.x a new VSImax method is introduced: VSImax v4.1. This methodology gives much better insight in system performance and scales to extremely large systems.

Starting the Analyzer

Description

Screenshot

Start the Login VSI Analyzer by clicking the Analyzer button in the bottom left corner of the Login VSI Management Console.

2013-11-04 01 52 16-Login VSI Management Console 4.0.7.162 - Express.png

The location of the cache and the location of the VSIshare will need to be specified if the Analyzer runs for the first time.

2013-11-04 01 52 21-Login VSI Analyzer.png

Specify the path to the VSIshare. Change the location of the cache if desired. Click Save to continue.

2013-11-04 01 52 53-Login VSI Analyzer.png

Select the test that you want to analyze and click Open to start the analysis.

2013-11-04 02 00 36-Load Login VSI Log.png

Analyzer preferences Login VSI 4.1.x

This analyzer overview is only available when the following workloads are used:

  • Task Worker
  • Knowledge Worker
  • Office Worker
  • Power Worker

The Analyzer preferences are configured by default and best practices. However, some cases do require some additional settings or modifications. This section will describe what each preference tab is used for and how to adjust them.

Analyzer preferences - General tab

Description

Screenshot

  • Show splash screen on start-up

Will show the Login VSI Analyzer splash screen on start-up when starting the Analyzer

  • Show log dialog on start-up

Will show the Login VSI Analyzer's ‘default’ log location at start-up. With ‘default’ we refer to the primary ‘VSIShare location’

  • VSIshare location

The location of the VSIshare to use for analyzing the logs

  • License file

The location of the Login VSI License file to use for the Analyzer (default Login VSI license location is: \\<hostname>\<VSIShare>\_VSI_Configuration\LoginVSI.lic)

Local Cache Settings

  • Local cache location

The local cache location used by the Analyzer

  • Overwrite cache entry

The overwrite cache entry method used -Always prompt user (default) -Never overwrite -Always overwrite

41AnalyzerPref1.PNG

Analyzer preferences – Chart Settings tab

Description

Screenshot

  • Chart window size

Select the chart window size

-Normal (default)

-Maximized

-Minimized


  • Graph line width

Select the line width value shown in the charts

-Selection from 1-100 (default is 1)


  • Automatically detect axis values

-This will automatically detect and set the best axis values for the charts (default). -Uncheck for manually axis values used to generate the charts

-Sessions Axis (X axis)
 -Minimum value shown
 -Maximum value shown
-Response Axis (Y axis)
 -Minimum value shown
 -Maximum value shown
  • Chart saving width

-The width of the chart(s) when saving chart(s) (default is 1024).

  • Chart saving height

-The height of the chart(s) when saving chart(s) (default is 768).

41AnalyzerPref3.PNG

Analyzer preferences – Custom Multipliers v4

Description

Screenshot

  • The multipliers (weights) v4 are optimized for the VSImax and v4.0.x workloads (set by default).

-The calculation method for VSImax v4.

-Check for manually custom multipliers used to calculate VSImax v4.


41AnalyzerPref4.PNG

Analyzer preferences – Custom Multipliers v4.1

Description

Screenshot

  • The multipliers (weights) v4.1 are optimized for the VSImax and v4.1.x workloads (set by default).

-The calculation method for VSImax v4.1.

-Check for manually custom multipliers used to calculate VSImax v4.1.


41AnalyzerPref5.PNG

Analyzer preferences – Custom Multipliers Storage

Description

Screenshot

  • The multipliers (weights) Storage are optimized for the Storage VSImax and Storage workload (set by default).

-The calculation method for Storage VSImax v4.1.

-Check for manually custom multipliers used to calculate Storage VSImax v4.1.



41AnalyzerPref6.PNG

Analyzer preferences – Custom Multipliers Storage

Description

Screenshot

  • The multipliers (weights) Storage are optimized for the Storage VSImax and Storage workload (set by default).

-The calculation method for Storage VSImax v4.1.

-Check for manually custom multipliers used to calculate Storage VSImax v4.1.



41AnalyzerPref6.PNG

Analyzer tabs Login VSI 4.1.x

This analyzer overview is only available when the following workloads are used:

  • Task Worker
  • Knowledge Worker
  • Office Worker
  • Power Worker

Description

Screenshot

The VSImax v4 tab is the main tab of the Login VSI analyzer. This tab shows the most important information.

VSI Analyzer 4.1.x VSImaxv4.JPG

Zooming and scaling. As of version 4.0.9 the controls have moved to a separate popup to control the scales of both the X-Axis and Y-Axis, so that there is a cleaner image when loading the charts. The zoom-checkbox is removed and there is a zoom-button with an icon instead.

Scale-pop-up.PNG
Zoom-scale-buttons.PNG

This section shows the following.

VSImax v4:

VSImax v4 shows the amount of sessions can be active on a system before the system is saturated. The blue X shows the point where VSImax was reached. This number gives you an indication of the scalability of the environment (higher is better).

VSIbase:

VSIbase is the best performance of the system during a test (The lowest response times). This number is used to determine what the performance threshold will be. VSIbase gives an indication of the base performance of the environment (lower is better). VSIbase is also indicated within the graph.

2013-11-04 02 01 49-Login VSI Analyzer 4.0.7.162 - Express - -Win7 O2013 v401 2vCPU S150 run1-.png

VSImax v4 threshold:

VSImax v4 threshold indicates at which point the environments saturation point is reached. It is based on VSIbase. VSImax v4 threshold is also indicated within the graph.

Stuck sessions:

How many sessions got stuck during the test. This number should be 0. Stuck sessions indicate a problem during the test. As stuck session do not generate load the VSImax score will be reduced by the number of stuck sessions.

2013-11-04 02 01 33-Login VSI Analyzer 4.0.7.162 - Express - -Win7 O2013 v401 2vCPU S150 run1-.png

Minimum Response:

Minimum response indicates the minimum response time for all the measurements taken when the indicated number of sessions on the X axis were active.

Average Response:

Average response indicates the average response time for all the measurements taken taken when the indicated number of sessions on the X axis were active.

Maximum Response:

Maximum response indicates the maximum response time for all the measurements taken when the indicated number of sessions on the X axis were active.

VSI Index Average:

VSI Index Average indicates the average value as calculated by VSI. The VSI Index Average differs from Average Response on the fact that Average Response is the pure average. VSI Index Average applies certain statistical rules to the average to avoid spikes from influencing the average too much.

2013-11-04 02 02 15-Login VSI Analyzer 4.0.7.162 - Express - -Win7 O2013 v401 2vCPU S150 run1-.png

VSImax v4 detailed

Description

Screenshot

The VSImax v4 detailed tab shows the individual measurements taken during a test in a combined graph. This graph shows the minimum, average and maximum response times for each individual measurement. There is also a Total metric that combines all of the metric into a single number. The minimum, average and maximum for this combined value is shown as well.

VSI Analyzer 4.1.x VSImaxv4 Detailed.JPG

The metrics are as follows.

Total:

The sum of all the metrics.

CPU:

Calculates a large array of random data.

FCTL:

File Copy Text Local. Copy a txt (plain text) file locally.

FCTS:

File Copy Text Share. Copy a txt (plain text) file locally.

NFP:

Notepad File Print. Open the print dialog in notepad.

NSLD:

Notepad Start/LoaD file. Start notepad by file type association, loading a text file.

NFO:

Notepad File Open. Open the file with the open dialog in notepad.

ZHC:

Zip High Compression. Zip a PST (Outlook Personal Folder) file, which is approximately 5 megabytes in size, using high compression.

ZLC:

Zip Low Compression. Zip a PST (Outlook Personal Folder) file, which is approximately 5 megabytes in size, using no compression.

IO:

Writes the Random CPU data array to disk.

UMEM:

Percentage of memory used by the sessions.

VSI Analyzer 4.1.x VSImaxv4 Detailed Metrics.JPG

VSImax v4 detailed weighted

Description

Screenshot

The VSImax v4 detailed weighted tab shows the individual measurements taken during a test in a combined graph where a multiplier is applied. This graph shows the minimum, average and maximum response times for each individual measurement. In this graph you can clearly see which measurement has an impact on your environment.

VSI Analyzer 4.1.x VSImaxv4 Detailed Weighted.jpg

The metrics are as follows.

CPU:

Calculates a large array of random data.

FCTL:

File Copy Text Local. Copy a txt (plain text) file locally.

FCTS:

File Copy Text Share. Copy a txt (plain text) file locally.

NFP:

Notepad File Print. Open the print dialog in notepad.

NSLD:

Notepad Start/LoaD file. Start notepad by file type association, loading a text file.

NFO:

Notepad File Open. Open the file with the open dialog in notepad.

ZHC:

Zip High Compression. Zip a PST (Outlook Personal Folder) file, which is approximately 5 megabytes in size, using high compression.

ZLC:

Zip Low Compression. Zip a PST (Outlook Personal Folder) file, which is approximately 5 megabytes in size, using no compression.

IO:

Writes the Random CPU data array to disk.

UMEM:

Percentage of memory used by the sessions.

VSI Analyzer 4.1.x VSImaxv4 Detailed Weighted Metrics.jpg

VSImax v4 Scatter

Description

Screenshot

The VSImax Scatter tab allows you to see the measurements based on time. Every tab before this tab shows the measurements consolidated by active session count. This tab allows you to see the data based on the time it was collected.

This is particularly useful to get an insight in trends after the sessions have finished logging on. The other tabs will consolidate all of the data collected after the last session has become active into a single data point. Namely the last active session count.

VSI Analyzer 4.1.x VSImaxv4 Scatter.jpg
VSI Analyzer 4.1.x VSImaxv4 Scatter ByTime.jpg

NSLD, NFO, ZHC, ZLC, CPU, IO, UMEM

Description

Screenshot

These tabs show information for the individual measurements taken during the test. These specific measurements are zoomed in to because they are used to calculate VSIbase and VSImax v4 threshold.


The tab is similar to the VSImax v4 detailed tab except that it, by default, will only show the measurement for the tab. The graphs scale has also been scaled for the individual measurement.

The tab will also display the baseline value for this measurement. This is the time it takes to complete this measurement during baseline measurements. These measurements are taken while the system is under no or very little load. It is used to see how the measurement trends from a system that isn’t under load.

VSI Analyzer 4.1.x NSLD.jpg
VSI Analyzer 4.1.x NFO.jpg
VSI Analyzer 4.1.x ZHC.jpg
VSI Analyzer 4.1.x ZLC.jpg
VSI Analyzer 4.1.x CPU.jpg
VSI Analyzer 4.1.x IO.jpg
VSI Analyzer 4.1.x UMEM.jpg

These tabs also allow you to add any of the other metrics. The baseline for the tabs specific metric will not disappear though.


LogonTimer

Description

Screenshot

The LogonTimer tab gives you an indication of the time it takes for a session to logon. The graph shows the trend of logon times during the test. The logon time is specified in milliseconds.

Please note that this is an indication of the logon time. VSI measures the time from the logon scripts running, shortly after group policy has been processed but before the shell has loaded (Windows Explorer), and the windows shell being loaded.

VSI Analyzer 4.1.x LogonTimer.jpg

CTXConnector

Description

Screenshot

The CTX connector shows a graph that contains the timings measured during the usage of the CTXconnector. It measures :

  • Starting IE
  • Loading Site
  • Logging in
  • Starting Resource
  • Closing IE
  • TotalTime
CTXconnector analyzer.JPG

Application Start times

Description

Screenshot

The AppStart (Application Start times) tab gives you an (indication) overview of the Application Start times during the test.

VSI Analyzer 4.1.x AppStart.jpg

Web Start times

Description

Screenshot

The WebStart (Websites Start times) tab gives you an (indication) overview of the Websites Start times during the test.

Webstart.PNG

Custom Timer

Description

Screenshot

Using the VSI_CustomTimer() command in the workloads you are able to measure custom actions. These are then reported in the analyzer. For this graph the following command was used: "VSI_CustomTimer("Timer1") and VSI_CustomTimerStop("Timer1","Random Copy Timer 1")"

CustomTimer Analyzer 413.JPG

VSImax v4 Data & Raw Data

Description

Screenshot

These tabs contain the raw and processed data used to create the graphs in the analyzer. You can use this data to run your own analysis on.

VSI Analyzer 4.1.x VSImaxv4 Data.jpg
VSI Analyzer 4.1.x Raw Data.jpg

Summary

Description

Screenshot

The Test information tab gives you an indication of the test scenario. The tables shown:

Test results review

This contains the results of the test. It tells the following things:

  • Number of sessions launched in no. of seconds
  • How many sessions failed during the test before VSImax was reached (or before maxmimum amount of users was reached)
  • No. of sessions that did not successfully launch
  • No. of sessions that failed to become active
  • No. of sessions that were active during the test
  • No. of sessions that got stuck during the test

Maximum Capacity:

This tells the user if the maximum capacity of the test was reached, and if so at which point. And what the baseline of the test is.

Configured Sessions:

Tells the user what the recommendation is concerning the configured amount of sessions.

Baseline:

Lets the user know what the quality of the baseline is.

VSI Analyzer 41 Summary.JPG

Settings

Description

Screenshot

This tab informs the user about the settings used for this test.

Description:

If a description has been added it can be found here.

Custom command line (CCL):

The configured CCL from the test displayed.

Workload settings:

Workload settings for now will display which, Office versions and Workload language are configured.

Scenario:

What workload is configured, how many sessions and sequential interval.

VSI Analyzer 41 Settings.JPG

Stuck Sessions

Description

Screenshot

This tab informs the user about the stuck sessions that appeared during the test. The list will show the:

  • Username
  • Computername
  • Date / time of last report
VSI Analyzer 41 StuckSessions.JPG

Login VSI 4.1 Analyzer Importing external data

In our latest version of VSI you can import External Performance data like ESXtop or Perfmon in the Login VSI analyzer.

All metrics are down/up scaled to be in the range of 0-100, by finding the max value and multiplying accordingly. This feature was expanded to be able to hold multiple sources and multiple imports of metrics with ability to create new tabs or merge to the existing external data tabs. This also adds the remaining external data series to moving box, and adds the source column in the data grid view.


Description

Screenshot

Open the Analyzer located in {VSIshare}\_VSI_Binaries\Analyzer\Login VSI Analyzer.exe or run the Analyzer from the Management Console.

Manual open analyzer.PNG
Auto open analyzer.PNG

Specify the {VSIshare} and click save.

2013-11-04 01 52 21-Login VSI Analyzer.png

Open a result.

Open result.PNG
Analyzer before import.PNG

Click file – import – external data or shortcut key [ctrl+x]

Analyzer import.PNG

Open the (external) CSV file, in this example we use Perfmon CSV file that is collected during the same Login VSI test.

Import xternal data.PNG

Select the metrics you want to import.

Select the metrics.PNG

A new tab is opened with the title External Data

All metrics are down/up scaled to be in the range of 0-100, by finding the max value and multiplying accordingly. This feature was expanded to be able to hold multiple sources and multiple imports of metrics with ability to create new tabs or merge to the existing external data tabs. This also adds the remaining external data series to moving box, and adds the source column in the data grid view.

The blue steps line in the picture is representing the active sessions. Everything is recalculated to time in seconds, so the X-axis is time in seconds.

External Performance Data Perfmon.PNG

Description 4.1

When analyzing Login VSI results the most important metric is VSImax as this informs you about the maximum amount of users you can get on an environment before it starts to slow down. Another important metric is the Baseline, this number indicates the performance of the system without stress.

This document will explain how the analyser is used to its maximum potential.

Baseline 4.1

In Login VSI 4.1 the baseline represents the fastest response times found in your environment during the complete test. The most important role of the baseline is that it is used to calculate the threshold for the test.

To collect the baseline select all response times, select the lowest 15 and from the lowest 15 remove the lowest 2, average the 13 remaining results.

Tolerance

Login VSI uses a fixed tolerance of 1000 ms that’s added to the baseline when a test is analyzed. Using this fixed tolerance but variable baseline allows for comparisons systems that have different performance characteristics (different storage, Hypervisor, Operating system, Antivirus etc.)

Growth ratio (Ratio of the Growth Margin)

Using the baseline and fixed tolerance it is clear to see that a faster system will always report the higher VSImax as the growth ratio for slower environments is much smaller where it’s likely that response times will increase faster.

ID

Scenario 1

Scenario 2

Baseline

960 ms
2150 ms

Tolerance

1000 ms
1000 ms

Baseline + Tolerance = Threshold

1960 ms
3150 ms

Growth ratio

104 %
46 %

Scenario 1 Has 1000 ms to grow, that is 104% of the current baseline therefor it has ‘more room’ to grow and will hit VSImax later than scenario 2.

Scenario 2 Has 1000 ms to grow, that is 46% of the current baseline therefor it has ‘less room’ to grow and will hit VSImax faster than scenario 1.

VSImax Calculation

The VSImax calculation has been changed from a percentage to time based. Login VSI looks back 90 seconds (when available else look forward) for an x amount of measuring points. Simply put instead of looking forward we look to the past.

Login VSI Timer

The VSI_Timer41 function measures the responsiveness of the system. It looks at the response of the system at multiple points within each workload. Within the Login VSI 4.1 timer, the timer mechanism has been made more reliable by removing erratic timers and adding more comprehensive timers.

For example: If we were to start an application we look at what kind of performance we can expect at that moment in time, e.g. how long does it take to start the application or open the document.

Changes from 4.0 to 4.1

Word Start Load Document (WSLD)

All previous versions of Login VSI used Microsoft Office Word within their timer mechanism to calculate the desktops performance. As published in Project VRC whitepaper “Phase 6: Microsoft Office Impact on VDI” the influence of Microsoft Office on the test results can be significant. Removing Microsoft Word from the timer improves the consistency of the test results across different environments, Microsoft Word will still be part of the generic workloads.

  • Independent from Office, able to run in environments without Microsoft Office
  • More reliable when using User Environment Managers (UEM)
  • More reliable when using application visualization solutions
  • More reliable when using antivirus scanners and office plugins

ZNC to ZLC

ZNC (Zip No Compression) has been changed to ZLC (Zip Low Compression). The reasons for this are:

  • ZNC was too fast, ZNC simply added a header and a footer to the file. This made it very fast requiring a large multiplier compensate this making small changes in response time relatively intense
  • ZNC with a virus scanner will increase the times a lot which meant that the results could not be compared to each other since ZNC and therefore was unreliable
  • ZLC uses a single random letter as password to prevent easy caching on a file system-file, this hardly generates any CPU overhead
  • ZLC uses a low compression: this consumes moderate CPU. ZHC is more CPU bottleneck prone then ZLC. ZLC is more IO prone the ZHC
  • Detailed weighted ZHC and ZLC is multi-threaded so with a 1 vCPU environment (VM) ZHC and ZLC almost doubles in response times in comparison to 2 vCPU's environments

NFP

NFP (Notepad File Print) is no longer used due to the amount of printers configured in an environment. The amount of printers could influence this number. If you have 15 printers it takes much longer to open the dialog compared to when you have 1 printer configured.

ZHC vs. ZLC

ZHC is more CPU prone and ZLC is more IO prone. The two graphs will most likely look the same but depending on which is higher and the difference between ZHC and ZLC you can see where the bottleneck of an environment lies, either IO or CPU.

CPU

CPU is a new introduced VSImax metric. CPU has proven to be very stable and resilient against applications and outside interactions. This metric shows us the performance of the CPU. CPU creates an array of data and is a purely compute process.

IO

IO has been added to Login VSI but not to the VSImax calculation. This graph will indicate when writes or throughput are a bottleneck. It is added to the analyzer because it can give you a good indication if IO is the bottle neck, it will not give you the exact reason but an indication. It is recommended that external performance data is used to confirm suspicions.

  • Writes multiple random files in different sizes to the storage.

UMEM

UMEM (Used Memory) is also added in 4.1 this shows the amount of available memory (in %) in a session (VM). This is interesting when the session is running out of memory and starts paging. This is recognized by decrease in IO performance. This graph will fluctuate naturally because segment 1 of the workload has little to no applications open and segment 4 has lots of applications open.

NSLD

NSLD (Notepad Start Load Document) is most of the time very quick and not IO intensive because most files opened by Notepad are cached by the environment or windows itself.

Used metrics to calculate VSImax 4.1

  • NSLD

Notepad starts and loads a 1500kb document

  • NFO

Measure how long it takes to show the file-open dialog in VSI notepad

  • ZHC

Create a zipfile with high compression

  • ZLC

Create a zipfile with low compression

  • CPU

Calculates a large array of random data

Analyzing

  • When IO becomes the bottleneck it will drag down CPU, so CPU will rise when IO starts to reach its maximum.
  • CPU will drag down IO when it becomes the bottleneck.
  • You can see in the IO / CPU tab which is the bottle neck by checking which of the two is presenting the most extreme results.
  • As always with every metric of Login VSI the values are all indications, without external performance data a true culprit cannot be found.
  • Compare results with a good test.
  • Run a number of tests and compare these with each other to see if the test results are reliable.

External Data

ESXTop, XenTop or PerfMon important data to import into Login VSI Analyzer.

  • Core Util Time (CPU)
  • Processor Time (CPU)

This can be 55% this means that the processor has a turbo mode (55%) and it is half since you have Hyper threading

  • Util time (CPU)
  • Ready time (CPU)
  • Commands per second (IO)
  • Megabytes per second (IO)
  • Reads (IO)
  • Writes (IO)
  • Throughput (IO)


The most probable bottleneck of an environment is most likely IOPS. This is because operating systems use lots of small files, this generates allot of IOPS.

  • Queue lengths – time between each IO
  • An IO latency of 10-20 is acceptable, everything above 20ms is considered bad and system degradation is imminent. However this is not the holy grail.
  • A true IO problem shows itself in (lots) of stuck sessions.

In short, there is no single metric in Login VSI or External data that points to the cause of the performance degradation. All of the metrics should support the suspected bottleneck, it is a joint venture of all metrics to confirm this.

CPU is an absolute metric, what this means is that CPU is always within 0% – 100%. Total Processor time is not useful since there are allot of background processes running during and this influences the results.

Scatter-chart

Set it “by time”. The scatter chart is very important. You can see here where within a test a problem arises. And since it is by time you can scan easily through the results. When there are gaps within this graph this means that at that point in time there were NO measurements. Most likely pointing to an over-saturated environment.

Calculating VSIMax 4.1

The simulated desktop workload is scripted in a 48 minute loop when a simulated Login VSI user is logged on, performing generic Office worker activities. After the loop is finished it will restart automatically. Within each loop the response times of five specific operations are measured in a regular interval: twelve times in within each loop. The response times of these five operations are used to determine VSImax.

The five operations from which the response times are measured are:

Notepad File Open (NFO)

Loading and initiating VSINotepad.exe and opening the openfile dialog. This operation is handled by the OS and by the VSINotepad.exe itself through execution. This operation seems almost instant from an end-user’s point of view.

Notepad Start Load (NSLD)

Loading and initiating VSINotepad.exe and opening a file. This operation is also handled by the OS and by the VSINotepad.exe itself through execution.This operation seems almost instant from an end-user’s point of view.

Zip High Compression (ZHC)

This action copy's a random file and compresses it (with 7zip) with high compression enabled. The compression will very briefly spike CPU and disk IO.

Zip Low Compression (ZLC)

This action copy's a random file and compresses it (with 7zip) with low compression enabled. The compression will very briefly disk IO and creates some load on the CPU aswell.

CPU

Calculates a large array of random data and spikes the CPU for a short period of time.

VSImax Metrics

Once the test is finished, VSImax v4.1 can be calculated. Previous VSImax models (Classic and Dynamic) needed Microsoft Office Word to function. With the new 4.1 timers this is no longer needed, we are therefore more flexible and applicable to a larger scale of scenario's.

The following actions are part of the VSImax v4.1 calculation and are weighted as follows (US notation):

  • Notepad File Open (NFO): 0.75
  • Notepad Start Load (NSLD): 0.2
  • Zip High Compression (ZHC): 0.125
  • Zip Low Compression (ZLC): 0.2
  • CPU: 0.75

This weighting is applied on the baseline and normal Login VSI response times.

VSImax Baseline

With the introduction of Login VSI 4.1 we also created a new method to calculate the basephase of an environment. With the new workloads (Taskworker, Powerworker, etc.) enabling 'basephase' for a more reliable baseline has become obsolete. The calculation is explained below. In total 15 VSI response time samples are taken from the entire test, the lowest 2 samples are removed and the 13 remaining samples are averaged. The result is the Baseline. In short:

  • Take the lowest 15 samples of the complete test
  • From those 15 samples remove the lowest 2
  • Average the 13 results that are left is the baseline

Login VSI 4.1 Timer Metrics

VSImax v4.1 shows the amount of sessions that can be active on a system before the system is saturated. The blue X shows the point where VSImax was reached. This number gives you an indication of the scalability of the environment (higher is better). VSImax v4.1 is determined by the measurements in the below table:

Measurement ID

Measurement Action

Measurement Action Detailed

Measures Related Resource

NSLD

Notepad starts and loads a 1500kb document

Notepad starts and loads a random local 1500kb document file that is copied from the content pool

CPU and IO

NFO

Measure how long it takes to show the file-open dialog in VSI notepad

VSI-Notepad file open [ctrl+o]

CPU RAM, and IO

ZHC*

Create a zipfile with high compression

Compress a local random .pst file that is copied from the content pool (5mb)

CPU and IO

ZLC*

Create a zipfile with low compression

Compress a local random .pst file that is copied from the content pool (5mb)

IO

CPU

Calculates a large array of random data

Creates a large array of random data that will be used in the IO timer

CPU

'*'compression is measured with 7zip


Other VSI 4.1 related measurements helping determine environmental issues (resources). Note, *the measurements in the table below are not used for the calculation of VSImax v4.1.

Measurement ID

Measurement Action

Measurement Action Detailed

Measures Related Resource

NFP

Press print open in VSI-Notepad

VSI-Notepad print open [ctrl+p]

CPU

UMEM

The percentage of memory used by the sessions

The average percentage of memory used by the active sessions

RAM

IO

Write the CPU random data file to disk

The file created by the CPU random data file is written to the local disk

IO

Other VSI related measurements helping determine environmental issues (resources). Note, *the measurements in the table below are not used for the calculation of VSImax v4.1.

Measurement ID

Measurement Action

Measurement Action Detailed

Measures Related Resource

FCDL

File copy document local

Copy a local random document from temp directory to home drive

IO

FCTL

File copy text local

Copy a local random text file from local temp directory to home drive

IO


Other VSI related measurements helping determine environmental issues (resources). Note, *the measurements in the table below are not used for the calculation of VSImax v4.1.

Measurement ID

Measurement Action

Measurement Action Detailed

Measures Related Resource

FCTS

File copy text share

Copy a text document from VSIshare to temp directory

IO and Network

Analyzing results with Login VSI 4.0.x

Analyzing results for or with version 4.0.x click on the link below.

Login VSI Analyzing Results 4.0.x