Login VSI Analyzing Results

From Login VSI Documentation
Revision as of 12:29, 26 October 2017 by John (talk | contribs) (Interpreting Test Results)

Jump to: navigation, search

The Login VSI Analyzer will process the data collected during a VSI workload. It will calculate if the target environment has reached its saturation point and if so, at how many concurrent sessions. This point is called VSImax.

Analyzing Results Video

Update September 3, 2014: Blog - Analyzing Login VSI Results

Interpreting Test Results

When analyzing the results of your Login VSI test, there are several things to take into consideration. The first thing to realize is that there is no silver bullet that will point to the cause of any performance degradation. You must take an overview of the whole test to find out where the problem lies. If your graphs look like Example A, then your system is working within normal parameters. However, any deviation from this result indicates there is an issue somewhere in the environment (Example B or C).

                                             Example A: VSImax v4.png Example B: Bad chart 1.png Example C: Bad-chart.png

Once you think you have discovered a suspected bottleneck, you should then check the rest of the metrics to see if they confirm the problem. For example, if you see the IO latency spiking (anything above 50ms), this indicates the storage platform that hosts the %temp% directory of the user, i.e. LoginVSI1/2/3 is the bottleneck. Anything higher than 100ms (but constant) can be the impact of a virus scanner. In this circumstance, please also check the VSImax v4 Detailed graph for the FCTS (VSIshare to %temp%) and FCTL (%temp% to C:) scores. If the FCTS score is spiking, this indicates that the network is your bottleneck, while FCTL spikes indicate that your local disk is the problem. However, if both IO latency and CPU scores are spiking, then it is more likely a CPU bottleneck. While a true IO problem also produces a large amount of stuck sessions.

We recommend checking the summary of the test first to see the results of VSImax (maximum number of users) and the Baseline (performance of the system without stress). You should then check the most important graphs to see if any patterns and/or clear performance bottlenecks are visible. The four most important graphs are:

  • VSImax v4: If the VSImax v4 graph reaches the VSImax point very early (VSImax = 1-20), this can (it’s never certain) indicate that there is an image management problem
  • IO: If the IO graph starts steadily but then fluctuates aggressively, then this can indicate that there is a problem with the storage platform
  • CPU: If the CPU graph starts with high peaks before settling down as time passes, then this can indicate that there is a power management problem
  • AppStart: This graph can tell you if a specific application is experiencing any problems

In addition to these four graphs you should also check the Raw Data table, which can help you focus your search. For example, you can sort the computername or total_response columns to see if there were any issues with individual VMs in relation to a specific host. We also recommend importing external performance data into the Analyzer from the host level (ESXTop, PerfMon or XenTop) to help you verify your suspicions behind any performance degradation.

Please be aware that many different factors can influence your test results in a negative way, such as hardware and/or software changes, anti-virus scans, backup jobs, or scheduled tasks. We also suggest that you run several tests and compare them with each other to see if the results are reliable. Statistically speaking, running ten consecutive tests gives enough data to get a reliable result. Once you have collated the results, it is best practice to remove the highest and lowest VSImax measurements (after sorting), and then average the rest to get the definitive VSImax value.

If you want in-depth help with analyzing your Login VSI test results, then please contact a member of Login VSI Support via: Media:support@loginvsi.com

Opening the Analyzer

Description

Screenshot

Start the Login VSI Analyzer by clicking the Analyzer button in the bottom left corner of the Login VSI Management Console.

Note: You can also open the Analyzer via the VSIshare and the shortcut on the desktop.

Analyzer1.png

The location of the cache and the location of the VSIshare will need to be specified when the Analyzer runs for the first time.

2013-11-04 01 52 21-Login VSI Analyzer.png

Specify the path to the VSIshare. Change the location of the cache if desired. Click Save to continue.

2013-11-04 01 52 53-Login VSI Analyzer.png

Select the test that you want to analyze and click Open to start the analysis.

Analyzer2.png

Analyzer Preferences

This analyzer overview is only available when the following workloads are used:

  • Task Worker
  • Knowledge Worker
  • Office Worker
  • Power Worker

These preferences are configured for best practices, by default. However, some cases do require some additional settings or modifications. This section will describe what each preference tab is used for and how to adjust them.

General

Description

Screenshot

  • Show splash screen on startup

Will show the Login VSI Analyzer splash screen when starting the Analyzer

  • Show log dialog on startup

Will show the Login VSI Analyzer's default log location at startup (with ‘default’ we refer to the primary ‘VSIShare location’)

  • VSIshare location

The location of the VSIshare to use for analyzing the logs

  • License file

The location of the Login VSI License file to use for the Analyzer (default Login VSI license location is: \\<hostname>\<VSIShare>\_VSI_Configuration\LoginVSI.lic)

Local Cache Settings

  • Local cache location

The local cache location used by the Analyzer

  • Overwrite cache entry

The overwrite cache entry method used: Always prompt user (default) / Never overwrite / Always overwrite

41AnalyzerPref1.PNG

Chart Settings

Description

Screenshot

  • Chart window size

Select the chart window size: Normal (default) / Maximized / Minimized

  • Graph line width

Select the line width value shown in the charts: Selection from 1-100 (default = 1)

  • Automatically detect axis values

This will automatically detect and set the best axis values for the charts (default)

Unmark the checkbox to manually change the axis values used to generate the charts:

 Sessions Axis (X)
 -Minimum value shown
 -Maximum value shown
 Response Axis (Y)
 -Minimum value shown
 -Maximum value shown
  • Chart saving width

The width of the chart(s) when saving chart(s) (default = 1024)

  • Chart saving height

The height of the chart(s) when saving chart(s) (default = 768)

41AnalyzerPref3.PNG

Analyzer Tabs: Troubleshooting Graphs

Description

Screenshot

These troubleshooting graphs are only available with Login VSI 4.1.25 and higher:

To enable them, simply mark the checkboxes next to each troubleshooting preference, click Apply, then click the OK button. Once a test has finished, you will see that the new graphs have been added to the list of tabs in the Analyzer.

Note: All the tabs are automatically enabled except for the three troubleshooting graphs. However, if you are running large-scale data tests and/or SOAK tests it might be useful to unmark (disable) the checkboxes of certain graphs that are not required.

Analyzer troubleshoot tabs.png

Analyzer Graphs

This analyzer overview is only available when the following workloads are used:

  • Taskworker
  • Knowledgeworker
  • Powerworker
  • Officeworker

Summary

Description

Screenshot

The Summary tab gives you an overview of the completed test, including the VSImax result.

The Test result review table summarizes the results of the test:

  • No. of sessions launched in (amount) of seconds
  • No. of failed sessions during the test before VSImax was reached (or before maximum amount of users was reached)
  • No. of sessions that did not successfully launch
  • No. of sessions that failed to become active
  • No. of sessions that were active during the test
  • No. of sessions that got stuck during the test (before VSImax threshold)

Maximum Capacity

Tells you if the maximum capacity of the test was reached, and if so, at which point, as well as what the baseline performance score is.

Configured Sessions

Tells you the recommendation concerning the configured amount of sessions.

Baseline

Lets you know the baseline performance rating:

  • 0-799 (Rating: Very Good)
  • 800-1199 (Rating: Good)
  • 1200-1599 (Rating: Fair)
  • 1600-1999 (Rating: Poor)
  • 2000-9999 (Rating: Very Poor)
SummaryO16.png

Settings

Description

Screenshot

This tab contains the settings that were used for the test.

Description

If a description was included, it can be found here.

Custom Command Line (CCL)

The configured CCL from the completed test.

Workload Settings

Displays the configured Office version and workload language.

Scenario

Displays the configured workload, number of sessions and sequential interval.

SettingsO16.png

VSImax v4

Description

Screenshot

The VSImax v4 tab is the main tab of the Login VSI Analyzer and shows the most important information.

Note: This line graph can also be displayed in scatter format.

VSImax v4.png

Reporting, Zooming and Scaling

To generate an overview of the test results, click Generate Report.

To zoom in, click the Zoom icon, then use your mouse to highlight the area of the graph you would like to take a closer look at.

To adjust the scales of both the X-axis and Y-axis, click the Scale icon to open the GraphControls pop-up.

Scale-pop-up.PNG
Scale.png

VSImax

VSImax shows the amount of sessions can be active on a system before the system is saturated. The red X shows the point where VSImax was reached. This number gives you an indication of the scalability of the environment (higher is better).

VSIbase

VSIbase is the best performance of the system during a test (the lowest response times). This number is used to determine what the performance threshold will be. VSIbase gives an indication of the base performance of the environment (lower is better).

VSImax v4 average

VSImax v4 average is calculated on the amount of active users that are logged on the system but removes the two highest and two lowest samples to provide a more accurate measurement.

VSImax v4 threshold

VSImax v4 threshold indicates at which point the environments saturation point is reached (based on VSIbase).

Stuck sessions

How many sessions got stuck during the test. This number should be zero. Stuck sessions indicate a problem during the test. As stuck sessions do not generate load, the VSImax score will be reduced by the amount of stuck sessions.

VSImax metrics.png

Average Response

Indicates the pure average response time for all the measurements taken taken when the indicated number of sessions on the X-axis were active.

Maximum Response

Indicates the maximum response time for all the measurements taken when the indicated number of sessions on the X-axis were active.

Minimum Response

Indicates the minimum response time for all the measurements taken when the indicated number of sessions on the X-axis were active.

VSI Index Average

Indicates the average value as calculated by Login VSI. The VSI Index Average differs from Average Response due to the fact that the Average Response is the pure average. The VSI Index Average applies certain statistical rules to the average to avoid any spikes from influencing the calculation.

VSImax legend.png

VSImax v4 Detailed

Description

Screenshot

The VSImax v4 detailed tab shows the individual measurements taken during a test in a combined graph. This graph shows the minimum, average and maximum response times for each individual measurement. There is also a Total metric that combines all of the metric into a single number. The minimum, average and maximum for this combined value is shown as well.

VSImax v4 detailed.png

The metrics are as follows.

Total:

The sum of all the metrics.

CPU:

Calculates a large array of random data.

FCTL:

File Copy Text Local. Copy a txt (plain text) file locally.

FCTS:

File Copy Text Share. Copy a txt (plain text) file locally.

NFP:

Notepad File Print. Open the print dialog in notepad.

NSLD:

Notepad Start/LoaD file. Start notepad by file type association, loading a text file.

NFO:

Notepad File Open. Open the file with the open dialog in notepad.

ZHC:

Zip High Compression. Zip a PST (Outlook Personal Folder) file, which is approximately 5 megabytes in size, using high compression.

ZLC:

Zip Low Compression. Zip a PST (Outlook Personal Folder) file, which is approximately 5 megabytes in size, using no compression.

IO:

Writes the Random CPU data array to disk.

UMEM:

Percentage of memory used by the sessions.

VSI Analyzer 4.1.x VSImaxv4 Detailed Metrics.JPG

The second graph compliments the VSImax v4 Detailed graph by showing the individual measurement response times correlated with the amount of time in seconds (instead of active sessions).

To switch between each version, use the dropdown box underneath the X-axis.

Detailed time.png

VSImax v4 Detailed Weighted

Description

Screenshot

The VSImax v4 detailed weighted tab shows the individual measurements taken during a test in a combined graph where a multiplier is applied. This graph shows the minimum, average and maximum response times for each individual measurement. In this graph you can clearly see which measurement has an impact on your environment.

VSImax v4 weighted.png

The metrics are as follows.

CPU:

Calculates a large array of random data.

FCTL:

File Copy Text Local. Copy a txt (plain text) file locally.

FCTS:

File Copy Text Share. Copy a txt (plain text) file locally.

NFP:

Notepad File Print. Open the print dialog in notepad.

NSLD:

Notepad Start/LoaD file. Start notepad by file type association, loading a text file.

NFO:

Notepad File Open. Open the file with the open dialog in notepad.

ZHC:

Zip High Compression. Zip a PST (Outlook Personal Folder) file, which is approximately 5 megabytes in size, using high compression.

ZLC:

Zip Low Compression. Zip a PST (Outlook Personal Folder) file, which is approximately 5 megabytes in size, using no compression.

IO:

Writes the Random CPU data array to disk.

UMEM:

Percentage of memory used by the sessions.

VSI Analyzer 4.1.x VSImaxv4 Detailed Weighted Metrics.jpg

The second graph compliments the VSImax v4 Detailed Weighted graph by showing the individual measurement response times correlated with the amount of time in seconds (instead of active sessions).

To switch between each version, use the dropdown box underneath the X-axis.

Weighted time.png

VSImax v4 Scatter

Description

Screenshot

The VSImax Scatter tab allows you to see the measurements based on time. Every tab before this tab shows the measurements consolidated by active session count. This tab allows you to see the data based on the time it was collected.

This is particularly useful to get an insight in trends after the sessions have finished logging on. The other tabs will consolidate all of the data collected after the last session has become active into a single data point. Namely the last active session count.

VSImax v4 scatter.png
Scatter time.png

UMEM, IO, CPU, ZLC, ZHC, NFP, NFO, NSLD

Description

Screenshot

These tabs show information for the individual measurements taken during the test. These specific measurements are zoomed in to because they are used to calculate VSIbase and VSImax v4 threshold.


They also display the baseline value for each individual measurement. This is the time it takes to complete this measurement during baseline measurements. These measurements are taken while the system is under no or very little load. It is used to see how the measurement trends from a system that isn’t under load.


All the individual measurement graphs are available in two versions. To switch between each version, simply open the dropdown box underneath the X-axis:


  • Detailed: Shows min/max/average response times correlated with active sessions (plus VSImax and Baseline average)
  • By time: Shows response times correlated with the amount of time (plus VSImax)


Note: NFP (Notepad File Print) is no longer used to calculate VSImax due to the amount of printers configured in an environment. The amount of printers could influence this number. If you have 15 printers it takes much longer to open the dialog compared to when you have 1 printer configured.

UMEM.png
UMEM by time.png
IO detailed.png
IO detailed by time.png
CPU detailed.png
CPU by time.png
ZLC detailed.png
ZHC detailed.png
NFP detailed.png
NFO detailed.png
NSLD detailed.png

AppStart

Description

Screenshot

The AppStart (application start times) tab gives you an (indication) overview of all the application start times taken during the test.

AppStart.png
AppStart time.png

LogonTimer

Description

Screenshot

The LogonTimer tab gives you an indication of the time it takes for a session to logon. The graph shows the trend of logon times during the test. The logon time is specified in milliseconds.

Please note that this is an indication of the logon time. VSI measures the time from the logon scripts running, shortly after group policy has been processed but before the shell has loaded (Windows Explorer), and the windows shell being loaded.

LogonTimer.png
LogonTimer by time.png

VSImax per Computer (ver4.1.25+)

Description

Screenshot

This graph shows the amount of active sessions per computer correlated with response times in milliseconds. Each computer is juxtaposed over one another. To see individual measurements, unmark the checkboxes underneath the graph as required. If there are any flat lines displayed, then this computer is not working properly and requires closer inspection.


VSImax per comp.png

Stuck Sessions Over Time (ver4.1.25+)

Description

Screenshot

This graph shows the amount of stuck sessions correlated with the current number of active sessions. If you see a stuck session indicator, please take further action as this session is not functioning properly.


Stuck sessions over time graph.png

Active Sessions Over Time (ver4.1.25+)

Description

Screenshot

This graph shows the amount of active sessions correlated with response times in milliseconds. Each blue dot represents an active session. If there are any large gaps in-between the blue dots, then some of these sessions did not activate properly.


Active session OT.png

VSImax v4 Data & Raw Data

Description

Screenshot

These tabs contain the raw and processed data used to create the graphs in the analyzer. You can use this data to run your own analysis on.

Raw Data columns explained:

  • Session_count = ActiveSessions: The number of sessions that were active when the the VSI timer started. The analyzer uses this to merge data.
  • Log_time: The moment that the data is written to the CSV file.
  • Total_Response: The total sum of the metrices from FCTS to UMEM.
  • LoopNumber: Counts from segment 1 to 4 which is 1 loop. The user starting in segment 3 and goes to 4 and 1 and 2 is counted as 2 loops.
  • CurrentSegment: The segment that the user is in at the time the session starts.
  • SegmentsRun: The total of segments the user has gone through.
VSImax v4 data.png
Raw data.png

Stuck Sessions / Failed to Become Active

Description

Screenshot

The Stuck sessions tab informs the user about any sessions that got stuck during the test. The list will show the:

  • Username
  • Computer
  • Date/time of last report



The Failed to become active tab informs the user about any sessions that did not activate during the test. The list will show the:

  • Username
  • Launcher
  • Date/time of last report


Note: Both of these tabs only appear in the Analyzer when the test contains stuck/inactive sessions.

VSI Analyzer 41 StuckSessions.JPG
Failed to become active.png

Importing External Data into the Analyzer

In our latest version of VSI you can import External Performance data like ESXtop or Perfmon in the Login VSI analyzer.

All metrics are down/up scaled to be in the range of 0-100, by finding the max value and multiplying accordingly. This feature was expanded to be able to hold multiple sources and multiple imports of metrics with ability to create new tabs or merge to the existing external data tabs. This also adds the remaining external data series to moving box, and adds the source column in the data grid view.


Description

Screenshot

Open the Analyzer located in {VSIshare}\_VSI_Binaries\Analyzer\Login VSI Analyzer.exe or run the Analyzer from the Management Console.

Manual open analyzer.PNG
Analyzer1.png

Specify the {VSIshare} and click save.

2013-11-04 01 52 21-Login VSI Analyzer.png

Open a result.

Open result.PNG
Analyzer before import.PNG

Click file – import – external data or shortcut key [ctrl+x]

Analyzer import.PNG

Open the (external) CSV file, in this example we use Perfmon CSV file that is collected during the same Login VSI test.

Import xternal data.PNG

Select the metrics you want to import.

Select the metrics.PNG

A new tab is opened with the title External Data

All metrics are down/up scaled to be in the range of 0-100, by finding the max value and multiplying accordingly. This feature was expanded to be able to hold multiple sources and multiple imports of metrics with ability to create new tabs or merge to the existing external data tabs. This also adds the remaining external data series to moving box, and adds the source column in the data grid view.

The blue steps line in the picture is representing the active sessions. Everything is recalculated to time in seconds, so the X-axis is time in seconds.

External Performance Data Perfmon.PNG

Additional Resources

Click on the links below to read related resources: