The following are a collection of observations identified by repeated
and long term running of a set of Phoronix Test Suite based tests orchestrated
under Phoromatic. No attempt has been
made to identify the underlying cause of the regressions, but merely to
illustrate the value and opportunities of broad and consistent testing
done in an automated fashion.
The results were pulled from the respective trackers by adjusting the
number of days to 180 and regenerating the page.
Some notes on Terminology
Regression
The term regression within this page is used consistently to refer to
an unexpected change in behaviour. Until analysis is conducted, a
regression may caused by either an intended and unintended change.
Obviously an intended change is no longer a regression, even if
the impact of the change is negative against a particular measure.
Further, an intended change that is negative impact, may also
have a positive impact elsewhere. Whereby the first impact is not
a regression, but the second one is.
The term positive and negative regression will also be used within the
discussion below. A negative regression is your typical
regression that most people would consider a regression. A positive regression is a regression
that has a beneficial impact. However, a beneficial impact that
is unintended usually means either someone has done something amazing
and you want some more of that, or someone has created a shortcut that
may have introduced a performance impact at the expense of some other
safeguard in the system. A similar dichotomy can be found in the
financial markets where a company that either misses their guidance in
a negative or strongly positive manner will invariably be punished by
the stock market for lack of understanding of their business.
System vs Subsystem Testing
There is strong debate within the community of people reading about and
analyzing results about the relative value of generic (broad tests) and
targeted subsystem tests. Some observations will be made at the
bottom of the page, however this fundamentally a religious issue and
most likely won't be resolved.
Ubuntu Tracker
http://www.phoromatic.com/ubuntu-tracker.php
The Phoromatic Ubuntu Tracker runs a set of tests everyday and is
targeted to test the very latest Ubuntu development packages. The
latest Ubuntu packages from the development repository (currently,
10.04 Lucid) is pulled by the test systems every morning at 2AM CST.
After a reboot, the autonomous testing begins and takes most of the day
for this daily-system-tracker test suite to execute after which the
results are uploaded to this tracker page.
Find/Fix Regression
The three graphs below show a regression within Ubuntu that was
resolved. This resolution may have been directed (ie: someone saw
a regression and resolved them) or undirected (upstream fix,
serendipity, etc).
Regression
These four graphs show a regression that appears to have been
introduced around the 7th of August, 2010. This particular
regression has an interesting pattern. For Urban Terror, it was
what I call a positive regression (ie: the performance increased, vs
decreased). However, for OpenArena and the GtkComboBox, the
regression was negative. What makes this even more interesting is
that within the gtkcombobox test, there has been another positive
regression which has actually improved gtkcombobox to a better level
than the primary negative regression, potentially quantifying and
resolving the primary regression will improve the gtkcombobox scores
even further. Likewise with gtkradiobutton.
Sensitivity to change
In some scenarios, a test that is sensitive to multiple subsystems
serves as an important measure. Contrasting the two tests below,
you can see that there was a change that affected the GtkRadioButton
test in isolation of the common regression towards the end. From
an analysis perspective, the GtkComboBox test may have a strong
correlation to a particular subsystem and serves the maintainers of
those subsystems well. However the GtkRadioButton is sensitive to
multiple subsystems and consequently is limited in direct value to a
particular subsystem, but serves as a valuable sentinel of a change,
that warrants further investigation into the broad collection of
changes that occurred around that time, or a closer look at other
results for a behavior changes that happened in sync with this
regression.
Kernel Regressions
http://www.phoromatic.com/ubuntu-tracker.php
This is a test schedule that runs daily after the test system(s)
automatically install the latest mainline Linux kernel. It is
created by downloading the ubuntu daily kernel rebooting and running
the test.
Find+Fix
A nice example of a regression that was found and resolved in a small
wind of time.
Possible Open Regression
Around the 17th of September, there was a regression introduced to the
kernel that can be seen here. Again, what is generally a
negative regression for the first two graphs is a slightly positive
regression for the last graph (dbench - client count 1). By
observing the other portions of the graphs, you can see that there have
been around three extra performance changes affecting these tests since
April (4/21, 5/20, 7/3). The different tests are affected to a
different level.
Regression
Here is another regression that was most likely introduced by the 4/21
issue identified above. However in the case of PostMark, the cost
of the regression has most likely been recovered through what appears
to be incremental changes. Another find/fix regression on the
AtomZ530 is visible too.
Same Test, Different System Behavior
If multiple differing systems can be made available, hidden behaviors
can be exposed. In this particular case, a regression occurred
only on the 64bit Atom 330 system. With the richness of systems
available globally, having a limited number of test systems can hide
multiple regressions.
Change in Consistency
This particular test shows a very interesting pattern, the test
exhibited some variability up until the 4th of June, consistent up
until the 12th of August and then variable once more. Although
not a performance regression if you looked at the standard deviation
from test to test, you would see a marked decrease in the system
consistency in the middle with high variability towards the end.
Some Observations and Thoughts
This page is intended to provoke thoughts and analysis only. The
results haven't been verified directly, and as mentioned above there
has been no attempt to isolate the individual cause of the regressions.
Some of the thoughts that came out in the preparation of this
page include.
If you have any particular insight from either the results presented on
this page, please feel free to contact matthew@phoronix.com and
michael@phoronix.com to give us your insight.
Identification of Regressions
Most of the regressions on this page go unfound for a period, and some
have been resolved through serendipity or targeted fixes.
Investing effort up front in systems that allow visualization of
data provide strong insights into changes within a system.
Once a change has been identified, a simple bisection of your ordered
set of changes will usually identify the regressing subsystem.
Unfortunately, most development environments don't invest in the
level of infrastructure needed to make this effective.
Sentinel Tests
A sentinel test that is sensitive to multiple subsystems are extremely
valuable at a system level. The indicate that there is a problem
and that further analysis is needed. An alternate consideration
is that sentinels serve as Canaries in a Coal Mine. They are
variable and sensitive, but a marked change in behavior warrants
immediate investigation.
Variability of Results
A test that provides variable results is generally considered bad.
However even these tests may be multi-modal (potentially a bad
thing), or may have an intrinsic variability that can still be
normalized through successive runs. Control chart theory provides
a good opportunity through Nelson Rules to extract value from these
tests. Even observing the changes in variability from run to run
can provide a high level of value when analysing the system.
Multiplicity of Systems and Tests
As can be seen from the analysis above, just a small number of systems
and a large number of tests can show multiple trends and behaviors that
provide insight into what is happening on a system as the code
changes.
Same Test Different Systems
Extending the observation above, with a larger number of systems you
will find a larger number of regressions. This is particularly
interesting if you consider ways of doing crowd-sourced testing for
distributions or kernels.
Matthew Tippett
matthew@phoronix.com
2010-10-01
