|
|
Scientific testing
In this thread I am going to make some comments related to problems with countermeasure testing that appear here. A few other folks on this board have posted on these issues and I hope they will contribute. The purpose of this thread is not to criticize but to educate. I hope that comments here will be considered for the Guys of Lidar (GOL) II tests coming up. At the end of the thread I think we should try to come up with a consensus protocol for the guys to follow.
Variability and repeatability and statistics
Lets say I take a normal 6 sided die in each hand and roll them. My left hand throws a 6 and my right hand throws a 1. I conclude that my left hand is better at rolling dice than my right hand.
OK, I'm sure everyone sees that this is incorrect. However this is basically the approach used by both the GOL and SML. User Fallen has commented on this in the past.
Playing dice really can be biased. How do we know whether the result described above is due to chance or due to characteristics of the dice or my hands or what? The answer is called statistics. When Fallen (and I think someone else) critisized results on this basis, they were told that this was just amateur testing, not people writing a thesis, etc. I hope that by consideration of the example above that people can see that it doesn't matter how much you care about the results or how much effort you can apply to the test, in some cases you just don't know anything more after the test than you did before hand. Statistics is how you decide whether you know something.
To answer the question about the dice, we would have to throw the dice a large number of times. Then statistical tests can be run to see the likelihood that the differences are meaningful. Then, if they are, we need to do a different test to see if the difference is the dice or the hands (for example; in this case we could just switch the dice between the hands).
bias and blinding
One of the biggest things scientists struggle with is bias. We don't want to bias our results, and we try very hard to look at things objectively. However we know that it is human nature to interpret experience in light of existing beliefs. This simply cannot be reliably overcome.
Most or all of the GOL expressed opinions, some very strongly, about the results of their testing *before* the tests. If they had not done so they would most likely have been simply either less honest or less vocal, since I'm sure we all have opinions about which things will work better. However this made me very concerned over their results to the point where I was little interested in reading them when they came out.
The way this problem is overcome is by "blinding" (no relation to the laser jammer!) the experimenter to the state of the variable being tested. In drug trials neither the patient nor the doctor should know whether the patient is getting the drug or the placebo. This is "double blind" since neither knows. I won't belabor this explanation since probably most people are familiar with it.
In the case of countermeasure testing I suggest that cars being tested must have all countermeasures potentiallyinstalled and that the operator of the laser gun must not know which is being operated on any given run.
For active jammers, all should mounted but only one turned on.
For laser shield this is nearly invisible, so could be slipped in or out of a license plate frame.
For Veil this is more difficult. Perhaps it would be acceptable to Veil pieces of plastic and insert them in front of the plate and tape them over the headlights.
change 1 variable at a time
SML has been appropriately taken to task for egregiously failing to follow this simple rule of experiments. Some of their mistakes are so bad that in all seriousness they should be sued by the manufacturers. It is in some cases possible to extract meaningful conclusions when altering more than 1 variable at a time, but much more difficult. This is certainly NOT the way to do countermeasure testing.
The GOL have already determined that they do not want to test Veil in the upcoming test. I believe this is due to an understanding of this issue. Their previous tests are difficult to interpret since there are few valuable A/B comparisons presented. Only Outrun's car was fitted with all the jammers, so this is really the only useful one.
Here is a list of some variables that I can think of that should be held constant unless being deliberately changed as a test:
Car
Jammer mounting position
Driver
LIDAR operator
Weather
Time of day?
LIDAR unit
Car speed
Street/locations
Does this make it more difficult? Yes, obviously. As a first approach the tests should concentrate on a single car. Everyone should be helping with this test, not doing something on another car etc.
What kind of car? I can't answer this easily, but I would tend to suggest a "hard" car like a truck/SUV with chrome and/or light paint. On the one hand this will likley tend to make Veil and laser shield look worse. On the other hand, by making it harder for the jammers it may help reveal difference between them.
If time allows then multiple cars could be used, probably on different days, or different times of day (e.g. car A on the mornings of Day 1 and Day 2, and car B on the afternoons of Days 1 and 2 would keep the time of day constant with the car).
There should be a designated driver and LIDAR operator for the whole test.
By alternating semi-randomly the lidar guns and the countermeasures, subtle changes in the time of day (OK, this may be overkill, but it is easily done) can be compensated for.
I suggest that one person makes a list of conditions of lidar gun and countermeasure. This person should be removed from both the car preparation area and the LIDAR-use area. This person will communicate by radio or phone to each group independently which test to perform for each run. In this way the testers are blind but all combinations will be performed. Testing runs might look like this:
Gun Countermeasure
A B
A A
B A
A B
B B
B B
A A
B A
This is a simple example for 2 conditions of each with 2 runs for each combination.
Fraud
OK, this isn't really about science, but it would seem obvious that there is a risk in using countermeasures provided by manufacturers expressly for this test. One of many examples is how Consumer Reports only tests products bought in secret from retail establishments.
For the AntiLaser, this is not for sale so I can see allowing this as long as it is clearly designated as a prototype. If it performs poorly, there is little reason to wait for it to be released in the US. If it performs well, we still don't know if it is good, but we might be willing to wait for it to be released and for valid tests performed on consumer units.
I don't see how the modified Lidatek units can be used in this test. Speaking for myself there is no way I am going to spend hundreds of dollars over the cost of a blinder if I have any doubt in the results of the test. This needs to be iron clad. Yes, once again I am asking someone else to do more work. This is always an easy thing to do, but please consider your goals in do the test. Is it to get the test done, or are you trying to learn something and provide knowledge to fellow consumers? You will simply have to use stock Lidateks bought in the open market and it will take longer to do the test.
Perhaps Lidatek could be enlisted for suggestions in a way to defeat this problem on a stock unit. For example maybe a user really can reprogram the unit. Can turning it off and on skip the "recycling"? (This would presumably allow a temporary speed measurement and a modified version of testing would have to be adopted to compensate, for example, perhaps LIDAR would only be on in a few distance ranges and the car could be cycling the Lidatek in the other ranges.) Two independent Lidateks with a shutter to control which is exposed to laser? (OK, just kidding.)
Please understand I have no reason to doubt Lidatek and this post is not meant as a criticism of them in any way, just that any manufacturer is subject to doubt in such cases.
If you are talking about our test team, it is called:
( Unless you are just talking about Sethy...)
Whoops! Sorry, no offense intended. Posts fixed above.Originally Posted by jimbonzzz
If you are talking about our test team, it is called:
Lemme guess... the OP said "Gays of Lidar"?If you are talking about our test team, it is called:
( Unless you are just talking about Sethy...
We learned on the last test that weather and the sun affected the laser use in many ways. It is just simply to difficult to test everything in all circumstances.
However, we did try to make side notes that you will see on the officail GOL's website, what type of weather it was.
There are TONS of factors that are involved in pbtaining a speed on a vehicle with a laser. The shape of the front, the color, the size of the headlights, even how far the side veiw mirror sticks out.
On the upcoming test, I would like to test VEIL along side with some of these jammers if it does not JTG. But I am sure we will be documenting the weather along with it. We are also limiting the testing to 2 vehicles.
Yes, this is why you have to pick the variables that you think are important and change only them.
Yes. What this means is that people reading your test will have to be satisfied with a comparison of countermeasure performance on your car and not theirs. Test as much as you can on one or two cars. We hope results are similar on two different cars and then cross our fingers that we can extrapolate to our cars.
Documentation is obviously great, and you guys kick SML's butt in that account by all the videos you produce. However I'm sure you know that documentation of something like the weather only helps to know what data to throw out if the weather is changing. I think you need to focus on developing an approach to make as many A/B measurements as possible in a short period of time.
In fact, I think I will revise my suggestion above.
People can correct me if I'm wrong, but I think the big question here is which countermeasure (or combination thereof) work(s) best. Therefore I would do a set of runs with the same LIDAR all in a row. (If you can hide the LIDAR from the driver that would be nice but probably not very important.) So a revised schedule would be:Testing runs might look like this:
Gun Countermeasure
A B
A A
B A
A B
Gun A:
Nothing
Blinder
Lidatek
Blinder
AntiLaser
Lidatek
Nothing...
Lunch break
Gun B:
...
Posting Permissions
Reply With Quote
Bookmarks