| |
| TandemLaunch is considering building a benchmarking website to make it easier for researchers to compare the performance of their algorithm to others. As an Image Processing researcher, we would love your feedback about this idea. Please take the next 10 minutes to fill out this survey; your knowledge and insight into this area is valuable. |
| |
|
|
|
|
|
|
|
| |
| Generally speaking, are you interested in comparing your performance results to other researchers doing similar work? Why or why not? | | |
|
|
|
|
| |
| What challenges arise when you are trying to benchmarking your work? | | |
|
|
|
|
| |
| How do you select methods (other research) to benchmark against? How do you access benchmarking content (images)? | | |
|
|
|
|
| |
| Do you know of any 3rd party benchmarking services (e.g. moderated competitions)? What are they? | | |
|
|
|
|
| |
| Do you typically reproduce algorithms you’d like to benchmark from the paper or do researchers give you access to their algorithms? | | |
|
|
|
|
| |
| Do you typically make your code available to other researchers for benchmarking purposes? Why or why not? | | |
|
|
|
|
| |
| If you (the researcher) were to maintain a benchmarking service, how would you ensure that: |
| |
|
|
|
| |
| a. the content was relevant to current problems in image processing; and | | | | b. the results of best-in-class methods were available? | | |
|
|
|
|
| |
| What would you like a benchmarking service to look like? | | |
|
|
|
|
| |
| Let me paint you a picture: The benchmarking service would allow researchers to download test images, run their method, and upload their results. These results would be curated, where you could click through the original test image, along with the various processed images. Would you use it? How could it be better? | | |
|
|
|
|
| |
| If the 3rd party benchmarking service was run by a company that was interested in commercializing technologies, would that change your interest in using the service? | | |
|
|
|
|
| |
| Growth Questions: Our aim is to become the leading benchmarking service for image processing; ideally our service would become the standard source for benchmarking in the image processing domain. I am curious as to how you find out about new benchmarking opportunities: |
| |
|
|
|
| |
| How do you search for benchmarks? Conferences or newsletters or a network of colleagues? | | |
|
|
|
|
| |
| Do you refer colleagues or students to benchmarking services? What channel would you use to communicate it? | | |
|
|
|
|
| |
| Specific Benchmark Questions: We are interested building a benchmarking service for upscaling and denoising algorithms within image processing. |
| |
|
|
|
| |
| What image processing problems are worth having a benchmark for? | | |
|
|
|
|
| |
| What are the key factors that an upscaling and denoising benchemark would measure? | | |
|
|
|
|
| |
| As we want to allow researchers to benchmark their methods against the best methods available, what do you currently consider the best-in-class upscaling and denoising methods? What are the most cutting-edge techniques? | | |
|
|
|
|
Loading...
