<html><head><meta http-equiv="Content-Type" content="text/html; charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class=""><h3 class="" style="orphans: 2; widows: 2; background-color: rgb(255, 255, 255); box-sizing: border-box; line-height: 1.1; margin-top: 0px; margin-bottom: 20px; font-variant-ligatures: normal;"><span style="font-size: 14px;" class=""><span style="color: rgb(34, 34, 34); font-family: arial, sans-serif; font-variant-ligatures: normal;" class="">BELIV 2018: Evaluation and Beyond - Methodological Approaches for Visualization</span><br style="color: rgb(34, 34, 34); font-family: arial, sans-serif; font-variant-ligatures: normal; font-weight: 400;" class=""><br style="color: rgb(34, 34, 34); font-family: arial, sans-serif; font-variant-ligatures: normal; font-weight: 400;" class=""><span style="color: rgb(34, 34, 34); font-family: arial, sans-serif; font-variant-ligatures: normal;" class="">Date:</span></span><span style="color: rgb(34, 34, 34); font-family: arial, sans-serif; font-size: small; font-variant-ligatures: normal;" class=""> </span><span style="color: rgb(34, 34, 34); font-family: arial, sans-serif; font-size: small; font-variant-ligatures: normal; font-weight: 400;" class="">October 21st, 2018. Berlin</span><br style="color: rgb(34, 34, 34); font-family: arial, sans-serif; font-size: small; font-variant-ligatures: normal; font-weight: 400;" class=""><br style="color: rgb(34, 34, 34); font-family: arial, sans-serif; font-size: small; font-variant-ligatures: normal; font-weight: 400;" class=""><span style="color: rgb(34, 34, 34); font-family: arial, sans-serif; font-size: small; font-variant-ligatures: normal; font-weight: 400;" class="">We invite contributions to BELIV 2018, the international forum to broadly discuss research methods in visualization. Our discussions span from a novel and not-yet-fully established evaluation methods for visualization tools and techniques to methods that more generally establish the validity and scope of acquired visualization knowledge. The 2018 call for papers includes an expanded scope from years past to reflect this newly broadened purview.</span><br style="color: rgb(34, 34, 34); font-family: arial, sans-serif; font-size: small; font-variant-ligatures: normal; font-weight: 400;" class=""><br style="color: rgb(34, 34, 34); font-family: arial, sans-serif; font-size: small; font-variant-ligatures: normal; font-weight: 400;" class=""><span style="color: rgb(34, 34, 34); font-family: arial, sans-serif; font-size: small; font-variant-ligatures: normal; font-weight: 400;" class="">New this year: Complementary to the broader scope on research methods more generally, 2018 will also include a focus topic—replication practices in visualization—which will shape half of the workshop. This call for papers thus also includes specific issues related to both the replication of quantitative and qualitative work, as well as opportunities for reflective assessment of unsuccessful replication efforts.</span><br style="color: rgb(34, 34, 34); font-family: arial, sans-serif; font-size: small; font-variant-ligatures: normal; font-weight: 400;" class=""><br style="color: rgb(34, 34, 34); font-family: arial, sans-serif; font-size: small; font-variant-ligatures: normal; font-weight: 400;" class=""><span style="color: rgb(34, 34, 34); font-family: arial, sans-serif; font-variant-ligatures: normal; font-size: 14px;" class="">More info:</span><span style="color: rgb(34, 34, 34); font-family: arial, sans-serif; font-size: small; font-variant-ligatures: normal; font-weight: 400;" class=""> <a href="https://beliv-workshop.github.io/" class="">https://beliv-workshop.github.io/</a></span></h3><h3 class="" style="orphans: 2; widows: 2; background-color: rgb(255, 255, 255); box-sizing: border-box; line-height: 1.1; margin-top: 0px; margin-bottom: 20px; font-variant-ligatures: normal;"><span style="color: rgb(34, 34, 34); font-family: arial, sans-serif; font-variant-ligatures: normal; font-size: 14px;" class="">**Topics of Interest**</span><br style="color: rgb(34, 34, 34); font-family: arial, sans-serif; font-size: small; font-variant-ligatures: normal; font-weight: 400;" class=""><span style="color: rgb(34, 34, 34); font-family: arial, sans-serif; font-size: small; font-variant-ligatures: normal; font-weight: 400;" class="">Best practices for supporting replication</span><br style="color: rgb(34, 34, 34); font-family: arial, sans-serif; font-size: small; font-variant-ligatures: normal; font-weight: 400;" class=""><span style="color: rgb(34, 34, 34); font-family: arial, sans-serif; font-size: small; font-variant-ligatures: normal; font-weight: 400;" class="">How do we, or how should we build on other’s work</span><br style="color: rgb(34, 34, 34); font-family: arial, sans-serif; font-size: small; font-variant-ligatures: normal; font-weight: 400;" class=""><span style="color: rgb(34, 34, 34); font-family: arial, sans-serif; font-size: small; font-variant-ligatures: normal; font-weight: 400;" class="">The value of replication in VIS</span><br style="color: rgb(34, 34, 34); font-family: arial, sans-serif; font-size: small; font-variant-ligatures: normal; font-weight: 400;" class=""><span style="color: rgb(34, 34, 34); font-family: arial, sans-serif; font-size: small; font-variant-ligatures: normal; font-weight: 400;" class="">Alternatives when replication is not possible</span><br style="color: rgb(34, 34, 34); font-family: arial, sans-serif; font-size: small; font-variant-ligatures: normal; font-weight: 400;" class=""><span style="color: rgb(34, 34, 34); font-family: arial, sans-serif; font-size: small; font-variant-ligatures: normal; font-weight: 400;" class="">Discussion of methods that support replicability in VIS, e.g. pre-registration</span><br style="color: rgb(34, 34, 34); font-family: arial, sans-serif; font-size: small; font-variant-ligatures: normal; font-weight: 400;" class=""><span style="color: rgb(34, 34, 34); font-family: arial, sans-serif; font-size: small; font-variant-ligatures: normal; font-weight: 400;" class="">Reflection on replication attempts, successes, and failures</span><br style="color: rgb(34, 34, 34); font-family: arial, sans-serif; font-size: small; font-variant-ligatures: normal; font-weight: 400;" class=""><span style="color: rgb(34, 34, 34); font-family: arial, sans-serif; font-size: small; font-variant-ligatures: normal; font-weight: 400;" class="">Dangers of replication</span><br style="color: rgb(34, 34, 34); font-family: arial, sans-serif; font-size: small; font-variant-ligatures: normal; font-weight: 400;" class=""><span style="color: rgb(34, 34, 34); font-family: arial, sans-serif; font-size: small; font-variant-ligatures: normal; font-weight: 400;" class="">Reviewing replicated work</span><br style="color: rgb(34, 34, 34); font-family: arial, sans-serif; font-size: small; font-variant-ligatures: normal; font-weight: 400;" class=""><span style="color: rgb(34, 34, 34); font-family: arial, sans-serif; font-size: small; font-variant-ligatures: normal; font-weight: 400;" class="">Meta-analysis of replication in visualization</span><br style="color: rgb(34, 34, 34); font-family: arial, sans-serif; font-size: small; font-variant-ligatures: normal; font-weight: 400;" class=""><span style="color: rgb(34, 34, 34); font-family: arial, sans-serif; font-size: small; font-variant-ligatures: normal; font-weight: 400;" class="">Evaluation in the visualization development lifecycle</span><br style="color: rgb(34, 34, 34); font-family: arial, sans-serif; font-size: small; font-variant-ligatures: normal; font-weight: 400;" class=""><span style="color: rgb(34, 34, 34); font-family: arial, sans-serif; font-size: small; font-variant-ligatures: normal; font-weight: 400;" class="">Novel evaluation methods</span><br style="color: rgb(34, 34, 34); font-family: arial, sans-serif; font-size: small; font-variant-ligatures: normal; font-weight: 400;" class=""><span style="color: rgb(34, 34, 34); font-family: arial, sans-serif; font-size: small; font-variant-ligatures: normal; font-weight: 400;" class="">Revisiting or extending existing research methods</span><br style="color: rgb(34, 34, 34); font-family: arial, sans-serif; font-size: small; font-variant-ligatures: normal; font-weight: 400;" class=""><span style="color: rgb(34, 34, 34); font-family: arial, sans-serif; font-size: small; font-variant-ligatures: normal; font-weight: 400;" class="">Synthetic data sets and benchmarks</span><br style="color: rgb(34, 34, 34); font-family: arial, sans-serif; font-size: small; font-variant-ligatures: normal; font-weight: 400;" class=""><span style="color: rgb(34, 34, 34); font-family: arial, sans-serif; font-size: small; font-variant-ligatures: normal; font-weight: 400;" class="">Taxonomy of tasks</span><br style="color: rgb(34, 34, 34); font-family: arial, sans-serif; font-size: small; font-variant-ligatures: normal; font-weight: 400;" class=""><span style="color: rgb(34, 34, 34); font-family: arial, sans-serif; font-size: small; font-variant-ligatures: normal; font-weight: 400;" class="">Benchmark development and repositories</span><br style="color: rgb(34, 34, 34); font-family: arial, sans-serif; font-size: small; font-variant-ligatures: normal; font-weight: 400;" class=""><span style="color: rgb(34, 34, 34); font-family: arial, sans-serif; font-size: small; font-variant-ligatures: normal; font-weight: 400;" class="">How to turn empirical research into heuristics or guidelines</span><br style="color: rgb(34, 34, 34); font-family: arial, sans-serif; font-size: small; font-variant-ligatures: normal; font-weight: 400;" class=""><span style="color: rgb(34, 34, 34); font-family: arial, sans-serif; font-size: small; font-variant-ligatures: normal; font-weight: 400;" class="">Reflections on evaluation practice in visualization in the past, present, and future</span><br style="color: rgb(34, 34, 34); font-family: arial, sans-serif; font-size: small; font-variant-ligatures: normal; font-weight: 400;" class=""><span style="color: rgb(34, 34, 34); font-family: arial, sans-serif; font-size: small; font-variant-ligatures: normal; font-weight: 400;" class="">Validity of studies and research results</span><br style="color: rgb(34, 34, 34); font-family: arial, sans-serif; font-size: small; font-variant-ligatures: normal; font-weight: 400;" class=""><span style="color: rgb(34, 34, 34); font-family: arial, sans-serif; font-size: small; font-variant-ligatures: normal; font-weight: 400;" class="">Evaluation criteria and methods for the paper reviewing process</span><br style="color: rgb(34, 34, 34); font-family: arial, sans-serif; font-size: small; font-variant-ligatures: normal; font-weight: 400;" class=""><span style="color: rgb(34, 34, 34); font-family: arial, sans-serif; font-size: small; font-variant-ligatures: normal; font-weight: 400;" class="">Design methods and methodologies</span><br style="color: rgb(34, 34, 34); font-family: arial, sans-serif; font-size: small; font-variant-ligatures: normal; font-weight: 400;" class=""><span style="color: rgb(34, 34, 34); font-family: arial, sans-serif; font-size: small; font-variant-ligatures: normal; font-weight: 400;" class="">Methodological frameworks and theories</span><br style="color: rgb(34, 34, 34); font-family: arial, sans-serif; font-size: small; font-variant-ligatures: normal; font-weight: 400;" class=""><span style="color: rgb(34, 34, 34); font-family: arial, sans-serif; font-size: small; font-variant-ligatures: normal; font-weight: 400;" class="">Rigor in the evaluation process</span><br style="color: rgb(34, 34, 34); font-family: arial, sans-serif; font-size: small; font-variant-ligatures: normal; font-weight: 400;" class=""><span style="color: rgb(34, 34, 34); font-family: arial, sans-serif; font-size: small; font-variant-ligatures: normal; font-weight: 400;" class="">How to best communicate the results of empirical research</span><br style="color: rgb(34, 34, 34); font-family: arial, sans-serif; font-size: small; font-variant-ligatures: normal; font-weight: 400;" class=""><br style="color: rgb(34, 34, 34); font-family: arial, sans-serif; font-size: small; font-variant-ligatures: normal; font-weight: 400;" class=""><span style="color: rgb(34, 34, 34); font-family: arial, sans-serif; font-variant-ligatures: normal; font-size: 14px;" class="">**Important Dates**</span><br style="color: rgb(34, 34, 34); font-family: arial, sans-serif; font-size: small; font-variant-ligatures: normal; font-weight: 400;" class=""><span style="color: rgb(34, 34, 34); font-family: arial, sans-serif; font-size: small; font-variant-ligatures: normal; font-weight: 400;" class="">Jun 30, 2018: Paper submission due</span><br style="color: rgb(34, 34, 34); font-family: arial, sans-serif; font-size: small; font-variant-ligatures: normal; font-weight: 400;" class=""><span style="color: rgb(34, 34, 34); font-family: arial, sans-serif; font-size: small; font-variant-ligatures: normal; font-weight: 400;" class="">Aug 1, 2018: First notification</span><br style="color: rgb(34, 34, 34); font-family: arial, sans-serif; font-size: small; font-variant-ligatures: normal; font-weight: 400;" class=""><span style="color: rgb(34, 34, 34); font-family: arial, sans-serif; font-size: small; font-variant-ligatures: normal; font-weight: 400;" class="">Sep 1, 2018: Revisions due</span><br style="color: rgb(34, 34, 34); font-family: arial, sans-serif; font-size: small; font-variant-ligatures: normal; font-weight: 400;" class=""><span style="color: rgb(34, 34, 34); font-family: arial, sans-serif; font-size: small; font-variant-ligatures: normal; font-weight: 400;" class="">Sept 15, 2018: Final notification</span><br style="color: rgb(34, 34, 34); font-family: arial, sans-serif; font-size: small; font-variant-ligatures: normal; font-weight: 400;" class=""><span style="color: rgb(34, 34, 34); font-family: arial, sans-serif; font-size: small; font-variant-ligatures: normal; font-weight: 400;" class="">Oct 21, 2018: BELIV workshop</span><br style="color: rgb(34, 34, 34); font-family: arial, sans-serif; font-size: small; font-variant-ligatures: normal; font-weight: 400;" class=""></h3><div class=""><br class=""></div></body></html>