Trustworthy Natural Language Generation with Communicative Goals

While recent work in large language models have made natural language generation fluent, these models suffer from content “hallucination”, where model-generated statements are not attributable to sources in communicative scenarios (e.g. summarization, question answering and responses in dialogue systems). Furthermore, evaluation of generation systems remains challenging: (1) human evaluation studies are not reproducible and there is a lack of common benchmarking for diverse tasks, and (2) popular automatic evaluation methods such as BLEU and ROUGE correlate poorly with human judgments. In this talk, I will present a body of work that attempts to solve the above problems for a variety of natural language generation problems and showcase a vision towards future work in this area.

Bio

Dipanjan Das is a Principal Scientist at Google Research, based in New York City. He leads teams of researchers distributed between New York, London, Berlin, Amsterdam and Seattle focused on language technologies. Dipanjan’s research focus is on natural language generation, grounding and attribution in generative models and model interpretability. Before joining Google, Dipanjan completed a Ph.D. from the Language Technologies Institute, School of Computer Science at Carnegie Mellon University in 2012. In 2005, he completed a B.Tech. in Computer Science and Engineering from IIT Kharagpur. His work has received paper awards at the ACL and EMNLP conferences; he serves as action editor for the Transactions of the Association for Computational Linguistics and the ACL Rolling Review and as senior committee member at various *ACL and machine learning venues.

Back

Dataflow for Dialog

In addition to querying and updating a knowledge base, a dialogue system must represent the state of the dialogue. We argue that dialogue state is best described by a dataflow graph of values, connected by computations relating those values to one another and to the knowledge base. I’ll motivate the dataflow formalism and present some of our work that maps from text to dataflow (understanding) and dataflow to text (generation). Both understanding and generation may pull new information into the dataflow graph. One technique that works well for both is to generate from large language models constrained by task-dependent grammars.

This is work by the team at Microsoft Semantic Machines.

Bio

Jason Eisner is Professor of Computer Science at Johns Hopkins University, as well as Director of Research at Microsoft Semantic Machines. He is a Fellow of the Association for Computational Linguistics. At Johns Hopkins, he is also affiliated with the Center for Language and Speech Processing, the Mathematical Institute for Data Science, and the Cognitive Science Department. His goal is to develop the probabilistic modeling, inference, and learning techniques needed for a unified model of all kinds of linguistic structure. His 150+ papers have presented various algorithms for parsing, machine translation, and weighted finite-state machines; formalizations, algorithms, theorems, and empirical results in computational phonology; and unsupervised or semi-supervised learning methods for syntax, morphology, and word-sense disambiguation. He is also the lead designer of Dyna, a new declarative programming language that provides an infrastructure for AI research. He has received two school-wide awards for excellence in teaching, as well as recent Best Paper Awards at ACL 2017, EMNLP 2019, and NAACL 2021 and an Outstanding Paper Award at ACL 2022.

Back

Rethinking benchmarking in AI: Evaluation-as-a-Service and Dynamic Adversarial Data Collection

The current benchmarking paradigm in AI has many issues: benchmarks saturate quickly, are susceptible to overfitting, contain exploitable annotator artifacts, have unclear or imperfect evaluation metrics, and do not measure what we really care about. I will talk about my work on trying to rethink the way we do benchmarking in AI. First, I’ll go into our work at Hugging Face on establishing better best practices for the comprehensive evaluation of data and models, through the open source Evaluate library and the Evaluation on the Hub project. Second, I’ll talk about Dynabench, a research platform that facilitates human and model in the loop data collection and evaluation, as well as the progress the team has been making in exploring the dynamic adversarial data collection paradigm.

Bio

Douwe Kiela is the Head of Research at Hugging Face. Before, he was a Research Scientist at Facebook AI Research. His current research interests lie in developing better models for (grounded, multi-agent) language understanding and better tools for evaluation and benchmarking. He received his PhD and MPhil from the University of Cambridge. Before that, he did a BSc in Liberal Arts & Sciences at Utrecht University with a double major in Cognitive Artificial Intelligence and Philosophy; and an MSc in Logic at the University of Amsterdam’s ILLC.

Back

Temporal and Multilingual Knowledge Graphs

The world is dynamic and Knowledge Graphs (KG) should reflect this time-varying nature. Temporal KGs provide a mechanism to represent this dynamism through time-scoped edges. While question answering over traditional KGs have received quite a bit of attention, QA over Temporal KGs is still in early stages. In the first part of the talk, I shall present the challenges and approaches for Temporal KGQA. In the second part of the talk, I shall outline the challenges with severe sparsity in KGs when going beyond English. I shall present the need along with an approach to link KG with text in a cross-lingual manner. I shall conclude the talk with open problems in these areas.

Bio

Partha is a Researcher at Google Research, Bangalore where he leads the NLP group. He is also a faculty member (on leave) at IISc Bangalore. Partha founded KENOME, an enterprise Knowledge graph company with the mission to help enterprises make sense of unstructured data. Previously, Partha was a Postdoctoral Fellow in the Machine Learning Department at Carnegie Mellon University, working with Tom Mitchell on the NELL project. Partha is broadly interested in NLP and Machine Learning, and in making language technologies more inclusive. Partha is a recipient of several awards, including an Outstanding Paper Award at ACL 2019. He is a co-author of a book on Graph-based Semi-Supervised Learning. Homepage: https://parthatalukdar.github.io

Back

Keeping LMs in sync with the real world

Our world is open-ended, non-stationary and constantly evolving; thus what we talk about and how we talk about it changes over time. This inherent dynamic nature of language comes in stark contrast to the established static paradigm of NLP. This staticness has led over the years to a number of peculiarities; our models are “stuck” to the time they were trained, our systems are not designed to be easily adaptive, and our benchmarks further perpetuate this vicious circle.

In this talk, I will describe our set of experiments and results on taking current state-of-the-art models when placing them in the realistic scenario of making predictions from beyond the models’ training period. I will talk about new streaming language modeling and question answering benchmarks created for this purpose and show how current Transformer-based models perform worse in this realistic setup. I will then present and contrast different ways to keep our models in sync with the world as new data arrive in our stream, either by continually updating the (monolithic) models’ parameters or by leveraging semi-parametric approaches that flexibly store and use knowledge in a modular way. Finally, towards more open-ended models that do not lag behind the ever-changing world, I will talk about a new family of models, internet-augmented models, that leverage the power of commercial search engines as a source of factual and up-to-date knowledge.

Bio

Angeliki Lazaridou is a Staff Research Scientist at DeepMind. She obtained her PhD from the University of Trento, where she worked on predictive grounded language learning. At DeepMind, she has worked on interactive methods for language learning that rely on multi-agent communication as a means of alleviating the use of supervised language data. Currently, she spends most of her time thinking and working on how to best make language models be in sync with the complex and ever-evolving world.

Back

Anti-vaccination arguments: a conceptual taxonomy and a machine-learning model

The proliferation of anti-vaccination arguments is a threat to the success of many immunisation programmes. Effective rebuttal of contrarian arguments requires an approach that goes beyond addressing flaws in the arguments, by also considering the attitude roots—that is, the underlying psychological attributes driving a person’s belief—of opposition to vaccines. Through a preregistered systematic literature review and thematic analysis of anti-vaccination arguments, we developed a hierarchical taxonomy that relates common arguments and themes to 11 attitude roots that explain why an individual might express opposition to vaccination. We further validated our taxonomy on COVID-19 anti-vaccination misinformation, through a combination of human coding and machine learning using natural language processing algorithms. Overall, the taxonomy serves as a theoretical framework to link expressed opposition of vaccines to their underlying psychological processes. This enables future work to develop targeted rebuttals and other interventions that address the underlying motives of anti-vaccination arguments.

Bio

Professor Stephan Lewandowsky is a cognitive scientist at the University of Bristol and the recipient of numerous awards and honours, including a Discovery Outstanding Researcher Award from the Australian Research Council, a Wolfson Research Merit Fellowship from the Royal Society, and a Humboldt Research Award from the Humboldt Foundation in Germany. He is a Fellow of the Academy of Social Science (UK) and a Fellow of the Association of Psychological Science. He was appointed a fellow of the Committee for Skeptical Inquiry for his commitment to science, rational inquiry and public education. He also holds a Guest Professorship at the University of Potsdam in Germany.

Back

Project Alexandria in Viva Topics: AKBC in practice

At AKBC 2019, we presented Project Alexandria as a solution to inferring a knowledge base completely automatically from unstructured data. Since then, we have built Alexandria into the heart of a new Microsoft product called Viva Topics, launched last year. Viva Topics automatically constructs a knowledge base from an organization’s documents and intranet pages, and surfaces it across a wide range of Microsoft applications including SharePoint, Teams, Outlook and more.

In this talk, I will explore all the challenges the Alexandria team has encountered in transitioning from an academic project to a high quality, scalable production system . These challenges include enabling continual expansion of the knowledge base as new data arrives, allowing people to edit the knowledge base as it is being constructed, bringing together knowledge from a wide variety of structured and unstructured sources, and integrating knowledge across languages. Today, the Alexandria system can discover hundreds of types of entities from tens of millions of documents whilst keeping these entities updated, and integrated with human edits, as the organization evolves. The result is a solution which brings organizational knowledge effortlessly to the fingertips of users across the Microsoft ecosystem.

Bio

Yordan is a research engineering manager at Microsoft Research in Cambridge, UK where he leads a team of talented machine learning scientists and developers. His research interests revolve around probabilistic modelling and knowledge discovery, and his passion is taking research projects to real-world applications. Examples of the latter include Microsoft’s cloud-based machine learning platform Azure ML, the predecessor of focused inbox – Clutter, the TrueSkill rating system used for online player matchmaking, and what has been Yordan’s focus lately – Viva Topics, which automatically organizes enterprise knowledge into shared topics.

Back

NLP for Science: Advances and Challenges

With over one million papers added every year to the PubMed biomedical index alone — the explosion of scholarly knowledge presents tremendous opportunities for accelerating research across the sciences. However, the complexity of scientific literature presents formidable challenges for existing AI and NLP technologies, limiting our ability to tap into this vast treasure trove of information. In this talk, I will present our recent work toward helping researchers and clinicians make use of knowledge embedded in the literature. I will highlight methods that help discover new directions and solutions to problems, generate hypotheses, make predictions and decisions, and build connections across different ideas and areas. This includes models that predict clinical outcomes of hospital patients and new links in biomedical knowledge graphs, a novel scientific information retrieval method that achieves state-of-the-art results, and challenging new datasets for scientific IE. I will also present recent exploratory search and recommendation engines we have developed, and discuss important challenges and opportunities toward AI-powered augmentation of human scientists.

Bio

Tom Hope is a new assistant professor at The Hebrew University of Jerusalem’s School of Computer Science and Engineering, and a visiting research scientist at The Allen Institute for AI (AI2). Tom was awarded the 2022 Azrieli Early Career Faculty Fellowship which is given to eight scientists across all fields of study. Prior to that he was a postdoctoral researcher at AI2 and the University of Washington (UW), working with Daniel Weld and Eric Horvitz. His work has received four best paper awards, appeared in top AI, NLP and HCI venues, and received coverage from Nature and Science.

Back

Data driven policy for pandemic response: data, knowledge, and action well outside of the ivory tower

The “central dogma” of informatics, it has been said, is turning data into information and information into knowledge. Some take this framework further- knowledge informs action. Moreover, in the Learning Health System model, action begets more real-world data, which can be converted into information, etc. In 2019, North Carolina’s Department of Health and Human Services created the NC DHHS Data Office in recognition of the data’s importance, and potential, to drive policy and decision making. With vast troves of data collected across siloed government services- locally, regionally, and nationally- there is great potential to turn that data into knowledge, and knowledge into action. What became clear, both before and after the start of the pandemic, was that there are many steps to climb in the real world before we can enter this virtuous cycle. The speaker will describe the view from the trenches of US state government pandemic response, early pandemic data strategy, and how machine learning was used, and not used, in this real world use case.

Bio

Jessie Tenenbaum serves as the Chief Data Officer (CDO) for DHHS, assisting the Department in developing a strategy to use information to inform and evaluate policy and improve the health and well-being of residents of North Carolina. Prior to taking on the role of CDO, Tenenbaum was a founding faculty member of the Division of Translational Biomedical Informatics within Duke University’s Department of Biostatistics and Bioinformatics. Her research applies expertise in data standards and electronic health records to stratify mental health disorders to enable precision medicine. She is also interested in ethical, legal and social issues around big data and precision medicine. Prior to her faculty role, Tenenbaum was Associate Director for Bioinformatics for the Duke Translational Medicine Institute, with a focus on data standards and enterprise data warehousing.

Back

Overcoming spurious correlations in natural language understanding

Model robustness and spurious correlations have received increasing attention in the NLP community from method development to model interpretation. The core question is how to build models that are “right for the right reasons” and not taking shortcuts. In this talk, I will go through both success stories and unexpected failures in the quest for robust natural language understanding. Then, I will come back to the very basic question of what spurious features mean in NLP tasks, and argue that different types of spurious features come with unique challenges and require different treatment. I will conclude with lessons learned and thoughts for the future.

Bio

He He is an Assistant Professor in the Department of Computer Science and the Center for Data Science at New York University. She obtained her PhD in Computer Science at the University of Maryland, College Park. Before joining NYU, she spent a year at AWS AI and was a post-doc at Stanford University before that. She is interested in building robust and trustworthy NLP systems in human-centered settings. Her recent research focus includes robust language understanding, collaborative text generation, and understanding capabilities and issues of large language models.

Back