autoiac (#294)

* autoiac

* update acceptance rate & title

source/_data/SymbioticLab.bib

@@ -1955,4 +1955,17 @@ @Article{mercury:arxiv24
 
 In this paper, we present Mercury, a QoS-aware tiered memory system that ensures predictable performance for coexisting memory-intensive applications with different SLOs. Mercury enables per-tier page reclamation for application-level resource management and uses a proactive admission control algorithm to satisfy SLOs via per-tier memory capacity allocation and intra- and inter-tier bandwidth interference mitigation. It reacts to dynamic requirement changes via real-time adaptation. Extensive evaluations show that Mercury improves application performance by up to 53.4% and 20.3% compared to TPP and Colloid, respectively.
   }
+}
+
+@InProceedings{autoiac:neurips24,
+  author    = {Patrick TJ Kon and Jiachen Liu and Yiming Qiu and Weijun Fan and Ting He and Lei Lin and Haoran Zhang and Owen M. Park and George Sajan Elengikal and Yuxin Kang and Ang Chen and Mosharaf Chowdhury and Myungjin Lee and Xinyu Wang},
+  title     = {{IaC-Eval}: A code generation benchmark for Infrastructure-as-Code programs},
+  year      = {2024},
+  publist_topic = {Systems + AI},
+  publist_confkey = {NeurIPS'24},
+  booktitle = {NeurIPS},
+  publist_link = {paper || autoiac-neurips24.pdf},
+  publist_link = {code || https://github.com/autoiac-project/iac-eval},
+  publist_abstract = {
+  Infrastructure-as-Code (IaC), an important component of cloud computing, allows the definition of cloud infrastructure in high-level programs. However, developing IaC programs is challenging, complicated by factors that include the burgeoning complexity of the cloud ecosystem (e.g., diversity of cloud services and workloads), and the relative scarcity of IaC-specific code examples and public repositories. While large language models (LLMs) have shown promise in general code generation and could potentially aid in IaC development, no benchmarks currently exist for evaluating their ability to generate IaC code. We present IaC-Eval, a first step in this research direction. IaC-Eval's dataset includes 458 human-curated scenarios covering a wide range of popular AWS services, at varying difficulty levels. Each scenario mainly comprises a natural language IaC problem description and an infrastructure intent specification. The former is fed as user input to the LLM, while the latter is a general notion used to verify if the generated IaC program conforms to the user's intent; by making explicit the problem's requirements that can encompass various cloud services, resources and internal infrastructure details. Our in-depth evaluation shows that contemporary LLMs perform poorly on IaC-Eval, with the top-performing model, GPT-4, obtaining a pass@1 accuracy of 19.36%. In contrast, it scores 86.6% on EvalPlus, a popular Python code generation benchmark, highlighting a need for advancements in this domain. We open-source the IaC-Eval dataset and evaluation framework at https://github.com/autoiac-project/iac-eval to enable future research on LLM-based IaC code generation.}
 }

source/publications/index.md

@@ -430,11 +430,19 @@ venues:
       name: ICLR 23 Workshop on Tackling Climate Change with Machine Learning
       date: 2023-05-04
       url: https://www.climatechange.ai/events/iclr2023
+  NeurIPS:
+    category: Conferences
+    occurrences:
+      - key: NeurIPS'24
+        name: The Thirty-eight Conference on Neural Information Processing Systems 
+        date: 2024-12-09
+        url: https://neurips.cc/Conferences/2024
+        acceptance: 25.8%
 {% endpublist %}
 
 ---
 
 {% note default %}
 #### Copyright notice
 The documents listed above have been provided by the contributing authors as a means to ensure timely dissemination of scholarly and technical work on a noncommercial basis. Copyright and all rights therein are maintained by the authors or by other copyright holders, notwithstanding that they have offered their works here electronically. It is understood that all persons copying this information will adhere to the terms and constraints invoked by each author’s copyright. These works may not be reposted without the explicit permission of the copyright holder.
-{% endnote %}
+{% endnote %}