Code Generation for Niche Programming Languages with Large Language Models

Abstract

Code generation is a prominent use-case for Large Language Models (LLMs). Specialized LLMs such as CodeLlama or Codestral are trained on a large variety of programming languages and achieve a strong performance on coding tasks. However, when applied to less common programming languages which are not included in their pre-training corpus, their performance decreases. In this work, we describe an approach to integrate a LLM in the context of a coding copilot for specific applications where code shall be generated in a niche general-purpose programming language. We study the use of an intermediate domain-specific-language to limit the scope to the application-specific needs, and to enable the LLM to reliably generate code in such an application-specific scenario. We evaluate this method on two use-cases: Generating constraints in the context of product configuration using the MiniZinc constraint language, and generating test specifications in the context of railway infrastructure using the Balise Telegram Test Language. Our results show that defining an intermediate scope-limited DSL improves the performance of an LLM in our evaluated application-specific code generation scenarios. However, we can not guarantee that the presented performance results are generalizable to all scenarios.