My research focus on how to adopt LLMs to a new domain. This is very costly on resources: the traditional training paradigm of pretrain and fine-tuning is not an option. The reason is that contemporary models containing billions of parameters require millions of USD to pretrain. Finetuning could cost the same amount of resources. The prevailing solution is to do in-context Learning instead of fine-tuning. My work is focusing on how to do this with reduced resources including trainable prompts design and applying parameter efficient module on LLMs. Prompts as short instructions for LLMs provide more information for the downstream task. Parameter efficient modules such as LoRA and Adapter are a small set of networks added aside of original LLMs. These methods may require a smaller downstream dataset (hundreds of items) and retraining 2-5% of the original model parameters, and reach a similar performance of finetuning.