⚡ Bolt: Optimize PromptOptimizer thread pool usage (#147)

* ⚡ Bolt: Reuse ThreadPoolExecutor in PromptOptimizer

💡 What:
Initialized `ThreadPoolExecutor` in `PromptOptimizer.__init__` and reused it in `evaluate_prompt`.

🎯 Why:
The previous implementation created a new `ThreadPoolExecutor` for every call to `evaluate_prompt`. Since `evaluate_prompt` is called repeatedly inside the `optimize` loop (and for every variation), this caused significant overhead from repeatedly creating and destroying thread pools.

📊 Impact:
Benchmark showed a reduction in execution time from ~5.36s to ~3.76s (~30% improvement) for 500 iterations with a mocked LLM.

🔬 Measurement:
Ran a benchmark script executing `evaluate_prompt` 500 times.
Before: 5.36s
After: 3.76s

* ⚡ Bolt: Reuse ThreadPoolExecutor in PromptOptimizer

💡 What:
Initialized `ThreadPoolExecutor` in `PromptOptimizer.__init__` and reused it in `evaluate_prompt`. Added a `shutdown` method for proper cleanup.

🎯 Why:
The previous implementation created a new `ThreadPoolExecutor` for every call to `evaluate_prompt`. Since `evaluate_prompt` is called repeatedly inside the `optimize` loop (and for every variation), this caused significant overhead from repeatedly creating and destroying thread pools.

📊 Impact:
Benchmark showed a reduction in execution time from ~5.36s to ~3.76s (~30% improvement) for 500 iterations with a mocked LLM.

🔬 Measurement:
Ran a benchmark script executing `evaluate_prompt` 500 times.
Before: 5.36s
After: 3.76s

* ⚡ Bolt: Reuse ThreadPoolExecutor in PromptOptimizer

💡 What:
Initialized `ThreadPoolExecutor` in `PromptOptimizer.__init__` and reused it in `evaluate_prompt`. Added a `shutdown` method and wrapped execution in `try...finally` for proper resource management.

🎯 Why:
The previous implementation created a new `ThreadPoolExecutor` for every call to `evaluate_prompt`. Since `evaluate_prompt` is called repeatedly inside the `optimize` loop (and for every variation), this caused significant overhead from repeatedly creating and destroying thread pools.

📊 Impact:
Benchmark showed a reduction in execution time from ~5.36s to ~3.76s (~30% improvement) for 500 iterations with a mocked LLM.

🔬 Measurement:
Ran a benchmark script executing `evaluate_prompt` 500 times.
Before: 5.36s
After: 3.76s

---------

Co-authored-by: google-labs-jules[bot] <161369871+google-labs-jules[bot]@users.noreply.github.com>

.jules/bolt.md

@@ -0,0 +1,3 @@
+## 2024-05-23 - Thread Pool Overhead in Iterative Tasks
+**Learning:** Recreating `ThreadPoolExecutor` inside a frequently called loop (like an optimization loop) introduces significant overhead, especially when the individual tasks are short-lived.
+**Action:** Initialize `ThreadPoolExecutor` once in the class `__init__` and reuse it across method calls to amortize the setup cost.

plugins/llm-application-dev/skills/prompt-engineering-patterns/scripts/optimize-prompt.py

@@ -25,6 +25,11 @@ class PromptOptimizer:
         self.client = llm_client
         self.test_suite = test_suite
         self.results_history = []
+        self.executor = ThreadPoolExecutor()
+
+    def shutdown(self):
+        """Shutdown the thread pool executor."""
+        self.executor.shutdown(wait=True)
 
     def evaluate_prompt(self, prompt_template: str, test_cases: List[TestCase] = None) -> Dict[str, float]:
         """Evaluate a prompt template against test cases in parallel."""
@@ -63,8 +68,7 @@ class PromptOptimizer:
             }
 
         # Run test cases in parallel
-        with ThreadPoolExecutor() as executor:
+        results = list(self.executor.map(process_test_case, test_cases))
-            results = list(executor.map(process_test_case, test_cases))
 
         # Aggregate metrics
         for result in results:
@@ -247,6 +251,7 @@ def main():
 
     optimizer = PromptOptimizer(MockLLMClient(), test_suite)
 
+    try:
         base_prompt = "Classify the sentiment of: {text}\nSentiment:"
 
         results = optimizer.optimize(base_prompt)
@@ -257,6 +262,8 @@ def main():
         print(f"Best Prompt:\n{results['best_prompt']}")
 
         optimizer.export_results('optimization_results.json')
+    finally:
+        optimizer.shutdown()
 
 
 if __name__ == '__main__':