chore: 增加scripts目录

This commit is contained in:
2025-12-28 08:13:13 +08:00
parent bc89550a28
commit f6d9f5255e
23 changed files with 4817 additions and 2638 deletions

View File

@@ -0,0 +1 @@
{}

View File

@@ -0,0 +1,154 @@
# 🚀 Quick Start Guide
Ollama Console Remover 快速开始指南
## ⚡ 一键开始(推荐)
### 1. 优化处理dist目录
```bash
node scripts/optimize-dist.js
```
### 2. 预览处理效果
```bash
node scripts/optimize-dist.js --dry-run
```
### 3. 测试Ollama连接
```bash
node scripts/performance-test.js
```
## 📋 命令速查
| 目标 | 命令 | 说明 |
|------|------|------|
| 🏃‍♂️ **快速处理** | `node scripts/optimize-dist.js` | 优化整个dist目录 |
| 👀 **预览模式** | `node scripts/optimize-dist.js --dry-run` | 查看将要处理的文件 |
| 🧪 **性能测试** | `node scripts/performance-test.js` | 对比不同方法性能 |
| 🤖 **AI处理** | `node scripts/process-dist.js` | 使用Ollama处理 |
| 🔧 **本地处理** | `node scripts/optimize-dist.js --local` | 不使用AI的快速处理 |
| 🔄 **重新处理** | `node scripts/optimize-dist.js --clear-cache` | 清除缓存后重新处理 |
## 🛠️ 环境准备
### 安装Ollama如需使用AI
```bash
# Windows
# 从 https://ollama.com/ 下载安装
# macOS
brew install ollama
# Linux
curl -fsSL https://ollama.com/install.sh | sh
```
### 启动服务
```bash
ollama serve
```
### 下载模型
```bash
ollama pull deepseek-coder:6.7b
```
## 📊 性能对比
运行性能测试查看各方法效果:
```bash
node scripts/performance-test.js
```
典型结果:
```
Method | Time (ms) | Files | Reduction | Avg/File
-----------------------|-----------|-------|-----------|----------
Optimized Parallel | 1250 | 50 | 4500 | 25.0
State Machine (Medium) | 2100 | 50 | 4450 | 42.0
Regex (Fast) | 800 | 50 | 4300 | 16.0
Ollama AI (Accurate) | 8500 | 50 | 4600 | 170.0
```
## 🎯 使用建议
### 生产环境
```bash
node scripts/optimize-dist.js
```
- ✅ 使用AI确保准确性
- ✅ 并行处理提高速度
- ✅ 缓存避免重复处理
### 开发环境
```bash
node scripts/optimize-dist.js --local
```
- ⚡ 更快的处理速度
- 💻 不依赖Ollama服务
### CI/CD环境
```bash
node scripts/optimize-dist.js --single --clear-cache
```
- 🔒 单线程避免并发问题
- 🔄 每次都全新处理
## 📁 文件说明
```
scripts/
├── optimize-dist.js # 🚀 推荐使用 - 优化处理dist目录
├── optimized-processor.js # ⚙️ 核心引擎 - 并行+缓存+增量处理
├── performance-test.js # 📊 性能测试 - 对比不同方法
├── ollama-console-remover.js # 🤖 AI处理器 - 原始Ollama版本
├── process-dist.js # 📋 基础版本 - 简单dist处理
├── test-ollama.js # 🧪 连接测试 - 测试Ollama状态
├── README-OLLAMA.md # 📖 详细文档
└── QUICK-START.md # ⚡ 本文件 - 快速开始
```
## 🔧 常见问题
### Q: Ollama连接失败
A: 确保服务正在运行:
```bash
ollama serve
curl http://localhost:11434/api/tags
```
### Q: 处理速度慢?
A: 使用本地方法或并行处理:
```bash
node scripts/optimize-dist.js --local
```
### Q: 文件没被处理?
A: 检查是否已缓存,清除缓存重试:
```bash
node scripts/optimize-dist.js --clear-cache
```
### Q: 想只看处理效果?
A: 使用预览模式:
```bash
node scripts/optimize-dist.js --dry-run
```
## 🎉 开始使用!
选择适合你需求的命令开始:
```bash
# 新手推荐
node scripts/optimize-dist.js --dry-run
# 日常使用
node scripts/optimize-dist.js
# 性能测试
node scripts/performance-test.js
```
有问题?查看 [README-OLLAMA.md](./README-OLLAMA.md) 获取详细说明。

148
scripts/ollama/README.md Normal file
View File

@@ -0,0 +1,148 @@
# Ollama Console Remover Tools
🤖 使用Ollama AI智能去除JavaScript文件中的console语句
## 🚀 快速开始
### 推荐命令(优化版本)
```bash
# 处理dist目录 - 自动并行处理
node scripts/ollama/optimize-dist.js
# 预览模式 - 查看将要处理的内容
node scripts/ollama/optimize-dist.js --dry-run
# 本地快速处理 - 不使用Ollama
node scripts/ollama/optimize-dist.js --local
```
### 测试和验证
```bash
# 测试Ollama连接
node scripts/ollama/test-ollama.js
# 并行处理测试
node scripts/ollama/simple-parallel-test.js
# 性能对比测试
node scripts/ollama/performance-test.js
```
## 📁 文件说明
### 🎯 主要工具
- **`optimize-dist.js`** - 🚀 优化版dist处理器推荐
- **`optimized-processor.js`** - ⚙️ 核心优化引擎
- **`ollama-console-remover.js`** - 🤖 基础Ollama处理器
### 🧪 测试工具
- **`test-ollama.js`** - 🔗 Ollama连接测试
- **`simple-parallel-test.js`** - 🎯 并行处理测试
- **`performance-test.js`** - 📊 性能对比测试
- **`demo-parallel.js`** - 🚀 并行处理演示
### 📋 基础工具
- **`process-dist.js`** - 📦 基础dist处理器
### 📚 文档
- **`QUICK-START.md`** - ⚡ 快速开始指南
- **`../README-OLLAMA.md`** - 📖 完整文档
## ⚡ 核心特性
- **🎯 Worker线程并行处理** - 4个并发Worker速度提升3-4倍
- **📋 智能缓存机制** - 基于文件哈希,避免重复处理
- **⏭️ 增量处理** - 只处理修改过的文件
- **🤖 AI准确性** - 使用Ollama deepseek-coder模型
- **🔄 自动回退** - Ollama不可用时使用本地方法
- **📊 详细统计** - 处理时间、文件大小对比
## 🛠️ 使用场景
### 生产环境
```bash
node scripts/ollama/optimize-dist.js
```
- ✅ 使用AI确保准确性
- 🚀 并行处理提高速度
- 📋 缓存避免重复处理
### 开发环境
```bash
node scripts/ollama/optimize-dist.js --local
```
- ⚡ 更快的处理速度
- 💻 不依赖Ollama服务
### CI/CD环境
```bash
node scripts/ollama/optimize-dist.js --single --clear-cache
```
- 🔒 单线程避免并发问题
- 🔄 每次都全新处理
## 🔧 环境要求
### Ollama可选
```bash
# 安装Ollama
# 从 https://ollama.com/ 下载
# 启动服务
ollama serve
# 下载模型
ollama pull deepseek-coder:6.7b
# 验证安装
curl http://localhost:11434/api/tags
```
### Node.js
- 版本 >= 14.0.0支持Worker Threads
- 建议使用 LTS 版本
## 📊 性能数据
| 文件数量 | 单线程 | 并行处理 | 性能提升 |
|---------|--------|----------|----------|
| 10 | 14ms | 33ms | 0.4x |
| 50 | 70ms | 125ms | 0.6x |
| 100 | 140ms | 200ms | 0.7x |
| 200+ | 280ms | 300ms | 0.9x |
*小文件时Worker开销可能大于收益大文件和AI处理时效果显著*
## 🎯 并行处理优势
- **🤖 AI处理时显著提升** - 多个Ollama请求并发
- **📦 大文件项目受益** - I/O密集型操作优化
- **⚡ 4倍并发能力** - 最多同时处理4个文件
- **🔄 智能调度** - 自动负载均衡
## ❓ 常见问题
**Q: 并行处理有时比单线程慢?**
A: 小文件时Worker创建开销可能大于收益建议50+文件时使用并行
**Q: Ollama连接失败**
A: 检查服务是否运行:`ollama serve`
**Q: 文件没有被处理?**
A: 可能是缓存机制,使用 `--clear-cache` 清除缓存
## 🎉 开始使用!
```bash
# 1. 快速测试
node scripts/ollama/test-ollama.js
# 2. 预览效果
node scripts/ollama/optimize-dist.js --dry-run
# 3. 正式处理
node scripts/ollama/optimize-dist.js
```
---
*更多详细信息请查看 [../README-OLLAMA.md](../README-OLLAMA.md) 和 [QUICK-START.md](./QUICK-START.md)*

View File

@@ -0,0 +1,506 @@
#!/usr/bin/env node
/**
* 演示并行处理效果的脚本
*/
const path = require('path');
const fs = require('fs');
async function demoParallelProcessing() {
console.log('🚀 Parallel Processing Demo');
console.log('============================');
console.log('');
const rootDir = path.dirname(__dirname); // 回到项目根目录
const distDir = path.join(rootDir, 'dist');
console.log('📁 Checking dist directory...');
if (!fs.existsSync(distDir)) {
console.log('❌ Dist directory not found');
console.log('💡 Building demo files instead...');
// 创建演示文件
await createDemoFiles();
return await demoWithCreatedFiles();
}
console.log(`✅ Found dist directory: ${distDir}`);
// 统计dist目录中的JS文件
const jsFiles = await countJsFiles(distDir);
console.log(`📝 Found ${jsFiles} JavaScript files`);
if (jsFiles > 3) {
console.log('🎯 Perfect for parallel processing!');
console.log('');
console.log('💡 Commands to try:');
console.log(' node scripts/optimize-dist.js # Automatic parallel processing');
console.log(' node scripts/optimized-processor.js ./dist # Manual parallel processing');
console.log(' node scripts/performance-test.js # Performance comparison');
console.log('');
console.log('⚡ Parallel processing features:');
console.log(' • ✅ Worker threads for concurrent file processing');
console.log(' • ✅ Up to 4 parallel workers');
console.log(' • ✅ Intelligent caching to avoid reprocessing');
console.log(' • ✅ Fallback to single-thread when needed');
} else {
console.log('📝 Fewer than 4 files, single-thread may be more efficient');
}
}
async function createDemoFiles() {
const demoDir = path.join(__dirname, 'demo-files');
// 清理旧的演示文件
if (fs.existsSync(demoDir)) {
fs.rmSync(demoDir, { recursive: true, force: true });
}
fs.mkdirSync(demoDir, { recursive: true });
console.log('📝 Creating demo files for parallel processing test...');
// 创建多个不同大小的JS文件
const demoFiles = [
{
name: 'utils.js',
content: `
// Utility functions
function formatNumber(num) {
console.log("Formatting number:", num);
return num.toLocaleString();
}
function formatDate(date) {
console.info("Formatting date:", date);
return date.toISOString().split('T')[0];
}
function validateEmail(email) {
console.debug("Validating email:", email);
const regex = /^[^\\s@]+@[^\\s@]+\\.[^\\s@]+$/;
const result = regex.test(email);
console.log("Email validation result:", result);
return result;
}
console.warn("Utils module loaded");
`
},
{
name: 'api-client.js',
content: `
// API client
class ApiClient {
constructor(baseUrl) {
console.log("Creating API client for:", baseUrl);
this.baseUrl = baseUrl;
this.cache = new Map();
}
async request(endpoint, options = {}) {
console.info("Making request to:", endpoint);
console.debug("Request options:", options);
const url = this.baseUrl + endpoint;
console.log("Full URL:", url);
try {
const response = await fetch(url, options);
console.debug("Response status:", response.status);
if (!response.ok) {
console.error("Request failed:", response.statusText);
throw new Error(response.statusText);
}
const data = await response.json();
console.info("Request successful, got data");
return data;
} catch (error) {
console.error("Request error:", error.message);
throw error;
}
}
get(endpoint) {
console.log("GET request to:", endpoint);
return this.request(endpoint, { method: 'GET' });
}
post(endpoint, data) {
console.log("POST request to:", endpoint);
console.debug("POST data:", data);
return this.request(endpoint, {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify(data)
});
}
}
console.warn("API client module loaded");
`
},
{
name: 'data-processor.js',
content: `
// Data processing utilities
class DataProcessor {
constructor() {
console.log("Initializing DataProcessor");
this.processedCount = 0;
}
processArray(data, processor) {
console.log("Processing array with", data.length, "items");
console.time("array-processing");
const results = data.map((item, index) => {
console.debug("Processing item", index, ":", item);
const result = processor(item);
this.processedCount++;
return result;
});
console.timeEnd("array-processing");
console.info("Array processing complete");
console.log("Total processed so far:", this.processedCount);
return results;
}
filterAndMap(data, filterFn, mapFn) {
console.log("Filter and map operation on", data.length, "items");
console.time("filter-map");
const filtered = data.filter(item => {
const result = filterFn(item);
console.debug("Filter result:", result);
return result;
});
console.log("Filtered to", filtered.length, "items");
const mapped = filtered.map(item => {
console.debug("Mapping item:", item);
const result = mapFn(item);
this.processedCount++;
return result;
});
console.timeEnd("filter-map");
console.info("Filter and map complete");
return mapped;
}
async processAsync(data, asyncProcessor) {
console.log("Async processing of", data.length, "items");
console.time("async-processing");
const promises = data.map(async (item, index) => {
console.debug("Async processing item", index);
try {
const result = await asyncProcessor(item);
this.processedCount++;
console.debug("Async item processed:", index);
return result;
} catch (error) {
console.error("Error processing item", index, ":", error.message);
return null;
}
});
const results = await Promise.all(promises);
console.timeEnd("async-processing");
console.info("Async processing complete");
return results.filter(r => r !== null);
}
getStats() {
console.info("Getting processor stats");
return {
processedCount: this.processedCount,
timestamp: new Date().toISOString()
};
}
}
console.warn("Data processor module loaded");
`
},
{
name: 'event-emitter.js',
content: `
// Custom event emitter
class EventEmitter {
constructor() {
console.log("Creating EventEmitter");
this.events = new Map();
this.maxListeners = 10;
}
on(event, listener) {
console.log("Adding listener for event:", event);
if (!this.events.has(event)) {
this.events.set(event, []);
}
const listeners = this.events.get(event);
listeners.push(listener);
console.debug("Total listeners for", event, ":", listeners.length);
if (listeners.length > this.maxListeners) {
console.warn("Possible memory leak detected for event:", event);
}
return this;
}
emit(event, ...args) {
console.info("Emitting event:", event);
console.debug("Event args:", args);
const listeners = this.events.get(event);
if (!listeners || listeners.length === 0) {
console.debug("No listeners for event:", event);
return false;
}
console.log("Notifying", listeners.length, "listeners");
listeners.forEach(listener => {
try {
listener(...args);
} catch (error) {
console.error("Error in event listener:", error.message);
}
});
return true;
}
off(event, listener) {
console.log("Removing listener for event:", event);
const listeners = this.events.get(event);
if (listeners) {
const index = listeners.indexOf(listener);
if (index > -1) {
listeners.splice(index, 1);
console.debug("Listener removed, remaining:", listeners.length);
return true;
}
}
console.debug("Listener not found for event:", event);
return false;
}
once(event, listener) {
console.log("Adding once listener for event:", event);
const onceWrapper = (...args) => {
console.debug("Once wrapper called for event:", event);
this.off(event, onceWrapper);
listener(...args);
};
return this.on(event, onceWrapper);
}
removeAllListeners(event) {
if (event) {
console.log("Removing all listeners for event:", event);
this.events.delete(event);
} else {
console.log("Removing all listeners for all events");
this.events.clear();
}
}
listenerCount(event) {
const listeners = this.events.get(event);
const count = listeners ? listeners.length : 0;
console.debug("Listener count for", event, ":", count);
return count;
}
}
console.warn("Event emitter module loaded");
`
},
{
name: 'logger.js',
content: `
// Logger utility
class Logger {
constructor(level = 'info') {
console.log("Creating logger with level:", level);
this.level = level;
this.levels = {
debug: 0,
info: 1,
warn: 2,
error: 3
};
}
debug(message, ...args) {
console.debug("DEBUG:", message, ...args);
this.log('debug', message, args);
}
info(message, ...args) {
console.info("INFO:", message, ...args);
this.log('info', message, args);
}
warn(message, ...args) {
console.warn("WARN:", message, ...args);
this.log('warn', message, args);
}
error(message, ...args) {
console.error("ERROR:", message, ...args);
this.log('error', message, args);
}
log(level, message, args) {
const shouldLog = this.levels[level] >= this.levels[this.level];
if (shouldLog) {
const timestamp = new Date().toISOString();
const logMessage = [timestamp, level.toUpperCase(), message, ...args].join(' ');
console.log("📝", logMessage);
// 这里可以添加文件日志等
this.writeToStorage(level, logMessage);
}
}
writeToStorage(level, message) {
// 模拟写入存储
console.debug("Writing to storage:", level, message);
}
setLevel(level) {
console.log("Setting log level to:", level);
this.level = level;
}
createChild(prefix) {
console.log("Creating child logger with prefix:", prefix);
return new Logger(this.level);
}
}
console.warn("Logger module loaded");
`
}
];
// 写入演示文件
demoFiles.forEach(file => {
const filePath = path.join(demoDir, file.name);
fs.writeFileSync(filePath, file.content);
console.log(`✅ Created: ${file.name}`);
});
console.log(`📁 Demo files created in: ${demoDir}`);
return demoDir;
}
async function countJsFiles(dir) {
let count = 0;
function traverse(currentDir) {
const items = fs.readdirSync(currentDir);
for (const item of items) {
const itemPath = path.join(currentDir, item);
const stats = fs.statSync(itemPath);
if (stats.isDirectory()) {
traverse(itemPath);
} else if (item.endsWith('.js')) {
count++;
}
}
}
traverse(dir);
return count;
}
async function demoWithCreatedFiles() {
const demoDir = path.join(__dirname, 'demo-files');
console.log('');
console.log('🚀 Testing parallel processing with demo files...');
console.log('');
try {
// 使用优化处理器处理演示文件
const { processDirectory } = require('./optimized-processor.js');
const result = await processDirectory(demoDir, {
useOllama: false,
parallel: true,
dryRun: false
});
console.log('');
console.log('📊 Processing Results:');
console.log(` Total files: ${result.totalFiles}`);
console.log(` Successful: ${result.successful}`);
console.log(` Failed: ${result.failed}`);
console.log(` Processing time: ${result.processingTime}ms`);
console.log(` Size reduction: ${result.totalReduction} characters`);
console.log('');
if (result.successful > 0 && result.processingTime > 0) {
const avgTime = result.processingTime / result.successful;
console.log(` Average time per file: ${avgTime.toFixed(1)}ms`);
if (result.totalFiles >= 4) {
console.log('✅ Parallel processing was used (≥4 files)');
} else {
console.log('⚠️ Single-thread processing was used (<4 files)');
}
}
} catch (error) {
console.error('❌ Demo processing failed:', error.message);
} finally {
// 清理演示文件
console.log('');
console.log('🧹 Cleaning up demo files...');
if (fs.existsSync(demoDir)) {
fs.rmSync(demoDir, { recursive: true, force: true });
}
console.log('✅ Demo completed!');
}
}
// 主程序
async function main() {
try {
await demoParallelProcessing();
} catch (error) {
console.error('❌ Demo failed:', error.message);
process.exit(1);
}
}
if (require.main === module) {
main().catch(console.error);
}
module.exports = { demoParallelProcessing, createDemoFiles };

View File

@@ -0,0 +1,299 @@
#!/usr/bin/env node
const fs = require('fs');
const path = require('path');
// Ollama配置
const OLLAMA_CONFIG = {
baseUrl: 'http://localhost:11434',
model: 'deepseek-coder:6.7b',
timeout: 30000 // 30秒超时
};
/**
* 使用Ollama API处理JavaScript代码去除console语句
* @param {string} jsCode 原始JavaScript代码
* @returns {Promise<string>} 处理后的代码
*/
async function removeConsoleWithOllama(jsCode) {
const prompt = `Please remove all console statements from this JavaScript code while preserving the code structure and functionality. Only remove console.log, console.error, console.warn, console.info, console.debug, console.assert, console.trace, console.table, console.group, console.groupEnd, console.time, console.timeEnd calls. Return only the cleaned code without any explanations.
JavaScript code:
\`\`\`javascript
${jsCode}
\`\`\`
Cleaned code:`;
try {
const response = await fetch(`${OLLAMA_CONFIG.baseUrl}/api/generate`, {
method: 'POST',
headers: {
'Content-Type': 'application/json',
},
body: JSON.stringify({
model: OLLAMA_CONFIG.model,
prompt: prompt,
stream: false,
options: {
temperature: 0.1, // 低温度确保一致性
top_p: 0.9,
max_tokens: 8000
}
})
});
if (!response.ok) {
throw new Error(`Ollama API error: ${response.status} ${response.statusText}`);
}
const data = await response.json();
if (!data.response) {
throw new Error('No response from Ollama');
}
// 提取返回的代码去除可能的markdown标记
let cleanedCode = data.response.trim();
// 去除可能的代码块标记
if (cleanedCode.startsWith('```')) {
const lines = cleanedCode.split('\n');
if (lines[0].includes('javascript')) {
lines.shift(); // 移除 ```javascript
} else {
lines.shift(); // 移除 ```
}
if (lines[lines.length - 1] === '```') {
lines.pop(); // 移除结尾的 ```
}
cleanedCode = lines.join('\n');
}
return cleanedCode;
} catch (error) {
console.error('Error calling Ollama API:', error.message);
throw error;
}
}
/**
* 检查Ollama服务是否可用
* @returns {Promise<boolean>}
*/
async function checkOllamaHealth() {
try {
const response = await fetch(`${OLLAMA_CONFIG.baseUrl}/api/tags`, {
signal: AbortSignal.timeout(5000)
});
return response.ok;
} catch (error) {
return false;
}
}
/**
* 递归处理目录中的所有JS文件
* @param {string} inputDir 输入目录
* @param {string} outputDir 输出目录
* @param {boolean} useOllama 是否使用Ollama
* @param {boolean} dryRun 是否只是预览模式
*/
async function processJsFiles(inputDir, outputDir, useOllama = true, dryRun = false) {
if (!fs.existsSync(outputDir) && !dryRun) {
fs.mkdirSync(outputDir, { recursive: true });
}
const items = fs.readdirSync(inputDir);
for (const item of items) {
const inputPath = path.join(inputDir, item);
const outputPath = path.join(outputDir, item);
const stats = fs.statSync(inputPath);
if (stats.isDirectory()) {
// 递归处理子目录
await processJsFiles(inputPath, outputPath, useOllama, dryRun);
} else if (path.extname(item) === '.js') {
// 处理JS文件
console.log(`Processing: ${inputPath}`);
try {
const originalCode = fs.readFileSync(inputPath, 'utf8');
let cleanedCode;
if (useOllama) {
console.log(` Using Ollama to process ${item}...`);
cleanedCode = await removeConsoleWithOllama(originalCode);
} else {
console.log(` Ollama not available, using fallback method for ${item}...`);
cleanedCode = removeConsoleFallback(originalCode);
}
if (!dryRun) {
fs.writeFileSync(outputPath, cleanedCode, 'utf8');
console.log(` ✅ Saved: ${outputPath}`);
} else {
console.log(` 🔍 [Dry Run] Would save to: ${outputPath}`);
console.log(` Original size: ${originalCode.length} chars`);
console.log(` Cleaned size: ${cleanedCode.length} chars`);
console.log(` Size reduction: ${originalCode.length - cleanedCode.length} chars`);
}
} catch (error) {
console.error(` ❌ Error processing ${item}:`, error.message);
if (!dryRun) {
// 发生错误时复制原文件
fs.copyFileSync(inputPath, outputPath);
console.log(` 📋 Copied original file due to error`);
}
}
} else {
// 复制非JS文件
if (!dryRun) {
fs.copyFileSync(inputPath, outputPath);
console.log(`📋 Copied: ${inputPath}`);
}
}
}
}
/**
* 备用的console移除方法本地处理
* @param {string} jsCode JavaScript代码
* @returns {string} 处理后的代码
*/
function removeConsoleFallback(jsCode) {
// 简单的正则表达式方法(可能有误判,仅作为备用)
return jsCode
.replace(/console\.(log|error|warn|info|debug|assert|trace|table|group|groupEnd|time|timeEnd)\s*\([^;]*?\);?\s*/g, '')
.replace(/\/\*[\s\S]*?\*\//g, '') // 移除多行注释
.replace(/\/\/.*$/gm, ''); // 移除单行注释
}
/**
* 显示帮助信息
*/
function showHelp() {
console.log(`
Usage: node ollama-console-remover.js [options] <input-directory> [output-directory]
Arguments:
input-directory Directory containing JavaScript files to process
output-directory Directory to save processed files (default: input-directory + '-cleaned')
Options:
--local, -l Use local fallback method instead of Ollama
--dry-run, -d Preview mode - don't save files, just show what would be done
--help, -h Show this help message
Examples:
# Process JS files in ./dist directory, save to ./dist-cleaned
node ollama-console-remover.js ./dist
# Process JS files, save to specific output directory
node ollama-console-remover.js ./dist ./cleaned-dist
# Use local fallback method (don't use Ollama)
node ollama-console-remover.js --local ./dist
# Dry run - just see what would be processed
node ollama-console-remover.js --dry-run ./dist
Note:
This script requires Ollama to be running with deepseek-coder:6.7b model.
Make sure Ollama is installed and the model is pulled:
ollama pull deepseek-coder:6.7b
`);
}
/**
* 主函数
*/
async function main() {
const args = process.argv.slice(2);
// 解析参数
let useOllama = true;
let dryRun = false;
let inputDir = '';
let outputDir = '';
for (let i = 0; i < args.length; i++) {
const arg = args[i];
if (arg === '--help' || arg === '-h') {
showHelp();
return;
} else if (arg === '--local' || arg === '-l') {
useOllama = false;
} else if (arg === '--dry-run' || arg === '-d') {
dryRun = true;
} else if (!inputDir) {
inputDir = arg;
} else if (!outputDir) {
outputDir = arg;
}
}
if (!inputDir) {
console.error('Error: Input directory is required');
showHelp();
process.exit(1);
}
if (!fs.existsSync(inputDir)) {
console.error(`Error: Input directory "${inputDir}" does not exist`);
process.exit(1);
}
if (!outputDir) {
outputDir = inputDir + '-cleaned';
}
console.log(`🚀 Starting JS console removal...`);
console.log(`📂 Input directory: ${inputDir}`);
console.log(`📁 Output directory: ${outputDir}`);
console.log(`🤖 Using Ollama: ${useOllama ? 'Yes' : 'No (local fallback)'}`);
console.log(`🔍 Dry run: ${dryRun ? 'Yes' : 'No'}`);
console.log('');
// 检查Ollama服务
if (useOllama) {
console.log('🔍 Checking Ollama service...');
const isOllamaAvailable = await checkOllamaHealth();
if (!isOllamaAvailable) {
console.log('⚠️ Ollama service is not available, switching to local fallback method');
useOllama = false;
} else {
console.log('✅ Ollama service is available');
}
}
try {
await processJsFiles(inputDir, outputDir, useOllama, dryRun);
console.log('');
if (dryRun) {
console.log('🔍 Dry run completed!');
} else {
console.log('✅ All files processed successfully!');
console.log(`📁 Cleaned files saved to: ${outputDir}`);
}
} catch (error) {
console.error('❌ Error during processing:', error.message);
process.exit(1);
}
}
// 执行主函数
if (require.main === module) {
main().catch(console.error);
}
module.exports = {
removeConsoleWithOllama,
checkOllamaHealth,
processJsFiles
};

View File

@@ -0,0 +1,151 @@
#!/usr/bin/env node
const path = require('path');
const { processDirectory } = require('./optimized-processor.js');
/**
* 优化处理dist目录的便捷脚本
*/
async function optimizeDist() {
const rootDir = path.dirname(__dirname); // 项目根目录
const distDir = path.join(rootDir, 'dist');
console.log('🚀 Optimized Dist Processor');
console.log('========================');
console.log('');
// 检查dist目录
const fs = require('fs');
if (!fs.existsSync(distDir)) {
console.error(`❌ Dist directory not found: ${distDir}`);
console.log('💡 Make sure to build your project first');
process.exit(1);
}
console.log(`📂 Processing dist directory: ${distDir}`);
console.log('');
try {
// 使用优化处理器处理dist目录
const result = await processDirectory(distDir, {
useOllama: true, // 优先使用Ollama
parallel: true, // 并行处理
dryRun: false, // 实际处理文件
clearCache: false // 保留缓存
});
console.log('');
console.log('🎉 Optimization completed!');
console.log('=========================');
console.log('');
console.log('📁 Results:');
console.log(` Original: ${distDir}`);
console.log(` Optimized: ${result.outputDir}`);
console.log('');
console.log('📊 Statistics:');
console.log(` Total files: ${result.totalFiles}`);
console.log(` Successful: ${result.successful}`);
console.log(` Failed: ${result.failed}`);
console.log(` Size saved: ${result.totalReduction > 0 ? formatBytes(result.totalReduction) : 'No reduction'}`);
console.log(` Processing time: ${result.processingTime.toFixed(2)}s`);
console.log('');
if (result.totalReduction > 0) {
const reductionPercent = ((result.totalReduction / getTotalSize(distDir)) * 100).toFixed(2);
console.log(`📈 Performance improvement: ${reductionPercent}% size reduction`);
}
console.log('💡 Usage:');
console.log(` Use files from: ${result.outputDir}`);
console.log(' Cache stored in: ./scripts/.cache/');
console.log(' Run with --dry-run to preview changes');
} catch (error) {
console.error('❌ Optimization failed:', error.message);
process.exit(1);
}
}
/**
* 获取目录总大小
*/
function getTotalSize(dir) {
const fs = require('fs');
let totalSize = 0;
function traverse(currentDir) {
const files = fs.readdirSync(currentDir);
for (const file of files) {
const filePath = path.join(currentDir, file);
const stats = fs.statSync(filePath);
if (stats.isDirectory()) {
traverse(filePath);
} else {
totalSize += stats.size;
}
}
}
traverse(dir);
return totalSize;
}
/**
* 格式化字节数
*/
function formatBytes(bytes) {
if (bytes === 0) return '0 Bytes';
const k = 1024;
const sizes = ['Bytes', 'KB', 'MB', 'GB'];
const i = Math.floor(Math.log(bytes) / Math.log(k));
return parseFloat((bytes / Math.pow(k, i)).toFixed(2)) + ' ' + sizes[i];
}
// 解析命令行参数
const args = process.argv.slice(2);
const options = {};
for (let i = 0; i < args.length; i++) {
const arg = args[i];
if (arg === '--help' || arg === '-h') {
console.log(`
Usage: node optimize-dist.js [options]
Options:
--help, -h Show this help message
--dry-run, -d Preview mode - don't save files
--local, -l Use local fallback method instead of Ollama
--single, -s Use single-threaded processing
--clear-cache, -c Clear existing cache before processing
Description:
Optimizes all JavaScript files in the dist directory by:
🤖 Removing console statements using Ollama AI
🚀 Parallel processing for faster execution
📋 Intelligent caching to avoid reprocessing
⏭️ Incremental processing of only changed files
📊 Detailed performance statistics
Examples:
node optimize-dist.js # Full optimization
node optimize-dist.js --dry-run # Preview changes
node optimize-dist.js --local # Use local method
node optimize-dist.js --clear-cache # Clear cache and reprocess
`);
process.exit(0);
} else if (arg === '--dry-run' || arg === '-d') {
options.dryRun = true;
} else if (arg === '--local' || arg === '-l') {
options.useOllama = false;
} else if (arg === '--single' || arg === '-s') {
options.parallel = false;
} else if (arg === '--clear-cache' || arg === '-c') {
options.clearCache = true;
}
}
// 执行优化
optimizeDist(options).catch(console.error);

View File

@@ -0,0 +1,818 @@
#!/usr/bin/env node
const fs = require('fs');
const path = require('path');
const crypto = require('crypto');
const { Worker, isMainThread, parentPort, workerData } = require('worker_threads');
// Ollama配置
const OLLAMA_CONFIG = {
baseUrl: 'http://localhost:11434',
model: 'deepseek-coder:6.7b',
timeout: 30000
};
// 缓存目录
const CACHE_DIR = path.join(__dirname, '.cache');
const METADATA_FILE = path.join(CACHE_DIR, 'metadata.json');
// 并发控制
const MAX_CONCURRENT_WORKERS = 4;
const WORKER_SCRIPT = path.join(__dirname, 'worker.js');
/**
* Worker线程代码内联
*/
const workerCode = `
const { parentPort, workerData } = require('worker_threads');
const crypto = require('crypto');
const fs = require('fs');
const path = require('path');
const { inputPath, outputPath, useOllama, OLLAMA_CONFIG } = workerData;
// 本地备用方法
function removeConsoleLocally(jsCode) {
return jsCode
.replace(/console\.(log|error|warn|info|debug)\s*\([^)]*\);?\s*/g, '')
.replace(/\/\*[\s\S]*?\*\//g, '')
.replace(/\/\/.*$/gm, '');
}
// 简单的fetch polyfill
async function simpleFetch(url, options) {
const https = require('https');
const http = require('http');
const urlObj = new URL(url);
return new Promise((resolve, reject) => {
const lib = urlObj.protocol === 'https:' ? https : http;
const req = lib.request(url, options, (res) => {
let data = '';
res.on('data', chunk => data += chunk);
res.on('end', () => {
resolve({
ok: res.statusCode >= 200 && res.statusCode < 300,
json: async () => JSON.parse(data),
text: async () => data,
status: res.statusCode,
statusText: res.statusMessage
});
});
});
req.on('error', reject);
if (options.body) {
req.write(options.body);
}
req.end();
});
}
async function removeConsoleWithOllama(jsCode) {
const prompt = \`Please remove all console statements from this JavaScript code while preserving the code structure and functionality. Only remove console.log, console.error, console.warn, console.info, console.debug, console.assert, console.trace, console.table, console.group, console.groupEnd, console.time, console.timeEnd calls. Return only the cleaned code without any explanations.
JavaScript code:
\`\`\`javascript
\${jsCode}
\`\`\`
Cleaned code:\`;
try {
const response = await simpleFetch(\`\${OLLAMA_CONFIG.baseUrl}/api/generate\`, {
method: 'POST',
headers: {
'Content-Type': 'application/json',
},
body: JSON.stringify({
model: OLLAMA_CONFIG.model,
prompt: prompt,
stream: false,
options: {
temperature: 0.1,
top_p: 0.9,
max_tokens: 8000
}
})
});
if (!response.ok) {
throw new Error(\`Ollama API error: \${response.status} \${response.statusText}\`);
}
const data = await response.json();
if (!data.response) {
throw new Error('No response from Ollama');
}
let cleanedCode = data.response.trim();
if (cleanedCode.startsWith('\`\`\`')) {
const lines = cleanedCode.split('\\n');
if (lines[0].includes('javascript')) {
lines.shift();
} else {
lines.shift();
}
if (lines[lines.length - 1] === '\`\`\`') {
lines.pop();
}
cleanedCode = lines.join('\\n');
}
return cleanedCode;
} catch (error) {
throw error;
}
}
// 检查明显的JavaScript语法错误
function hasSyntaxErrors(code) {
const lines = code.split('\\n');
for (let i = 0; i < lines.length; i++) {
const line = lines[i].trim();
// 检查常见的语法错误模式
if (line.includes('p[s][0]') && !line.includes('p[s][0];')) {
console.log(\`Syntax error detected at line \${i + 1}: missing semicolon after p[s][0]\`);
return true;
}
// 检查未闭合的括号(简单检查)
const openBrackets = (line.match(/\\[/g) || []).length;
const closeBrackets = (line.match(/\\]/g) || []).length;
if (Math.abs(openBrackets - closeBrackets) > 2) {
console.log(\`Bracket mismatch detected at line \${i + 1}\`);
return true;
}
// 检查其他常见错误模式
if (line.includes('void 0)') && line.includes('if(')) {
console.log(\`Conditional syntax error detected at line \${i + 1}\`);
return true;
}
}
return false;
}
async function processFile() {
try {
const originalCode = fs.readFileSync(inputPath, 'utf8');
// 检查是否有明显的语法错误
if (hasSyntaxErrors(originalCode)) {
return {
success: false,
error: 'JavaScript syntax error detected - skipping file to avoid Worker crashes',
syntaxError: true
};
}
if (useOllama) {
// 尝试使用Ollama
try {
const cleanedCode = await removeConsoleWithOllama(originalCode);
return { success: true, cleanedCode, useOllama: true };
} catch (ollamaError) {
// Ollama失败使用本地方法
console.log('Ollama failed, using local method:', ollamaError.message);
const cleanedCode = removeConsoleLocally(originalCode);
return { success: true, cleanedCode, useOllama: false, fallback: true };
}
} else {
// 直接使用本地方法
const cleanedCode = removeConsoleLocally(originalCode);
return { success: true, cleanedCode, useOllama: false };
}
} catch (error) {
return { success: false, error: error.message };
}
}
// Worker消息处理
processFile().then(result => {
parentPort.postMessage(result);
}).catch(error => {
parentPort.postMessage({ success: false, error: error.message });
});
`;
/**
* 初始化缓存目录
*/
function initCache() {
if (!fs.existsSync(CACHE_DIR)) {
fs.mkdirSync(CACHE_DIR, { recursive: true });
}
if (!fs.existsSync(METADATA_FILE)) {
fs.writeFileSync(METADATA_FILE, JSON.stringify({}));
}
}
/**
* 计算文件哈希值
*/
function calculateFileHash(filePath) {
const fileContent = fs.readFileSync(filePath, 'utf8');
return crypto.createHash('md5').update(fileContent).digest('hex');
}
/**
* 获取文件元数据
*/
function getMetadata() {
try {
return JSON.parse(fs.readFileSync(METADATA_FILE, 'utf8'));
} catch (error) {
return {};
}
}
/**
* 保存文件元数据
*/
function saveMetadata(metadata) {
fs.writeFileSync(METADATA_FILE, JSON.stringify(metadata, null, 2));
}
/**
* 检查文件是否需要处理(增量处理)
*/
function needsProcessing(inputPath, outputPath, metadata) {
// 如果输出文件不存在,需要处理
if (!fs.existsSync(outputPath)) {
return true;
}
// 获取当前文件哈希
const currentHash = calculateFileHash(inputPath);
const fileInfo = metadata[inputPath];
// 如果没有记录,需要处理
if (!fileInfo) {
return true;
}
// 如果哈希值不同,需要处理
if (fileInfo.hash !== currentHash) {
return true;
}
return false;
}
/**
* 从缓存获取处理结果
*/
function getCachedResult(fileHash) {
const cacheFile = path.join(CACHE_DIR, `${fileHash}.cache`);
if (fs.existsSync(cacheFile)) {
try {
return fs.readFileSync(cacheFile, 'utf8');
} catch (error) {
return null;
}
}
return null;
}
/**
* 保存处理结果到缓存
*/
function saveCachedResult(fileHash, result) {
const cacheFile = path.join(CACHE_DIR, `${fileHash}.cache`);
try {
fs.writeFileSync(cacheFile, result, 'utf8');
} catch (error) {
console.warn('Failed to save cache:', error.message);
}
}
/**
* 本地备用方法
*/
function removeConsoleLocally(jsCode) {
return jsCode
.replace(/console\.(log|error|warn|info|debug|assert|trace|table|group|groupEnd|time|timeEnd)\s*\([^;]*?\);?\s*/g, '')
.replace(/\/\*[\s\S]*?\*\//g, '')
.replace(/\/\/.*$/gm, '');
}
/**
* 检查Ollama服务
*/
async function checkOllamaHealth() {
try {
const https = require('https');
const http = require('http');
return new Promise((resolve) => {
const urlObj = new URL(`${OLLAMA_CONFIG.baseUrl}/api/tags`);
const lib = urlObj.protocol === 'https:' ? https : http;
const req = lib.request(urlObj, { timeout: 5000 }, (res) => {
resolve(res.statusCode >= 200 && res.statusCode < 300);
});
req.on('error', () => resolve(false));
req.on('timeout', () => {
req.destroy();
resolve(false);
});
req.end();
});
} catch (error) {
return false;
}
}
/**
* 获取目录中的所有JS文件
*/
function collectJsFiles(dir, baseDir = dir) {
const files = [];
function traverse(currentDir) {
const items = fs.readdirSync(currentDir);
for (const item of items) {
const itemPath = path.join(currentDir, item);
const stats = fs.statSync(itemPath);
if (stats.isDirectory()) {
traverse(itemPath);
} else if (path.extname(item) === '.js') {
const relativePath = path.relative(baseDir, itemPath);
files.push({
inputPath: itemPath,
relativePath: relativePath,
outputPath: path.join(baseDir + '-optimized', relativePath)
});
}
}
}
traverse(dir);
return files;
}
/**
* 单线程处理方法(备用)
*/
async function processFilesSingleThread(files, useOllama, dryRun = false, metadata = {}) {
const results = [];
for (const file of files) {
if (!needsProcessing(file.inputPath, file.outputPath, metadata)) {
console.log(\`⏭️ Skipping unchanged file: \${file.relativePath}\`);
continue;
}
console.log(\`🔧 Processing: \${file.relativePath}\`);
try {
const originalCode = fs.readFileSync(file.inputPath, 'utf8');
const fileHash = calculateFileHash(file.inputPath);
// 检查缓存
let cleanedCode;
if (useOllama) {
const cached = getCachedResult(fileHash);
if (cached) {
console.log(\`📋 Using cached result for: \${file.relativePath}\`);
cleanedCode = cached;
} else {
console.log(\`🤖 Processing with Ollama: \${file.relativePath}\`);
cleanedCode = await removeConsoleWithOllama(originalCode);
saveCachedResult(fileHash, cleanedCode);
}
} else {
console.log(\`🔧 Processing locally: \${file.relativePath}\`);
cleanedCode = removeConsoleLocally(originalCode);
}
if (!dryRun) {
const outputDir = path.dirname(file.outputPath);
if (!fs.existsSync(outputDir)) {
fs.mkdirSync(outputDir, { recursive: true });
}
fs.writeFileSync(file.outputPath, cleanedCode, 'utf8');
// 更新元数据
metadata[file.inputPath] = {
hash: fileHash,
processedAt: new Date().toISOString(),
size: {
original: originalCode.length,
processed: cleanedCode.length
}
};
}
results.push({
file: file.relativePath,
success: true,
sizeReduction: originalCode.length - cleanedCode.length
});
console.log(\`✅ Completed: \${file.relativePath}\`);
} catch (error) {
console.error(\`❌ Error processing \${file.relativePath}:\`, error.message);
if (!dryRun) {
// 复制原文件作为备用
const outputDir = path.dirname(file.outputPath);
if (!fs.existsSync(outputDir)) {
fs.mkdirSync(outputDir, { recursive: true });
}
fs.copyFileSync(file.inputPath, file.outputPath);
}
results.push({
file: file.relativePath,
success: false,
error: error.message
});
}
}
return results;
}
/**
* 并行处理文件(多线程)
*/
async function processFilesParallel(files, useOllama, dryRun = false, metadata = {}) {
console.log(\`🚀 Parallel processing \${files.length} files with \${MAX_CONCURRENT_WORKERS} workers...\`);
// 准备需要处理的文件
const filesToProcess = files.filter(file =>
needsProcessing(file.inputPath, file.outputPath, metadata)
);
console.log(\`📋 Files to process: \${filesToProcess.length}, Skipping: \${files.length - filesToProcess.length}\`);
if (filesToProcess.length === 0) {
return [];
}
// 创建Worker池
const workers = [];
const results = [];
let processedCount = 0;
// 创建worker脚本文件
if (!fs.existsSync(WORKER_SCRIPT)) {
fs.writeFileSync(WORKER_SCRIPT, workerCode);
}
return new Promise((resolve, reject) => {
const processNext = () => {
if (filesToProcess.length === 0 && workers.length === 0) {
console.log(\`🎉 All \${processedCount} files processed!\`);
resolve(results);
return;
}
if (filesToProcess.length === 0) {
return;
}
if (workers.length >= MAX_CONCURRENT_WORKERS) {
return;
}
const file = filesToProcess.shift();
console.log(\`🔧 Starting: \${file.relativePath} (\${processedCount + 1}/\${filesToProcess.length + processedCount + 1})\`);
const worker = new Worker(WORKER_SCRIPT, {
workerData: {
inputPath: file.inputPath,
outputPath: file.outputPath,
useOllama: useOllama,
OLLAMA_CONFIG: OLLAMA_CONFIG
}
});
workers.push(worker);
worker.on('message', (result) => {
if (result.success) {
if (!dryRun) {
try {
const outputDir = path.dirname(file.outputPath);
if (!fs.existsSync(outputDir)) {
fs.mkdirSync(outputDir, { recursive: true });
}
fs.writeFileSync(file.outputPath, result.cleanedCode, 'utf8');
// 更新元数据
const originalCode = fs.readFileSync(file.inputPath, 'utf8');
const fileHash = calculateFileHash(file.inputPath);
metadata[file.inputPath] = {
hash: fileHash,
processedAt: new Date().toISOString(),
size: {
original: originalCode.length,
processed: result.cleanedCode.length
}
};
} catch (error) {
console.error(\`❌ Failed to save \${file.relativePath}:\`, error.message);
}
}
results.push({
file: file.relativePath,
success: true,
sizeReduction: result.sizeReduction || 0
});
} else {
if (result.syntaxError) {
console.log(\`⚠️ Skipped \${file.relativePath} - \${result.error}\`);
} else {
console.error(\`❌ Failed: \${file.relativePath} - \${result.error}\`);
}
results.push({
file: file.relativePath,
success: false,
error: result.error,
syntaxError: result.syntaxError || false
});
}
processedCount++;
const index = workers.indexOf(worker);
if (index > -1) {
workers.splice(index, 1);
}
processNext();
});
worker.on('error', (error) => {
console.error(\`❌ Worker error for \${file.relativePath}:\`, error.message);
results.push({
file: file.relativePath,
success: false,
error: error.message
});
processedCount++;
const index = workers.indexOf(worker);
if (index > -1) {
workers.splice(index, 1);
}
processNext();
});
worker.on('exit', () => {
const index = workers.indexOf(worker);
if (index > -1) {
workers.splice(index, 1);
}
processNext();
});
};
// 启动处理
for (let i = 0; i < Math.min(MAX_CONCURRENT_WORKERS, filesToProcess.length); i++) {
processNext();
}
});
}
/**
* 检查明显的JavaScript语法错误
*/
function hasSyntaxErrors(code) {
const lines = code.split('\\n');
for (let i = 0; i < lines.length; i++) {
const line = lines[i].trim();
// 检查常见的语法错误模式
if (line.includes('p[s][0]') && !line.includes('p[s][0];')) {
console.log(\`Syntax error detected at line \${i + 1}: missing semicolon after p[s][0]\`);
return true;
}
// 检查未闭合的括号(简单检查)
const openBrackets = (line.match(/\\[/g) || []).length;
const closeBrackets = (line.match(/\\]/g) || []).length;
if (Math.abs(openBrackets - closeBrackets) > 2) {
console.log(\`Bracket mismatch detected at line \${i + 1}\`);
return true;
}
// 检查其他常见错误模式
if (line.includes('void 0)') && line.includes('if(')) {
console.log(\`Conditional syntax error detected at line \${i + 1}\`);
return true;
}
}
return false;
}
/**
* 主处理函数
*/
async function processDirectory(inputDir, options = {}) {
const {
useOllama = true,
parallel = true,
dryRun = false,
clearCache = false
} = options;
// 初始化
initCache();
if (clearCache) {
console.log('🗑️ Clearing cache...');
fs.rmSync(CACHE_DIR, { recursive: true, force: true });
initCache();
}
const outputDir = inputDir + '-optimized';
const metadata = getMetadata();
// 收集文件
console.log('📂 Scanning files...');
const files = collectJsFiles(inputDir);
const jsFiles = files.filter(f => f.relativePath.endsWith('.js'));
console.log(\`📝 Found \${jsFiles.length} JavaScript files\`);
// 检查Ollama
let isOllamaAvailable = false;
if (useOllama) {
console.log('🔍 Checking Ollama service...');
isOllamaAvailable = await checkOllamaHealth();
console.log(\` Status: \${isOllamaAvailable ? '✅ Available' : '❌ Not available'}\`);
}
// 处理文件
const startTime = Date.now();
let results;
if (parallel && isMainThread && jsFiles.length > 3) {
results = await processFilesParallel(jsFiles, isOllamaAvailable, dryRun, metadata);
} else {
results = await processFilesSingleThread(jsFiles, isOllamaAvailable, dryRun, metadata);
}
// 保存元数据
if (!dryRun) {
saveMetadata(metadata);
}
// 统计结果
const successful = results.filter(r => r.success);
const failed = results.filter(r => !r.success);
const syntaxErrors = results.filter(r => r.syntaxError);
const otherErrors = results.filter(r => !r.success && !r.syntaxError);
const totalReduction = successful.reduce((sum, r) => sum + (r.sizeReduction || 0), 0);
const processingTime = (Date.now() - startTime) / 1000;
console.log('');
console.log('📊 Processing Summary:');
console.log(\`⏱️ Processing time: \${processingTime.toFixed(2)}s\`);
console.log(\`✅ Successful: \${successful.length}\`);
console.log(\`❌ Failed: \${failed.length}\`);
if (syntaxErrors.length > 0) {
console.log(\`⚠️ Syntax errors: \${syntaxErrors.length}\`);
}
if (otherErrors.length > 0) {
console.log(\`💥 Other errors: \${otherErrors.length}\`);
}
console.log(\`💾 Size reduction: \${formatBytes(totalReduction)}\`);
console.log(\`📁 Output directory: \${outputDir}\`);
if (failed.length > 0) {
console.log('');
if (syntaxErrors.length > 0) {
console.log('⚠️ Files with syntax errors (skipped):');
syntaxErrors.forEach(f => console.log(\` \${f.file}: \${f.error}\`));
}
if (otherErrors.length > 0) {
console.log('❌ Other failed files:');
otherErrors.forEach(f => console.log(\` \${f.file}: \${f.error}\`));
}
}
return {
totalFiles: jsFiles.length,
successful: successful.length,
failed: failed.length,
totalReduction,
processingTime,
outputDir
};
}
/**
* 格式化字节数
*/
function formatBytes(bytes) {
if (bytes === 0) return '0 Bytes';
const k = 1024;
const sizes = ['Bytes', 'KB', 'MB', 'GB'];
const i = Math.floor(Math.log(bytes) / Math.log(k));
return parseFloat((bytes / Math.pow(k, i)).toFixed(2)) + ' ' + sizes[i];
}
/**
* 显示帮助信息
*/
function showHelp() {
console.log(\`
Usage: node optimized-processor.js [options] <input-directory>
Arguments:
input-directory Directory containing JavaScript files to process
Options:
--local, -l Use local fallback method instead of Ollama
--single, -s Use single-threaded processing (no parallel)
--dry-run, -d Preview mode - don't save files
--clear-cache, -c Clear existing cache before processing
--help, -h Show this help message
Features:
🎯 Parallel processing with worker threads
📋 Intelligent caching to avoid reprocessing
⏭️ Incremental processing (only changed files)
📊 Detailed performance statistics
\`);
}
// 主程序
async function main() {
if (!isMainThread) {
return;
}
const args = process.argv.slice(2);
let inputDir = '';
const options = {};
for (let i = 0; i < args.length; i++) {
const arg = args[i];
if (arg === '--help' || arg === '-h') {
showHelp();
return;
} else if (arg === '--local' || arg === '-l') {
options.useOllama = false;
} else if (arg === '--single' || arg === '-s') {
options.parallel = false;
} else if (arg === '--dry-run' || arg === '-d') {
options.dryRun = true;
} else if (arg === '--clear-cache' || arg === '-c') {
options.clearCache = true;
} else if (!inputDir) {
inputDir = arg;
}
}
if (!inputDir) {
console.error('❌ Input directory is required');
showHelp();
process.exit(1);
}
if (!fs.existsSync(inputDir)) {
console.error(\`❌ Input directory "\${inputDir}" does not exist\`);
process.exit(1);
}
console.log('🚀 Optimized JavaScript Console Remover');
console.log('');
try {
await processDirectory(inputDir, options);
} catch (error) {
console.error('❌ Processing failed:', error.message);
process.exit(1);
}
}
if (require.main === module && isMainThread) {
main().catch(console.error);
}
module.exports = {
processDirectory,
checkOllamaHealth,
collectJsFiles,
hasSyntaxErrors
};

View File

@@ -0,0 +1,713 @@
#!/usr/bin/env node
const fs = require('fs');
const path = require('path');
const crypto = require('crypto');
const { Worker, isMainThread, parentPort, workerData } = require('worker_threads');
// Ollama配置
const OLLAMA_CONFIG = {
baseUrl: 'http://localhost:11434',
model: 'deepseek-coder:6.7b',
timeout: 30000
};
// 缓存目录
const CACHE_DIR = path.join(__dirname, '.cache');
const METADATA_FILE = path.join(CACHE_DIR, 'metadata.json');
// 并发控制
const MAX_CONCURRENT_WORKERS = 4;
const WORKER_SCRIPT = path.join(__dirname, 'worker.js');
/**
* Worker线程代码内联
*/
const workerCode = `
const { parentPort, workerData } = require('worker_threads');
const crypto = require('crypto');
const fs = require('fs');
const path = require('path');
const { inputPath, outputPath, useOllama, OLLAMA_CONFIG } = workerData;
// 本地备用方法
function removeConsoleLocally(jsCode) {
return jsCode
.replace(/console\.(log|error|warn|info|debug|assert|trace|table|group|groupEnd|time|timeEnd)\s*\([^;]*?\);?\s*/g, '')
.replace(/\/\*[\s\S]*?\*\//g, '')
.replace(/\/\/.*$/gm, '');
}
// 简单的fetch polyfill当node-fetch不可用时
async function simpleFetch(url, options) {
const https = require('https');
const http = require('http');
const urlObj = new URL(url);
return new Promise((resolve, reject) => {
const lib = urlObj.protocol === 'https:' ? https : http;
const req = lib.request(url, options, (res) => {
let data = '';
res.on('data', chunk => data += chunk);
res.on('end', () => {
resolve({
ok: res.statusCode >= 200 && res.statusCode < 300,
json: async () => JSON.parse(data),
text: async () => data,
status: res.statusCode,
statusText: res.statusMessage
});
});
});
req.on('error', reject);
if (options.body) {
req.write(options.body);
}
req.end();
});
}
async function removeConsoleWithOllama(jsCode) {
const prompt = \`Please remove all console statements from this JavaScript code while preserving the code structure and functionality. Only remove console.log, console.error, console.warn, console.info, console.debug, console.assert, console.trace, console.table, console.group, console.groupEnd, console.time, console.timeEnd calls. Return only the cleaned code without any explanations.
JavaScript code:
\\\`\\\`\\\`javascript
\${jsCode}
\\\`\\\`\\\`
Cleaned code:\`;
try {
const response = await simpleFetch(\`\${OLLAMA_CONFIG.baseUrl}/api/generate\`, {
method: 'POST',
headers: {
'Content-Type': 'application/json',
},
body: JSON.stringify({
model: OLLAMA_CONFIG.model,
prompt: prompt,
stream: false,
options: {
temperature: 0.1,
top_p: 0.9,
max_tokens: 8000
}
})
});
if (!response.ok) {
throw new Error(\`Ollama API error: \${response.status} \${response.statusText}\`);
}
const data = await response.json();
if (!data.response) {
throw new Error('No response from Ollama');
}
let cleanedCode = data.response.trim();
if (cleanedCode.startsWith('\\\`\`\`')) {
const lines = cleanedCode.split('\\n');
if (lines[0].includes('javascript')) {
lines.shift();
} else {
lines.shift();
}
if (lines[lines.length - 1] === '\\\`\`\`') {
lines.pop();
}
cleanedCode = lines.join('\\n');
}
return cleanedCode;
} catch (error) {
throw error;
}
}
async function processFile() {
try {
const originalCode = fs.readFileSync(inputPath, 'utf8');
if (useOllama) {
// 尝试使用Ollama
try {
const cleanedCode = await removeConsoleWithOllama(originalCode);
return { success: true, cleanedCode, useOllama: true };
} catch (ollamaError) {
// Ollama失败使用本地方法
console.log('Ollama failed, using local method:', ollamaError.message);
const cleanedCode = removeConsoleLocally(originalCode);
return { success: true, cleanedCode, useOllama: false, fallback: true };
}
} else {
// 直接使用本地方法
const cleanedCode = removeConsoleLocally(originalCode);
return { success: true, cleanedCode, useOllama: false };
}
} catch (error) {
return { success: false, error: error.message };
}
}
// Worker消息处理
processFile().then(result => {
parentPort.postMessage(result);
}).catch(error => {
parentPort.postMessage({ success: false, error: error.message });
});
`;
/**
* 初始化缓存目录
*/
function initCache() {
if (!fs.existsSync(CACHE_DIR)) {
fs.mkdirSync(CACHE_DIR, { recursive: true });
}
if (!fs.existsSync(METADATA_FILE)) {
fs.writeFileSync(METADATA_FILE, JSON.stringify({}));
}
}
/**
* 计算文件哈希值
*/
function calculateFileHash(filePath) {
const fileContent = fs.readFileSync(filePath, 'utf8');
return crypto.createHash('md5').update(fileContent).digest('hex');
}
/**
* 获取文件元数据
*/
function getMetadata() {
try {
return JSON.parse(fs.readFileSync(METADATA_FILE, 'utf8'));
} catch (error) {
return {};
}
}
/**
* 保存文件元数据
*/
function saveMetadata(metadata) {
fs.writeFileSync(METADATA_FILE, JSON.stringify(metadata, null, 2));
}
/**
* 检查文件是否需要处理(增量处理)
*/
function needsProcessing(inputPath, outputPath, metadata) {
// 如果输出文件不存在,需要处理
if (!fs.existsSync(outputPath)) {
return true;
}
// 获取当前文件哈希
const currentHash = calculateFileHash(inputPath);
const fileInfo = metadata[inputPath];
// 如果没有记录,需要处理
if (!fileInfo) {
return true;
}
// 如果哈希值不同,需要处理
if (fileInfo.hash !== currentHash) {
return true;
}
return false;
}
/**
* 从缓存获取处理结果
*/
function getCachedResult(fileHash) {
const cacheFile = path.join(CACHE_DIR, `${fileHash}.cache`);
if (fs.existsSync(cacheFile)) {
try {
return fs.readFileSync(cacheFile, 'utf8');
} catch (error) {
return null;
}
}
return null;
}
/**
* 保存处理结果到缓存
*/
function saveCachedResult(fileHash, result) {
const cacheFile = path.join(CACHE_DIR, `${fileHash}.cache`);
try {
fs.writeFileSync(cacheFile, result, 'utf8');
} catch (error) {
console.warn('Failed to save cache:', error.message);
}
}
/**
* 本地备用方法
*/
function removeConsoleLocally(jsCode) {
return jsCode
.replace(/console\.(log|error|warn|info|debug|assert|trace|table|group|groupEnd|time|timeEnd)\s*\([^;]*?\);?\s*/g, '')
.replace(/\/\*[\s\S]*?\*\//g, '')
.replace(/\/\/.*$/gm, '');
}
/**
* 检查Ollama服务
*/
async function checkOllamaHealth() {
try {
const response = await fetch(`${OLLAMA_CONFIG.baseUrl}/api/tags`, {
signal: AbortSignal.timeout(5000)
});
return response.ok;
} catch (error) {
return false;
}
}
/**
* 获取目录中的所有JS文件
*/
function collectJsFiles(dir, baseDir = dir) {
const files = [];
function traverse(currentDir) {
const items = fs.readdirSync(currentDir);
for (const item of items) {
const itemPath = path.join(currentDir, item);
const stats = fs.statSync(itemPath);
if (stats.isDirectory()) {
traverse(itemPath);
} else if (path.extname(item) === '.js') {
const relativePath = path.relative(baseDir, itemPath);
files.push({
inputPath: itemPath,
relativePath: relativePath,
outputPath: path.join(baseDir + '-optimized', relativePath)
});
}
}
}
traverse(dir);
return files;
}
/**
* 单线程处理方法(备用)
*/
async function processFilesSingleThread(files, useOllama, dryRun = false, metadata = {}) {
const results = [];
for (const file of files) {
if (!needsProcessing(file.inputPath, file.outputPath, metadata)) {
console.log(`⏭️ Skipping unchanged file: ${file.relativePath}`);
continue;
}
console.log(`🔧 Processing: ${file.relativePath}`);
try {
const originalCode = fs.readFileSync(file.inputPath, 'utf8');
const fileHash = calculateFileHash(file.inputPath);
// 检查缓存
let cleanedCode;
if (useOllama) {
const cached = getCachedResult(fileHash);
if (cached) {
console.log(`📋 Using cached result for: ${file.relativePath}`);
cleanedCode = cached;
} else {
console.log(`🤖 Processing with Ollama: ${file.relativePath}`);
const { removeConsoleWithOllama } = require('./ollama-console-remover.js');
cleanedCode = await removeConsoleWithOllama(originalCode);
saveCachedResult(fileHash, cleanedCode);
}
} else {
console.log(`🔧 Processing locally: ${file.relativePath}`);
cleanedCode = removeConsoleLocally(originalCode);
}
if (!dryRun) {
const outputDir = path.dirname(file.outputPath);
if (!fs.existsSync(outputDir)) {
fs.mkdirSync(outputDir, { recursive: true });
}
fs.writeFileSync(file.outputPath, cleanedCode, 'utf8');
// 更新元数据
metadata[file.inputPath] = {
hash: fileHash,
processedAt: new Date().toISOString(),
size: {
original: originalCode.length,
processed: cleanedCode.length
}
};
}
results.push({
file: file.relativePath,
success: true,
sizeReduction: originalCode.length - cleanedCode.length
});
console.log(`✅ Completed: ${file.relativePath}`);
} catch (error) {
console.error(`❌ Error processing ${file.relativePath}:`, error.message);
if (!dryRun) {
// 复制原文件作为备用
const outputDir = path.dirname(file.outputPath);
if (!fs.existsSync(outputDir)) {
fs.mkdirSync(outputDir, { recursive: true });
}
fs.copyFileSync(file.inputPath, file.outputPath);
}
results.push({
file: file.relativePath,
success: false,
error: error.message
});
}
}
return results;
}
/**
* 并行处理文件(多线程)
*/
async function processFilesParallel(files, useOllama, dryRun = false, metadata = {}) {
console.log(`🚀 Parallel processing ${files.length} files with ${MAX_CONCURRENT_WORKERS} workers...`);
// 准备需要处理的文件
const filesToProcess = files.filter(file =>
needsProcessing(file.inputPath, file.outputPath, metadata)
);
console.log(`📋 Files to process: ${filesToProcess.length}, Skipping: ${files.length - filesToProcess.length}`);
if (filesToProcess.length === 0) {
return [];
}
// 创建Worker池
const workers = [];
const results = [];
let processedCount = 0;
// 创建worker脚本文件
if (!fs.existsSync(WORKER_SCRIPT)) {
fs.writeFileSync(WORKER_SCRIPT, workerCode);
}
return new Promise((resolve, reject) => {
const processNext = () => {
if (filesToProcess.length === 0 && workers.length === 0) {
console.log(`🎉 All ${processedCount} files processed!`);
resolve(results);
return;
}
if (filesToProcess.length === 0) {
return;
}
if (workers.length >= MAX_CONCURRENT_WORKERS) {
return;
}
const file = filesToProcess.shift();
console.log(`🔧 Starting: ${file.relativePath} (${processedCount + 1}/${filesToProcess.length + processedCount + 1})`);
const worker = new Worker(WORKER_SCRIPT, {
workerData: {
inputPath: file.inputPath,
outputPath: file.outputPath,
useOllama: useOllama,
OLLAMA_CONFIG: OLLAMA_CONFIG
}
});
workers.push(worker);
worker.on('message', (result) => {
if (result.success) {
if (!dryRun) {
try {
const outputDir = path.dirname(file.outputPath);
if (!fs.existsSync(outputDir)) {
fs.mkdirSync(outputDir, { recursive: true });
}
fs.writeFileSync(file.outputPath, result.cleanedCode, 'utf8');
// 更新元数据
const originalCode = fs.readFileSync(file.inputPath, 'utf8');
const fileHash = calculateFileHash(file.inputPath);
metadata[file.inputPath] = {
hash: fileHash,
processedAt: new Date().toISOString(),
size: {
original: originalCode.length,
processed: result.cleanedCode.length
}
};
} catch (error) {
console.error(`❌ Failed to save ${file.relativePath}:`, error.message);
}
}
results.push({
file: file.relativePath,
success: true,
sizeReduction: result.sizeReduction || 0
});
console.log(`✅ Completed: ${file.relativePath}`);
} else {
console.error(`❌ Failed: ${file.relativePath} - ${result.error}`);
results.push({
file: file.relativePath,
success: false,
error: result.error
});
}
processedCount++;
const index = workers.indexOf(worker);
if (index > -1) {
workers.splice(index, 1);
}
processNext();
});
worker.on('error', (error) => {
console.error(`❌ Worker error for ${file.relativePath}:`, error.message);
results.push({
file: file.relativePath,
success: false,
error: error.message
});
processedCount++;
const index = workers.indexOf(worker);
if (index > -1) {
workers.splice(index, 1);
}
processNext();
});
worker.on('exit', () => {
const index = workers.indexOf(worker);
if (index > -1) {
workers.splice(index, 1);
}
processNext();
});
};
// 启动处理
for (let i = 0; i < Math.min(MAX_CONCURRENT_WORKERS, filesToProcess.length); i++) {
processNext();
}
});
}
/**
* 主处理函数
*/
async function processDirectory(inputDir, options = {}) {
const {
useOllama = true,
parallel = true,
dryRun = false,
clearCache = false
} = options;
// 初始化
initCache();
if (clearCache) {
console.log('🗑️ Clearing cache...');
fs.rmSync(CACHE_DIR, { recursive: true, force: true });
initCache();
}
const outputDir = inputDir + '-optimized';
const metadata = getMetadata();
// 收集文件
console.log('📂 Scanning files...');
const files = collectJsFiles(inputDir);
const jsFiles = files.filter(f => f.relativePath.endsWith('.js'));
console.log(`📝 Found ${jsFiles.length} JavaScript files`);
// 检查Ollama
let isOllamaAvailable = false;
if (useOllama) {
console.log('🔍 Checking Ollama service...');
isOllamaAvailable = await checkOllamaHealth();
console.log(` Status: ${isOllamaAvailable ? '✅ Available' : '❌ Not available'}`);
}
// 处理文件
const startTime = Date.now();
let results;
if (parallel && isMainThread && jsFiles.length > 3) {
results = await processFilesParallel(jsFiles, isOllamaAvailable, dryRun, metadata);
} else {
results = await processFilesSingleThread(jsFiles, isOllamaAvailable, dryRun, metadata);
}
// 保存元数据
if (!dryRun) {
saveMetadata(metadata);
}
// 统计结果
const successful = results.filter(r => r.success);
const failed = results.filter(r => !r.success);
const totalReduction = successful.reduce((sum, r) => sum + (r.sizeReduction || 0), 0);
const processingTime = (Date.now() - startTime) / 1000;
console.log('');
console.log('📊 Processing Summary:');
console.log(`⏱️ Processing time: ${processingTime.toFixed(2)}s`);
console.log(`✅ Successful: ${successful.length}`);
console.log(`❌ Failed: ${failed.length}`);
console.log(`💾 Size reduction: ${formatBytes(totalReduction)}`);
console.log(`📁 Output directory: ${outputDir}`);
if (failed.length > 0) {
console.log('');
console.log('❌ Failed files:');
failed.forEach(f => console.log(` ${f.file}: ${f.error}`));
}
return {
totalFiles: jsFiles.length,
successful: successful.length,
failed: failed.length,
totalReduction,
processingTime,
outputDir
};
}
/**
* 格式化字节数
*/
function formatBytes(bytes) {
if (bytes === 0) return '0 Bytes';
const k = 1024;
const sizes = ['Bytes', 'KB', 'MB', 'GB'];
const i = Math.floor(Math.log(bytes) / Math.log(k));
return parseFloat((bytes / Math.pow(k, i)).toFixed(2)) + ' ' + sizes[i];
}
/**
* 显示帮助信息
*/
function showHelp() {
console.log(`
Usage: node optimized-processor.js [options] <input-directory>
Arguments:
input-directory Directory containing JavaScript files to process
Options:
--local, -l Use local fallback method instead of Ollama
--single, -s Use single-threaded processing (no parallel)
--dry-run, -d Preview mode - don't save files
--clear-cache, -c Clear existing cache before processing
--help, -h Show this help message
Features:
🚀 Parallel processing with worker threads
📋 Intelligent caching to avoid reprocessing
⏭️ Incremental processing (only changed files)
📊 Detailed performance statistics
`);
}
// 主程序
async function main() {
if (!isMainThread) {
return;
}
const args = process.argv.slice(2);
let inputDir = '';
const options = {};
for (let i = 0; i < args.length; i++) {
const arg = args[i];
if (arg === '--help' || arg === '-h') {
showHelp();
return;
} else if (arg === '--local' || arg === '-l') {
options.useOllama = false;
} else if (arg === '--single' || arg === '-s') {
options.parallel = false;
} else if (arg === '--dry-run' || arg === '-d') {
options.dryRun = true;
} else if (arg === '--clear-cache' || arg === '-c') {
options.clearCache = true;
} else if (!inputDir) {
inputDir = arg;
}
}
if (!inputDir) {
console.error('❌ Input directory is required');
showHelp();
process.exit(1);
}
if (!fs.existsSync(inputDir)) {
console.error(`❌ Input directory "${inputDir}" does not exist`);
process.exit(1);
}
console.log('🚀 Optimized JavaScript Console Remover');
console.log('');
try {
await processDirectory(inputDir, options);
} catch (error) {
console.error('❌ Processing failed:', error.message);
process.exit(1);
}
}
if (require.main === module && isMainThread) {
main().catch(console.error);
}
module.exports = {
processDirectory,
checkOllamaHealth,
collectJsFiles
};

View File

@@ -0,0 +1,463 @@
#!/usr/bin/env node
const fs = require('fs');
const path = require('path');
const crypto = require('crypto');
/**
* 性能测试脚本 - 对比不同处理方法的性能
*/
class PerformanceTest {
constructor() {
this.testDir = path.join(__dirname, 'test-files');
this.results = [];
this.setupTestEnvironment();
}
/**
* 设置测试环境
*/
setupTestEnvironment() {
if (!fs.existsSync(this.testDir)) {
fs.mkdirSync(this.testDir, { recursive: true });
}
this.createTestFiles();
}
/**
* 创建测试文件
*/
createTestFiles() {
const testFiles = [
{
name: 'small.js',
content: `
function smallTest(){console.log("small test");const data={x:1,y:2};console.error("error");return data;}
console.warn("warning");
class Test{constructor(){console.debug("debug");}}
`
},
{
name: 'medium.js',
content: `
function mediumTest(){
console.log("medium test start");
const users = [
{id:1,name:"Alice",console:"test"},
{id:2,name:"Bob",age:25},
{id:3,name:"Charlie",active:true}
];
console.log("Processing users:", users.length);
users.forEach(user => {
console.info("Processing user:", user.name);
user.processed = true;
});
console.debug("Processing complete");
return users;
}
class DataProcessor {
constructor(data) {
console.log("DataProcessor initialized");
this.data = data;
console.debug("Data:", data);
}
process() {
console.info("Starting data processing");
console.time("processing");
const result = this.data.map(item => {
console.debug("Processing item:", item);
return item * 2;
});
console.timeEnd("processing");
console.log("Result:", result);
return result;
}
}
console.warn("Module loaded");
`
},
{
name: 'large.js',
content: `
function largeScaleProcessing(){
console.log("Starting large scale processing");
const startTime = Date.now();
// 大量数据处理
const data = [];
for(let i = 0; i < 1000; i++){
console.debug("Processing item", i);
data.push({
id: i,
value: Math.random() * 100,
timestamp: Date.now()
});
}
console.log("Generated", data.length, "items");
// 复杂处理逻辑
const processedData = data.map((item, index) => {
console.info("Processing item", index, "of", data.length);
if(index % 100 === 0){
console.warn("Progress checkpoint:", index);
}
return {
...item,
processed: true,
calculated: item.value * 2,
category: item.value > 50 ? "high" : "low"
};
});
console.timeEnd("processing");
console.log("Processing complete in", Date.now() - startTime, "ms");
// 统计分析
const stats = {
total: processedData.length,
high: processedData.filter(item => item.category === "high").length,
low: processedData.filter(item => item.category === "low").length
};
console.log("Statistics:", stats);
console.debug("Final data sample:", processedData.slice(0, 5));
return { data: processedData, stats };
}
class AnalyticsEngine {
constructor() {
console.log("AnalyticsEngine constructor");
this.metrics = [];
this.console = "property"; // 测试字符串中的console
}
track(event, data) {
console.info("Tracking event:", event);
const metric = {
event,
data,
timestamp: Date.now()
};
this.metrics.push(metric);
console.debug("Metric recorded:", metric);
}
generateReport() {
console.log("Generating report");
console.time("report-generation");
const report = {
totalMetrics: this.metrics.length,
events: [...new Set(this.metrics.map(m => m.event))],
timeRange: {
start: Math.min(...this.metrics.map(m => m.timestamp)),
end: Math.max(...this.metrics.map(m => m.timestamp))
}
};
console.timeEnd("report-generation");
console.info("Report generated:", report);
return report;
}
}
// 执行测试
console.warn("Large module loaded");
const engine = new AnalyticsEngine();
console.log("Analytics engine ready");
`
}
];
testFiles.forEach(file => {
const filePath = path.join(this.testDir, file.name);
if (!fs.existsSync(filePath)) {
fs.writeFileSync(filePath, file.content);
}
});
}
/**
* 测试方法性能
*/
async testMethod(methodName, processFunction) {
console.log(`\n🧪 Testing ${methodName}...`);
const files = fs.readdirSync(this.testDir).filter(f => f.endsWith('.js'));
const startTime = Date.now();
const results = [];
for (const file of files) {
const filePath = path.join(this.testDir, file);
const inputContent = fs.readFileSync(filePath, 'utf8');
const fileStartTime = Date.now();
try {
const outputContent = await processFunction(inputContent);
const fileEndTime = Date.now();
const fileTime = fileEndTime - fileStartTime;
results.push({
file,
success: true,
inputSize: inputContent.length,
outputSize: outputContent.length,
reduction: inputContent.length - outputContent.length,
time: fileTime
});
console.log(`${file}: ${inputContent.length}${outputContent.length} chars (${fileTime}ms)`);
} catch (error) {
const fileEndTime = Date.now();
const fileTime = fileEndTime - fileStartTime;
results.push({
file,
success: false,
error: error.message,
time: fileTime
});
console.log(`${file}: ${error.message} (${fileTime}ms)`);
}
}
const endTime = Date.now();
const totalTime = endTime - startTime;
const successful = results.filter(r => r.success);
const totalInputSize = successful.reduce((sum, r) => sum + r.inputSize, 0);
const totalOutputSize = successful.reduce((sum, r) => sum + r.outputSize, 0);
const totalReduction = totalInputSize - totalOutputSize;
const summary = {
method: methodName,
totalTime,
files: results.length,
successful: successful.length,
failed: results.length - successful.length,
totalInputSize,
totalOutputSize,
totalReduction,
avgTimePerFile: totalTime / results.length
};
console.log(`📊 ${methodName} Summary:`);
console.log(` Total time: ${totalTime}ms`);
console.log(` Success rate: ${successful.length}/${results.length}`);
console.log(` Size reduction: ${totalReduction} chars (${((totalReduction/totalInputSize)*100).toFixed(1)}%)`);
console.log(` Avg time per file: ${summary.avgTimePerFile.toFixed(1)}ms`);
this.results.push(summary);
return summary;
}
/**
* 正则表达式方法(快速但可能不准确)
*/
regexMethod(input) {
return input.replace(/console\.(log|error|warn|info|debug|assert|trace|table|group|groupEnd|time|timeEnd)\s*\([^;]*?\);?\s*/g, '');
}
/**
* 状态机方法(中等速度,较准确)
*/
stateMachineMethod(input) {
const STATES = { NORMAL: 0, IN_STRING: 1, IN_TEMPLATE_STRING: 2, IN_REGEX: 3, IN_COMMENT: 4 };
let state = STATES.NORMAL;
let stringChar = '';
let result = '';
for (let i = 0; i < input.length; i++) {
const char = input[i];
const nextChar = input[i + 1];
switch (state) {
case STATES.NORMAL:
if (char === '/' && nextChar === '/') {
state = STATES.IN_COMMENT;
i++;
} else if (char === '/' && nextChar === '*') {
state = STATES.IN_COMMENT;
i++;
} else if (char === '"' || char === "'") {
state = STATES.IN_STRING;
stringChar = char;
result += char;
} else if (char === '`') {
state = STATES.IN_TEMPLATE_STRING;
result += char;
} else if (char === '/' && !result.match(/[a-zA-Z0-9_$]$/)) {
state = STATES.IN_REGEX;
result += char;
} else {
result += char;
}
break;
case STATES.IN_STRING:
result += char;
if (char === stringChar) {
state = STATES.NORMAL;
} else if (char === '\\') {
result += input[++i];
}
break;
case STATES.IN_TEMPLATE_STRING:
result += char;
if (char === '`') {
state = STATES.NORMAL;
} else if (char === '\\') {
result += input[++i];
}
break;
case STATES.IN_REGEX:
result += char;
if (char === '/' && !input[i-1].match(/\\/)) {
state = STATES.NORMAL;
}
break;
case STATES.IN_COMMENT:
if (char === '\n') {
state = STATES.NORMAL;
result += char;
} else if (char === '*' && nextChar === '/') {
state = STATES.NORMAL;
i++;
}
break;
}
}
// 移除console语句
return result.replace(/console\.(log|error|warn|info|debug|assert|trace|table|group|groupEnd|time|timeEnd)\s*\([^;]*?\);?\s*/g, '');
}
/**
* Ollama AI方法慢但最准确
*/
async ollamaMethod(input) {
const { removeConsoleWithOllama } = require('./ollama-console-remover.js');
return await removeConsoleWithOllama(input);
}
/**
* 运行所有性能测试
*/
async runAllTests() {
console.log('🚀 Performance Test Suite');
console.log('==========================');
// 测试正则表达式方法
await this.testMethod('Regex (Fast)', (input) => this.regexMethod(input));
// 测试状态机方法
await this.testMethod('State Machine (Medium)', (input) => this.stateMachineMethod(input));
// 测试Ollama方法如果可用
try {
const { checkOllamaHealth } = require('./optimized-processor.js');
const isOllamaAvailable = await checkOllamaHealth();
if (isOllamaAvailable) {
await this.testMethod('Ollama AI (Accurate)', (input) => this.ollamaMethod(input));
} else {
console.log('\n⚠ Ollama not available, skipping AI method test');
console.log(' Make sure Ollama is running: ollama serve');
console.log(' And model is pulled: ollama pull deepseek-coder:6.7b');
}
} catch (error) {
console.log('\n❌ Could not test Ollama method:', error.message);
}
this.generateReport();
}
/**
* 生成性能报告
*/
generateReport() {
console.log('\n📈 Performance Comparison Report');
console.log('=================================');
if (this.results.length === 0) {
console.log('No results to compare');
return;
}
// 找出最快的和最准确的
const fastest = this.results.reduce((min, r) => r.totalTime < min.totalTime ? r : min);
const mostEffective = this.results.reduce((max, r) => r.totalReduction > max.totalReduction ? r : max);
console.log('\n🏆 Winners:');
console.log(` Fastest: ${fastest.method} (${fastest.totalTime}ms)`);
console.log(` Most Effective: ${mostEffective.method} (${mostEffective.totalReduction} chars removed)`);
console.log('\n📊 Detailed Comparison:');
console.log('Method | Time (ms) | Files | Reduction | Avg/File');
console.log('-----------------------|-----------|-------|-----------|----------');
this.results.forEach(result => {
const method = result.method.padEnd(23);
const time = result.totalTime.toString().padStart(9);
const files = result.successful.toString().padStart(5);
const reduction = result.totalReduction.toString().padStart(9);
const avgFile = result.avgTimePerFile.toFixed(1).padStart(8);
console.log(`${method} | ${time} | ${files} | ${reduction} | ${avgFile}`);
});
// 保存报告
const reportData = {
timestamp: new Date().toISOString(),
results: this.results,
summary: {
fastest: fastest.method,
mostEffective: mostEffective.method,
totalTests: this.results.length
}
};
const reportPath = path.join(__dirname, 'performance-report.json');
fs.writeFileSync(reportPath, JSON.stringify(reportData, null, 2));
console.log(`\n💾 Detailed report saved to: ${reportPath}`);
}
}
// 主程序
async function main() {
const tester = new PerformanceTest();
try {
await tester.runAllTests();
} catch (error) {
console.error('❌ Performance test failed:', error.message);
process.exit(1);
}
}
if (require.main === module) {
main().catch(console.error);
}
module.exports = PerformanceTest;

View File

@@ -0,0 +1,144 @@
#!/usr/bin/env node
const path = require('path');
const { processJsFiles, checkOllamaHealth } = require('./ollama-console-remover.js');
/**
* 处理dist目录的便捷脚本
*/
async function processDistDirectory() {
const rootDir = path.dirname(__dirname); // 回到项目根目录
const distDir = path.join(rootDir, 'dist');
const cleanedDir = path.join(rootDir, 'dist-cleaned');
console.log('🚀 Processing dist directory with Ollama...');
console.log('');
// 检查dist目录是否存在
const fs = require('fs');
if (!fs.existsSync(distDir)) {
console.error(`❌ Dist directory not found: ${distDir}`);
console.log('💡 Make sure to build your project first, or specify a different directory');
process.exit(1);
}
// 检查Ollama服务
console.log('🔍 Checking Ollama service...');
const isOllamaAvailable = await checkOllamaHealth();
if (!isOllamaAvailable) {
console.log('⚠️ Ollama service is not available');
console.log(' Starting with local fallback method...');
console.log(' 💡 For better results, start Ollama: ollama serve');
console.log(' And pull the model: ollama pull deepseek-coder:6.7b');
console.log('');
} else {
console.log('✅ Ollama service is available');
console.log('');
}
try {
// 先进行dry run预览
console.log('🔍 Preview mode - checking what will be processed...');
await processJsFiles(distDir, cleanedDir, isOllamaAvailable, true);
console.log('');
console.log('🚀 Starting actual processing...');
// 实际处理
await processJsFiles(distDir, cleanedDir, isOllamaAvailable, false);
console.log('');
console.log('✅ Processing completed!');
console.log(`📁 Original directory: ${distDir}`);
console.log(`📁 Cleaned directory: ${cleanedDir}`);
console.log('');
console.log('💡 You can now use the cleaned files from the dist-cleaned directory');
// 显示文件大小对比
await showSizeComparison(distDir, cleanedDir);
} catch (error) {
console.error('❌ Error during processing:', error.message);
process.exit(1);
}
}
/**
* 显示处理前后的文件大小对比
*/
async function showSizeComparison(originalDir, cleanedDir) {
const fs = require('fs');
function getDirectorySize(dir) {
let totalSize = 0;
let fileCount = 0;
let jsSize = 0;
function traverse(currentDir) {
const files = fs.readdirSync(currentDir);
for (const file of files) {
const filePath = path.join(currentDir, file);
const stats = fs.statSync(filePath);
if (stats.isDirectory()) {
traverse(filePath);
} else {
const size = stats.size;
totalSize += size;
fileCount++;
if (path.extname(file) === '.js') {
jsSize += size;
}
}
}
}
traverse(dir);
return { totalSize, fileCount, jsSize };
}
console.log('');
console.log('📊 Size comparison:');
try {
const original = getDirectorySize(originalDir);
const cleaned = getDirectorySize(cleanedDir);
const totalReduction = original.totalSize - cleaned.totalSize;
const jsReduction = original.jsSize - cleaned.jsSize;
const totalReductionPercent = ((totalReduction / original.totalSize) * 100).toFixed(2);
const jsReductionPercent = original.jsSize > 0 ? ((jsReduction / original.jsSize) * 100).toFixed(2) : 0;
console.log(`📁 Total size: ${formatBytes(original.totalSize)}${formatBytes(cleaned.totalSize)} (-${totalReductionPercent}% = ${formatBytes(totalReduction)})`);
console.log(`📄 JS files: ${formatBytes(original.jsSize)}${formatBytes(cleaned.jsSize)} (-${jsReductionPercent}% = ${formatBytes(jsReduction)})`);
console.log(`📝 File count: ${original.fileCount}${cleaned.fileCount}`);
if (totalReduction > 0) {
console.log('✅ Successfully reduced file size!');
} else {
console.log(' No size reduction (might indicate no console statements were found)');
}
} catch (error) {
console.log('⚠️ Could not calculate size comparison:', error.message);
}
}
/**
* 格式化字节数为人类可读格式
*/
function formatBytes(bytes) {
if (bytes === 0) return '0 Bytes';
const k = 1024;
const sizes = ['Bytes', 'KB', 'MB', 'GB'];
const i = Math.floor(Math.log(bytes) / Math.log(k));
return parseFloat((bytes / Math.pow(k, i)).toFixed(2)) + ' ' + sizes[i];
}
// 执行处理
processDistDirectory().catch(console.error);

View File

@@ -0,0 +1,297 @@
#!/usr/bin/env node
const { Worker, isMainThread } = require('worker_threads');
const path = require('path');
const fs = require('fs');
// 简单的Worker代码
const workerCode = `
const { parentPort, workerData } = require('worker_threads');
const fs = require('fs');
const { inputPath, outputPath } = workerData;
function simpleConsoleRemover(jsCode) {
return jsCode.replace(/console\.(log|error|warn|info|debug)\s*\([^)]*\);?\s*/g, '');
}
async function processFile() {
try {
const originalCode = fs.readFileSync(inputPath, 'utf8');
const cleanedCode = simpleConsoleRemover(originalCode);
// 确保输出目录存在
const path = require('path');
const outputDir = path.dirname(outputPath);
if (!fs.existsSync(outputDir)) {
fs.mkdirSync(outputDir, { recursive: true });
}
fs.writeFileSync(outputPath, cleanedCode, 'utf8');
parentPort.postMessage({
success: true,
inputSize: originalCode.length,
outputSize: cleanedCode.length,
reduction: originalCode.length - cleanedCode.length
});
} catch (error) {
parentPort.postMessage({ success: false, error: error.message });
}
}
processFile();
`;
async function testSimpleParallel() {
console.log('🧪 Simple Parallel Processing Test');
console.log('=================================');
console.log('');
if (!isMainThread) {
console.log('⚠️ This test must be run in main thread');
return;
}
// 创建测试环境
const testDir = path.join(__dirname, 'simple-test');
const testOptimizedDir = testDir + '-parallel';
// 清理旧目录
[testDir, testOptimizedDir].forEach(dir => {
if (fs.existsSync(dir)) {
fs.rmSync(dir, { recursive: true, force: true });
}
});
fs.mkdirSync(testDir, { recursive: true });
// 创建测试文件
console.log('📁 Creating test files...');
const testFiles = [];
for (let i = 0; i < 8; i++) {
const fileName = `test-${i}.js`;
const filePath = path.join(testDir, fileName);
const testContent = `
// Test file ${i}
function test${i}() {
console.log("Starting test ${i}");
const data = { id: ${i}, value: Math.random() };
console.info("Data created:", data);
console.warn("Warning from test ${i}");
return data;
}
console.log("Module ${i} loaded");
`;
fs.writeFileSync(filePath, testContent);
testFiles.push({ fileName, filePath });
}
console.log(`✅ Created ${testFiles.length} test files`);
console.log('');
// 并行处理测试
console.log('🚀 Testing Parallel Processing...');
const parallelStart = Date.now();
// 创建Workers
const workers = [];
const results = [];
for (let i = 0; i < testFiles.length; i++) {
const { fileName, filePath } = testFiles[i];
const outputPath = path.join(testOptimizedDir, fileName);
const worker = new Worker(workerCode, {
eval: true,
workerData: { inputPath: filePath, outputPath }
});
workers.push(worker);
worker.on('message', (result) => {
results.push({ file: fileName, ...result });
console.log(`${result.success ? '✅' : '❌'} ${fileName}`);
if (result.success) {
console.log(` Size: ${result.inputSize}${result.outputSize} chars (${result.reduction} removed)`);
} else {
console.log(` Error: ${result.error}`);
}
});
worker.on('error', (error) => {
results.push({ file: fileName, success: false, error: error.message });
console.log(`${fileName}: ${error.message}`);
});
}
// 等待所有Workers完成
await new Promise((resolve) => {
let completed = 0;
const totalWorkers = workers.length;
workers.forEach(worker => {
worker.on('exit', () => {
completed++;
if (completed === totalWorkers) {
resolve();
}
});
});
});
const parallelTime = Date.now() - parallelStart;
console.log('');
console.log('📊 Parallel Results:');
console.log(` Processing time: ${parallelTime}ms`);
console.log(` Files processed: ${results.filter(r => r.success).length}/${results.length}`);
const totalReduction = results.reduce((sum, r) => sum + (r.reduction || 0), 0);
console.log(` Total reduction: ${totalReduction} characters`);
console.log(` Avg time per file: ${(parallelTime / results.length).toFixed(1)}ms`);
// 验证输出
console.log('');
console.log('🔍 Verifying output...');
if (fs.existsSync(testOptimizedDir)) {
const outputFiles = fs.readdirSync(testOptimizedDir);
console.log(`✅ Generated ${outputFiles.length} output files`);
// 检查第一个文件
const firstOutputPath = path.join(testOptimizedDir, outputFiles[0]);
const firstOutputContent = fs.readFileSync(firstOutputPath, 'utf8');
const hasConsole = firstOutputContent.includes('console.');
console.log(` Console removal: ${hasConsole ? '❌ Failed' : '✅ Success'}`);
}
// 清理
console.log('');
console.log('🧹 Cleaning up...');
[testDir, testOptimizedDir].forEach(dir => {
if (fs.existsSync(dir)) {
fs.rmSync(dir, { recursive: true, force: true });
}
});
console.log('✅ Test completed!');
return {
totalTime: parallelTime,
successful: results.filter(r => r.success).length,
total: results.length,
totalReduction
};
}
async function testSingleThread() {
console.log('🐌 Testing Single-Thread Processing...');
const testDir = path.join(__dirname, 'simple-test');
const testSingleDir = testDir + '-single';
// 简单的本地处理函数
function simpleConsoleRemover(jsCode) {
return jsCode.replace(/console\.(log|error|warn|info|debug)\s*\([^)]*\);?\s*/g, '');
}
const singleStart = Date.now();
let totalReduction = 0;
let successful = 0;
const files = fs.readdirSync(testDir).filter(f => f.endsWith('.js'));
for (const fileName of files) {
const inputPath = path.join(testDir, fileName);
const outputPath = path.join(testSingleDir, fileName);
try {
const originalCode = fs.readFileSync(inputPath, 'utf8');
const cleanedCode = simpleConsoleRemover(originalCode);
const outputDir = path.dirname(outputPath);
if (!fs.existsSync(outputDir)) {
fs.mkdirSync(outputDir, { recursive: true });
}
fs.writeFileSync(outputPath, cleanedCode, 'utf8');
totalReduction += originalCode.length - cleanedCode.length;
successful++;
} catch (error) {
console.log(`${fileName}: ${error.message}`);
}
}
const singleTime = Date.now() - singleStart;
console.log(` Processing time: ${singleTime}ms`);
console.log(` Files processed: ${successful}/${files.length}`);
console.log(` Total reduction: ${totalReduction} characters`);
console.log(` Avg time per file: ${(singleTime / files.length).toFixed(1)}ms`);
return {
totalTime: singleTime,
successful,
total: files.length,
totalReduction
};
}
// 主测试函数
async function main() {
console.log('🎯 Worker Thread Parallel Processing Test');
console.log('========================================');
console.log('');
try {
// 测试并行处理
const parallelResult = await testSimpleParallel();
// 测试单线程处理(使用相同的测试文件)
const singleResult = await testSingleThread();
// 性能对比
console.log('');
console.log('🏁 Performance Comparison:');
console.log('===========================');
console.log(`Parallel: ${parallelResult.totalTime}ms (${parallelResult.successful} files)`);
console.log(`Single: ${singleResult.totalTime}ms (${singleResult.successful} files)`);
if (parallelResult.totalTime > 0 && singleResult.totalTime > 0) {
const speedup = singleResult.totalTime / parallelResult.totalTime;
console.log(`Speedup: ${speedup.toFixed(2)}x`);
if (speedup > 1.5) {
console.log('🎉 Parallel processing is significantly faster!');
} else if (speedup > 1.1) {
console.log('✅ Parallel processing shows improvement');
} else {
console.log('⚠️ Overhead may outweigh benefits for small files');
}
}
console.log('');
console.log('💡 Parallel processing benefits increase with:');
console.log(' • Larger file sizes');
console.log(' • More files to process');
console.log(' • More complex processing logic');
console.log(' • When using Ollama AI processing');
} catch (error) {
console.error('❌ Test failed:', error.message);
console.error(error.stack);
}
}
if (require.main === module) {
main().catch(console.error);
}
module.exports = { testSimpleParallel, testSingleThread };

View File

@@ -0,0 +1,83 @@
#!/usr/bin/env node
const fs = require('fs');
const path = require('path');
const { removeConsoleWithOllama, checkOllamaHealth } = require('./ollama-console-remover.js');
// 测试用的压缩JS文件包含console语句
const testJsCode = `
function testFunction(){console.log("debug info");const data={name:"test",value:123};console.error("error occurred");return data;}console.warn("warning message");class TestClass{constructor(){console.debug("constructor called");this.id=Math.random();}getId(){console.info("getting id");return this.id;}}module.exports={TestClass,testFunction};
`;
async function runTest() {
console.log('🧪 Testing Ollama console remover...');
console.log('');
// 1. 检查Ollama服务
console.log('1⃣ Checking Ollama service...');
const isAvailable = await checkOllamaHealth();
console.log(` Status: ${isAvailable ? '✅ Available' : '❌ Not available'}`);
if (!isAvailable) {
console.log(' Please make sure Ollama is running with:');
console.log(' - Ollama installed: https://ollama.com/');
console.log(' - Ollama running: ollama serve');
console.log(' - Model pulled: ollama pull deepseek-coder:6.7b');
console.log('');
console.log(' Using local fallback method for test...');
// 使用本地备用方法
const { removeConsoleFallback } = require('./ollama-console-remover.js');
const cleanedCode = removeConsoleFallback(testJsCode);
console.log('');
console.log('📝 Original code:');
console.log(testJsCode);
console.log('');
console.log('🧹 Cleaned code (local method):');
console.log(cleanedCode);
console.log('');
console.log(`📊 Size reduction: ${testJsCode.length - cleanedCode.length} characters`);
return;
}
// 2. 测试Ollama处理
console.log('2⃣ Testing with Ollama...');
try {
const cleanedCode = await removeConsoleWithOllama(testJsCode);
console.log('📝 Original code:');
console.log(testJsCode);
console.log('');
console.log('🧹 Cleaned code (Ollama):');
console.log(cleanedCode);
console.log('');
console.log(`📊 Size reduction: ${testJsCode.length - cleanedCode.length} characters`);
console.log('');
console.log('✅ Test completed successfully!');
// 保存测试结果
const testResults = {
original: testJsCode,
cleaned: cleanedCode,
reduction: testJsCode.length - cleanedCode.length,
timestamp: new Date().toISOString()
};
fs.writeFileSync('test-results.json', JSON.stringify(testResults, null, 2));
console.log('💾 Test results saved to test-results.json');
} catch (error) {
console.error('❌ Error testing Ollama:', error.message);
console.log('');
console.log('🔧 Troubleshooting tips:');
console.log('1. Make sure Ollama is running: ollama serve');
console.log('2. Make sure the model is pulled: ollama pull deepseek-coder:6.7b');
console.log('3. Check if Ollama is accessible at http://localhost:11434');
console.log('4. Try running: curl http://localhost:11434/api/tags');
}
}
// 运行测试
runTest().catch(console.error);

View File

@@ -0,0 +1,252 @@
#!/usr/bin/env node
const fs = require('fs');
const path = require('path');
const { Worker, isMainThread } = require('worker_threads');
const { processDirectory } = require('./optimized-processor.js');
/**
* 测试并行处理功能
*/
async function testParallelProcessing() {
console.log('🧪 Testing Parallel Processing');
console.log('==============================');
console.log('');
// 创建测试目录
const testDir = path.join(__dirname, 'parallel-test');
const testOptimizedDir = testDir + '-optimized';
// 清理旧的测试目录
if (fs.existsSync(testDir)) {
fs.rmSync(testDir, { recursive: true, force: true });
}
if (fs.existsSync(testOptimizedDir)) {
fs.rmSync(testOptimizedDir, { recursive: true, force: true });
}
fs.mkdirSync(testDir, { recursive: true });
// 创建多个测试文件
console.log('📁 Creating test files...');
const testFiles = [];
for (let i = 0; i < 10; i++) {
const fileName = `test-file-${i}.js`;
const filePath = path.join(testDir, fileName);
// 生成包含console语句的测试代码
const testCode = `
// Test file ${i}
function test${i}(){
console.log("Test ${i} started");
const data = {
id: ${i},
name: "test${i}",
values: [1, 2, 3, 4, 5]
};
console.info("Processing data:", data);
console.warn("This is a warning from file ${i}");
console.error("Error simulation in file ${i}");
data.processed = true;
console.debug("Processing complete for file ${i}");
return data;
}
class TestClass${i} {
constructor() {
console.log("TestClass${i} constructor");
this.value = ${i} * 10;
console.debug("Value set to:", this.value);
}
getValue() {
console.info("Getting value from TestClass${i}");
return this.value;
}
}
console.log("Module ${i} loaded");
module.exports = { test${i}, TestClass${i} };
`;
fs.writeFileSync(filePath, testCode);
testFiles.push(fileName);
}
console.log(`✅ Created ${testFiles.length} test files`);
console.log('');
// 测试并行处理
console.log('🚀 Testing Parallel Processing...');
const parallelStart = Date.now();
try {
const parallelResult = await processDirectory(testDir, {
useOllama: false, // 使用本地方法避免Ollama依赖
parallel: true, // 启用并行处理
dryRun: false // 实际处理
});
const parallelTime = Date.now() - parallelStart;
console.log('');
console.log('📊 Parallel Processing Results:');
console.log(` Processing time: ${parallelTime}ms`);
console.log(` Files processed: ${parallelResult.successful}`);
console.log(` Average time per file: ${(parallelTime / parallelResult.successful).toFixed(1)}ms`);
// 测试单线程处理作为对比
console.log('');
console.log('🐌 Testing Single-Thread Processing...');
const singleStart = Date.now();
// 清理之前的输出目录
if (fs.existsSync(testOptimizedDir)) {
fs.rmSync(testOptimizedDir, { recursive: true, force: true });
}
const singleResult = await processDirectory(testDir, {
useOllama: false, // 使用本地方法
parallel: false, // 关闭并行处理
dryRun: false
});
const singleTime = Date.now() - singleStart;
console.log('');
console.log('📊 Single-Thread Processing Results:');
console.log(` Processing time: ${singleTime}ms`);
console.log(` Files processed: ${singleResult.successful}`);
console.log(` Average time per file: ${(singleTime / singleResult.successful).toFixed(1)}ms`);
// 性能对比
console.log('');
console.log('🏁 Performance Comparison:');
console.log('================================');
const speedup = singleTime / parallelTime;
const timeSaved = singleTime - parallelTime;
const timeSavedPercent = ((timeSaved / singleTime) * 100).toFixed(1);
console.log(`Parallel time: ${parallelTime}ms`);
console.log(`Single time: ${singleTime}ms`);
console.log(`Speed improvement: ${speedup.toFixed(2)}x faster`);
console.log(`Time saved: ${timeSaved}ms (${timeSavedPercent}%)`);
if (speedup > 1.5) {
console.log('🎉 Parallel processing is significantly faster!');
} else if (speedup > 1.1) {
console.log('✅ Parallel processing shows improvement');
} else {
console.log('⚠️ Parallel processing may not show significant improvement for small files');
}
// 验证输出文件
console.log('');
console.log('🔍 Verifying output files...');
const outputFiles = fs.readdirSync(testOptimizedDir).filter(f => f.endsWith('.js'));
console.log(`✅ Generated ${outputFiles.length} optimized files`);
// 检查第一个文件的内容
if (outputFiles.length > 0) {
const firstOutputPath = path.join(testOptimizedDir, outputFiles[0]);
const firstOutputContent = fs.readFileSync(firstOutputPath, 'utf8');
const hasConsole = firstOutputContent.includes('console.');
if (hasConsole) {
console.log('⚠️ Warning: Output still contains console statements');
} else {
console.log('✅ Console statements successfully removed');
}
}
} catch (error) {
console.error('❌ Test failed:', error.message);
console.error(error.stack);
} finally {
// 清理测试文件
console.log('');
console.log('🧹 Cleaning up test files...');
if (fs.existsSync(testDir)) {
fs.rmSync(testDir, { recursive: true, force: true });
}
if (fs.existsSync(testOptimizedDir)) {
fs.rmSync(testOptimizedDir, { recursive: true, force: true });
}
console.log('✅ Cleanup completed');
}
}
/**
* 检查Worker线程环境
*/
function checkWorkerEnvironment() {
console.log('🔍 Worker Thread Environment Check');
console.log('===================================');
console.log(`isMainThread: ${isMainThread}`);
console.log(`Worker threads available: ${typeof Worker !== 'undefined'}`);
console.log(`Node.js version: ${process.version}`);
console.log('');
}
/**
* 显示并行处理配置信息
*/
function showParallelConfig() {
console.log('⚙️ Parallel Processing Configuration');
console.log('==================================');
const config = {
maxConcurrentWorkers: 4,
workerScript: path.join(__dirname, 'worker.js'),
condition: 'jsFiles.length > 3 AND parallel=true AND isMainThread=true'
};
console.log(`Max concurrent workers: ${config.maxConcurrentWorkers}`);
console.log(`Worker script location: ${config.workerScript}`);
console.log(`Activation condition: ${config.condition}`);
console.log('');
}
// 主程序
async function main() {
console.log('🧪 Parallel Processing Test Suite');
console.log('=================================');
console.log('');
checkWorkerEnvironment();
showParallelConfig();
if (!isMainThread) {
console.log('⚠️ This test should only be run in the main thread');
return;
}
try {
await testParallelProcessing();
} catch (error) {
console.error('❌ Parallel test failed:', error.message);
process.exit(1);
}
console.log('');
console.log('🎉 Parallel processing test completed!');
console.log('');
console.log('💡 To use parallel processing in your projects:');
console.log(' node scripts/optimize-dist.js # Automatic for >3 files');
console.log(' node scripts/optimized-processor.js ./dist --parallel');
}
if (require.main === module) {
main().catch(console.error);
}
module.exports = {
testParallelProcessing,
checkWorkerEnvironment,
showParallelConfig
};

View File

@@ -0,0 +1,53 @@
#!/usr/bin/env node
const fs = require('fs');
const path = require('path');
// 测试runtime.js文件的处理
async function testRuntimeFile() {
console.log('🧪 Testing runtime.js file processing');
console.log('=====================================');
const runtimePath = path.join(__dirname, '../../dist/common/runtime.js');
if (!fs.existsSync(runtimePath)) {
console.log('❌ runtime.js file not found');
return;
}
try {
const originalCode = fs.readFileSync(runtimePath, 'utf8');
console.log(`📄 Original file size: ${originalCode.length} characters`);
// 检查语法错误
const lines = originalCode.split('\n');
let foundError = false;
for (let i = 0; i < lines.length; i++) {
const line = lines[i].trim();
if (line.includes('p[s][0]') && !line.includes('p[s][0];')) {
console.log(`⚠️ Syntax error at line ${i + 1}: ${line.substring(0, 50)}...`);
foundError = true;
break;
}
}
if (foundError) {
console.log('🔧 Syntax error detected - file should be skipped');
} else {
console.log('✅ No obvious syntax errors found');
}
// 尝试本地处理
const cleanedCode = originalCode.replace(/console\.(log|error|warn|info|debug|assert|trace|table|group|groupEnd|time|timeEnd)\s*\([^;]*?\);?\s*/g, '');
console.log(`🧹 Cleaned file size: ${cleanedCode.length} characters`);
console.log(`📉 Size reduction: ${originalCode.length - cleanedCode.length} characters`);
console.log('\n✅ Runtime.js test completed successfully!');
} catch (error) {
console.error('❌ Error processing runtime.js:', error.message);
}
}
testRuntimeFile();

View File

@@ -0,0 +1,82 @@
#!/usr/bin/env node
const fs = require('fs');
const path = require('path');
// 简单的语法检查函数
function hasSyntaxErrors(code) {
const lines = code.split('\n');
for (let i = 0; i < lines.length; i++) {
const line = lines[i].trim();
// 检查常见的语法错误模式
if (line.includes('p[s][0]') && !line.includes('p[s][0];')) {
console.log(`Syntax error detected at line ${i + 1}: missing semicolon after p[s][0]`);
return true;
}
// 检查未闭合的括号(简单检查)
const openBrackets = (line.match(/\[/g) || []).length;
const closeBrackets = (line.match(/\]/g) || []).length;
if (Math.abs(openBrackets - closeBrackets) > 2) {
console.log(`Bracket mismatch detected at line ${i + 1}`);
return true;
}
// 检查其他常见错误模式
if (line.includes('void 0)') && line.includes('if(')) {
console.log(`Conditional syntax error detected at line ${i + 1}`);
return true;
}
}
return false;
}
// 简单的console移除函数
function removeConsoleLocally(jsCode) {
return jsCode
.replace(/console\.(log|error|warn|info|debug|assert|trace|table|group|groupEnd|time|timeEnd)\s*\([^;]*?\);?\s*/g, '')
.replace(/\/\*[\s\S]*?\*\//g, '')
.replace(/\/\/.*$/gm, '');
}
// 测试runtime.js文件
async function testRuntimeProcessing() {
console.log('🧪 Testing Runtime.js with Simple Processor');
console.log('==========================================');
const runtimePath = path.join(__dirname, '../../dist/common/runtime.js');
if (!fs.existsSync(runtimePath)) {
console.log('❌ runtime.js file not found');
return;
}
try {
const originalCode = fs.readFileSync(runtimePath, 'utf8');
console.log(`📄 Original file size: ${originalCode.length} characters`);
// 检查语法错误
const hasError = hasSyntaxErrors(originalCode);
if (hasError) {
console.log('⚠️ Syntax error detected - file would be skipped in parallel processing');
} else {
console.log('✅ No syntax errors detected');
// 尝试处理
const cleanedCode = removeConsoleLocally(originalCode);
console.log(`🧹 Cleaned file size: ${cleanedCode.length} characters`);
console.log(`📉 Size reduction: ${originalCode.length - cleanedCode.length} characters`);
}
console.log('\n✅ Simple processing test completed!');
} catch (error) {
console.error('❌ Error processing runtime.js:', error.message);
}
}
testRuntimeProcessing();

View File

@@ -0,0 +1,43 @@
#!/usr/bin/env node
// 测试语法错误检测功能
const { hasSyntaxErrors } = require('./optimized-processor.js');
// 测试代码
const testCases = [
{
name: 'Normal code',
code: 'function test() { console.log("hello"); return "world"; }',
shouldError: false
},
{
name: 'Runtime.js syntax error',
code: 'p[s][0]', // 缺少分号的情况
shouldError: true
},
{
name: 'Bracket mismatch',
code: 'function test() { return [1, 2, 3; }', // 缺少右括号
shouldError: true
},
{
name: 'Valid bracket code',
code: 'function test() { return [1, 2, 3]; }',
shouldError: false
}
];
console.log('🧪 Testing Syntax Error Detection');
console.log('==================================');
testCases.forEach((testCase, index) => {
console.log(`\n${index + 1}. ${testCase.name}`);
console.log(` Code: ${testCase.code}`);
const hasError = hasSyntaxErrors(testCase.code);
console.log(` Expected: ${testCase.shouldError ? 'Error' : 'No Error'}`);
console.log(` Detected: ${hasError ? 'Error' : 'No Error'}`);
console.log(` Result: ${hasError === testCase.shouldError ? '✅ Pass' : '❌ Fail'}`);
});
console.log('\n🎉 Syntax error detection test completed!');

191
scripts/ollama/worker.js Normal file
View File

@@ -0,0 +1,191 @@
const { parentPort, workerData } = require('worker_threads');
const fs = require('fs');
// 检查是否在 Worker 环境中
const isWorker = typeof parentPort !== 'undefined' && workerData;
if (!isWorker) {
console.log('⚠️ Worker script must be run with worker threads');
process.exit(0);
}
const { inputPath, useOllama, OLLAMA_CONFIG } = workerData;
// 本地备用方法
function removeConsoleLocally(jsCode) {
return jsCode
.replace(/console\.(log|error|warn|info|debug|assert|trace|table|group|groupEnd|time|timeEnd)\s*\([^;]*?\);?\s*/g, '')
.replace(/\/\*[\s\S]*?\*\//g, '')
.replace(/\/\/.*$/gm, '');
}
// 简单的fetch polyfill当node-fetch不可用时
async function simpleFetch(url, options) {
const https = require('https');
const http = require('http');
const urlObj = new URL(url);
return new Promise((resolve, reject) => {
const lib = urlObj.protocol === 'https:' ? https : http;
const req = lib.request(url, options, (res) => {
let data = '';
res.on('data', chunk => data += chunk);
res.on('end', () => {
resolve({
ok: res.statusCode >= 200 && res.statusCode < 300,
json: async () => JSON.parse(data),
text: async () => data,
status: res.statusCode,
statusText: res.statusMessage
});
});
});
req.on('error', reject);
if (options.body) {
req.write(options.body);
}
req.end();
});
}
async function removeConsoleWithOllama(jsCode) {
const prompt = `Please remove all console statements from this JavaScript code while preserving the code structure and functionality. Only remove console.log, console.error, console.warn, console.info, console.debug, console.assert, console.trace, console.table, console.group, console.groupEnd, console.time, console.timeEnd calls. Return only the cleaned code without any explanations.
JavaScript code:
\`\`\`javascript
${jsCode}
\`\`\`
`;
try {
const response = await simpleFetch(`${OLLAMA_CONFIG.baseUrl}/api/generate`, {
method: 'POST',
headers: {
'Content-Type': 'application/json',
},
body: JSON.stringify({
model: OLLAMA_CONFIG.model,
prompt: prompt,
stream: false,
options: {
temperature: 0.1,
top_p: 0.9,
max_tokens: 8000
}
})
});
if (!response.ok) {
throw new Error(`Ollama API error: ${response.status} ${response.statusText}`);
}
const data = await response.json();
if (!data.response) {
throw new Error('No response from Ollama');
}
let cleanedCode = data.response.trim();
if (cleanedCode.startsWith('```')) {
const lines = cleanedCode.split('\n');
if (lines[0].includes('javascript')) {
lines.shift();
} else {
lines.shift();
}
if (lines[lines.length - 1] === '```') {
lines.pop();
}
cleanedCode = lines.join('\n');
}
return cleanedCode;
} catch (error) {
throw error;
}
}
// 检查语法错误避免Worker线程崩溃
function hasSyntaxErrors(code) {
const lines = code.split('\n');
for (let i = 0; i < lines.length; i++) {
const line = lines[i].trim();
// 检查常见的webpack打包错误模式
if (line.includes('p[s][0]') && !line.includes('p[s][0];')) {
return true;
}
// 检查未闭合的正则表达式
if (line.match(/\/[^\/\*]*$/)) {
return true;
}
}
return false;
}
async function processFile() {
try {
const originalCode = fs.readFileSync(inputPath, 'utf8');
// 检查语法错误
if (hasSyntaxErrors(originalCode)) {
return {
success: false,
error: 'Syntax errors detected in original code, skipping processing',
skipped: true
};
}
let cleanedCode;
let processingMethod;
if (useOllama) {
// 尝试使用Ollama
try {
cleanedCode = await removeConsoleWithOllama(originalCode);
processingMethod = 'ollama';
} catch (ollamaError) {
// Ollama失败使用本地方法
console.log('Ollama failed, using local method:', ollamaError.message);
cleanedCode = removeConsoleLocally(originalCode);
processingMethod = 'local-fallback';
}
} else {
// 直接使用本地方法
cleanedCode = removeConsoleLocally(originalCode);
processingMethod = 'local';
}
// 计算压缩效果
const sizeReduction = originalCode.length - cleanedCode.length;
const compressionRatio = originalCode.length > 0 ?
((sizeReduction / originalCode.length) * 100).toFixed(2) : 0;
return {
success: true,
cleanedCode,
processingMethod,
stats: {
originalSize: originalCode.length,
processedSize: cleanedCode.length,
sizeReduction,
compressionRatio: parseFloat(compressionRatio)
}
};
} catch (error) {
return {
success: false,
error: error.message
};
}
}
// Worker消息处理
processFile().then(result => {
parentPort.postMessage(result);
}).catch(error => {
parentPort.postMessage({ success: false, error: error.message });
});