在本文中我們將研究以下主題
我們將從建立 React hook 開始，它將執行所有操作，例如 startRecording、stopRecording、建立 Audio Blob、錯誤處理等。
在我們進入正題之前，還有一些其他事情要注意
const VOICE_MIN_DECIBELS = -35 const DELAY_BETWEEN_DIALOGUE = 2000
讓我們將我們的鉤子命名為useAudioInput.ts，我們將使用瀏覽器api，如navigator.mediaDevices.getUserMedia、MediaRecorder和AudioContext。 AudioContext 將幫助我們識別輸入音訊是否高於被視為輸入所需的最小分貝，因此我們將從以下變數和道具開始
const defaultConfig = {
audio: true
};
type Payload = Blob;
type Config = {
audio: boolean;
timeSlice?: number
timeInMillisToStopRecording?: number
onStop: () => void;
onDataReceived: (payload: Payload) => void
};
export const useAudioInput = (config: Config = defaultConfig) => {
const mediaChunks = useRef<Blob[]>([]);
const [isRecording, setIsRecording] = useState(false);
const mediaRecorder = useRef<MediaRecorder | null>(null);
const [error, setError] = useState<Error| null>(null);
let requestId: number;
let timer: ReturnType<typeof setTimeout>;
const createBlob = () => {
const [chunk] = mediaChunks.current;
const blobProperty = { type: chunk.type };
return new Blob(mediaChunks.current, blobProperty)
}
...
}
在上面的程式碼中，我們將使用 mediaChunks 作為變數來保存輸入 blob 和 mediaRecorder 來擁有新 MediaRecorder 的實例，該實例將流作為來自 navigator.mediaDevices.getUserMedia 的輸入。接下來讓我們處理 getUserMedia 不可用的情況
...
useEffect(() => {
if(!navigator.mediaDevices || !navigator.mediaDevices.getUserMedia) {
const notAvailable = new Error('Your browser does not support Audio Input')
setError(notAvailable)
}
},[]);
...
我們將開始編寫鉤子的實際功能，其中包括各種函數，如 setupMediaRecorder、setupAudioContext、onRecordingStart、onRecordingActive、startRecording、stopRecording 等。
const onRecordingStart = () => mediaChunks.current = [];
const onRecordingActive = useCallback(({data}: BlobEvent) => {
if(data) {
mediaChunks.current.push(data);
config?.onDataReceived?.(createBlob())
}
},[config]);
const startTimer = () => {
timer = setTimeout(() => {
stopRecording();
}, config.timeInMillisToStopRecording)
};
const setupMediaRecorder = ({stream}:{stream: MediaStream}) => {
mediaRecorder.current = new MediaRecorder(stream)
mediaRecorder.current.ondataavailable = onRecordingActive
mediaRecorder.current.onstop = onRecordingStop
mediaRecorder.current.onstart = onRecordingStart
mediaRecorder.current.start(config.timeSlice)
};
const setupAudioContext = ({stream}:{stream: MediaStream}) => {
const audioContext = new AudioContext();
const audioStreamSource = audioContext.createMediaStreamSource(stream);
const analyser = audioContext.createAnalyser();
analyser.minDecibels = VOICE_MIN_DECIBELS;
audioStreamSource.connect(analyser);
const bufferLength = analyser.frequencyBinCount;
const domainData = new Uint8Array(bufferLength)
return {
domainData,
bufferLength,
analyser
}
};
const startRecording = async () => {
setIsRecording(true);
await navigator.mediaDevices
.getUserMedia({
audio: config.audio
})
.then((stream) => {
setupMediaRecorder({stream});
if(config.timeSlice) {
const { domainData, analyser, bufferLength } = setupAudioContext({ stream });
startTimer()
}
})
.catch(e => {
setError(e);
setIsRecording(false)
})
};
const stopRecording = () => {
mediaRecorder.current?.stop();
clearTimeout(timer);
window.cancelAnimationFrame(requestId);
setIsRecording(false);
onRecordingStop()
};
const createBlob = () => {
const [chunk] = mediaChunks.current;
const blobProperty = { type: chunk.type };
return new Blob(mediaChunks.current, blobProperty)
}
const onRecordingStop = () => config?.onStop?.();
透過上面的程式碼我們已經差不多完成了鉤子，唯一懸而未決的事情是識別用戶是否已經停止說話，如果2 沒有輸入，我們將使用DELAY_BETWEEN_DIALOGUE 作為我們要等待的時間秒後，我們將假設用戶已停止講話並將點擊語音轉文字端點。
...
const detectSound = ({
recording,
analyser,
bufferLength,
domainData
}: {
recording: boolean
analyser: AnalyserNode
bufferLength: number
domainData: Uint8Array
}) => {
let lastDetectedTime = performance.now();
let anySoundDetected = false;
const compute = () => {
if (!recording) {
return;
}
const currentTime = performance.now();
const timeBetweenTwoDialog =
anySoundDetected === true && currentTime - lastDetectedTime > DELAY_BETWEEN_DIALOGUE;
if (timeBetweenTwoDialog) {
stopRecording();
return;
}
analyser.getByteFrequencyData(domainData);
for (let i = 0; i < bufferLength; i += 1) {
if (domainData[i] > 0) {
anySoundDetected = true;
lastDetectedTime = performance.now();
}
}
requestId = window.requestAnimationFrame(compute);
};
compute();
}
...
const startRecording = async () => {
...
detectSound()
...
}
在上面的程式碼中，我們使用 requestAnimationFrame 來偵測使用者音訊輸入，這樣我們就完成了鉤子，現在可以開始在各個地方使用鉤子了。
例如
const onDataReceived = async (data: BodyInit) => {
const rawResponse = await fetch('https://backend-endpoint', {
method: 'POST',
body: data
});
const response = await rawResponse.json();
setText(response)
};
const { isRecording, startRecording, error } = useAudioInput({
audio: true,
timeInMillisToStopRecording: 2000,
timeSlice: 400,
onDataReceived
})
第二部分是連接一個節點伺服器，它可以與Google語音到文字 API 進行通信，我已經附上了我在創建節點方面時引用的文檔。
https://codelabs.developers.google.com/codelabs/cloud-speech-text-node。
// demo node server which connects with google speech to text api endpoint
const express = require('express');
const cors = require('cors');
const speech = require('@google-cloud/speech');
const client = new speech.SpeechClient();
async function convert(audioBlob) {
const request = {
config: {
encoding: 'WEBM_OPUS', // Ensure this matches the format of the audio being sent
sampleRateHertz: 48000, // This should match the sample rate of your recording
languageCode: 'en-US'
},
audio: {
content: audioBlob
}
};
const [response] = await client.recognize(request);
const transcription = response.results
.map(result => result.alternatives[0].transcript)
.join('\n');
return transcription;
}
const app = express();
app.use(cors())
app.use(express.json());
app.post('/upload', express.raw({ type: '*/*' }), async (req, res) => {
const audioBlob = req.body;
const response = await convert(audioBlob);
res.json(response);
});
app.listen(4000,'0.0.0.0', () => {
console.log('Example app listening on port 4000!');
});
在本文中，我介紹瞭如何將音訊內容或 blob 發送到 google 語音轉文字端點，我們還可以發送 blob uri 而不是內容，唯一的變化是有效負載
// sending url as part of audio object to speech to text api
...
audio: {url: audioUrl} or audio: {content: audioBlob}
...
與本文相關的程式碼存在於 Github 中。
以上是透過 Google Speech to Text 進行音訊轉文字輸入的詳細內容。更多資訊請關注PHP中文網其他相關文章！
