[파이썬/Python] 한국어 STT, CER과 WER 계산

음성 인식에서 WER과 CER은 정확도를 판별하는 지표이다. 계산 방법과 코드는 각각 아래와 같다.

 

① WER(Word Error Rate)

D : 음성 인식된 텍스트에 잘못 삭제된 단어 수

S : 음성 인식된 텍스트에 잘못 대체된 단어 수

I : 음성 인식된 텍스트에 잘못 추가된 단어 수

N : 정답 텍스트의 단어 수

 

단어 에러 비율(WER) = (S+D+I)/N

 

② CER(Chatacter Error Rate)

D : 음성 인식된 텍스트에 잘못 삭제된 음절 수

S : 음성 인식된 텍스트에 잘못 대체된 음절 수

I : 음성 인식된 텍스트에 잘못 추가된 음절 수

N : 정답 텍스트의 음절 수

 

음절 에러 비율(CER) = (S+D+I)/N

 

import Levenshtein as Lev

def wer(ref, hyp ,debug=False):
    r = ref.split()
    h = hyp.split()
    #costs will holds the costs, like in the Levenshtein distance algorithm
    costs = [[0 for inner in range(len(h)+1)] for outer in range(len(r)+1)]
    # backtrace will hold the operations we've done.
    # so we could later backtrace, like the WER algorithm requires us to.
    backtrace = [[0 for inner in range(len(h)+1)] for outer in range(len(r)+1)]

    OP_OK = 0
    OP_SUB = 1
    OP_INS = 2
    OP_DEL = 3

    DEL_PENALTY=1 # Tact
    INS_PENALTY=1 # Tact
    SUB_PENALTY=1 # Tact
    # First column represents the case where we achieve zero
    # hypothesis words by deleting all reference words.
    for i in range(1, len(r)+1):
        costs[i][0] = DEL_PENALTY*i
        backtrace[i][0] = OP_DEL

    # First row represents the case where we achieve the hypothesis
    # by inserting all hypothesis words into a zero-length reference.
    for j in range(1, len(h) + 1):
        costs[0][j] = INS_PENALTY * j
        backtrace[0][j] = OP_INS

    # computation
    for i in range(1, len(r)+1):
        for j in range(1, len(h)+1):
            if r[i-1] == h[j-1]:
                costs[i][j] = costs[i-1][j-1]
                backtrace[i][j] = OP_OK
            else:
                substitutionCost = costs[i-1][j-1] + SUB_PENALTY # penalty is always 1
                insertionCost    = costs[i][j-1] + INS_PENALTY   # penalty is always 1
                deletionCost     = costs[i-1][j] + DEL_PENALTY   # penalty is always 1

                costs[i][j] = min(substitutionCost, insertionCost, deletionCost)
                if costs[i][j] == substitutionCost:
                    backtrace[i][j] = OP_SUB
                elif costs[i][j] == insertionCost:
                    backtrace[i][j] = OP_INS
                else:
                    backtrace[i][j] = OP_DEL

    # back trace though the best route:
    i = len(r)
    j = len(h)
    numSub = 0
    numDel = 0
    numIns = 0
    numCor = 0
    if debug:
        print("OP\tREF\tHYP")
        lines = []
    while i > 0 or j > 0:
        if backtrace[i][j] == OP_OK:
            numCor += 1
            i-=1
            j-=1
            if debug:
                lines.append("OK\t" + r[i]+"\t"+h[j])
        elif backtrace[i][j] == OP_SUB:
            numSub +=1
            i-=1
            j-=1
            if debug:
                lines.append("SUB\t" + r[i]+"\t"+h[j])
        elif backtrace[i][j] == OP_INS:
            numIns += 1
            j-=1
            if debug:
                lines.append("INS\t" + "****" + "\t" + h[j])
        elif backtrace[i][j] == OP_DEL:
            numDel += 1
            i-=1
            if debug:
                lines.append("DEL\t" + r[i]+"\t"+"****")
    if debug:
        lines = reversed(lines)
        for line in lines:
            print(line)
        print("Ncor " + str(numCor))
        print("Nsub " + str(numSub))
        print("Ndel " + str(numDel))
        print("Nins " + str(numIns))
    return numCor, numSub, numDel, numIns, (numSub + numDel + numIns) / (float) (len(r))
    
# Run:
# ref='Tuan anh mot ha chin'
# hyp='tuan anh mot hai ba bon chin'
# wer(ref, hyp ,debug=True)

# OP	REF	     HYP
# SUB	Tuan     tuan
# OK	anh	     anh
# OK	mot	     mot
# INS	****	 hai
# INS	****	 ba
# SUB	ha	     bon
# OK	chin	 chin
# Ncor 3
# Nsub 2
# Ndel 0
# Nins 2
# 0.8

def cer(ref, hyp):
    ref = ref.replace(' ', '')
    hyp = hyp.replace(' ', '')
    dist = Lev.distance(hyp, ref)
    length = len(ref)
    return dist, length, dist/length

비율만 출력하려면 return 부분을 좀 손보면 된다.