public-archive/multi-speaker-tacotron-tensorflow
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity

initial commit

Browse source
This commit is contained in:
Taehoon Kim 2017-10-15 23:00:17 +09:00
commit 0a6f6db0b1
56 changed files with 49395 additions and 0 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

132
.gitignore vendored Normal file
View file

@ -0,0 +1,132 @@
# Text
*.png
*.txt
*.json
*.csv
# Data
logs
*.npy
*.npz
*.tar
*.tar.gz
# Media
*.mp4
*.mp3
*.flac
*.wav
*.ts
.DS_Store
# Created by https://www.gitignore.io/api/python,vim
### Python ###
# Byte-compiled / optimized / DLL files
__pycache__/
*.py[cod]
*$py.class
# C extensions
*.so
# Distribution / packaging
.Python
env/
build/
develop-eggs/
dist/
downloads/
eggs/
.eggs/
lib64/
parts/
sdist/
var/
wheels/
*.egg-info/
.installed.cfg
*.egg
# PyInstaller
# Usually these files are written by a python script from a template
# before PyInstaller builds the exe, so as to inject date/other infos into it.
*.manifest
*.spec
# Installer logs
pip-log.txt
pip-delete-this-directory.txt
# Unit test / coverage reports
htmlcov/
.tox/
.coverage
.coverage.*
.cache
nosetests.xml
coverage.xml
*,cover
.hypothesis/
# Translations
*.mo
*.pot
# Django stuff:
*.log
local_settings.py
# Flask stuff:
instance/
.webassets-cache
# Scrapy stuff:
.scrapy
# Sphinx documentation
docs/_build/
# PyBuilder
target/
# Jupyter Notebook
.ipynb_checkpoints
# pyenv
.python-version
# celery beat schedule file
celerybeat-schedule
# dotenv
.env
# virtualenv
.venv/
venv/
ENV/
# Spyder project settings
.spyderproject
# Rope project settings
.ropeproject
### Vim ###
# swap
[._]*.s[a-v][a-z]
[._]*.sw[a-p]
[._]s[a-v][a-z]
[._]sw[a-p]
# session
Session.vim
# temporary
.netrwhist
*~
# auto-generated tag files
tags
# End of https://www.gitignore.io/api/python,vim

3
DISCLAIMER Normal file
View file

@ -0,0 +1,3 @@
This is not an official [DEVSISTERS](http://devsisters.com/) product and is not responsible for misuse or for any damage that you may cause. You agree that you use this software at your own risk.
이것은 [데브시스터즈](http://devsisters.com/)의 공식적인 제품이 아닙니다. [데브시스터즈](http://devsisters.com )는 이 코드를 잘못 사용했을 시 발생한 문제나 이슈에 대한 책임을 지지 않으며 이 소프트웨어의 사용은 사용자 자신에>게 전적으로 책임이 있습니다.

40
LICENSE Normal file
View file

@ -0,0 +1,40 @@
Copyright (c) 2017 Devsisters
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in
all copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
THE SOFTWARE.
Copyright (c) 2017 Keith Ito
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in
all copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
THE SOFTWARE.

168
README.md Normal file
View file

@ -0,0 +1,168 @@
# Multi-Speaker Tacotron in TensorFlow
[[한국어 가이드](./README_ko.md)]
TensorFlow implementation of:
- [Deep Voice 2: Multi-Speaker Neural Text-to-Speech](https://arxiv.org/abs/1705.08947)
- [Listening while Speaking: Speech Chain by Deep Learning](https://arxiv.org/abs/1707.04879)
- [Tacotron: Towards End-to-End Speech Synthesis](https://arxiv.org/abs/1703.10135)
Samples audios (in Korean) can be found [here](http://carpedm20.github.io/tacotron/en.html).
![model](./assets/model.png)
## Prerequisites
- Python 3.6+
- [Tensorflow 1.3](https://www.tensorflow.org/install/)
## Usage
### 1. Install prerequisites
After preparing [Tensorflow](https://www.tensorflow.org/install/), install prerequisites with:
pip3 install -r requirements.txt
If you want to synthesize a speech in Korean dicrectly, follow [2-3. Download pre-trained models](#2-3-download-pre-trained-models).
### 2-1. Generate custom datasets
The `datasets` directory should look like:
datasets
├── jtbc
│ ├── alignment.json
│ └── audio
│ ├── 1.mp3
│ ├── 2.mp3
│ ├── 3.mp3
│ └── ...
└── YOUR_DATASET
├── alignment.json
└── audio
├── 1.mp3
├── 2.mp3
├── 3.mp3
└── ...
and `YOUR_DATASET/alignment.json` should look like:
{
"./datasets/YOUR_DATASET/audio/001.mp3": "My name is Taehoon Kim.",
"./datasets/YOUR_DATASET/audio/002.mp3": "The buses aren't the problem.",
"./datasets/YOUR_DATASET/audio/003.mp3": "They have discovered a new particle.",
}
After you prepare as described, you should genearte preprocessed data with:
python -m datasets.generate_data ./datasets/YOUR_DATASET/alignment.json
### 2-2. Generate Korean datasets
You can generate datasets for 3 public Korean figures including:
1. [Sohn Suk-hee](https://en.wikipedia.org/wiki/Sohn_Suk-hee): anchor and president of JTBC
2. [Park Geun-hye](https://en.wikipedia.org/wiki/Park_Geun-hye): a former President of South Korea
3. [Moon Jae-in](https://en.wikipedia.org/wiki/Moon_Jae-in): the current President of South Korea
Each dataset can be generated with following scripts:
./scripts/prepare_son.sh # Sohn Suk-hee
./scripts/prepare_park.sh # Park Geun-hye
./scripts/prepare_moon.sh # Moon Jae-in
Each script execute below commands. (explain with `son` dataset)
0. To automate an alignment between sounds and texts, prepare `GOOGLE_APPLICATION_CREDENTIALS` to use [Google Speech Recognition API](https://cloud.google.com/speech/). To get credentials, read [this](https://developers.google.com/identity/protocols/application-default-credentials).
export GOOGLE_APPLICATION_CREDENTIALS="YOUR-GOOGLE.CREDENTIALS.json"
1. Download speech(or video) and text.
python -m datasets.son.download
2. Segment all audios on silence.
python -m audio.silence --audio_pattern "./datasets/son/audio/*.wav" --method=pydub
3. By using [Google Speech Recognition API](https://cloud.google.com/speech/), we predict sentences for all segmented audios. (this is optional for `moon` and `park` because they already have `alignment.json`)
python -m recognition.google --audio_pattern "./datasets/son/audio/*.*.wav"
4. By comparing original text and recognised text, save `audio<->text` pair information into `./datasets/son/alignment.json`.
python -m recognition.alignment --recognition_path "./datasets/son/recognition.json" --score_threshold=0.5
5. Finally, generated numpy files which will be used in training.
python3 -m datasets.synthesizer_data ./datasets/son/alignment.json
Because the automatic generation is extremely naive, the dataset is noisy. However, if you have enough datasets (20+ hours with random initialization or 5+ hours with pretrained model initialization), you can expect an acceptable quality of audio synthesis.
### 2-3. Download pre-trained models
You can download a pre-trained models or generate audio. Available models are:
1. Single speaker model for [Sohn Suk-hee](https://en.wikipedia.org/wiki/Sohn_Suk-hee).
python3 download.py son
2. Single speaker model for [Park Geun-hye](https://en.wikipedia.org/wiki/Park_Geun-hye).
python3 download.py park
After you donwload pre-trained models, you can generate voices as follows:
python3 synthesizer.py --load_path logs/son-20171015 --text "이거 실화냐?"
python3 synthesizer.py --load_path logs/park-20171015 --text "이거 실화냐?"
**WARNING: The two pre-trained models are being made available for research purpose only.**
### 3. Train a model
To train a single-speaker model:
python train.py --data_path=datasets/jtbc
python train.py --data_path=datasets/park --initialize_path=PATH_TO_CHECKPOINT
To train a multi-speaker model:
python train.py --data_path=datasets/jtbc,datasets/park
If you don't have good and enough (10+ hours) dataset, it would be better to use `--initialize_path` to use a well-trained model as initial parameters.
### 4. Synthesize audio
You can train your own models with:
python3 app.py --load_path logs/park-20171015 --num_speakers=1
or generate audio directly with:
python3 synthesizer.py --load_path logs/park-20171015 --text "이거 실화냐?"
## Disclaimer
This is not an official [DEVSISTERS](http://devsisters.com/) product. This project is not responsible for misuse or for any damage that you may cause. You agree that you use this software at your own risk.
## References
- [Keith Ito](https://github.com/keithito)'s [tacotron](https://github.com/keithito/tacotron)
- [DEVIEW 2017 presentation](https://deview.kr/2017/schedule/182) (Korean)
## Author
Taehoon Kim / [@carpedm20](http://carpedm20.github.io/)

169
README_ko.md Normal file
View file

@ -0,0 +1,169 @@
# D.Voice: 오픈소스 딥러닝 음성 합성 엔진
[[English Guide](./README.md)]
D.Voice는 TensorFlow로 구현된 오픈소스 딥러닝 음성 합성 엔진입니다. 이 프로젝트는:
- [Deep Voice 2: Multi-Speaker Neural Text-to-Speech](https://arxiv.org/abs/1705.08947)
- [Listening while Speaking: Speech Chain by Deep Learning](https://arxiv.org/abs/1707.04879)
- [Tacotron: Towards End-to-End Speech Synthesis](https://arxiv.org/abs/1703.10135)
위 세 논문의 모델들의 구현을 포함하고 있습니다. 음성 데모는 [여기](http://carpedm20.github.io/tacotron/)서 들어보실 수 있습니다.
![model](./assets/model.png)
## Prerequisites
- Python 3.6+
- [Tensorflow 1.3](https://www.tensorflow.org/install/)
## 사용 방법
### 1. 필수 라이브러리 설치
[Tensorflow 1.3](https://www.tensorflow.org/install/)를 설치한 후, 아래 명령어로 필수 라이브러리를 설치합니다.
pip3 install -r requirements.txt
바로 음성을 만들고 싶으면 [2-3. 학습된 모델 다운받기](#2-4-미리-학습된-모델-다운받기)를 따라하시면 됩니다.
### 2-1. 학습할 데이터 준비하기
`datasets` 디렉토리는 다음과 같이 구성되어야 합니다:
datasets
├── son
│ ├── alignment.json
│ └── audio
│ ├── 1.mp3
│ ├── 2.mp3
│ ├── 3.mp3
│ └── ...
└── 아무개
├── alignment.json
└── audio
├── 1.mp3
├── 2.mp3
├── 3.mp3
└── ...
그리고 `아무개/alignment.json`는 아래와 같은 포멧으로 `json` 형태로 준비해 주세요.
{
"./datasets/아무개/audio/001.mp3": "존경하는 국민 여러분",
"./datasets/아무개/audio/002.mp3": "국회의장과 국회의원 여러분",
"./datasets/아무개/audio/003.mp3": "저는 오늘",
}
`datasets`와 `아무개/alignment.json`가 준비되면, 아래 명령어로 학습 데이터를 만드시면 됩니다:
python3 -m datasets.synthesizer_data ./datasets/아무개/alignment.json
### 2-2. {손석희, 문재인, 박근혜} 데이터 만들기
만약 음성 데이터가 없으시다면, 3명의 한국인 음성 데이터를 만드실 수 있습니다:
1. [손석희](https://ko.wikipedia.org/wiki/%EC%86%90%EC%84%9D%ED%9D%AC)
2. [박근혜](https://ko.wikipedia.org/wiki/%EB%B0%95%EA%B7%BC%ED%98%9C)
3. [문재인](https://ko.wikipedia.org/wiki/%EB%AC%B8%EC%9E%AC%EC%9D%B8)
각각의 데이터는 아래 스크립트로 만들 수 있으며,
./scripts/prepare_son.sh # 손석희
./scripts/prepare_park.sh # 박근혜
./scripts/prepare_moon.sh # 문재인
각 스크립트는 아래와 같은 명령어를 실행합니다. (son 기준으로 설명합니다)
0. 자동으로 `음성<->텍스트` 페어를 만들기 위해 [구글 음성 인식 API](https://cloud.google.com/speech/)를 사용하며, `GOOGLE_APPLICATION_CREDENTIALS`를 준비해야 합니다. `GOOGLE_APPLICATION_CREDENTIALS`를 얻기 위해서는 [여기](https://developers.google.com/identity/protocols/application-default-credentials)를 참고해 주세요.
export GOOGLE_APPLICATION_CREDENTIALS="YOUR-GOOGLE.CREDENTIALS.json"
1. 음성(혹은 영상)과 텍스트 데이터를 다운로드 받습니다.
python -m datasets.son.download
2. 음성을 정적을 기준으로 분리합니다.
python -m audio.silence --audio_pattern "./datasets/son/audio/*.wav" --method=pydub
3. 작게 분리된 음성들을 [Google Speech Recognition API](https://cloud.google.com/speech/)를 사용해 대략적인 문장들을 예측합니다.
python -m recognition.google --audio_pattern "./datasets/son/audio/*.*.wav"
4. 기존의 텍스트와 음성 인식으로 예측된 텍스트를 비교해 `음성<->텍스트` 쌍 정보를 `./datasets/son/alignment.json`에 저장합니다. (`moon`과 `park` 데이터셋은 `alignment.json`이 이미 있기 때문에 이 과정은 생략하셔도 됩니다.)
python -m recognition.alignment --recognition_path "./datasets/son/recognition.json" --score_threshold=0.5
5. 마지막으로 학습에 사용될 numpy 파일들을 만듭니다.
python3 -m datasets.synthesizer_data ./datasets/son/alignment.json
자동화 과정이 굉장히 간단하기 때문에, 데이터에 노이즈가 많이 존재합니다. 하지만 오디오와 텍스트가 충분히 많이 있다면 (처음부터 학습시 20시간 이상, 미리 학습된 모델에서 학습시 5+시간 이상) 적당한 퀄리티의 음성 합성을 기대할 수 있습니다.
### 2-4. 미리 학습된 모델 다운받기
미리 학습된 모델들을 사용해 음성을 만들거나 모델을 학습시킬 수 있습니다. 아래 모델 중 하나를 다운로드 받으시고:
1. 단일 화자 모델 - [손석희](https://ko.wikipedia.org/wiki/%EC%86%90%EC%84%9D%ED%9D%AC)
python3 download.py son
2. 단일 화자 모델 - [박근혜](https://ko.wikipedia.org/wiki/%EB%B0%95%EA%B7%BC%ED%98%9C)
python3 download.py park
학습된 모델을 다운받으시고, 아래 명령어로 음성을 만들어 낼 수 있습니다:
python3 synthesizer.py --load_path logs/son-20171015 --text "이거 실화냐?"
python3 synthesizer.py --load_path logs/park-20171015 --text "이거 실화냐?"
**주의: 학습된 모델을 연구 이외의 목적으로 사용하는 것을 금지합니다.**
### 3. 모델 학습하기
단일 화자 모델을 학습하려면:
python3 train.py --data_path=datasets/son
python3 train.py --data_path=datasets/park --initialize_path logs/son-20171015
다중 화자 모델을 학습하려면:
python3 train.py --data_path=datasets/son,datasets/park
학습 데이터가 좋지 않다면 `--initialize_path`로 이미 학습된 모델의 파라미터로 초기화 해서 학습하시는 것이 좋습니다.
### 4. 음성 만들기
모델을 학습시킨 후 웹 데모를 통해 음성을 만들거나:
python app.py --load_path logs/park-20171015 --num_speakers=1
아래 명령어로 음성을 만들 수 있습니다:
python3 synthesizer.py --load_path logs/park-20171015 --text "이거 실화냐?"
## Disclaimer
이것은 [데브시스터즈](http://devsisters.com/)의 공식적인 제품이 아닙니다. [데브시스터즈](http://devsisters.com/)는 이 코드를 잘못 사용했을 시 발생한 문제나 이슈에 대한 책임을 지지 않으며 이 소프트웨어의 사용은 사용자 자신에게 전적으로 책임이 있습니다.
## References
- [Keith Ito](https://github.com/keithito)'s [tacotron](https://github.com/keithito/tacotron)
- [DEVIEW 2017 발표 자료](https://www.slideshare.net/carpedm20/deview-2017-80824162)
## Author
Taehoon Kim / [@carpedm20](http://carpedm20.github.io/)

133
app.py Normal file
View file

@ -0,0 +1,133 @@
#!flask/bin/python
import os
import hashlib
import argparse
from flask_cors import CORS
from flask import Flask, request, render_template, jsonify, \
send_from_directory, make_response, send_file
from hparams import hparams
from audio import load_audio
from synthesizer import Synthesizer
from utils import str2bool, prepare_dirs, makedirs, add_postfix
ROOT_PATH = "web"
AUDIO_DIR = "audio"
AUDIO_PATH = os.path.join(ROOT_PATH, AUDIO_DIR)
base_path = os.path.dirname(os.path.realpath(__file__))
static_path = os.path.join(base_path, 'web/static')
global_config = None
synthesizer = Synthesizer()
app = Flask(__name__, root_path=ROOT_PATH, static_url_path='')
CORS(app)
def match_target_amplitude(sound, target_dBFS):
change_in_dBFS = target_dBFS - sound.dBFS
return sound.apply_gain(change_in_dBFS)
def amplify(path, keep_silence=300):
sound = AudioSegment.from_file(path)
nonsilent_ranges = pydub.silence.detect_nonsilent(
sound, silence_thresh=-50, min_silence_len=300)
new_sound = None
for idx, (start_i, end_i) in enumerate(nonsilent_ranges):
if idx == len(nonsilent_ranges) - 1:
end_i = None
amplified_sound = \
match_target_amplitude(sound[start_i:end_i], -20.0)
if idx == 0:
new_sound = amplified_sound
else:
new_sound = new_sound.append(amplified_sound)
if idx < len(nonsilent_ranges) - 1:
new_sound = new_sound.append(sound[end_i:nonsilent_ranges[idx+1][0]])
return new_sound.export("out.mp3", format="mp3")
def generate_audio_response(text, speaker_id):
global global_config
model_name = os.path.basename(global_config.load_path)
hashed_text = hashlib.md5(text.encode('utf-8')).hexdigest()
relative_dir_path = os.path.join(AUDIO_DIR, model_name)
relative_audio_path = os.path.join(
relative_dir_path, "{}.{}.wav".format(hashed_text, speaker_id))
real_path = os.path.join(ROOT_PATH, relative_audio_path)
makedirs(os.path.dirname(real_path))
if not os.path.exists(add_postfix(real_path, 0)):
try:
audio = synthesizer.synthesize(
[text], paths=[real_path], speaker_ids=[speaker_id],
attention_trim=True)[0]
except:
return jsonify(success=False), 400
return send_file(
add_postfix(relative_audio_path, 0),
mimetype="audio/wav",
as_attachment=True,
attachment_filename=hashed_text + ".wav")
response = make_response(audio)
response.headers['Content-Type'] = 'audio/wav'
response.headers['Content-Disposition'] = 'attachment; filename=sound.wav'
return response
@app.route('/')
def index():
text = request.args.get('text') or "듣고 싶은 문장을 입력해 주세요."
return render_template('index.html', text=text)
@app.route('/generate')
def view_method():
text = request.args.get('text')
speaker_id = int(request.args.get('speaker_id'))
if text:
return generate_audio_response(text, speaker_id)
else:
return {}
@app.route('/js/<path:path>')
def send_js(path):
return send_from_directory(
os.path.join(static_path, 'js'), path)
@app.route('/css/<path:path>')
def send_css(path):
return send_from_directory(
os.path.join(static_path, 'css'), path)
@app.route('/audio/<path:path>')
def send_audio(path):
return send_from_directory(
os.path.join(static_path, 'audio'), path)
if __name__ == '__main__':
parser = argparse.ArgumentParser()
parser.add_argument('--load_path', required=True)
parser.add_argument('--checkpoint_step', default=None, type=int)
parser.add_argument('--num_speakers', default=1, type=int)
parser.add_argument('--port', default=5000, type=int)
parser.add_argument('--debug', default=False, type=str2bool)
config = parser.parse_args()
if os.path.exists(config.load_path):
prepare_dirs(config, hparams)
global_config = config
synthesizer.load(config.load_path, config.num_speakers, config.checkpoint_step)
else:
print(" [!] load_path not found: {}".format(config.load_path))
app.run(host='0.0.0.0', port=config.port, debug=config.debug)

BIN
assets/model.png Normal file
View file

Binary file not shown.
Side by side

After

Width:  |  Height:  |  Size: 625 KiB

168
audio/__init__.py Normal file
View file

@ -0,0 +1,168 @@
# Code based on https://github.com/keithito/tacotron/blob/master/util/audio.py
import math
import numpy as np
import tensorflow as tf
from scipy import signal
from hparams import hparams
import librosa
import librosa.filters
def load_audio(path, pre_silence_length=0, post_silence_length=0):
audio = librosa.core.load(path, sr=hparams.sample_rate)[0]
if pre_silence_length > 0 or post_silence_length > 0:
audio = np.concatenate([
get_silence(pre_silence_length),
audio,
get_silence(post_silence_length),
])
return audio
def save_audio(audio, path, sample_rate=None):
audio *= 32767 / max(0.01, np.max(np.abs(audio)))
librosa.output.write_wav(path, audio.astype(np.int16),
hparams.sample_rate if sample_rate is None else sample_rate)
print(" [*] Audio saved: {}".format(path))
def resample_audio(audio, target_sample_rate):
return librosa.core.resample(
audio, hparams.sample_rate, target_sample_rate)
def get_duration(audio):
return librosa.core.get_duration(audio, sr=hparams.sample_rate)
def frames_to_hours(n_frames):
return sum((n_frame for n_frame in n_frames)) * \
hparams.frame_shift_ms / (3600 * 1000)
def get_silence(sec):
return np.zeros(hparams.sample_rate * sec)
def spectrogram(y):
D = _stft(_preemphasis(y))
S = _amp_to_db(np.abs(D)) - hparams.ref_level_db
return _normalize(S)
def inv_spectrogram(spectrogram):
S = _db_to_amp(_denormalize(spectrogram) + hparams.ref_level_db) # Convert back to linear
return inv_preemphasis(_griffin_lim(S ** hparams.power)) # Reconstruct phase
def inv_spectrogram_tensorflow(spectrogram):
S = _db_to_amp_tensorflow(_denormalize_tensorflow(spectrogram) + hparams.ref_level_db)
return _griffin_lim_tensorflow(tf.pow(S, hparams.power))
def melspectrogram(y):
D = _stft(_preemphasis(y))
S = _amp_to_db(_linear_to_mel(np.abs(D)))
return _normalize(S)
def inv_melspectrogram(melspectrogram):
S = _mel_to_linear(_db_to_amp(_denormalize(melspectrogram))) # Convert back to linear
return inv_preemphasis(_griffin_lim(S ** hparams.power)) # Reconstruct phase
# Based on https://github.com/librosa/librosa/issues/434
def _griffin_lim(S):
angles = np.exp(2j * np.pi * np.random.rand(*S.shape))
S_complex = np.abs(S).astype(np.complex)
y = _istft(S_complex * angles)
for i in range(hparams.griffin_lim_iters):
angles = np.exp(1j * np.angle(_stft(y)))
y = _istft(S_complex * angles)
return y
def _griffin_lim_tensorflow(S):
with tf.variable_scope('griffinlim'):
S = tf.expand_dims(S, 0)
S_complex = tf.identity(tf.cast(S, dtype=tf.complex64))
y = _istft_tensorflow(S_complex)
for i in range(hparams.griffin_lim_iters):
est = _stft_tensorflow(y)
angles = est / tf.cast(tf.maximum(1e-8, tf.abs(est)), tf.complex64)
y = _istft_tensorflow(S_complex * angles)
return tf.squeeze(y, 0)
def _stft(y):
n_fft, hop_length, win_length = _stft_parameters()
return librosa.stft(y=y, n_fft=n_fft, hop_length=hop_length, win_length=win_length)
def _istft(y):
_, hop_length, win_length = _stft_parameters()
return librosa.istft(y, hop_length=hop_length, win_length=win_length)
def _stft_tensorflow(signals):
n_fft, hop_length, win_length = _stft_parameters()
return tf.contrib.signal.stft(signals, win_length, hop_length, n_fft, pad_end=False)
def _istft_tensorflow(stfts):
n_fft, hop_length, win_length = _stft_parameters()
return tf.contrib.signal.inverse_stft(stfts, win_length, hop_length, n_fft)
def _stft_parameters():
n_fft = (hparams.num_freq - 1) * 2
hop_length = int(hparams.frame_shift_ms / 1000 * hparams.sample_rate)
win_length = int(hparams.frame_length_ms / 1000 * hparams.sample_rate)
return n_fft, hop_length, win_length
# Conversions:
_mel_basis = None
_inv_mel_basis = None
def _linear_to_mel(spectrogram):
global _mel_basis
if _mel_basis is None:
_mel_basis = _build_mel_basis()
return np.dot(_mel_basis, spectrogram)
def _mel_to_linear(mel_spectrogram):
global _inv_mel_basis
if _inv_mel_basis is None:
_inv_mel_basis = np.linalg.pinv(_build_mel_basis())
return np.maximum(1e-10, np.dot(_inv_mel_basis, mel_spectrogram))
def _build_mel_basis():
n_fft = (hparams.num_freq - 1) * 2
return librosa.filters.mel(hparams.sample_rate, n_fft, n_mels=hparams.num_mels)
def _amp_to_db(x):
return 20 * np.log10(np.maximum(1e-5, x))
def _db_to_amp(x):
return np.power(10.0, x * 0.05)
def _db_to_amp_tensorflow(x):
return tf.pow(tf.ones(tf.shape(x)) * 10.0, x * 0.05)
def _preemphasis(x):
return signal.lfilter([1, -hparams.preemphasis], [1], x)
def inv_preemphasis(x):
return signal.lfilter([1], [1, -hparams.preemphasis], x)
def _normalize(S):
return np.clip((S - hparams.min_level_db) / -hparams.min_level_db, 0, 1)
def _denormalize(S):
return (np.clip(S, 0, 1) * -hparams.min_level_db) + hparams.min_level_db
def _denormalize_tensorflow(S):
return (tf.clip_by_value(S, 0, 1) * -hparams.min_level_db) + hparams.min_level_db

71
audio/get_duration.py Normal file
View file

@ -0,0 +1,71 @@
import os
import datetime
from glob import glob
from tqdm import tqdm
from tinytag import TinyTag
from collections import defaultdict
from multiprocessing.dummy import Pool
from utils import load_json
def second_to_hour(sec):
return str(datetime.timedelta(seconds=int(sec)))
def get_duration(path):
filename = os.path.basename(path)
candidates = filename.split('.')[0].split('_')
dataset = candidates[0]
if not os.path.exists(path):
print(" [!] {} not found".format(path))
return dataset, 0
if True: # tinytag
tag = TinyTag.get(path)
duration = tag.duration
else: # librosa
y, sr = librosa.load(path)
duration = librosa.get_duration(y=y, sr=sr)
return dataset, duration
def get_durations(paths, print_detail=True):
duration_all = 0
duration_book = defaultdict(list)
pool = Pool()
iterator = pool.imap_unordered(get_duration, paths)
for dataset, duration in tqdm(iterator, total=len(paths)):
duration_all += duration
duration_book[dataset].append(duration)
total_count = 0
for book, duration in duration_book.items():
if book:
time = second_to_hour(sum(duration))
file_count = len(duration)
total_count += file_count
if print_detail:
print(" [*] Duration of {}: {} (file #: {})". \
format(book, time, file_count))
print(" [*] Total Duration : {} (file #: {})". \
format(second_to_hour(duration_all), total_count))
print()
return duration_all
if __name__ == '__main__':
import argparse
parser = argparse.ArgumentParser()
parser.add_argument('--audio-pattern', default=None) # datasets/krbook/audio/*.wav
parser.add_argument('--data-path', default=None) # datasets/jtbc/alignment.json
config, unparsed = parser.parse_known_args()
if config.audio_pattern is not None:
duration = get_durations(get_paths_by_pattern(config.data_dir))
elif config.data_path is not None:
paths = load_json(config.data_path).keys()
duration = get_durations(paths)

520
audio/google_speech.py Normal file
View file

@ -0,0 +1,520 @@
import io
import os
import sys
import json
import string
import argparse
import operator
import numpy as np
from glob import glob
from tqdm import tqdm
from nltk import ngrams
from difflib import SequenceMatcher
from collections import defaultdict
from google.cloud import speech
from google.cloud.speech import enums
from google.cloud.speech import types
from utils import parallel_run
from text import text_to_sequence
####################################################
# When one or two audio is missed in the middle
####################################################
def get_continuous_audio_paths(paths, debug=False):
audio_ids = get_audio_ids_from_paths(paths)
min_id, max_id = min(audio_ids), max(audio_ids)
if int(max_id) - int(min_id) + 1 != len(audio_ids):
base_path = paths[0].replace(min_id, "{:0" + str(len(max_id)) + "d}")
new_paths = [
base_path.format(audio_id) \
for audio_id in range(int(min_id), int(max_id) + 1)]
if debug: print("Missing audio : {} -> {}".format(paths, new_paths))
return new_paths
else:
return paths
def get_argmax_key(info, with_value=False):
max_key = max(info.keys(), key=(lambda k: info[k]))
if with_value:
return max_key, info[max_key]
else:
return max_key
def similarity(text_a, text_b):
text_a = "".join(remove_puncuations(text_a.strip()).split())
text_b = "".join(remove_puncuations(text_b.strip()).split())
score = SequenceMatcher(None, text_a, text_b).ratio()
#score = 1 / (distance(decompose_ko_text(text_a), decompose_ko_text(text_b)) + 1e-5)
#score = SequenceMatcher(None,
# decompose_ko_text(text_a), decompose_ko_text(text_b)).ratio()
if len(text_a) < len(text_b):
return -1 + score
else:
return score
def get_key_value_sorted(data):
keys = list(data.keys())
keys.sort()
values = [data[key] for key in keys]
return keys, values
def replace_pred_with_book(
path, book_path=None, threshold=0.9, max_candidate_num=5,
min_post_char_check=2, max_post_char_check=7, max_n=5,
max_allow_missing_when_matching=4, debug=False):
#######################################
# find text book from pred
#######################################
if book_path is None:
book_path = path.replace("speech", "text").replace("json", "txt")
data = json.loads(open(path).read())
keys, preds = get_key_value_sorted(data)
book_words = [word for word in open(book_path).read().split() if word != "=="]
book_texts = [text.replace('\n', '') for text in open(book_path).readlines()]
loc = 0
prev_key = None
force_stop = False
prev_end_loc = -1
prev_sentence_ended = True
prev_empty_skip = False
prev_not_found_skip = False
black_lists = ["160.{:04d}".format(audio_id) for audio_id in range(20, 36)]
new_preds = {}
for key, pred in zip(keys, preds):
if debug: print(key, pred)
if pred == "" or key in black_lists:
prev_empty_skip = True
continue
width, counter = 1, 0
sim_dict, loc_dict = {}, {}
while True:
words = book_words[loc:loc + width]
if len(words) == 0:
print("Force stop. Left {}, Del {} {}". \
format(len(preds) - len(new_preds), new_preds[prev_key], prev_key))
new_preds.pop(prev_key, None)
force_stop = True
break
candidate_candidates = {}
for _pred in list(set([pred, koreanize_numbers(pred)])):
max_skip = 0 if has_number(_pred[0]) or \
_pred[0] in """"'“”’‘’""" else len(words)
end_sims = []
for idx in range(min(max_skip, 10)):
text = " ".join(words[idx:])
################################################
# Score of trailing sentence is also important
################################################
for jdx in range(min_post_char_check,
max_post_char_check):
sim = similarity(
"".join(_pred.split())[-jdx:],
"".join(text.split())[-jdx:])
end_sims.append(sim)
candidate_candidates[text] = similarity(_pred, text)
candidate, sim = get_argmax_key(
candidate_candidates, with_value=True)
if sim > threshold or max(end_sims + [-1]) > threshold - 0.2 or \
len(sim_dict) > 0:
sim_dict[candidate] = sim
loc_dict[candidate] = loc + width
if len(sim_dict) > 0:
counter += 1
if counter > max_candidate_num:
break
width += 1
if width - len(_pred.split()) > 5:
break
if force_stop:
break
if len(sim_dict) != 0:
#############################################################
# Check missing words between prev pred and current pred
#############################################################
if prev_key is not None:
cur_idx = int(key.rsplit('.', 2)[-2])
prev_idx = int(prev_key.rsplit('.', 2)[-2])
if cur_idx - prev_idx > 10:
force_stop = True
break
# word alinged based on prediction but may contain missing words
# because google speech recognition sometimes skip one or two word
# ex. ('오누이는 서로 자기가 할 일을 정했다.', '서로 자기가 할 일을 정했다.')
original_candidate = new_candidate = get_argmax_key(sim_dict)
word_to_find = original_candidate.split()[0]
if not prev_empty_skip:
search_idx = book_words[prev_end_loc:].index(word_to_find) \
if word_to_find in book_words[prev_end_loc:] else -1
if 0 < search_idx < 4 and not prev_sentence_ended:
words_to_check = book_words[prev_end_loc:prev_end_loc + search_idx]
if ends_with_punctuation(words_to_check[0]) == True:
tmp = " ".join([new_preds[prev_key]] + words_to_check[:1])
if debug: print(prev_key, tmp, new_preds[prev_key])
new_preds[prev_key] = tmp
prev_end_loc += 1
prev_sentence_ended = True
search_idx = book_words[prev_end_loc:].index(word_to_find) \
if word_to_find in book_words[prev_end_loc:] else -1
if 0 < search_idx < 4 and prev_sentence_ended:
words_to_check = book_words[prev_end_loc:prev_end_loc + search_idx]
if not any(ends_with_punctuation(word) for word in words_to_check):
new_candidate = " ".join(words_to_check + [original_candidate])
if debug: print(key, new_candidate, original_candidate)
new_preds[key] = new_candidate
prev_sentence_ended = ends_with_punctuation(new_candidate)
loc = loc_dict[original_candidate]
prev_key = key
prev_not_found_skip = False
else:
loc += len(_pred.split()) - 1
prev_sentence_ended = True
prev_not_found_skip = True
prev_end_loc = loc
prev_empty_skip = False
if debug:
print("=", pred)
print("=", new_preds[key], loc)
if force_stop:
print(" [!] Force stop: {}".format(path))
align_diff = loc - len(book_words)
if abs(align_diff) > 10:
print(" => Align result of {}: {} - {} = {}".format(path, loc, len(book_words), align_diff))
#######################################
# find exact match of n-gram of pred
#######################################
finished_ids = []
keys, preds = get_key_value_sorted(new_preds)
if abs(align_diff) > 10:
keys, preds = keys[:-30], preds[:-30]
unfinished_ids = range(len(keys))
text_matches = []
for n in range(max_n, 1, -1):
ngram_preds = ngrams(preds, n)
for n_allow_missing in range(0, max_allow_missing_when_matching + 1):
unfinished_ids = list(set(unfinished_ids) - set(finished_ids))
existing_ngram_preds = []
for ngram in ngram_preds:
for text in book_texts:
candidates = [
" ".join(text.split()[:-n_allow_missing]),
" ".join(text.split()[n_allow_missing:]),
]
for tmp_text in candidates:
if " ".join(ngram) == tmp_text:
existing_ngram_preds.append(ngram)
break
tmp_keys = []
cur_ngram = []
ngram_idx = 0
ngram_found = False
for id_idx in unfinished_ids:
key, pred = keys[id_idx], preds[id_idx]
if ngram_idx >= len(existing_ngram_preds):
break
cur_ngram = existing_ngram_preds[ngram_idx]
if pred in cur_ngram:
ngram_found = True
tmp_keys.append(key)
finished_ids.append(id_idx)
if len(tmp_keys) == len(cur_ngram):
if debug: print(n_allow_missing, tmp_keys, cur_ngram)
tmp_keys = get_continuous_audio_paths(tmp_keys, debug)
text_matches.append(
[[" ".join(cur_ngram)], tmp_keys]
)
ngram_idx += 1
tmp_keys = []
cur_ngram = []
else:
if pred == cur_ngram[-1]:
ngram_idx += 1
tmp_keys = []
cur_ngram = []
else:
if len(tmp_keys) > 0:
ngram_found = False
tmp_keys = []
cur_ngram = []
for id_idx in range(len(keys)):
if id_idx not in finished_ids:
key, pred = keys[id_idx], preds[id_idx]
text_matches.append(
[[pred], [key]]
)
##############################################################
# ngram again for just in case after adding missing words
##############################################################
max_keys = [max(get_audio_ids_from_paths(item[1], as_int=True)) for item in text_matches]
sorted_text_matches = \
[item for _, item in sorted(zip(max_keys, text_matches))]
preds = [item[0][0] for item in sorted_text_matches]
keys = [item[1] for item in sorted_text_matches]
def book_sentence_idx_search(query, book_texts):
for idx, text in enumerate(book_texts):
if query in text:
return idx, text
return False, False
text_matches = []
idx, book_cursor_idx = 0, 0
if len(preds) == 0:
return []
while True:
tmp_texts = book_texts[book_cursor_idx:]
jdx = 0
tmp_pred = preds[idx]
idxes_to_merge = [idx]
prev_sent_idx, prev_sent = book_sentence_idx_search(tmp_pred, tmp_texts)
while idx + jdx + 1 < len(preds):
jdx += 1
tmp_pred = preds[idx + jdx]
sent_idx, sent = book_sentence_idx_search(tmp_pred, tmp_texts)
if not sent_idx:
if debug: print(" [!] NOT FOUND: {}".format(tmp_pred))
break
if prev_sent_idx == sent_idx:
idxes_to_merge.append(idx + jdx)
else:
break
new_keys = get_continuous_audio_paths(
sum([keys[jdx] for jdx in idxes_to_merge], []))
text_matches.append([ [tmp_texts[prev_sent_idx]], new_keys ])
if len(new_keys) > 1:
book_cursor_idx += 1
book_cursor_idx = max(book_cursor_idx, sent_idx)
if idx == len(preds) - 1:
break
idx = idx + jdx
# Counter([len(i) for i in text_matches.values()])
return text_matches
def get_text_from_audio_batch(paths, multi_process=False):
results = {}
items = parallel_run(get_text_from_audio, paths,
desc="get_text_from_audio_batch")
for item in items:
results.update(item)
return results
def get_text_from_audio(path):
error_count = 0
txt_path = path.replace('flac', 'txt')
if os.path.exists(txt_path):
with open(txt_path) as f:
out = json.loads(open(txt_path).read())
return out
out = {}
while True:
try:
client = speech.SpeechClient()
with io.open(path, 'rb') as audio_file:
content = audio_file.read()
audio = types.RecognitionAudio(content=content)
config = types.RecognitionConfig(
encoding=enums.RecognitionConfig.AudioEncoding.FLAC,
sample_rate_hertz=16000,
language_code='ko-KR')
response = client.recognize(config, audio)
if len(response.results) > 0:
alternatives = response.results[0].alternatives
results = [alternative.transcript for alternative in alternatives]
assert len(results) == 1, "More than 1 results: {}".format(results)
out = { path: "" if len(results) == 0 else results[0] }
print(results[0])
break
break
except:
error_count += 1
print("Skip warning for {} for {} times". \
format(path, error_count))
if error_count > 5:
break
else:
continue
with open(txt_path, 'w') as f:
json.dump(out, f, indent=2, ensure_ascii=False)
return out
if __name__ == '__main__':
parser = argparse.ArgumentParser()
parser.add_argument('--asset-dir', type=str, default='assets')
parser.add_argument('--data-dir', type=str, default='audio')
parser.add_argument('--pattern', type=str, default="audio/*.flac")
parser.add_argument('--metadata', type=str, default="metadata.json")
config, unparsed = parser.parse_known_args()
paths = glob(config.pattern)
paths.sort()
paths = paths
book_ids = list(set([
os.path.basename(path).split('.', 1)[0] for path in paths]))
book_ids.sort()
def get_finished_ids():
finished_paths = glob(os.path.join(
config.asset_dir, "speech-*.json"))
finished_ids = list(set([
os.path.basename(path).split('.', 1)[0].replace("speech-", "") for path in finished_paths]))
finished_ids.sort()
return finished_ids
finished_ids = get_finished_ids()
print("# Finished : {}/{}".format(len(finished_ids), len(book_ids)))
book_ids_to_parse = list(set(book_ids) - set(finished_ids))
book_ids_to_parse.sort()
assert os.path.exists(config.asset_dir), "assert_dir not found"
pbar = tqdm(book_ids_to_parse, "[1] google_speech",
initial=len(finished_ids), total=len(book_ids))
for book_id in pbar:
current_paths = glob(config.pattern.replace("*", "{}.*".format(book_id)))
pbar.set_description("[1] google_speech : {}".format(book_id))
results = get_text_from_audio_batch(current_paths)
filename = "speech-{}.json".format(book_id)
path = os.path.join(config.asset_dir, filename)
with open(path, "w") as f:
json.dump(results, f, indent=2, ensure_ascii=False)
finished_ids = get_finished_ids()
for book_id in tqdm(finished_ids, "[2] text_match"):
filename = "speech-{}.json".format(book_id)
path = os.path.join(config.asset_dir, filename)
clean_path = path.replace("speech", "clean-speech")
if os.path.exists(clean_path):
print(" [*] Skip {}".format(clean_path))
else:
results = replace_pred_with_book(path)
with open(clean_path, "w") as f:
json.dump(results, f, indent=2, ensure_ascii=False)
# Dummy
if False:
match_paths = get_paths_by_pattern(
config.asset_dir, 'clean-speech-*.json')
metadata_path = os.path.join(config.data_dir, config.metadata)
print(" [3] Merge clean-speech-*.json into {}".format(metadata_path))
merged_data = []
for path in match_paths:
with open(path) as f:
merged_data.extend(json.loads(f.read()))
import ipdb; ipdb.set_trace()
with open(metadata_path, 'w') as f:
json.dump(merged_data, f, indent=2, ensure_ascii=False)

143
audio/silence.py Normal file
View file

@ -0,0 +1,143 @@
import os
import re
import sys
import json
import librosa
import argparse
import numpy as np
from tqdm import tqdm
from glob import glob
from pydub import silence
from pydub import AudioSegment
from functools import partial
from hparams import hparams
from utils import parallel_run, add_postfix
from audio import load_audio, save_audio, get_duration, get_silence
def abs_mean(x):
return abs(x).mean()
def remove_breath(audio):
edges = librosa.effects.split(
audio, top_db=40, frame_length=128, hop_length=32)
for idx in range(len(edges)):
start_idx, end_idx = edges[idx][0], edges[idx][1]
if start_idx < len(audio):
if abs_mean(audio[start_idx:end_idx]) < abs_mean(audio) - 0.05:
audio[start_idx:end_idx] = 0
return audio
def split_on_silence_with_librosa(
audio_path, top_db=40, frame_length=1024, hop_length=256,
skip_idx=0, out_ext="wav",
min_segment_length=3, max_segment_length=8,
pre_silence_length=0, post_silence_length=0):
filename = os.path.basename(audio_path).split('.', 1)[0]
in_ext = audio_path.rsplit(".")[1]
audio = load_audio(audio_path)
edges = librosa.effects.split(audio,
top_db=top_db, frame_length=frame_length, hop_length=hop_length)
new_audio = np.zeros_like(audio)
for idx, (start, end) in enumerate(edges[skip_idx:]):
new_audio[start:end] = remove_breath(audio[start:end])
save_audio(new_audio, add_postfix(audio_path, "no_breath"))
audio = new_audio
edges = librosa.effects.split(audio,
top_db=top_db, frame_length=frame_length, hop_length=hop_length)
audio_paths = []
for idx, (start, end) in enumerate(edges[skip_idx:]):
segment = audio[start:end]
duration = get_duration(segment)
if duration <= min_segment_length or duration >= max_segment_length:
continue
output_path = "{}/{}.{:04d}.{}".format(
os.path.dirname(audio_path), filename, idx, out_ext)
padded_segment = np.concatenate([
get_silence(pre_silence_length),
segment,
get_silence(post_silence_length),
])
save_audio(padded_segment, output_path)
audio_paths.append(output_path)
return audio_paths
def read_audio(audio_path):
return AudioSegment.from_file(audio_path)
def split_on_silence_with_pydub(
audio_path, skip_idx=0, out_ext="wav",
silence_thresh=-40, min_silence_len=400,
silence_chunk_len=100, keep_silence=100):
filename = os.path.basename(audio_path).split('.', 1)[0]
in_ext = audio_path.rsplit(".")[1]
audio = read_audio(audio_path)
not_silence_ranges = silence.detect_nonsilent(
audio, min_silence_len=silence_chunk_len,
silence_thresh=silence_thresh)
edges = [not_silence_ranges[0]]
for idx in range(1, len(not_silence_ranges)-1):
cur_start = not_silence_ranges[idx][0]
prev_end = edges[-1][1]
if cur_start - prev_end < min_silence_len:
edges[-1][1] = not_silence_ranges[idx][1]
else:
edges.append(not_silence_ranges[idx])
audio_paths = []
for idx, (start_idx, end_idx) in enumerate(edges[skip_idx:]):
start_idx = max(0, start_idx - keep_silence)
end_idx += keep_silence
target_audio_path = "{}/{}.{:04d}.{}".format(
os.path.dirname(audio_path), filename, idx, out_ext)
audio[start_idx:end_idx].export(target_audio_path, out_ext)
audio_paths.append(target_audio_path)
return audio_paths
def split_on_silence_batch(audio_paths, method, **kargv):
audio_paths.sort()
method = method.lower()
if method == "librosa":
fn = partial(split_on_silence_with_librosa, **kargv)
elif method == "pydub":
fn = partial(split_on_silence_with_pydub, **kargv)
parallel_run(fn, audio_paths,
desc="Split on silence", parallel=False)
if __name__ == "__main__":
parser = argparse.ArgumentParser()
parser.add_argument('--audio_pattern', required=True)
parser.add_argument('--out_ext', default='wav')
parser.add_argument('--method', choices=['librosa', 'pydub'], required=True)
config = parser.parse_args()
audio_paths = glob(config.audio_pattern)
split_on_silence_batch(
audio_paths, config.method,
out_ext=config.out_ext,
)

0
datasets/__init__.py Normal file
View file

328
datasets/datafeeder.py Normal file
View file

@ -0,0 +1,328 @@
import os
import time
import pprint
import random
import threading
import traceback
import numpy as np
from glob import glob
import tensorflow as tf
from collections import defaultdict
import text
from utils.infolog import log
from utils import parallel_run, remove_file
from audio import frames_to_hours
from audio.get_duration import get_durations
_pad = 0
def get_frame(path):
data = np.load(path)
n_frame = data["linear"].shape[0]
n_token = len(data["tokens"])
return (path, n_frame, n_token)
def get_path_dict(
data_dirs, hparams, config,
data_type, n_test=None,
rng=np.random.RandomState(123)):
# Load metadata:
path_dict = {}
for data_dir in data_dirs:
paths = glob("{}/*.npz".format(data_dir))
if data_type == 'train':
rng.shuffle(paths)
if not config.skip_path_filter:
items = parallel_run(
get_frame, paths, desc="filter_by_min_max_frame_batch", parallel=True)
min_n_frame = hparams.reduction_factor * hparams.min_iters
max_n_frame = hparams.reduction_factor * hparams.max_iters - hparams.reduction_factor
new_items = [(path, n) for path, n, n_tokens in items \
if min_n_frame <= n <= max_n_frame and n_tokens >= hparams.min_tokens]
if any(check in data_dir for check in ["son", "yuinna"]):
blacklists = [".0000.", ".0001.", "NB11479580.0001"]
new_items = [item for item in new_items \
if any(check not in item[0] for check in blacklists)]
new_paths = [path for path, n in new_items]
new_n_frames = [n for path, n in new_items]
hours = frames_to_hours(new_n_frames)
log(' [{}] Loaded metadata for {} examples ({:.2f} hours)'. \
format(data_dir, len(new_n_frames), hours))
log(' [{}] Max length: {}'.format(data_dir, max(new_n_frames)))
log(' [{}] Min length: {}'.format(data_dir, min(new_n_frames)))
else:
new_paths = paths
if data_type == 'train':
new_paths = new_paths[:-n_test]
elif data_type == 'test':
new_paths = new_paths[-n_test:]
else:
raise Exception(" [!] Unkown data_type: {}".format(data_type))
path_dict[data_dir] = new_paths
return path_dict
class DataFeeder(threading.Thread):
'''Feeds batches of data into a queue on a background thread.'''
def __init__(self, coordinator, data_dirs,
hparams, config, batches_per_group, data_type, batch_size):
super(DataFeeder, self).__init__()
self._coord = coordinator
self._hp = hparams
self._cleaner_names = [x.strip() for x in hparams.cleaners.split(',')]
self._step = 0
self._offset = defaultdict(lambda: 2)
self._batches_per_group = batches_per_group
self.rng = np.random.RandomState(config.random_seed)
self.data_type = data_type
self.batch_size = batch_size
self.min_tokens = hparams.min_tokens
self.min_n_frame = hparams.reduction_factor * hparams.min_iters
self.max_n_frame = hparams.reduction_factor * hparams.max_iters - hparams.reduction_factor
self.skip_path_filter = config.skip_path_filter
# Load metadata:
self.path_dict = get_path_dict(
data_dirs, self._hp, config, self.data_type,
n_test=self.batch_size, rng=self.rng)
self.data_dirs = list(self.path_dict.keys())
self.data_dir_to_id = {
data_dir: idx for idx, data_dir in enumerate(self.data_dirs)}
data_weight = {
data_dir: 1. for data_dir in self.data_dirs
}
if self._hp.main_data_greedy_factor > 0 and \
any(main_data in data_dir for data_dir in self.data_dirs \
for main_data in self._hp.main_data):
for main_data in self._hp.main_data:
for data_dir in self.data_dirs:
if main_data in data_dir:
data_weight[data_dir] += self._hp.main_data_greedy_factor
weight_Z = sum(data_weight.values())
self.data_ratio = {
data_dir: weight / weight_Z for data_dir, weight in data_weight.items()
}
log("="*40)
log(pprint.pformat(self.data_ratio, indent=4))
log("="*40)
#audio_paths = [path.replace("/data/", "/audio/"). \
# replace(".npz", ".wav") for path in self.data_paths]
#duration = get_durations(audio_paths, print_detail=False)
# Create placeholders for inputs and targets. Don't specify batch size because we want to
# be able to feed different sized batches at eval time.
self._placeholders = [
tf.placeholder(tf.int32, [None, None], 'inputs'),
tf.placeholder(tf.int32, [None], 'input_lengths'),
tf.placeholder(tf.float32, [None], 'loss_coeff'),
tf.placeholder(tf.float32, [None, None, hparams.num_mels], 'mel_targets'),
tf.placeholder(tf.float32, [None, None, hparams.num_freq], 'linear_targets'),
]
# Create queue for buffering data:
dtypes = [tf.int32, tf.int32, tf.float32, tf.float32, tf.float32]
self.is_multi_speaker = len(self.data_dirs) > 1
if self.is_multi_speaker:
self._placeholders.append(
tf.placeholder(tf.int32, [None], 'inputs'),
)
dtypes.append(tf.int32)
num_worker = 8 if self.data_type == 'train' else 1
queue = tf.FIFOQueue(num_worker, dtypes, name='input_queue')
self._enqueue_op = queue.enqueue(self._placeholders)
if self.is_multi_speaker:
self.inputs, self.input_lengths, self.loss_coeff, \
self.mel_targets, self.linear_targets, self.speaker_id = queue.dequeue()
else:
self.inputs, self.input_lengths, self.loss_coeff, \
self.mel_targets, self.linear_targets = queue.dequeue()
self.inputs.set_shape(self._placeholders[0].shape)
self.input_lengths.set_shape(self._placeholders[1].shape)
self.loss_coeff.set_shape(self._placeholders[2].shape)
self.mel_targets.set_shape(self._placeholders[3].shape)
self.linear_targets.set_shape(self._placeholders[4].shape)
if self.is_multi_speaker:
self.speaker_id.set_shape(self._placeholders[5].shape)
else:
self.speaker_id = None
if self.data_type == 'test':
examples = []
while True:
for data_dir in self.data_dirs:
examples.append(self._get_next_example(data_dir))
#print(data_dir, text.sequence_to_text(examples[-1][0], False, True))
if len(examples) >= self.batch_size:
break
if len(examples) >= self.batch_size:
break
self.static_batches = [examples for _ in range(self._batches_per_group)]
else:
self.static_batches = None
def start_in_session(self, session, start_step):
self._step = start_step
self._session = session
self.start()
def run(self):
try:
while not self._coord.should_stop():
self._enqueue_next_group()
except Exception as e:
traceback.print_exc()
self._coord.request_stop(e)
def _enqueue_next_group(self):
start = time.time()
# Read a group of examples:
n = self.batch_size
r = self._hp.reduction_factor
if self.static_batches is not None:
batches = self.static_batches
else:
examples = []
for data_dir in self.data_dirs:
if self._hp.initial_data_greedy:
if self._step < self._hp.initial_phase_step and \
any("krbook" in data_dir for data_dir in self.data_dirs):
data_dir = [data_dir for data_dir in self.data_dirs if "krbook" in data_dir][0]
if self._step < self._hp.initial_phase_step:
example = [self._get_next_example(data_dir) \
for _ in range(int(n * self._batches_per_group // len(self.data_dirs)))]
else:
example = [self._get_next_example(data_dir) \
for _ in range(int(n * self._batches_per_group * self.data_ratio[data_dir]))]
examples.extend(example)
examples.sort(key=lambda x: x[-1])
batches = [examples[i:i+n] for i in range(0, len(examples), n)]
self.rng.shuffle(batches)
log('Generated %d batches of size %d in %.03f sec' % (len(batches), n, time.time() - start))
for batch in batches:
feed_dict = dict(zip(self._placeholders, _prepare_batch(batch, r, self.rng, self.data_type)))
self._session.run(self._enqueue_op, feed_dict=feed_dict)
self._step += 1
def _get_next_example(self, data_dir):
'''Loads a single example (input, mel_target, linear_target, cost) from disk'''
data_paths = self.path_dict[data_dir]
while True:
if self._offset[data_dir] >= len(data_paths):
self._offset[data_dir] = 0
if self.data_type == 'train':
self.rng.shuffle(data_paths)
data_path = data_paths[self._offset[data_dir]]
self._offset[data_dir] += 1
try:
if os.path.exists(data_path):
data = np.load(data_path)
else:
continue
except:
remove_file(data_path)
continue
if not self.skip_path_filter:
break
if self.min_n_frame <= data["linear"].shape[0] <= self.max_n_frame and \
len(data["tokens"]) > self.min_tokens:
break
input_data = data['tokens']
mel_target = data['mel']
if 'loss_coeff' in data:
loss_coeff = data['loss_coeff']
else:
loss_coeff = 1
linear_target = data['linear']
return (input_data, loss_coeff, mel_target, linear_target,
self.data_dir_to_id[data_dir], len(linear_target))
def _prepare_batch(batch, reduction_factor, rng, data_type=None):
if data_type == 'train':
rng.shuffle(batch)
inputs = _prepare_inputs([x[0] for x in batch])
input_lengths = np.asarray([len(x[0]) for x in batch], dtype=np.int32)
loss_coeff = np.asarray([x[1] for x in batch], dtype=np.float32)
mel_targets = _prepare_targets([x[2] for x in batch], reduction_factor)
linear_targets = _prepare_targets([x[3] for x in batch], reduction_factor)
if len(batch[0]) == 6:
speaker_id = np.asarray([x[4] for x in batch], dtype=np.int32)
return (inputs, input_lengths, loss_coeff,
mel_targets, linear_targets, speaker_id)
else:
return (inputs, input_lengths, loss_coeff, mel_targets, linear_targets)
def _prepare_inputs(inputs):
max_len = max((len(x) for x in inputs))
return np.stack([_pad_input(x, max_len) for x in inputs])
def _prepare_targets(targets, alignment):
max_len = max((len(t) for t in targets)) + 1
return np.stack([_pad_target(t, _round_up(max_len, alignment)) for t in targets])
def _pad_input(x, length):
return np.pad(x, (0, length - x.shape[0]), mode='constant', constant_values=_pad)
def _pad_target(t, length):
return np.pad(t, [(0, length - t.shape[0]), (0,0)], mode='constant', constant_values=_pad)
def _round_up(x, multiple):
remainder = x % multiple
return x if remainder == 0 else x + multiple - remainder

191
datasets/generate_data.py Normal file
View file

@ -0,0 +1,191 @@
# Code based on https://github.com/keithito/tacotron/blob/master/datasets/ljspeech.py
import os
import re
import sys
import json
import argparse
import numpy as np
from tqdm import tqdm
from glob import glob
from functools import partial
from collections import Counter, defaultdict
from concurrent.futures import ProcessPoolExecutor
import matplotlib
matplotlib.use('agg')
import matplotlib.pyplot as plt
from hparams import hparams
from text import text_to_sequence
from utils import makedirs, remove_file, warning
from audio import load_audio, spectrogram, melspectrogram, frames_to_hours
def one(x=None):
return 1
def build_from_path(config):
warning("Sampling rate: {}".format(hparams.sample_rate))
executor = ProcessPoolExecutor(max_workers=config.num_workers)
futures = []
index = 1
base_dir = os.path.dirname(config.metadata_path)
data_dir = os.path.join(base_dir, config.data_dirname)
makedirs(data_dir)
loss_coeff = defaultdict(one)
if config.metadata_path.endswith("json"):
with open(config.metadata_path) as f:
content = f.read()
info = json.loads(content)
elif config.metadata_path.endswith("csv"):
with open(config.metadata_path) as f:
info = {}
for line in f:
path, text = line.strip().split('|')
info[path] = text
else:
raise Exception(" [!] Unkown metadata format: {}".format(config.metadata_path))
new_info = {}
for path in info.keys():
if not os.path.exists(path):
new_path = os.path.join(base_dir, path)
if not os.path.exists(new_path):
print(" [!] Audio not found: {}".format([path, new_path]))
continue
else:
new_path = path
new_info[new_path] = info[path]
info = new_info
for path in info.keys():
if type(info[path]) == list:
if hparams.ignore_recognition_level == 1 and len(info[path]) == 1 or \
hparams.ignore_recognition_level == 2:
loss_coeff[path] = hparams.recognition_loss_coeff
info[path] = info[path][0]
ignore_description = {
0: "use all",
1: "ignore only unmatched_alignment",
2: "fully ignore recognitio",
}
print(" [!] Skip recognition level: {} ({})". \
format(hparams.ignore_recognition_level,
ignore_description[hparams.ignore_recognition_level]))
for audio_path, text in info.items():
if hparams.ignore_recognition_level > 0 and loss_coeff[audio_path] != 1:
continue
if base_dir not in audio_path:
audio_path = os.path.join(base_dir, audio_path)
try:
tokens = text_to_sequence(text)
except:
continue
fn = partial(
_process_utterance,
audio_path, data_dir, tokens, loss_coeff[audio_path])
futures.append(executor.submit(fn))
n_frames = [future.result() for future in tqdm(futures)]
n_frames = [n_frame for n_frame in n_frames if n_frame is not None]
hours = frames_to_hours(n_frames)
print(' [*] Loaded metadata for {} examples ({:.2f} hours)'.format(len(n_frames), hours))
print(' [*] Max length: {}'.format(max(n_frames)))
print(' [*] Min length: {}'.format(min(n_frames)))
plot_n_frames(n_frames, os.path.join(
base_dir, "n_frames_before_filter.png"))
min_n_frame = hparams.reduction_factor * hparams.min_iters
max_n_frame = hparams.reduction_factor * hparams.max_iters - hparams.reduction_factor
n_frames = [n for n in n_frames if min_n_frame <= n <= max_n_frame]
hours = frames_to_hours(n_frames)
print(' [*] After filtered: {} examples ({:.2f} hours)'.format(len(n_frames), hours))
print(' [*] Max length: {}'.format(max(n_frames)))
print(' [*] Min length: {}'.format(min(n_frames)))
plot_n_frames(n_frames, os.path.join(
base_dir, "n_frames_after_filter.png"))
def plot_n_frames(n_frames, path):
labels, values = list(zip(*Counter(n_frames).most_common()))
values = [v for _, v in sorted(zip(labels, values))]
labels = sorted(labels)
indexes = np.arange(len(labels))
width = 1
fig, ax = plt.subplots(figsize=(len(labels) / 2, 5))
plt.bar(indexes, values, width)
plt.xticks(indexes + width * 0.5, labels)
plt.tight_layout()
plt.savefig(path)
def _process_utterance(audio_path, data_dir, tokens, loss_coeff):
audio_name = os.path.basename(audio_path)
filename = audio_name.rsplit('.', 1)[0] + ".npz"
numpy_path = os.path.join(data_dir, filename)
if not os.path.exists(numpy_path):
wav = load_audio(audio_path)
linear_spectrogram = spectrogram(wav).astype(np.float32)
mel_spectrogram = melspectrogram(wav).astype(np.float32)
data = {
"linear": linear_spectrogram.T,
"mel": mel_spectrogram.T,
"tokens": tokens,
"loss_coeff": loss_coeff,
}
n_frame = linear_spectrogram.shape[1]
if hparams.skip_inadequate:
min_n_frame = hparams.reduction_factor * hparams.min_iters
max_n_frame = hparams.reduction_factor * hparams.max_iters - hparams.reduction_factor
if min_n_frame <= n_frame <= max_n_frame and len(tokens) >= hparams.min_tokens:
return None
np.savez(numpy_path, **data, allow_pickle=False)
else:
try:
data = np.load(numpy_path)
n_frame = data["linear"].shape[0]
except:
remove_file(numpy_path)
return _process_utterance(audio_path, data_dir, tokens, loss_coeff)
return n_frame
if __name__ == '__main__':
parser = argparse.ArgumentParser(description='spectrogram')
parser.add_argument('metadata_path', type=str)
parser.add_argument('--data_dirname', type=str, default="data")
parser.add_argument('--num_workers', type=int, default=None)
config = parser.parse_args()
build_from_path(config)

59
datasets/moon/download.py Normal file
View file

@ -0,0 +1,59 @@
import os
import youtube_dl
from pydub import AudioSegment
from utils import makedirs, remove_file
base_dir = os.path.dirname(os.path.realpath(__file__))
def get_mili_sec(text):
minute, second = text.strip().split(':')
return (int(minute) * 60 + int(second)) * 1000
class Data(object):
def __init__(
self, text_path, video_url, title, start_time, end_time):
self.text_path = text_path
self.video_url = video_url
self.title = title
self.start = get_mili_sec(start_time)
self.end = get_mili_sec(end_time)
def read_csv(path):
with open(path) as f:
data = []
for line in f:
text_path, video_url, title, start_time, end_time = line.split('|')
data.append(Data(text_path, video_url, title, start_time, end_time))
return data
def download_audio_with_urls(data, out_ext="wav"):
for d in data:
original_path = os.path.join(base_dir, 'audio',
os.path.basename(d.text_path)).replace('.txt', '.original.mp3')
out_path = os.path.join(base_dir, 'audio',
os.path.basename(d.text_path)).replace('.txt', '.wav')
options = {
'format': 'bestaudio/best',
'outtmpl': original_path,
'postprocessors': [{
'key': 'FFmpegExtractAudio',
'preferredcodec': 'mp3',
'preferredquality': '320',
}],
}
with youtube_dl.YoutubeDL(options) as ydl:
ydl.download([d.video_url])
audio = AudioSegment.from_file(original_path)
audio[d.start:d.end].export(out_path, out_ext)
remove_file(original_path)
if __name__ == '__main__':
makedirs(os.path.join(base_dir, "audio"))
data = read_csv(os.path.join(base_dir, "metadata.csv"))
download_audio_with_urls(data)

12
datasets/moon/metadata.csv Normal file
View file

@ -0,0 +1,12 @@
assets/001.txt|https://www.youtube.com/watch?v=_YWqWHe8LwE|국회 시정연설|0:56|30:05
assets/002.txt|https://www.youtube.com/watch?v=p0iokDQy1sQ|유엔총회 기조연설|0:00|21:55
assets/003.txt|https://www.youtube.com/watch?v=eU4xI0OR9yQ|베를린 한반도 평화구상 연설|0:00|25:06
assets/004.txt|https://www.youtube.com/watch?v=PQXSzswJDyU|동방경제포럼 기조연설|0:00|17:58
assets/005.txt|https://www.youtube.com/watch?v=dOYaWLddRbU|취임사1|0:01|0:37
assets/006.txt|https://www.youtube.com/watch?v=dOYaWLddRbU|취임사2|1:09|12:45
assets/007.txt|https://www.youtube.com/watch?v=05yqIiwpqGw|6·15 남북공동선언 17주년 기념식 축사|0:05|12:14
assets/008.txt|https://www.youtube.com/watch?v=etwb4AR5hg4|현충일 추념사|0:00|12:05
assets/009.txt|https://www.youtube.com/watch?v=TGZeC52r8WM|바다의 날 기념사|0:00|12:20
assets/010.txt|https://www.youtube.com/watch?v=T2ANoBtp1p8|제72주년 광복절 경축사|0:13|29:26
assets/011.txt|https://www.youtube.com/watch?v=HRCTTRWAbNA|남북정상회담 17주년 기념식 축사|0:07|12:03
assets/012.txt|https://www.youtube.com/watch?v=Md5219iWdbs|2차 AIIB 연차총회 개회식 축사|0:07|15:55
1 assets/001.txt https://www.youtube.com/watch?v=_YWqWHe8LwE 국회 시정연설 0:56 30:05
2 assets/002.txt https://www.youtube.com/watch?v=p0iokDQy1sQ 유엔총회 기조연설 0:00 21:55
3 assets/003.txt https://www.youtube.com/watch?v=eU4xI0OR9yQ 베를린 한반도 평화구상 연설 0:00 25:06
4 assets/004.txt https://www.youtube.com/watch?v=PQXSzswJDyU 동방경제포럼 기조연설 0:00 17:58
5 assets/005.txt https://www.youtube.com/watch?v=dOYaWLddRbU 취임사1 0:01 0:37
6 assets/006.txt https://www.youtube.com/watch?v=dOYaWLddRbU 취임사2 1:09 12:45
7 assets/007.txt https://www.youtube.com/watch?v=05yqIiwpqGw 6·15 남북공동선언 17주년 기념식 축사 0:05 12:14
8 assets/008.txt https://www.youtube.com/watch?v=etwb4AR5hg4 현충일 추념사 0:00 12:05
9 assets/009.txt https://www.youtube.com/watch?v=TGZeC52r8WM 바다의 날 기념사 0:00 12:20
10 assets/010.txt https://www.youtube.com/watch?v=T2ANoBtp1p8 제72주년 광복절 경축사 0:13 29:26
11 assets/011.txt https://www.youtube.com/watch?v=HRCTTRWAbNA 남북정상회담 17주년 기념식 축사 0:07 12:03
12 assets/012.txt https://www.youtube.com/watch?v=Md5219iWdbs 2차 AIIB 연차총회 개회식 축사 0:07 15:55

4879
datasets/moon/recognition.json Normal file
View file

File diff suppressed because it is too large Load diff

59
datasets/park/download.py Normal file
View file

@ -0,0 +1,59 @@
import os
import youtube_dl
from pydub import AudioSegment
from utils import makedirs, remove_file
base_dir = os.path.dirname(os.path.realpath(__file__))
def get_mili_sec(text):
minute, second = text.strip().split(':')
return (int(minute) * 60 + int(second)) * 1000
class Data(object):
def __init__(
self, text_path, video_url, title, start_time, end_time):
self.text_path = text_path
self.video_url = video_url
self.title = title
self.start = get_mili_sec(start_time)
self.end = get_mili_sec(end_time)
def read_csv(path):
with open(path) as f:
data = []
for line in f:
text_path, video_url, title, start_time, end_time = line.split('|')
data.append(Data(text_path, video_url, title, start_time, end_time))
return data
def download_audio_with_urls(data, out_ext="wav"):
for d in data:
original_path = os.path.join(base_dir, 'audio',
os.path.basename(d.text_path)).replace('.txt', '.original.mp3')
out_path = os.path.join(base_dir, 'audio',
os.path.basename(d.text_path)).replace('.txt', '.wav')
options = {
'format': 'bestaudio/best',
'outtmpl': original_path,
'postprocessors': [{
'key': 'FFmpegExtractAudio',
'preferredcodec': 'mp3',
'preferredquality': '320',
}],
}
with youtube_dl.YoutubeDL(options) as ydl:
ydl.download([d.video_url])
audio = AudioSegment.from_file(original_path)
audio[d.start:d.end].export(out_path, out_ext)
remove_file(original_path)
if __name__ == '__main__':
makedirs(os.path.join(base_dir, "audio"))
data = read_csv(os.path.join(base_dir, "metadata.csv"))
download_audio_with_urls(data)

21
datasets/park/metadata.csv Normal file
View file

@ -0,0 +1,21 @@
assets/001.txt|https://www.youtube.com/watch?v=jn_Re6tW5Uo|개성공단 국회 연설|0:04|26:00
assets/002.txt|https://www.youtube.com/watch?v=56WKAcps8uM|2015 대국민 담화|0:05|24:13
assets/003.txt|https://www.youtube.com/watch?v=_Fym6railzc|2016 대국민 담화|0:11|30:47
assets/004.txt|https://www.youtube.com/watch?v=vBYXDJkW5eY|제97주년 3ㆍ1절 기념축사|0:01|18:36
assets/005.txt|https://www.youtube.com/watch?v=__37IbJeb4I|건군 68주년 국군의 날 기념식 축사|0:00|16:36
assets/006.txt|https://www.youtube.com/watch?v=A_Fyx2wZB30|최순실 사건 대국민 담화 발표|0:02|8:50
assets/007.txt|https://www.youtube.com/watch?v=8eKgE5sRsko|2016 현충일 추념사|0:00|8:28
assets/008.txt|https://www.youtube.com/watch?v=xbrMCJn4OfQ|2014 현충일 추념사|0:00|6:45
assets/009.txt|https://www.youtube.com/watch?v=ONBO3A6YGw8|제70차 유엔총회 기조연설|0:21|23:03
assets/010.txt|https://www.youtube.com/watch?v=rl1lTwD5-CU|2014 신년 기자회견|0:05|99:00
assets/011.txt|https://www.youtube.com/watch?v=iI-K6B3u-a8|2016 서해 수호의 날 기념사|0:09|8:13
assets/012.txt|https://www.youtube.com/watch?v=SuOJEZMPGqE|연설문 사전 유출 대국민사과|0:26|1:48
assets/013.txt|https://www.youtube.com/watch?v=BVQMycTnmAU|2017 예산안 설명 국회 시정연설|0:48|36:43
assets/014.txt|https://www.youtube.com/watch?v=-buLcCLNeTA|2016 20대 국회 개원 연설|0:00|27:32
assets/015.txt|https://www.youtube.com/watch?v=5G4o-v8QfFw|2014 독일 드레스덴 연설|0:00|22:29
assets/016.txt|https://www.youtube.com/watch?v=qczKAq9gA-k|70주년 광복절 경축사|0:09|25:33
assets/017.txt|https://www.youtube.com/watch?v=T_29pBDIfDQ|71주년 광복절 경축사|0:06|26:27
assets/018.txt|https://www.youtube.com/watch?v=P9Rf1ERW7pE|아프리카연합(AU) 특별연설|0:07|20:04
assets/019.txt|https://www.youtube.com/watch?v=P7K9oVBdqe0|2014 예산안 시정연설|0:01|35:14
assets/020.txt|https://www.youtube.com/watch?v=Enuo-yOjT9M|2013 예산안 시정연설|0:00|29:15
assets/021.txt|https://www.youtube.com/watch?v=GYHtSjMi3DU|69주년 광복절 경축사|0:00|24:19
1 assets/001.txt https://www.youtube.com/watch?v=jn_Re6tW5Uo 개성공단 국회 연설 0:04 26:00
2 assets/002.txt https://www.youtube.com/watch?v=56WKAcps8uM 2015 대국민 담화 0:05 24:13
3 assets/003.txt https://www.youtube.com/watch?v=_Fym6railzc 2016 대국민 담화 0:11 30:47
4 assets/004.txt https://www.youtube.com/watch?v=vBYXDJkW5eY 제97주년 3ㆍ1절 기념축사 0:01 18:36
5 assets/005.txt https://www.youtube.com/watch?v=__37IbJeb4I 건군 68주년 국군의 날 기념식 축사 0:00 16:36
6 assets/006.txt https://www.youtube.com/watch?v=A_Fyx2wZB30 최순실 사건 대국민 담화 발표 0:02 8:50
7 assets/007.txt https://www.youtube.com/watch?v=8eKgE5sRsko 2016 현충일 추념사 0:00 8:28
8 assets/008.txt https://www.youtube.com/watch?v=xbrMCJn4OfQ 2014 현충일 추념사 0:00 6:45
9 assets/009.txt https://www.youtube.com/watch?v=ONBO3A6YGw8 제70차 유엔총회 기조연설 0:21 23:03
10 assets/010.txt https://www.youtube.com/watch?v=rl1lTwD5-CU 2014 신년 기자회견 0:05 99:00
11 assets/011.txt https://www.youtube.com/watch?v=iI-K6B3u-a8 2016 서해 수호의 날 기념사 0:09 8:13
12 assets/012.txt https://www.youtube.com/watch?v=SuOJEZMPGqE 연설문 사전 유출 대국민사과 0:26 1:48
13 assets/013.txt https://www.youtube.com/watch?v=BVQMycTnmAU 2017 예산안 설명 국회 시정연설 0:48 36:43
14 assets/014.txt https://www.youtube.com/watch?v=-buLcCLNeTA 2016 20대 국회 개원 연설 0:00 27:32
15 assets/015.txt https://www.youtube.com/watch?v=5G4o-v8QfFw 2014 독일 드레스덴 연설 0:00 22:29
16 assets/016.txt https://www.youtube.com/watch?v=qczKAq9gA-k 70주년 광복절 경축사 0:09 25:33
17 assets/017.txt https://www.youtube.com/watch?v=T_29pBDIfDQ 71주년 광복절 경축사 0:06 26:27
18 assets/018.txt https://www.youtube.com/watch?v=P9Rf1ERW7pE 아프리카연합(AU) 특별연설 0:07 20:04
19 assets/019.txt https://www.youtube.com/watch?v=P7K9oVBdqe0 2014 예산안 시정연설 0:01 35:14
20 assets/020.txt https://www.youtube.com/watch?v=Enuo-yOjT9M 2013 예산안 시정연설 0:00 29:15
21 assets/021.txt https://www.youtube.com/watch?v=GYHtSjMi3DU 69주년 광복절 경축사 0:00 24:19

0
datasets/park/process.py Normal file
View file

8776
datasets/park/recognition.json Normal file
View file

File diff suppressed because it is too large Load diff

150
datasets/son/download.py Normal file
View file

@ -0,0 +1,150 @@
import re
import os
import sys
import m3u8
import json
import requests
import subprocess
from functools import partial
from bs4 import BeautifulSoup
from utils import get_encoder_name, parallel_run, makedirs
API_URL = 'http://api.jtbc.joins.com/ad/pre/NV10173083'
BASE_URL = 'http://nsvc.jtbc.joins.com/API/News/Newapp/Default.aspx'
def soupify(text):
return BeautifulSoup(text, "html.parser")
def get_news_ids(page_id):
params = {
'NJC': 'NJC300',
'CAID': 'NC10011174',
'PGI': page_id,
}
response = requests.request(
method='GET', url=BASE_URL, params=params,
)
soup = soupify(response.text)
return [item.text for item in soup.find_all('news_id')]
def download_news_video_and_content(
news_id, base_dir, chunk_size=32*1024,
video_dir="video", asset_dir="assets", audio_dir="audio"):
video_dir = os.path.join(base_dir, video_dir)
asset_dir = os.path.join(base_dir, asset_dir)
audio_dir = os.path.join(base_dir, audio_dir)
makedirs(video_dir)
makedirs(asset_dir)
makedirs(audio_dir)
text_path = os.path.join(asset_dir, "{}.txt".format(news_id))
original_text_path = os.path.join(asset_dir, "original-{}.txt".format(news_id))
video_path = os.path.join(video_dir, "{}.ts".format(news_id))
audio_path = os.path.join(audio_dir, "{}.wav".format(news_id))
params = {
'NJC': 'NJC400',
'NID': news_id, # NB11515152
'CD': 'A0100',
}
response = requests.request(
method='GET', url=BASE_URL, params=params,
)
soup = soupify(response.text)
article_contents = soup.find_all('article_contents')
assert len(article_contents) == 1, \
"# of <article_contents> of {} should be 1: {}".format(news_id, response.text)
text = soupify(article_contents[0].text).get_text() # remove <div>
with open(original_text_path, "w") as f:
f.write(text)
with open(text_path, "w") as f:
from nltk import sent_tokenize
text = re.sub(r'\[.{0,80} :\s.+]', '', text) # remove quote
text = re.sub(r'☞.+http.+\)', '', text) # remove quote
text = re.sub(r'\(https?:\/\/.*[\r\n]*\)', '', text) # remove url
sentences = sent_tokenize(text)
sentences = [sent for sentence in sentences for sent in sentence.split('\n') if sent]
new_texts = []
for sent in sentences:
sent = sent.strip()
sent = re.sub(r'\([^)]*\)', '', sent)
#sent = re.sub(r'\<.{0,80}\>', '', sent)
sent = sent.replace('…', '.')
new_texts.append(sent)
f.write("\n".join([sent for sent in new_texts if sent]))
vod_paths = soup.find_all('vod_path')
assert len(vod_paths) == 1, \
"# of <vod_path> of {} should be 1: {}".format(news_id, response.text)
if not os.path.exists(video_path):
redirect_url = soup.find_all('vod_path')[0].text
list_url = m3u8.load(redirect_url).playlists[0].absolute_uri
video_urls = [segment.absolute_uri for segment in m3u8.load(list_url).segments]
with open(video_path, "wb") as f:
for url in video_urls:
response = requests.get(url, stream=True)
total_size = int(response.headers.get('content-length', 0))
for chunk in response.iter_content(chunk_size):
if chunk: # filter out keep-alive new chunks
f.write(chunk)
if not os.path.exists(audio_path):
encoder = get_encoder_name()
command = "{} -y -loglevel panic -i {} -ab 160k -ac 2 -ar 44100 -vn {}".\
format(encoder, video_path, audio_path)
subprocess.call(command, shell=True)
return True
if __name__ == '__main__':
news_ids = []
page_idx = 1
base_dir = os.path.dirname(os.path.realpath(__file__))
news_id_path = os.path.join(base_dir, "news_ids.json")
if not os.path.exists(news_id_path):
while True:
tmp_ids = get_news_ids(page_idx)
if len(tmp_ids) == 0:
break
news_ids.extend(tmp_ids)
print(" [*] Download page {}: {}/{}".format(page_idx, len(tmp_ids), len(news_ids)))
page_idx += 1
with open(news_id_path, "w") as f:
json.dump(news_ids, f, indent=2, ensure_ascii=False)
else:
with open(news_id_path) as f:
news_ids = json.loads(f.read())
exceptions = ["NB10830162"]
news_ids = list(set(news_ids) - set(exceptions))
fn = partial(download_news_video_and_content, base_dir=base_dir)
results = parallel_run(
fn, news_ids, desc="Download news video+text", parallel=True)

59
datasets/yuinna/download.py Normal file
View file

@ -0,0 +1,59 @@
import os
import sys
import json
import argparse
import requests
from bs4 import BeautifulSoup
from functools import partial
from utils import download_with_url, makedirs, parallel_run
base_path = os.path.dirname(os.path.realpath(__file__))
RSS_URL = "http://enabler.kbs.co.kr/api/podcast_channel/feed.xml?channel_id=R2010-0440"
def itunes_download(item):
audio_dir = os.path.join(base_path, "audio")
date, url = item
path = os.path.join(audio_dir, "{}.mp4".format(date))
if not os.path.exists(path):
download_with_url(url, path)
def download_all(config):
audio_dir = os.path.join(base_path, "audio")
makedirs(audio_dir)
soup = BeautifulSoup(requests.get(RSS_URL).text, "html5lib")
items = [item for item in soup.find_all('item')]
titles = [item.find('title').text[9:-3] for item in items]
guids = [item.find('guid').text for item in items]
accept_list = ['친절한 인나씨', '반납예정일', '귀욤열매 드세요']
new_guids = [guid for title, guid in zip(titles, guids) \
if any(accept in title for accept in accept_list) and '-' not in title]
new_titles = [title for title, _ in zip(titles, guids) \
if any(accept in title for accept in accept_list) and '-' not in title]
for idx, title in enumerate(new_titles):
print(" [{:3d}] {}, {}".format(idx + 1, title,
os.path.basename(new_guids[idx]).split('_')[2]))
if idx == config.max_num: print("="*30)
urls = {
os.path.basename(guid).split('_')[2]: guid \
for guid in new_guids[:config.max_num]
}
parallel_run(itunes_download, urls.items(),
desc=" [*] Itunes download", parallel=True)
if __name__ == '__main__':
parser = argparse.ArgumentParser()
parser.add_argument('--max_num', default=100, type=int)
config, unparsed = parser.parse_known_args()
download_all(config)

29165
datasets/yuinna/recognition.json Normal file
View file

File diff suppressed because it is too large Load diff

122
download.py Normal file
View file

@ -0,0 +1,122 @@
# Code based on https://github.com/carpedm20/DCGAN-tensorflow/blob/master/download.py
from __future__ import print_function
import os
import sys
import gzip
import json
import tarfile
import zipfile
import argparse
import requests
from tqdm import tqdm
from six.moves import urllib
from utils import query_yes_no
parser = argparse.ArgumentParser(description='Download model checkpoints.')
parser.add_argument('checkpoints', metavar='N', type=str, nargs='+', choices=['son', 'park'],
help='name of checkpoints to download [son, park]')
def download(url, dirpath):
filename = url.split('/')[-1]
filepath = os.path.join(dirpath, filename)
u = urllib.request.urlopen(url)
f = open(filepath, 'wb')
filesize = int(u.headers["Content-Length"])
print("Downloading: %s Bytes: %s" % (filename, filesize))
downloaded = 0
block_sz = 8192
status_width = 70
while True:
buf = u.read(block_sz)
if not buf:
print('')
break
else:
print('', end='\r')
downloaded += len(buf)
f.write(buf)
status = (("[%-" + str(status_width + 1) + "s] %3.2f%%") %
('=' * int(float(downloaded) / filesize * status_width) + '>', downloaded * 100. / filesize))
print(status, end='')
sys.stdout.flush()
f.close()
return filepath
def download_file_from_google_drive(id, destination):
URL = "https://docs.google.com/uc?export=download"
session = requests.Session()
response = session.get(URL, params={ 'id': id }, stream=True)
token = get_confirm_token(response)
if token:
params = { 'id' : id, 'confirm' : token }
response = session.get(URL, params=params, stream=True)
save_response_content(response, destination)
def get_confirm_token(response):
for key, value in response.cookies.items():
if key.startswith('download_warning'):
return value
return None
def save_response_content(response, destination, chunk_size=32*1024):
total_size = int(response.headers.get('content-length', 0))
with open(destination, "wb") as f:
for chunk in tqdm(response.iter_content(chunk_size), total=total_size,
unit='B', unit_scale=True, desc=destination):
if chunk: # filter out keep-alive new chunks
f.write(chunk)
def unzip(filepath):
print("Extracting: " + filepath)
dirpath = os.path.dirname(filepath)
with zipfile.ZipFile(filepath) as zf:
zf.extractall(dirpath)
os.remove(filepath)
def download_checkpoint(checkpoint):
if checkpoint == "son":
save_path, drive_id = "son-20171015.tar.gz", "0B_7wC-DuR6ORcmpaY1A5V1AzZUU"
elif checkpoint == "park":
save_path, drive_id = "park-20171015.tar.gz", "0B_7wC-DuR6ORYjhlekl5bVlkQ2c"
else:
raise Exception(" [!] Unknown checkpoint: {}".format(checkpoint))
if os.path.exists(save_path):
print('[*] {} already exists'.format(save_path))
else:
download_file_from_google_drive(drive_id, save_path)
if save_path.endswith(".zip"):
zip_dir = ''
with zipfile.ZipFile(save_path) as zf:
zip_dir = zf.namelist()[0]
zf.extractall(dirpath)
os.remove(save_path)
os.rename(os.path.join(dirpath, zip_dir), os.path.join(dirpath, data_dir))
elif save_path.endswith("tar.gz"):
tar = tarfile.open(save_path, "r:gz")
tar.extractall()
tar.close()
elif save_path.endswith("tar"):
tar = tarfile.open(save_path, "r:")
tar.extractall()
tar.close()
if __name__ == '__main__':
args = parser.parse_args()
print(" [!] The pre-trained models are being made available for research purpose only")
print(" [!] 학습된 모델을 연구 이외의 목적으로 사용하는 것을 금지합니다.")
print()
if query_yes_no(" [?] Are you agree on this? 이에 동의하십니까?"):
if 'park' in args.checkpoints:
download_checkpoint('park')
if 'son' in args.checkpoints:
download_checkpoint('son')

138
eval.py Normal file
View file

@ -0,0 +1,138 @@
import os
import re
import math
import argparse
from glob import glob
from synthesizer import Synthesizer
from train import create_batch_inputs_from_texts
from utils import makedirs, str2bool, backup_file
from hparams import hparams, hparams_debug_string
texts = [
'텍스트를 음성으로 읽어주는 "음성합성" 기술은 시각 장애인을 위한 오디오북, 음성 안내 시스템, 대화 인공지능 등 많은 분야에 활용할 수 있습니다.',
"하지만 개인이 원하는 목소리로 음성합성 엔진을 만들기에는 어려움이 많았고 소수의 기업만이 기술을 보유하고 있었습니다.",
"최근 딥러닝 기술의 발전은 음성합성 기술의 진입 장벽을 많이 낮췄고 이제는 누구나 손쉽게 음성합성 엔진을 만들 수 있게 되었습니다.",
"본 세션에서는 딥러닝을 활용한 음성합성 기술을 소개하고 개발 경험과 그 과정에서 얻었던 팁을 공유하고자 합니다.",
"음성합성 엔진을 구현하는데 사용한 세 가지 연구를 소개하고 각각의 기술이 얼마나 자연스러운 목소리를 만들 수 있는지를 공유합니다.",
# Harry Potter
"그리고 헤르미온느는 겁에 질려 마룻바닥에 쓰러져 있었다.",
"그러자 론은 요술지팡이를 꺼냈다. 무엇을 할지도 모르면서 그는 머리에 처음으로 떠오른 주문을 외치고 있었다.",
"윙가르디움 레비오우사.... 하지만, 그렇게 소리쳤다.",
"그러자 그 방망이가 갑자기 트롤의 손에서 벗어나, 저 위로 올라가더니 탁하며 그 주인의 머리 위에 떨어졌다.",
"그러자 트롤이 그 자리에서 비틀거리더니 방 전체를 흔들어버릴 것 같은 커다란 소리를 내며 쿵 하고 넘어졌다. ",
"그러자 조그맣게 펑 하는 소리가 나면서 가장 가까이 있는 가로등이 꺼졌다.",
"그리고 그가 다시 찰깍하자 그 다음 가로등이 깜박거리며 나가 버렸다.",
#"그가 그렇게 가로등 끄기를 열두번 하자, 이제 그 거리에 남아 있는 불빛이라곤, ",
#"바늘로 꼭 질러둔 것처럼 작게 보이는 멀리서 그를 지켜보고 있는 고양이의 두 눈뿐이었다.",
#"프리벳가 4번지에 살고 있는 더즐리 부부는 자신들이 정상적이라는 것을 아주 자랑스럽게 여기는 사람들이었다. ",
#"그들은 기이하거나 신비스런 일과는 전혀 무관해 보였다.",
#"아니, 그런 터무니없는 것은 도저히 참아내지 못했다.",
#"더즐리 씨는 그루닝스라는 드릴제작회사의 중역이었다.",
#"그는 목이 거의 없을 정도로 살이 뒤룩뒤룩 찐 몸집이 큰 사내로, 코밑에는 커다란 콧수염을 기르고 있었다.",
#"더즐리 부인은 마른 체구의 금발이었고, 목이 보통사람보다 두 배는 길어서, 담 너머로 고개를 쭉 배고 이웃 사람들을 몰래 훔쳐보는 그녀의 취미에는 더없이 제격이었다.",
# From Yoo Inna's Audiobook (http://campaign.happybean.naver.com/yooinna_audiobook):
#'16세기 중엽 어느 가을날 옛 런던 시의 가난한 캔티 집안에 사내아이 하나가 태어났다.',
#'그런데 그 집안에서는 그 사내아이를 별로 반기지 않았다.',
#'바로 같은 날 또 한 명의 사내아이가 영국의 부유한 튜터 가문에서 태어났다.',
#'그런데 그 가문에서는 그 아이를 무척이나 반겼다.',
#'온 영국이 다 함께 그 아이를 반겼다.',
## From NAVER's Audiobook (http://campaign.happybean.naver.com/yooinna_audiobook):
#'부랑자 패거리는 이른 새벽에 일찍 출발하여 길을 떠났다.',
#'하늘은 찌푸렸고, 발밑의 땅은 질퍽거렸으며, 겨울의 냉기가 공기 중에 감돌았다.',
#'지난밤의 흥겨움은 온데간데없이 사라졌다.',
#'시무룩하게 말이 없는 사람들도 있었고, 안달복달하며 조바심을 내는 사람들도 있었지만, 기분이 좋은 사람은 하나도 없었다.',
## From NAVER's nVoice example (https://www.facebook.com/naverlabs/videos/422780217913446):
#'감사합니다. Devsisters 김태훈 님의 발표였습니다.',
#'이것으로 금일 마련된 track 2의 모든 세션이 종료되었습니다.',
#'장시간 끝까지 참석해주신 개발자 여러분들께 진심으로 감사의 말씀을 드리며,',
#'잠시 후 5시 15분부터 특정 주제에 관심 있는 사람들이 모여 자유롭게 이야기하는 오프미팅이 진행될 예정이므로',
#'참여신청을 해주신 분들은 진행 요원의 안내에 따라 이동해주시기 바랍니다.',
## From Kakao's Son Seok hee example (https://www.youtube.com/watch?v=ScfdAH2otrY):
#'소설가 마크 트웨인이 말했습니다.',
#'인생에 가장 중요한 이틀이 있는데, 하나는 세상에 태어난 날이고 다른 하나는 왜 이 세상에 왔는가를 깨닫는 날이다.',
#'그런데 그 첫번째 날은 누구나 다 알지만 두번째 날은 참 어려운 것 같습니다.',
#'누구나 그 두번째 날을 만나기 위해 애쓰는게 삶인지도 모르겠습니다.',
#'뉴스룸도 그런 면에서 똑같습니다.',
#'저희들도 그 두번째의 날을 만나고 기억하기 위해 매일 매일 최선을 다하겠습니다.',
]
def get_output_base_path(load_path, eval_dirname="eval"):
if not os.path.isdir(load_path):
base_dir = os.path.dirname(load_path)
else:
base_dir = load_path
base_dir = os.path.join(base_dir, eval_dirname)
if os.path.exists(base_dir):
backup_file(base_dir)
makedirs(base_dir)
m = re.compile(r'.*?\.ckpt\-([0-9]+)').match(load_path)
base_path = os.path.join(base_dir,
'eval-%d' % int(m.group(1)) if m else 'eval')
return base_path
def run_eval(args):
print(hparams_debug_string())
load_paths = glob(args.load_path_pattern)
for load_path in load_paths:
if not os.path.exists(os.path.join(load_path, "checkpoint")):
print(" [!] Skip non model directory: {}".format(load_path))
continue
synth = Synthesizer()
synth.load(load_path)
for speaker_id in range(synth.num_speakers):
base_path = get_output_base_path(load_path, "eval-{}".format(speaker_id))
inputs, input_lengths = create_batch_inputs_from_texts(texts)
for idx in range(math.ceil(len(inputs) / args.batch_size)):
start_idx, end_idx = idx*args.batch_size, (idx+1)*args.batch_size
cur_texts = texts[start_idx:end_idx]
cur_inputs = inputs[start_idx:end_idx]
synth.synthesize(
texts=cur_texts,
speaker_ids=[speaker_id] * len(cur_texts),
tokens=cur_inputs,
base_path="{}-{}".format(base_path, idx),
manual_attention_mode=args.manual_attention_mode,
base_alignment_path=args.base_alignment_path,
)
synth.close()
def main():
parser = argparse.ArgumentParser()
parser.add_argument('--batch_size', default=16)
parser.add_argument('--load_path_pattern', required=True)
parser.add_argument('--base_alignment_path', default=None)
parser.add_argument('--manual_attention_mode', default=0, type=int,
help="0: None, 1: Argmax, 2: Sharpening, 3. Pruning")
parser.add_argument('--hparams', default='',
help='Hyperparameter overrides as a comma-separated list of name=value pairs')
args = parser.parse_args()
#hparams.max_iters = 100
#hparams.parse(args.hparams)
run_eval(args)
if __name__ == '__main__':
main()

156
hparams.py Normal file
View file

@ -0,0 +1,156 @@
import tensorflow as tf
SCALE_FACTOR = 1
def f(num):
return num // SCALE_FACTOR
basic_params = {
# Comma-separated list of cleaners to run on text prior to training and eval. For non-English
# text, you may want to use "basic_cleaners" or "transliteration_cleaners" See TRAINING_DATA.md.
'cleaners': 'korean_cleaners',
}
basic_params.update({
# Audio
'num_mels': 80,
'num_freq': 1025,
'sample_rate': 20000,
'frame_length_ms': 50,
'frame_shift_ms': 12.5,
'preemphasis': 0.97,
'min_level_db': -100,
'ref_level_db': 20,
})
if True:
basic_params.update({
'sample_rate': 24000,
})
basic_params.update({
# Model
'model_type': 'single', # [single, simple, deepvoice]
'speaker_embedding_size': f(16),
'embedding_size': f(256),
'dropout_prob': 0.5,
# Encoder
'enc_prenet_sizes': [f(256), f(128)],
'enc_bank_size': 16,
'enc_bank_channel_size': f(128),
'enc_maxpool_width': 2,
'enc_highway_depth': 4,
'enc_rnn_size': f(128),
'enc_proj_sizes': [f(128), f(128)],
'enc_proj_width': 3,
# Attention
'attention_type': 'bah_mon', # ntm2-5
'attention_size': f(256),
'attention_state_size': f(256),
# Decoder recurrent network
'dec_layer_num': 2,
'dec_rnn_size': f(256),
# Decoder
'dec_prenet_sizes': [f(256), f(128)],
'post_bank_size': 8,
'post_bank_channel_size': f(256),
'post_maxpool_width': 2,
'post_highway_depth': 4,
'post_rnn_size': f(128),
'post_proj_sizes': [f(256), 80], # num_mels=80
'post_proj_width': 3,
'reduction_factor': 4,
})
if False: # Deep Voice 2
basic_params.update({
'dropout_prob': 0.8,
'attention_size': f(512),
'dec_prenet_sizes': [f(256), f(128), f(64)],
'post_bank_channel_size': f(512),
'post_rnn_size': f(256),
'reduction_factor': 4,
})
elif True: # Deep Voice 2
basic_params.update({
'dropout_prob': 0.8,
#'attention_size': f(512),
#'dec_prenet_sizes': [f(256), f(128)],
#'post_bank_channel_size': f(512),
'post_rnn_size': f(256),
'reduction_factor': 4,
})
elif False: # Single Speaker
basic_params.update({
'dropout_prob': 0.5,
'attention_size': f(128),
'post_bank_channel_size': f(128),
#'post_rnn_size': f(128),
'reduction_factor': 4,
})
elif False: # Single Speaker with generalization
basic_params.update({
'dropout_prob': 0.8,
'attention_size': f(256),
'dec_prenet_sizes': [f(256), f(128), f(64)],
'post_bank_channel_size': f(128),
'post_rnn_size': f(128),
'reduction_factor': 4,
})
basic_params.update({
# Training
'batch_size': 16,
'adam_beta1': 0.9,
'adam_beta2': 0.999,
'use_fixed_test_inputs': False,
'initial_learning_rate': 0.002,
'decay_learning_rate_mode': 0,
'initial_data_greedy': True,
'initial_phase_step': 8000,
'main_data_greedy_factor': 0,
'main_data': [''],
'prioritize_loss': False,
'recognition_loss_coeff': 0.2,
'ignore_recognition_level': 1, # 0: use all, 1: ignore only unmatched_alignment, 2: fully ignore recognition
# Eval
'min_tokens': 50,
'min_iters': 30,
'max_iters': 200,
'skip_inadequate': False,
'griffin_lim_iters': 60,
'power': 1.5, # Power to raise magnitudes to prior to Griffin-Lim
})
# Default hyperparameters:
hparams = tf.contrib.training.HParams(**basic_params)
def hparams_debug_string():
values = hparams.values()
hp = [' %s: %s' % (name, values[name]) for name in sorted(values)]
return 'Hyperparameters:\n' + '\n'.join(hp)

17
models/__init__.py Normal file
View file

@ -0,0 +1,17 @@
import os
from glob import glob
from .tacotron import Tacotron
def create_model(hparams):
return Tacotron(hparams)
def get_most_recent_checkpoint(checkpoint_dir):
checkpoint_paths = [path for path in glob("{}/*.ckpt-*.data-*".format(checkpoint_dir))]
idxes = [int(os.path.basename(path).split('-')[1].split('.')[0]) for path in checkpoint_paths]
max_idx = max(idxes)
lastest_checkpoint = os.path.join(checkpoint_dir, "model.ckpt-{}".format(max_idx))
print(" [*] Found lastest checkpoint: {}".format(lastest_checkpoint))
return lastest_checkpoint

73
models/helpers.py Normal file
View file

@ -0,0 +1,73 @@
# Code based on https://github.com/keithito/tacotron/blob/master/models/tacotron.py
import numpy as np
import tensorflow as tf
from tensorflow.contrib.seq2seq import Helper
# Adapted from tf.contrib.seq2seq.GreedyEmbeddingHelper
class TacoTestHelper(Helper):
def __init__(self, batch_size, output_dim, r):
with tf.name_scope('TacoTestHelper'):
self._batch_size = batch_size
self._output_dim = output_dim
self._end_token = tf.tile([0.0], [output_dim * r])
@property
def batch_size(self):
return self._batch_size
def initialize(self, name=None):
return (tf.tile([False], [self._batch_size]), _go_frames(self._batch_size, self._output_dim))
def sample(self, time, outputs, state, name=None):
return tf.tile([0], [self._batch_size]) # Return all 0; we ignore them
def next_inputs(self, time, outputs, state, sample_ids, name=None):
'''Stop on EOS. Otherwise, pass the last output as the next input and pass through state.'''
with tf.name_scope('TacoTestHelper'):
finished = tf.reduce_all(tf.equal(outputs, self._end_token), axis=1)
# Feed last output frame as next input. outputs is [N, output_dim * r]
next_inputs = outputs[:, -self._output_dim:]
return (finished, next_inputs, state)
class TacoTrainingHelper(Helper):
def __init__(self, inputs, targets, output_dim, r, rnn_decoder_test_mode=False):
# inputs is [N, T_in], targets is [N, T_out, D]
with tf.name_scope('TacoTrainingHelper'):
self._batch_size = tf.shape(inputs)[0]
self._output_dim = output_dim
self._rnn_decoder_test_mode = rnn_decoder_test_mode
# Feed every r-th target frame as input
self._targets = targets[:, r-1::r, :]
# Use full length for every target because we don't want to mask the padding frames
num_steps = tf.shape(self._targets)[1]
self._lengths = tf.tile([num_steps], [self._batch_size])
@property
def batch_size(self):
return self._batch_size
def initialize(self, name=None):
return (tf.tile([False], [self._batch_size]), _go_frames(self._batch_size, self._output_dim))
def sample(self, time, outputs, state, name=None):
return tf.tile([0], [self._batch_size]) # Return all 0; we ignore them
def next_inputs(self, time, outputs, state, sample_ids, name=None):
with tf.name_scope(name or 'TacoTrainingHelper'):
finished = (time + 1 >= self._lengths)
if self._rnn_decoder_test_mode:
next_inputs = outputs[:, -self._output_dim:]
else:
next_inputs = self._targets[:, time, :]
return (finished, next_inputs, state)
def _go_frames(batch_size, output_dim):
'''Returns all-zero <GO> frames for a given batch size and output dimension'''
return tf.tile([[0.0]], [batch_size, output_dim])

131
models/modules.py Normal file
View file

@ -0,0 +1,131 @@
# Code based on https://github.com/keithito/tacotron/blob/master/models/tacotron.py
import tensorflow as tf
from tensorflow.contrib.rnn import GRUCell
from tensorflow.python.layers import core
from tensorflow.contrib.seq2seq.python.ops.attention_wrapper \
import _bahdanau_score, _BaseAttentionMechanism, BahdanauAttention, \
AttentionWrapper, AttentionWrapperState
def get_embed(inputs, num_inputs, embed_size, name):
embed_table = tf.get_variable(
name, [num_inputs, embed_size], dtype=tf.float32,
initializer=tf.truncated_normal_initializer(stddev=0.1))
return tf.nn.embedding_lookup(embed_table, inputs)
def prenet(inputs, is_training, layer_sizes, drop_prob, scope=None):
x = inputs
drop_rate = drop_prob if is_training else 0.0
with tf.variable_scope(scope or 'prenet'):
for i, size in enumerate(layer_sizes):
dense = tf.layers.dense(x, units=size, activation=tf.nn.relu, name='dense_%d' % (i+1))
x = tf.layers.dropout(dense, rate=drop_rate, name='dropout_%d' % (i+1))
return x
def cbhg(inputs, input_lengths, is_training,
bank_size, bank_channel_size,
maxpool_width, highway_depth, rnn_size,
proj_sizes, proj_width, scope,
before_highway=None, encoder_rnn_init_state=None):
batch_size = tf.shape(inputs)[0]
with tf.variable_scope(scope):
with tf.variable_scope('conv_bank'):
# Convolution bank: concatenate on the last axis
# to stack channels from all convolutions
conv_fn = lambda k: \
conv1d(inputs, k, bank_channel_size,
tf.nn.relu, is_training, 'conv1d_%d' % k)
conv_outputs = tf.concat(
[conv_fn(k) for k in range(1, bank_size+1)], axis=-1,
)
# Maxpooling:
maxpool_output = tf.layers.max_pooling1d(
conv_outputs,
pool_size=maxpool_width,
strides=1,
padding='same')
# Two projection layers:
proj_out = maxpool_output
for idx, proj_size in enumerate(proj_sizes):
activation_fn = None if idx == len(proj_sizes) - 1 else tf.nn.relu
proj_out = conv1d(
proj_out, proj_width, proj_size, activation_fn,
is_training, 'proj_{}'.format(idx + 1))
# Residual connection:
if before_highway is not None:
expanded_before_highway = tf.expand_dims(before_highway, [1])
tiled_before_highway = tf.tile(
expanded_before_highway, [1, tf.shape(proj_out)[1], 1])
highway_input = proj_out + inputs + tiled_before_highway
else:
highway_input = proj_out + inputs
# Handle dimensionality mismatch:
if highway_input.shape[2] != rnn_size:
highway_input = tf.layers.dense(highway_input, rnn_size)
# 4-layer HighwayNet:
for idx in range(highway_depth):
highway_input = highwaynet(highway_input, 'highway_%d' % (idx+1))
rnn_input = highway_input
# Bidirectional RNN
if encoder_rnn_init_state is not None:
initial_state_fw, initial_state_bw = \
tf.split(encoder_rnn_init_state, 2, 1)
else:
initial_state_fw, initial_state_bw = None, None
cell_fw, cell_bw = GRUCell(rnn_size), GRUCell(rnn_size)
outputs, states = tf.nn.bidirectional_dynamic_rnn(
cell_fw, cell_bw,
rnn_input,
sequence_length=input_lengths,
initial_state_fw=initial_state_fw,
initial_state_bw=initial_state_bw,
dtype=tf.float32)
return tf.concat(outputs, axis=2) # Concat forward and backward
def batch_tile(tensor, batch_size):
expaneded_tensor = tf.expand_dims(tensor, [0])
return tf.tile(expaneded_tensor, \
[batch_size] + [1 for _ in tensor.get_shape()])
def highwaynet(inputs, scope):
highway_dim = int(inputs.get_shape()[-1])
with tf.variable_scope(scope):
H = tf.layers.dense(
inputs,
units=highway_dim,
activation=tf.nn.relu,
name='H')
T = tf.layers.dense(
inputs,
units=highway_dim,
activation=tf.nn.sigmoid,
name='T',
bias_initializer=tf.constant_initializer(-1.0))
return H * T + inputs * (1.0 - T)
def conv1d(inputs, kernel_size, channels, activation, is_training, scope):
with tf.variable_scope(scope):
conv1d_output = tf.layers.conv1d(
inputs,
filters=channels,
kernel_size=kernel_size,
activation=activation,
padding='same')
return tf.layers.batch_normalization(conv1d_output, training=is_training)

418
models/rnn_wrappers.py Normal file
View file

@ -0,0 +1,418 @@
import numpy as np
import tensorflow as tf
from tensorflow.contrib.rnn import RNNCell
from tensorflow.python.ops import rnn_cell_impl
from tensorflow.contrib.data.python.util import nest
from tensorflow.contrib.seq2seq.python.ops.attention_wrapper \
import _bahdanau_score, _BaseAttentionMechanism, BahdanauAttention, \
AttentionWrapperState, AttentionMechanism
from .modules import prenet
_zero_state_tensors = rnn_cell_impl._zero_state_tensors
class AttentionWrapper(RNNCell):
"""Wraps another `RNNCell` with attention.
"""
def __init__(self,
cell,
attention_mechanism,
is_manual_attention,
manual_alignments,
attention_layer_size=None,
alignment_history=False,
cell_input_fn=None,
output_attention=True,
initial_cell_state=None,
name=None):
"""Construct the `AttentionWrapper`.
Args:
cell: An instance of `RNNCell`.
attention_mechanism: A list of `AttentionMechanism` instances or a single
instance.
attention_layer_size: A list of Python integers or a single Python
integer, the depth of the attention (output) layer(s). If None
(default), use the context as attention at each time step. Otherwise,
feed the context and cell output into the attention layer to generate
attention at each time step. If attention_mechanism is a list,
attention_layer_size must be a list of the same length.
alignment_history: Python boolean, whether to store alignment history
from all time steps in the final output state (currently stored as a
time major `TensorArray` on which you must call `stack()`).
cell_input_fn: (optional) A `callable`. The default is:
`lambda inputs, attention: array_tf.concat([inputs, attention], -1)`.
output_attention: Python bool. If `True` (default), the output at each
time step is the attention value. This is the behavior of Luong-style
attention mechanisms. If `False`, the output at each time step is
the output of `cell`. This is the beahvior of Bhadanau-style
attention mechanisms. In both cases, the `attention` tensor is
propagated to the next time step via the state and is used there.
This flag only controls whether the attention mechanism is propagated
up to the next cell in an RNN stack or to the top RNN output.
initial_cell_state: The initial state value to use for the cell when
the user calls `zero_state()`. Note that if this value is provided
now, and the user uses a `batch_size` argument of `zero_state` which
does not match the batch size of `initial_cell_state`, proper
behavior is not guaranteed.
name: Name to use when creating tf.
Raises:
TypeError: `attention_layer_size` is not None and (`attention_mechanism`
is a list but `attention_layer_size` is not; or vice versa).
ValueError: if `attention_layer_size` is not None, `attention_mechanism`
is a list, and its length does not match that of `attention_layer_size`.
"""
super(AttentionWrapper, self).__init__(name=name)
self.is_manual_attention = is_manual_attention
self.manual_alignments = manual_alignments
if isinstance(attention_mechanism, (list, tuple)):
self._is_multi = True
attention_mechanisms = attention_mechanism
for attention_mechanism in attention_mechanisms:
if not isinstance(attention_mechanism, AttentionMechanism):
raise TypeError(
"attention_mechanism must contain only instances of "
"AttentionMechanism, saw type: %s"
% type(attention_mechanism).__name__)
else:
self._is_multi = False
if not isinstance(attention_mechanism, AttentionMechanism):
raise TypeError(
"attention_mechanism must be an AttentionMechanism or list of "
"multiple AttentionMechanism instances, saw type: %s"
% type(attention_mechanism).__name__)
attention_mechanisms = (attention_mechanism,)
if cell_input_fn is None:
cell_input_fn = (
lambda inputs, attention: tf.concat([inputs, attention], -1))
else:
if not callable(cell_input_fn):
raise TypeError(
"cell_input_fn must be callable, saw type: %s"
% type(cell_input_fn).__name__)
if attention_layer_size is not None:
attention_layer_sizes = tuple(
attention_layer_size
if isinstance(attention_layer_size, (list, tuple))
else (attention_layer_size,))
if len(attention_layer_sizes) != len(attention_mechanisms):
raise ValueError(
"If provided, attention_layer_size must contain exactly one "
"integer per attention_mechanism, saw: %d vs %d"
% (len(attention_layer_sizes), len(attention_mechanisms)))
self._attention_layers = tuple(
layers_core.Dense(
attention_layer_size, name="attention_layer", use_bias=False)
for attention_layer_size in attention_layer_sizes)
self._attention_layer_size = sum(attention_layer_sizes)
else:
self._attention_layers = None
self._attention_layer_size = sum(
attention_mechanism.values.get_shape()[-1].value
for attention_mechanism in attention_mechanisms)
self._cell = cell
self._attention_mechanisms = attention_mechanisms
self._cell_input_fn = cell_input_fn
self._output_attention = output_attention
self._alignment_history = alignment_history
with tf.name_scope(name, "AttentionWrapperInit"):
if initial_cell_state is None:
self._initial_cell_state = None
else:
final_state_tensor = nest.flatten(initial_cell_state)[-1]
state_batch_size = (
final_state_tensor.shape[0].value
or tf.shape(final_state_tensor)[0])
error_message = (
"When constructing AttentionWrapper %s: " % self._base_name +
"Non-matching batch sizes between the memory "
"(encoder output) and initial_cell_state. Are you using "
"the BeamSearchDecoder? You may need to tile your initial state "
"via the tf.contrib.seq2seq.tile_batch function with argument "
"multiple=beam_width.")
with tf.control_dependencies(
self._batch_size_checks(state_batch_size, error_message)):
self._initial_cell_state = nest.map_structure(
lambda s: tf.identity(s, name="check_initial_cell_state"),
initial_cell_state)
def _batch_size_checks(self, batch_size, error_message):
return [tf.assert_equal(batch_size,
attention_mechanism.batch_size,
message=error_message)
for attention_mechanism in self._attention_mechanisms]
def _item_or_tuple(self, seq):
"""Returns `seq` as tuple or the singular element.
Which is returned is determined by how the AttentionMechanism(s) were passed
to the constructor.
Args:
seq: A non-empty sequence of items or generator.
Returns:
Either the values in the sequence as a tuple if AttentionMechanism(s)
were passed to the constructor as a sequence or the singular element.
"""
t = tuple(seq)
if self._is_multi:
return t
else:
return t[0]
@property
def output_size(self):
if self._output_attention:
return self._attention_layer_size
else:
return self._cell.output_size
@property
def state_size(self):
return AttentionWrapperState(
cell_state=self._cell.state_size,
time=tf.TensorShape([]),
attention=self._attention_layer_size,
alignments=self._item_or_tuple(
a.alignments_size for a in self._attention_mechanisms),
alignment_history=self._item_or_tuple(
() for _ in self._attention_mechanisms)) # sometimes a TensorArray
def zero_state(self, batch_size, dtype):
with tf.name_scope(type(self).__name__ + "ZeroState", values=[batch_size]):
if self._initial_cell_state is not None:
cell_state = self._initial_cell_state
else:
cell_state = self._cell.zero_state(batch_size, dtype)
error_message = (
"When calling zero_state of AttentionWrapper %s: " % self._base_name +
"Non-matching batch sizes between the memory "
"(encoder output) and the requested batch size. Are you using "
"the BeamSearchDecoder? If so, make sure your encoder output has "
"been tiled to beam_width via tf.contrib.seq2seq.tile_batch, and "
"the batch_size= argument passed to zero_state is "
"batch_size * beam_width.")
with tf.control_dependencies(
self._batch_size_checks(batch_size, error_message)):
cell_state = nest.map_structure(
lambda s: tf.identity(s, name="checked_cell_state"),
cell_state)
return AttentionWrapperState(
cell_state=cell_state,
time=tf.zeros([], dtype=tf.int32),
attention=_zero_state_tensors(self._attention_layer_size, batch_size, dtype),
alignments=self._item_or_tuple(
attention_mechanism.initial_alignments(batch_size, dtype)
for attention_mechanism in self._attention_mechanisms),
alignment_history=self._item_or_tuple(
tf.TensorArray(dtype=dtype, size=0, dynamic_size=True)
if self._alignment_history else ()
for _ in self._attention_mechanisms))
def call(self, inputs, state):
"""Perform a step of attention-wrapped RNN.
- Step 1: Mix the `inputs` and previous step's `attention` output via
`cell_input_fn`.
- Step 2: Call the wrapped `cell` with this input and its previous state.
- Step 3: Score the cell's output with `attention_mechanism`.
- Step 4: Calculate the alignments by passing the score through the
`normalizer`.
- Step 5: Calculate the context vector as the inner product between the
alignments and the attention_mechanism's values (memory).
- Step 6: Calculate the attention output by concatenating the cell output
and context through the attention layer (a linear layer with
`attention_layer_size` outputs).
Args:
inputs: (Possibly nested tuple of) Tensor, the input at this time step.
state: An instance of `AttentionWrapperState` containing
tensors from the previous time step.
Returns:
A tuple `(attention_or_cell_output, next_state)`, where:
- `attention_or_cell_output` depending on `output_attention`.
- `next_state` is an instance of `AttentionWrapperState`
containing the state calculated at this time step.
Raises:
TypeError: If `state` is not an instance of `AttentionWrapperState`.
"""
if not isinstance(state, AttentionWrapperState):
raise TypeError("Expected state to be instance of AttentionWrapperState. "
"Received type %s instead." % type(state))
# Step 1: Calculate the true inputs to the cell based on the
# previous attention value.
cell_inputs = self._cell_input_fn(inputs, state.attention)
cell_state = state.cell_state
cell_output, next_cell_state = self._cell(cell_inputs, cell_state)
cell_batch_size = (
cell_output.shape[0].value or tf.shape(cell_output)[0])
error_message = (
"When applying AttentionWrapper %s: " % self.name +
"Non-matching batch sizes between the memory "
"(encoder output) and the query (decoder output). Are you using "
"the BeamSearchDecoder? You may need to tile your memory input via "
"the tf.contrib.seq2seq.tile_batch function with argument "
"multiple=beam_width.")
with tf.control_dependencies(
self._batch_size_checks(cell_batch_size, error_message)):
cell_output = tf.identity(
cell_output, name="checked_cell_output")
if self._is_multi:
previous_alignments = state.alignments
previous_alignment_history = state.alignment_history
else:
previous_alignments = [state.alignments]
previous_alignment_history = [state.alignment_history]
all_alignments = []
all_attentions = []
all_histories = []
for i, attention_mechanism in enumerate(self._attention_mechanisms):
attention, alignments = _compute_attention(
attention_mechanism, cell_output, previous_alignments[i],
self._attention_layers[i] if self._attention_layers else None,
self.is_manual_attention, self.manual_alignments, state.time)
alignment_history = previous_alignment_history[i].write(
state.time, alignments) if self._alignment_history else ()
all_alignments.append(alignments)
all_histories.append(alignment_history)
all_attentions.append(attention)
attention = tf.concat(all_attentions, 1)
next_state = AttentionWrapperState(
time=state.time + 1,
cell_state=next_cell_state,
attention=attention,
alignments=self._item_or_tuple(all_alignments),
alignment_history=self._item_or_tuple(all_histories))
if self._output_attention:
return attention, next_state
else:
return cell_output, next_state
def _compute_attention(
attention_mechanism, cell_output, previous_alignments,
attention_layer, is_manual_attention, manual_alignments, time):
computed_alignments = attention_mechanism(
cell_output, previous_alignments=previous_alignments)
batch_size, max_time = \
tf.shape(computed_alignments)[0], tf.shape(computed_alignments)[1]
alignments = tf.cond(
is_manual_attention,
lambda: manual_alignments[:, time, :],
lambda: computed_alignments,
)
#alignments = tf.one_hot(tf.zeros((batch_size,), dtype=tf.int32), max_time, dtype=tf.float32)
# Reshape from [batch_size, memory_time] to [batch_size, 1, memory_time]
expanded_alignments = tf.expand_dims(alignments, 1)
# Context is the inner product of alignments and values along the
# memory time dimension.
# alignments shape is
# [batch_size, 1, memory_time]
# attention_mechanism.values shape is
# [batch_size, memory_time, memory_size]
# the batched matmul is over memory_time, so the output shape is
# [batch_size, 1, memory_size].
# we then squeeze out the singleton dim.
context = tf.matmul(expanded_alignments, attention_mechanism.values)
context = tf.squeeze(context, [1])
if attention_layer is not None:
attention = attention_layer(tf.concat([cell_output, context], 1))
else:
attention = context
return attention, alignments
class DecoderPrenetWrapper(RNNCell):
'''Runs RNN inputs through a prenet before sending them to the cell.'''
def __init__(
self, cell, embed_to_concat,
is_training, prenet_sizes, dropout_prob):
super(DecoderPrenetWrapper, self).__init__()
self._is_training = is_training
self._cell = cell
self._embed_to_concat = embed_to_concat
self.prenet_sizes = prenet_sizes
self.dropout_prob = dropout_prob
@property
def state_size(self):
return self._cell.state_size
@property
def output_size(self):
return self._cell.output_size
def call(self, inputs, state):
prenet_out = prenet(
inputs, self._is_training,
self.prenet_sizes, self.dropout_prob, scope='decoder_prenet')
if self._embed_to_concat is not None:
concat_out = tf.concat(
[prenet_out, self._embed_to_concat],
axis=-1, name='speaker_concat')
return self._cell(concat_out, state)
else:
return self._cell(prenet_out, state)
def zero_state(self, batch_size, dtype):
return self._cell.zero_state(batch_size, dtype)
class ConcatOutputAndAttentionWrapper(RNNCell):
'''Concatenates RNN cell output with the attention context vector.
This is expected to wrap a cell wrapped with an AttentionWrapper constructed with
attention_layer_size=None and output_attention=False. Such a cell's state will include an
"attention" field that is the context vector.
'''
def __init__(self, cell, embed_to_concat):
super(ConcatOutputAndAttentionWrapper, self).__init__()
self._cell = cell
self._embed_to_concat = embed_to_concat
@property
def state_size(self):
return self._cell.state_size
@property
def output_size(self):
return self._cell.output_size + self._cell.state_size.attention
def call(self, inputs, state):
output, res_state = self._cell(inputs, state)
if self._embed_to_concat is not None:
tensors = [
output, res_state.attention,
self._embed_to_concat,
]
return tf.concat(tensors, axis=-1), res_state
else:
return tf.concat([output, res_state.attention], axis=-1), res_state
def zero_state(self, batch_size, dtype):
return self._cell.zero_state(batch_size, dtype)

343
models/tacotron.py Normal file
View file

@ -0,0 +1,343 @@
# Code based on https://github.com/keithito/tacotron/blob/master/models/tacotron.py
import numpy as np
import tensorflow as tf
from tensorflow.contrib.seq2seq import BasicDecoder, BahdanauAttention, BahdanauMonotonicAttention
from tensorflow.contrib.rnn import GRUCell, MultiRNNCell, OutputProjectionWrapper, ResidualWrapper
from utils.infolog import log
from text.symbols import symbols
from .modules import *
from .helpers import TacoTestHelper, TacoTrainingHelper
from .rnn_wrappers import AttentionWrapper, DecoderPrenetWrapper, ConcatOutputAndAttentionWrapper
class Tacotron():
def __init__(self, hparams):
self._hparams = hparams
def initialize(
self, inputs, input_lengths, num_speakers, speaker_id,
mel_targets=None, linear_targets=None, loss_coeff=None,
rnn_decoder_test_mode=False, is_randomly_initialized=False,
):
is_training = linear_targets is not None
self.is_randomly_initialized = is_randomly_initialized
with tf.variable_scope('inference') as scope:
hp = self._hparams
batch_size = tf.shape(inputs)[0]
# Embeddings
char_embed_table = tf.get_variable(
'embedding', [len(symbols), hp.embedding_size], dtype=tf.float32,
initializer=tf.truncated_normal_initializer(stddev=0.5))
# [N, T_in, embedding_size]
char_embedded_inputs = \
tf.nn.embedding_lookup(char_embed_table, inputs)
self.num_speakers = num_speakers
if self.num_speakers > 1:
if hp.speaker_embedding_size != 1:
speaker_embed_table = tf.get_variable(
'speaker_embedding',
[self.num_speakers, hp.speaker_embedding_size], dtype=tf.float32,
initializer=tf.truncated_normal_initializer(stddev=0.5))
# [N, T_in, speaker_embedding_size]
speaker_embed = tf.nn.embedding_lookup(speaker_embed_table, speaker_id)
if hp.model_type == 'deepvoice':
if hp.speaker_embedding_size == 1:
before_highway = get_embed(
speaker_id, self.num_speakers,
hp.enc_prenet_sizes[-1], "before_highway")
encoder_rnn_init_state = get_embed(
speaker_id, self.num_speakers,
hp.enc_rnn_size * 2, "encoder_rnn_init_state")
attention_rnn_init_state = get_embed(
speaker_id, self.num_speakers,
hp.attention_state_size, "attention_rnn_init_state")
decoder_rnn_init_states = [get_embed(
speaker_id, self.num_speakers,
hp.dec_rnn_size, "decoder_rnn_init_states{}".format(idx + 1)) \
for idx in range(hp.dec_layer_num)]
else:
deep_dense = lambda x, dim: \
tf.layers.dense(x, dim, activation=tf.nn.softsign)
before_highway = deep_dense(
speaker_embed, hp.enc_prenet_sizes[-1])
encoder_rnn_init_state = deep_dense(
speaker_embed, hp.enc_rnn_size * 2)
attention_rnn_init_state = deep_dense(
speaker_embed, hp.attention_state_size)
decoder_rnn_init_states = [deep_dense(
speaker_embed, hp.dec_rnn_size) for _ in range(hp.dec_layer_num)]
speaker_embed = None # deepvoice does not use speaker_embed directly
elif hp.model_type == 'simple':
before_highway = None
encoder_rnn_init_state = None
attention_rnn_init_state = None
decoder_rnn_init_states = None
else:
raise Exception(" [!] Unkown multi-speaker model type: {}".format(hp.model_type))
else:
speaker_embed = None
before_highway = None
encoder_rnn_init_state = None
attention_rnn_init_state = None
decoder_rnn_init_states = None
##############
# Encoder
##############
# [N, T_in, enc_prenet_sizes[-1]]
prenet_outputs = prenet(char_embedded_inputs, is_training,
hp.enc_prenet_sizes, hp.dropout_prob,
scope='prenet')
encoder_outputs = cbhg(
prenet_outputs, input_lengths, is_training,
hp.enc_bank_size, hp.enc_bank_channel_size,
hp.enc_maxpool_width, hp.enc_highway_depth, hp.enc_rnn_size,
hp.enc_proj_sizes, hp.enc_proj_width,
scope="encoder_cbhg",
before_highway=before_highway,
encoder_rnn_init_state=encoder_rnn_init_state)
##############
# Attention
##############
# For manaul control of attention
self.is_manual_attention = tf.placeholder(
tf.bool, shape=(), name='is_manual_attention',
)
self.manual_alignments = tf.placeholder(
tf.float32, shape=[None, None, None], name="manual_alignments",
)
dec_prenet_outputs = DecoderPrenetWrapper(
GRUCell(hp.attention_state_size),
speaker_embed,
is_training, hp.dec_prenet_sizes, hp.dropout_prob)
if hp.attention_type == 'bah_mon':
attention_mechanism = BahdanauMonotonicAttention(
hp.attention_size, encoder_outputs)
elif hp.attention_type == 'bah_norm':
attention_mechanism = BahdanauAttention(
hp.attention_size, encoder_outputs, normalize=True)
elif hp.attention_type == 'luong_scaled':
attention_mechanism = LuongAttention(
hp.attention_size, encoder_outputs, scale=True)
elif hp.attention_type == 'luong':
attention_mechanism = LuongAttention(
hp.attention_size, encoder_outputs)
elif hp.attention_type == 'bah':
attention_mechanism = BahdanauAttention(
hp.attention_size, encoder_outputs)
elif hp.attention_type.startswith('ntm2'):
shift_width = int(hp.attention_type.split('-')[-1])
attention_mechanism = NTMAttention2(
hp.attention_size, encoder_outputs, shift_width=shift_width)
else:
raise Exception(" [!] Unkown attention type: {}".format(hp.attention_type))
attention_cell = AttentionWrapper(
dec_prenet_outputs,
attention_mechanism,
self.is_manual_attention,
self.manual_alignments,
initial_cell_state=attention_rnn_init_state,
alignment_history=True,
output_attention=False
)
# Concatenate attention context vector and RNN cell output into a 512D vector.
# [N, T_in, attention_size+attention_state_size]
concat_cell = ConcatOutputAndAttentionWrapper(
attention_cell, embed_to_concat=speaker_embed)
# Decoder (layers specified bottom to top):
cells = [OutputProjectionWrapper(concat_cell, hp.dec_rnn_size)]
for _ in range(hp.dec_layer_num):
cells.append(ResidualWrapper(GRUCell(hp.dec_rnn_size)))
# [N, T_in, 256]
decoder_cell = MultiRNNCell(cells, state_is_tuple=True)
# Project onto r mel spectrograms (predict r outputs at each RNN step):
output_cell = OutputProjectionWrapper(
decoder_cell, hp.num_mels * hp.reduction_factor)
decoder_init_state = output_cell.zero_state(
batch_size=batch_size, dtype=tf.float32)
if hp.model_type == "deepvoice":
# decoder_init_state[0] : AttentionWrapperState
# = cell_state + attention + time + alignments + alignment_history
# decoder_init_state[0][0] = attention_rnn_init_state (already applied)
decoder_init_state = list(decoder_init_state)
for idx, cell in enumerate(decoder_rnn_init_states):
shape1 = decoder_init_state[idx + 1].get_shape().as_list()
shape2 = cell.get_shape().as_list()
if shape1 != shape2:
raise Exception(" [!] Shape {} and {} should be equal". \
format(shape1, shape2))
decoder_init_state[idx + 1] = cell
decoder_init_state = tuple(decoder_init_state)
if is_training:
helper = TacoTrainingHelper(
inputs, mel_targets, hp.num_mels, hp.reduction_factor,
rnn_decoder_test_mode)
else:
helper = TacoTestHelper(
batch_size, hp.num_mels, hp.reduction_factor)
(decoder_outputs, _), final_decoder_state, _ = \
tf.contrib.seq2seq.dynamic_decode(
BasicDecoder(output_cell, helper, decoder_init_state),
maximum_iterations=hp.max_iters)
# [N, T_out, M]
mel_outputs = tf.reshape(
decoder_outputs, [batch_size, -1, hp.num_mels])
# Add post-processing CBHG:
# [N, T_out, 256]
#post_outputs = post_cbhg(mel_outputs, hp.num_mels, is_training)
post_outputs = cbhg(
mel_outputs, None, is_training,
hp.post_bank_size, hp.post_bank_channel_size,
hp.post_maxpool_width, hp.post_highway_depth, hp.post_rnn_size,
hp.post_proj_sizes, hp.post_proj_width,
scope='post_cbhg')
if speaker_embed is not None and hp.model_type == 'simple':
expanded_speaker_emb = tf.expand_dims(speaker_embed, [1])
tiled_speaker_embedding = tf.tile(
expanded_speaker_emb, [1, tf.shape(post_outputs)[1], 1])
# [N, T_out, 256 + alpha]
post_outputs = \
tf.concat([tiled_speaker_embedding, post_outputs], axis=-1)
linear_outputs = tf.layers.dense(post_outputs, hp.num_freq) # [N, T_out, F]
# Grab alignments from the final decoder state:
alignments = tf.transpose(
final_decoder_state[0].alignment_history.stack(), [1, 2, 0])
self.inputs = inputs
self.speaker_id = speaker_id
self.input_lengths = input_lengths
self.loss_coeff = loss_coeff
self.mel_outputs = mel_outputs
self.linear_outputs = linear_outputs
self.alignments = alignments
self.mel_targets = mel_targets
self.linear_targets = linear_targets
self.final_decoder_state = final_decoder_state
log('='*40)
log(' model_type: %s' % hp.model_type)
log('='*40)
log('Initialized Tacotron model. Dimensions: ')
log(' embedding: %d' % char_embedded_inputs.shape[-1])
if speaker_embed is not None:
log(' speaker embedding: %d' % speaker_embed.shape[-1])
else:
log(' speaker embedding: None')
log(' prenet out: %d' % prenet_outputs.shape[-1])
log(' encoder out: %d' % encoder_outputs.shape[-1])
log(' attention out: %d' % attention_cell.output_size)
log(' concat attn & out: %d' % concat_cell.output_size)
log(' decoder cell out: %d' % decoder_cell.output_size)
log(' decoder out (%d frames): %d' % (hp.reduction_factor, decoder_outputs.shape[-1]))
log(' decoder out (1 frame): %d' % mel_outputs.shape[-1])
log(' postnet out: %d' % post_outputs.shape[-1])
log(' linear out: %d' % linear_outputs.shape[-1])
def add_loss(self):
'''Adds loss to the model. Sets "loss" field. initialize must have been called.'''
with tf.variable_scope('loss') as scope:
hp = self._hparams
mel_loss = tf.abs(self.mel_targets - self.mel_outputs)
l1 = tf.abs(self.linear_targets - self.linear_outputs)
expanded_loss_coeff = tf.expand_dims(
tf.expand_dims(self.loss_coeff, [-1]), [-1])
if hp.prioritize_loss:
# Prioritize loss for frequencies.
upper_priority_freq = int(5000 / (hp.sample_rate * 0.5) * hp.num_freq)
lower_priority_freq = int(165 / (hp.sample_rate * 0.5) * hp.num_freq)
l1_priority= l1[:,:,lower_priority_freq:upper_priority_freq]
self.loss = tf.reduce_mean(mel_loss * expanded_loss_coeff) + \
0.5 * tf.reduce_mean(l1 * expanded_loss_coeff) + \
0.5 * tf.reduce_mean(l1_priority * expanded_loss_coeff)
self.linear_loss = tf.reduce_mean(
0.5 * (tf.reduce_mean(l1) + tf.reduce_mean(l1_priority)))
else:
self.loss = tf.reduce_mean(mel_loss * expanded_loss_coeff) + \
tf.reduce_mean(l1 * expanded_loss_coeff)
self.linear_loss = tf.reduce_mean(l1)
self.mel_loss = tf.reduce_mean(mel_loss)
self.loss_without_coeff = self.mel_loss + self.linear_loss
def add_optimizer(self, global_step):
'''Adds optimizer. Sets "gradients" and "optimize" fields. add_loss must have been called.
Args:
global_step: int32 scalar Tensor representing current global step in training
'''
with tf.variable_scope('optimizer') as scope:
hp = self._hparams
step = tf.cast(global_step + 1, dtype=tf.float32)
if hp.decay_learning_rate_mode == 0:
if self.is_randomly_initialized:
warmup_steps = 4000.0
else:
warmup_steps = 40000.0
self.learning_rate = hp.initial_learning_rate * warmup_steps**0.5 * \
tf.minimum(step * warmup_steps**-1.5, step**-0.5)
elif hp.decay_learning_rate_mode == 1:
self.learning_rate = hp.initial_learning_rate * \
tf.train.exponential_decay(1., step, 3000, 0.95)
optimizer = tf.train.AdamOptimizer(self.learning_rate, hp.adam_beta1, hp.adam_beta2)
gradients, variables = zip(*optimizer.compute_gradients(self.loss))
self.gradients = gradients
clipped_gradients, _ = tf.clip_by_global_norm(gradients, 1.0)
# Add dependency on UPDATE_OPS; otherwise batchnorm won't work correctly. See:
# https://github.com/tensorflow/tensorflow/issues/1122
with tf.control_dependencies(tf.get_collection(tf.GraphKeys.UPDATE_OPS)):
self.optimize = optimizer.apply_gradients(zip(clipped_gradients, variables),
global_step=global_step)
def get_dummy_feed_dict(self):
feed_dict = {
self.is_manual_attention: False,
self.manual_alignments: np.zeros([1, 1, 1]),
}
return feed_dict

181
recognition/alignment.py Normal file
View file

@ -0,0 +1,181 @@
import os
import string
import argparse
import operator
from functools import partial
from difflib import SequenceMatcher
from audio.get_duration import get_durations
from text import remove_puncuations, text_to_sequence
from utils import load_json, write_json, parallel_run, remove_postfix, backup_file
def plain_text(text):
return "".join(remove_puncuations(text.strip()).split())
def add_punctuation(text):
if text.endswith('다'):
return text + "."
else:
return text
def similarity(text_a, text_b):
text_a = plain_text(text_a)
text_b = plain_text(text_b)
score = SequenceMatcher(None, text_a, text_b).ratio()
return score
def first_word_combined_words(text):
words = text.split()
if len(words) > 1:
first_words = [words[0], words[0]+words[1]]
else:
first_words = [words[0]]
return first_words
def first_word_combined_texts(text):
words = text.split()
if len(words) > 1:
if len(words) > 2:
text2 = " ".join([words[0]+words[1]] + words[2:])
else:
text2 = words[0]+words[1]
texts = [text, text2]
else:
texts = [text]
return texts
def search_optimal(found_text, recognition_text):
# 1. found_text is usually more accurate
# 2. recognition_text can have more or less word
optimal = None
if plain_text(recognition_text) in plain_text(found_text):
optimal = recognition_text
else:
found = False
for tmp_text in first_word_combined_texts(found_text):
for recognition_first_word in first_word_combined_words(recognition_text):
if recognition_first_word in tmp_text:
start_idx = tmp_text.find(recognition_first_word)
if tmp_text != found_text:
found_text = found_text[max(0, start_idx-1):].strip()
else:
found_text = found_text[start_idx:].strip()
found = True
break
if found:
break
recognition_last_word = recognition_text.split()[-1]
if recognition_last_word in found_text:
end_idx = found_text.find(recognition_last_word)
punctuation = ""
if len(found_text) > end_idx + len(recognition_last_word):
punctuation = found_text[end_idx + len(recognition_last_word)]
if punctuation not in string.punctuation:
punctuation = ""
found_text = found_text[:end_idx] + recognition_last_word + punctuation
found = True
if found:
optimal = found_text
return optimal
def align_text_for_jtbc(
item, score_threshold, debug=False):
audio_path, recognition_text = item
audio_dir = os.path.dirname(audio_path)
base_dir = os.path.dirname(audio_dir)
news_path = remove_postfix(audio_path.replace("audio", "assets"))
news_path = os.path.splitext(news_path)[0] + ".txt"
strip_fn = lambda line: line.strip().replace('"', '').replace("'", "")
candidates = [strip_fn(line) for line in open(news_path).readlines()]
scores = { candidate: similarity(candidate, recognition_text) \
for candidate in candidates}
sorted_scores = sorted(scores.items(), key=operator.itemgetter(1))[::-1]
first, second = sorted_scores[0], sorted_scores[1]
if first[1] > second[1] and first[1] >= score_threshold:
found_text, score = first
aligned_text = search_optimal(found_text, recognition_text)
if debug:
print(" ", audio_path)
print(" ", recognition_text)
print("=> ", found_text)
print("==>", aligned_text)
print("="*30)
if aligned_text is not None:
result = { audio_path: add_punctuation(aligned_text) }
elif abs(len(text_to_sequence(found_text)) - len(text_to_sequence(recognition_text))) > 10:
result = {}
else:
result = { audio_path: [add_punctuation(found_text), recognition_text] }
else:
result = {}
if len(result) == 0:
result = { audio_path: [recognition_text] }
return result
def align_text_batch(config):
if "jtbc" in config.recognition_path.lower():
align_text = partial(align_text_for_jtbc,
score_threshold=config.score_threshold)
else:
raise Exception(" [!] find_related_texts for `{}` is not defined". \
format(config.recognition_path))
results = {}
data = load_json(config.recognition_path)
items = parallel_run(
align_text, data.items(),
desc="align_text_batch", parallel=True)
for item in items:
results.update(item)
found_count = sum([type(value) == str for value in results.values()])
print(" [*] # found: {:.5f}% ({}/{})".format(
len(results)/len(data), len(results), len(data)))
print(" [*] # exact match: {:.5f}% ({}/{})".format(
found_count/len(items), found_count, len(items)))
return results
if __name__ == '__main__':
parser = argparse.ArgumentParser()
parser.add_argument('--recognition_path', required=True)
parser.add_argument('--alignment_filename', default="alignment.json")
parser.add_argument('--score_threshold', default=0.4, type=float)
config, unparsed = parser.parse_known_args()
results = align_text_batch(config)
base_dir = os.path.dirname(config.recognition_path)
alignment_path = \
os.path.join(base_dir, config.alignment_filename)
if os.path.exists(alignment_path):
backup_file(alignment_path)
write_json(alignment_path, results)
duration = get_durations(results.keys(), print_detail=False)

126
recognition/google.py Normal file
View file

@ -0,0 +1,126 @@
import io
import os
import json
import argparse
import numpy as np
from glob import glob
from functools import partial
from utils import parallel_run, remove_file, backup_file, write_json
from audio import load_audio, save_audio, resample_audio, get_duration
def text_recognition(path, config):
root, ext = os.path.splitext(path)
txt_path = root + ".txt"
if os.path.exists(txt_path):
with open(txt_path) as f:
out = json.loads(open(txt_path).read())
return out
from google.cloud import speech
from google.cloud.speech import enums
from google.cloud.speech import types
out = {}
error_count = 0
tmp_path = os.path.splitext(path)[0] + ".tmp.wav"
while True:
try:
client = speech.SpeechClient()
content = load_audio(
path, pre_silence_length=config.pre_silence_length,
post_silence_length=config.post_silence_length)
max_duration = config.max_duration - \
config.pre_silence_length - config.post_silence_length
audio_duration = get_duration(content)
if audio_duration >= max_duration:
print(" [!] Skip {} because of duration: {} > {}". \
format(path, audio_duration, max_duration))
return {}
content = resample_audio(content, config.sample_rate)
save_audio(content, tmp_path, config.sample_rate)
with io.open(tmp_path, 'rb') as f:
audio = types.RecognitionAudio(content=f.read())
config = types.RecognitionConfig(
encoding=enums.RecognitionConfig.AudioEncoding.LINEAR16,
sample_rate_hertz=config.sample_rate,
language_code='ko-KR')
response = client.recognize(config, audio)
if len(response.results) > 0:
alternatives = response.results[0].alternatives
results = [alternative.transcript for alternative in alternatives]
assert len(results) == 1, "More than 1 results: {}".format(results)
out = { path: "" if len(results) == 0 else results[0] }
print(path, results[0])
break
break
except Exception as err:
raise Exception("OS error: {0}".format(err))
error_count += 1
print("Skip warning for {} for {} times". \
format(path, error_count))
if error_count > 5:
break
else:
continue
remove_file(tmp_path)
with open(txt_path, 'w') as f:
json.dump(out, f, indent=2, ensure_ascii=False)
return out
def text_recognition_batch(paths, config):
paths.sort()
results = {}
items = parallel_run(
partial(text_recognition, config=config), paths,
desc="text_recognition_batch", parallel=True)
for item in items:
results.update(item)
return results
if __name__ == '__main__':
parser = argparse.ArgumentParser()
parser.add_argument('--audio_pattern', required=True)
parser.add_argument('--recognition_filename', default="recognition.json")
parser.add_argument('--sample_rate', default=16000, type=int)
parser.add_argument('--pre_silence_length', default=1, type=int)
parser.add_argument('--post_silence_length', default=1, type=int)
parser.add_argument('--max_duration', default=60, type=int)
config, unparsed = parser.parse_known_args()
audio_dir = os.path.dirname(config.audio_pattern)
for tmp_path in glob(os.path.join(audio_dir, "*.tmp.*")):
remove_file(tmp_path)
paths = glob(config.audio_pattern)
paths.sort()
results = text_recognition_batch(paths, config)
base_dir = os.path.dirname(audio_dir)
recognition_path = \
os.path.join(base_dir, config.recognition_filename)
if os.path.exists(recognition_path):
backup_file(recognition_path)
write_json(recognition_path, results)

75
requirements.txt Normal file
View file

@ -0,0 +1,75 @@
appnope==0.1.0
audioread==2.1.5
bleach==1.5.0
certifi==2017.7.27.1
chardet==3.0.4
click==6.7
cycler==0.10.0
Cython==0.26.1
decorator==4.0.11
entrypoints==0.2.3
Flask==0.12.2
Flask-Cors==3.0.3
gTTS==1.2.2
gTTS-token==1.1.1
html5lib==0.9999999
idna==2.6
imageio==2.1.2
ipdb==0.10.3
ipykernel==4.6.1
ipython==6.1.0
ipython-genutils==0.2.0
ipywidgets==7.0.1
itsdangerous==0.24
jamo==0.4.0
jedi==0.10.2
Jinja2==2.9.6
joblib==0.11
jsonschema==2.6.0
jupyter-client==5.1.0
jupyter-core==4.3.0
librosa==0.5.1
llvmlite==0.20.0
Markdown==2.6.9
MarkupSafe==1.0
matplotlib==2.0.2
mistune==0.7.4
moviepy==0.2.3.2
nbconvert==5.3.1
nbformat==4.4.0
nltk==3.2.4
notebook==5.1.0
numba==0.35.0
numpy==1.13.3
olefile==0.44
pandocfilters==1.4.2
pexpect==4.2.1
pickleshare==0.7.4
Pillow==4.3.0
prompt-toolkit==1.0.15
protobuf==3.4.0
ptyprocess==0.5.2
pydub==0.20.0
Pygments==2.2.0
pyparsing==2.2.0
python-dateutil==2.6.1
pytz==2017.2
pyzmq==16.0.2
requests==2.18.4
resampy==0.2.0
scikit-learn==0.19.0
scipy==0.19.1
simplegeneric==0.8.1
six==1.11.0
tensorflow==1.3.0
tensorflow-tensorboard==0.1.6
terminado==0.6
testpath==0.3.1
tornado==4.5.2
tqdm==4.11.2
traitlets==4.3.2
urllib3==1.22
wcwidth==0.1.7
Werkzeug==0.12.2
widgetsnbextension==3.0.3
youtube-dl==2017.10.7

8
run.sh Normal file
View file

@ -0,0 +1,8 @@
#!/bin/sh
CUDA_VISIBLE_DEVICES= python app.py --load_path logs/deepvoice2-256-256-krbook-bah-mon-22000-no-priority --dataname=krbook --num_speakers=1
CUDA_VISIBLE_DEVICES= python app.py --load_path logs/jtbc_2017-09-25_11-49-23 --dataname=krbook --num_speakers=1 --port=5002
CUDA_VISIBLE_DEVICES= python app.py --load_path logs/krbook_2017-09-27_17-02-44 --dataname=krbook --num_speakers=1 --port=5001
CUDA_VISIBLE_DEVICES= python app.py --load_path logs/krfemale_2017-10-10_20-37-38 --dataname=krbook --num_speakers=1 --port=5003
CUDA_VISIBLE_DEVICES= python app.py --load_path logs/krmale_2017-10-10_17-49-49 --dataname=krbook --num_speakers=1 --port=5005
CUDA_VISIBLE_DEVICES= python app.py --load_path logs/park+moon+krbook_2017-10-09_20-43-53 --dataname=krbook --num_speakers=3 --port=5004

16
scripts/prepare_jtbc.sh Executable file
View file

@ -0,0 +1,16 @@
#!/bin/sh
# 1. Download and extract audio and texts
python -m datasets.jtbc.download
# 2. Split audios on silence
python -m audio.silence --audio_pattern "./datasets/jtbc/audio/*.wav" --method=pydub
# 3. Run Google Speech Recognition
python -m recognition.google --audio_pattern "./datasets/jtbc/audio/*.*.wav"
# 4. Run heuristic text-audio pair search (any improvement on this is welcome)
python -m recognition.alignment --recognition_path "./datasets/jtbc/recognition.json" --score_threshold=0.5
# 5. Remove intro music
rm datasets/jtbc/data/*.0000.npz

13
scripts/prepare_moon.sh Executable file
View file

@ -0,0 +1,13 @@
#!/bin/sh
# 1. Download and extract audio and texts
python -m datasets.moon.download
# 2. Split audios on silence
python -m audio.silence --audio_pattern "./datasets/moon/audio/*.wav" --method=pydub
# 3. Run Google Speech Recognition
python -m recognition.google --audio_pattern "./datasets/moon/audio/*.*.wav"
# 4. Run heuristic text-audio pair search (any improvement on this is welcome)
python -m recognition.alignment --recognition_path "./datasets/moon/recognition.json" --score_threshold=0.5

13
scripts/prepare_park.sh Executable file
View file

@ -0,0 +1,13 @@
#!/bin/sh
# 1. Download and extract audio and texts
python -m datasets.park.download
# 2. Split audios on silence
python -m audio.silence --audio_pattern "./datasets/park/audio/*.wav" --method=pydub
# 3. Run Google Speech Recognition
python -m recognition.google --audio_pattern "./datasets/park/audio/*.*.wav"
# 4. Run heuristic text-audio pair search (any improvement on this is welcome)
python -m recognition.alignment --recognition_path "./datasets/park/recognition.json" --score_threshold=0.5

389
synthesizer.py Normal file
View file

@ -0,0 +1,389 @@
import io
import os
import re
import librosa
import argparse
import numpy as np
from glob import glob
from tqdm import tqdm
import tensorflow as tf
from functools import partial
from hparams import hparams
from models import create_model, get_most_recent_checkpoint
from audio import save_audio, inv_spectrogram, inv_preemphasis, \
inv_spectrogram_tensorflow
from utils import plot, PARAMS_NAME, load_json, load_hparams, \
add_prefix, add_postfix, get_time, parallel_run, makedirs
from text.korean import tokenize
from text import text_to_sequence, sequence_to_text
class Synthesizer(object):
def close(self):
tf.reset_default_graph()
self.sess.close()
def load(self, checkpoint_path, num_speakers=2, checkpoint_step=None, model_name='tacotron'):
self.num_speakers = num_speakers
if os.path.isdir(checkpoint_path):
load_path = checkpoint_path
checkpoint_path = get_most_recent_checkpoint(checkpoint_path, checkpoint_step)
else:
load_path = os.path.dirname(checkpoint_path)
print('Constructing model: %s' % model_name)
inputs = tf.placeholder(tf.int32, [None, None], 'inputs')
input_lengths = tf.placeholder(tf.int32, [None], 'input_lengths')
batch_size = tf.shape(inputs)[0]
speaker_id = tf.placeholder_with_default(
tf.zeros([batch_size], dtype=tf.int32), [None], 'speaker_id')
load_hparams(hparams, load_path)
with tf.variable_scope('model') as scope:
self.model = create_model(hparams)
self.model.initialize(
inputs, input_lengths,
self.num_speakers, speaker_id)
self.wav_output = \
inv_spectrogram_tensorflow(self.model.linear_outputs)
print('Loading checkpoint: %s' % checkpoint_path)
sess_config = tf.ConfigProto(
allow_soft_placement=True,
intra_op_parallelism_threads=1,
inter_op_parallelism_threads=2)
sess_config.gpu_options.allow_growth = True
self.sess = tf.Session(config=sess_config)
self.sess.run(tf.global_variables_initializer())
saver = tf.train.Saver()
saver.restore(self.sess, checkpoint_path)
def synthesize(self,
texts=None, tokens=None,
base_path=None, paths=None, speaker_ids=None,
start_of_sentence=None, end_of_sentence=True,
pre_word_num=0, post_word_num=0,
pre_surplus_idx=0, post_surplus_idx=1,
use_short_concat=False,
manual_attention_mode=0,
base_alignment_path=None,
librosa_trim=False,
attention_trim=True):
# Possible inputs:
# 1) text=text
# 2) text=texts
# 3) tokens=tokens, texts=texts # use texts as guide
if type(texts) == str:
texts = [texts]
if texts is not None and tokens is None:
sequences = [text_to_sequence(text) for text in texts]
elif tokens is not None:
sequences = tokens
if paths is None:
paths = [None] * len(sequences)
if texts is None:
texts = [None] * len(sequences)
time_str = get_time()
def plot_and_save_parallel(
wavs, alignments, use_manual_attention):
items = list(enumerate(zip(
wavs, alignments, paths, texts, sequences)))
fn = partial(
plot_graph_and_save_audio,
base_path=base_path,
start_of_sentence=start_of_sentence, end_of_sentence=end_of_sentence,
pre_word_num=pre_word_num, post_word_num=post_word_num,
pre_surplus_idx=pre_surplus_idx, post_surplus_idx=post_surplus_idx,
use_short_concat=use_short_concat,
use_manual_attention=use_manual_attention,
librosa_trim=librosa_trim,
attention_trim=attention_trim,
time_str=time_str)
return parallel_run(fn, items,
desc="plot_graph_and_save_audio", parallel=False)
input_lengths = np.argmax(np.array(sequences) == 1, 1)
fetches = [
#self.wav_output,
self.model.linear_outputs,
self.model.alignments,
]
feed_dict = {
self.model.inputs: sequences,
self.model.input_lengths: input_lengths,
}
if base_alignment_path is None:
feed_dict.update({
self.model.manual_alignments: np.zeros([1, 1, 1]),
self.model.is_manual_attention: False,
})
else:
manual_alignments = []
alignment_path = os.path.join(
base_alignment_path,
os.path.basename(base_path))
for idx in range(len(sequences)):
numpy_path = "{}.{}.npy".format(alignment_path, idx)
manual_alignments.append(np.load(numpy_path))
alignments_T = np.transpose(manual_alignments, [0, 2, 1])
feed_dict.update({
self.model.manual_alignments: alignments_T,
self.model.is_manual_attention: True,
})
if speaker_ids is not None:
if type(speaker_ids) == dict:
speaker_embed_table = sess.run(
self.model.speaker_embed_table)
speaker_embed = [speaker_ids[speaker_id] * \
speaker_embed_table[speaker_id] for speaker_id in speaker_ids]
feed_dict.update({
self.model.speaker_embed_table: np.tile()
})
else:
feed_dict[self.model.speaker_id] = speaker_ids
wavs, alignments = \
self.sess.run(fetches, feed_dict=feed_dict)
results = plot_and_save_parallel(
wavs, alignments, True)
if manual_attention_mode > 0:
# argmax one hot
if manual_attention_mode == 1:
alignments_T = np.transpose(alignments, [0, 2, 1]) # [N, E, D]
new_alignments = np.zeros_like(alignments_T)
for idx in range(len(alignments)):
argmax = alignments[idx].argmax(1)
new_alignments[idx][(argmax, range(len(argmax)))] = 1
# sharpening
elif manual_attention_mode == 2:
new_alignments = np.transpose(alignments, [0, 2, 1]) # [N, E, D]
for idx in range(len(alignments)):
var = np.var(new_alignments[idx], 1)
mean_var = var[:input_lengths[idx]].mean()
new_alignments = np.pow(new_alignments[idx], 2)
# prunning
elif manual_attention_mode == 3:
new_alignments = np.transpose(alignments, [0, 2, 1]) # [N, E, D]
for idx in range(len(alignments)):
argmax = alignments[idx].argmax(1)
new_alignments[idx][(argmax, range(len(argmax)))] = 1
feed_dict.update({
self.model.manual_alignments: new_alignments,
self.model.is_manual_attention: True,
})
new_wavs, new_alignments = \
self.sess.run(fetches, feed_dict=feed_dict)
results = plot_and_save_parallel(
new_wavs, new_alignments, True)
return results
def plot_graph_and_save_audio(args,
base_path=None,
start_of_sentence=None, end_of_sentence=None,
pre_word_num=0, post_word_num=0,
pre_surplus_idx=0, post_surplus_idx=1,
use_short_concat=False,
use_manual_attention=False, save_alignment=False,
librosa_trim=False, attention_trim=False,
time_str=None):
idx, (wav, alignment, path, text, sequence) = args
if base_path:
plot_path = "{}/{}.png".format(base_path, get_time())
elif path:
plot_path = path.rsplit('.', 1)[0] + ".png"
else:
plot_path = None
#plot_path = add_prefix(plot_path, time_str)
if use_manual_attention:
plot_path = add_postfix(plot_path, "manual")
if plot_path:
plot.plot_alignment(alignment, plot_path, text=text)
if use_short_concat:
wav = short_concat(
wav, alignment, text,
start_of_sentence, end_of_sentence,
pre_word_num, post_word_num,
pre_surplus_idx, post_surplus_idx)
if attention_trim and end_of_sentence:
end_idx_counter = 0
attention_argmax = alignment.argmax(0)
end_idx = min(len(sequence) - 1, max(attention_argmax))
max_counter = min((attention_argmax == end_idx).sum(), 5)
for jdx, attend_idx in enumerate(attention_argmax):
if len(attention_argmax) > jdx + 1:
if attend_idx == end_idx:
end_idx_counter += 1
if attend_idx == end_idx and attention_argmax[jdx + 1] > end_idx:
break
if end_idx_counter >= max_counter:
break
else:
break
spec_end_idx = hparams.reduction_factor * jdx + 3
wav = wav[:spec_end_idx]
audio_out = inv_spectrogram(wav.T)
if librosa_trim and end_of_sentence:
yt, index = librosa.effects.trim(audio_out,
frame_length=5120, hop_length=256, top_db=50)
audio_out = audio_out[:index[-1]]
if save_alignment:
alignment_path = "{}/{}.npy".format(base_path, idx)
np.save(alignment_path, alignment, allow_pickle=False)
if path or base_path:
if path:
current_path = add_postfix(path, idx)
elif base_path:
current_path = plot_path.replace(".png", ".wav")
save_audio(audio_out, current_path)
return True
else:
io_out = io.BytesIO()
save_audio(audio_out, io_out)
result = io_out.getvalue()
return result
def get_most_recent_checkpoint(checkpoint_dir, checkpoint_step=None):
if checkpoint_step is None:
checkpoint_paths = [path for path in glob("{}/*.ckpt-*.data-*".format(checkpoint_dir))]
idxes = [int(os.path.basename(path).split('-')[1].split('.')[0]) for path in checkpoint_paths]
max_idx = max(idxes)
else:
max_idx = checkpoint_step
lastest_checkpoint = os.path.join(checkpoint_dir, "model.ckpt-{}".format(max_idx))
print(" [*] Found lastest checkpoint: {}".format(lastest_checkpoint))
return lastest_checkpoint
def short_concat(
wav, alignment, text,
start_of_sentence, end_of_sentence,
pre_word_num, post_word_num,
pre_surplus_idx, post_surplus_idx):
# np.array(list(decomposed_text))[attention_argmax]
attention_argmax = alignment.argmax(0)
if not start_of_sentence and pre_word_num > 0:
surplus_decomposed_text = decompose_ko_text("".join(text.split()[0]))
start_idx = len(surplus_decomposed_text) + 1
for idx, attend_idx in enumerate(attention_argmax):
if attend_idx == start_idx and attention_argmax[idx - 1] < start_idx:
break
wav_start_idx = hparams.reduction_factor * idx - 1 - pre_surplus_idx
else:
wav_start_idx = 0
if not end_of_sentence and post_word_num > 0:
surplus_decomposed_text = decompose_ko_text("".join(text.split()[-1]))
end_idx = len(decomposed_text.replace(surplus_decomposed_text, '')) - 1
for idx, attend_idx in enumerate(attention_argmax):
if attend_idx == end_idx and attention_argmax[idx + 1] > end_idx:
break
wav_end_idx = hparams.reduction_factor * idx + 1 + post_surplus_idx
else:
if True: # attention based split
if end_of_sentence:
end_idx = min(len(decomposed_text) - 1, max(attention_argmax))
else:
surplus_decomposed_text = decompose_ko_text("".join(text.split()[-1]))
end_idx = len(decomposed_text.replace(surplus_decomposed_text, '')) - 1
while True:
if end_idx in attention_argmax:
break
end_idx -= 1
end_idx_counter = 0
for idx, attend_idx in enumerate(attention_argmax):
if len(attention_argmax) > idx + 1:
if attend_idx == end_idx:
end_idx_counter += 1
if attend_idx == end_idx and attention_argmax[idx + 1] > end_idx:
break
if end_idx_counter > 5:
break
else:
break
wav_end_idx = hparams.reduction_factor * idx + 1 + post_surplus_idx
else:
wav_end_idx = None
wav = wav[wav_start_idx:wav_end_idx]
if end_of_sentence:
wav = np.lib.pad(wav, ((0, 20), (0, 0)), 'constant', constant_values=0)
else:
wav = np.lib.pad(wav, ((0, 10), (0, 0)), 'constant', constant_values=0)
if __name__ == "__main__":
parser = argparse.ArgumentParser()
parser.add_argument('--load_path', required=True)
parser.add_argument('--sample_path', default="samples")
parser.add_argument('--text', required=True)
parser.add_argument('--num_speakers', default=1, type=int)
parser.add_argument('--speaker_id', default=0, type=int)
parser.add_argument('--checkpoint_step', default=None, type=int)
config = parser.parse_args()
makedirs(config.sample_path)
synthesizer = Synthesizer()
synthesizer.load(config.load_path, config.num_speakers, config.checkpoint_step)
audio = synthesizer.synthesize(
texts=[config.text],
base_path=config.sample_path,
speaker_ids=[config.speaker_id],
attention_trim=False)[0]

101
text/__init__.py Normal file
View file

@ -0,0 +1,101 @@
import re
import string
import numpy as np
from text import cleaners
from hparams import hparams
from text.symbols import symbols, PAD, EOS
from text.korean import jamo_to_korean
# Mappings from symbol to numeric ID and vice versa:
_symbol_to_id = {s: i for i, s in enumerate(symbols)}
_id_to_symbol = {i: s for i, s in enumerate(symbols)}
# Regular expression matching text enclosed in curly braces:
_curly_re = re.compile(r'(.*?)\{(.+?)\}(.*)')
puncuation_table = str.maketrans({key: None for key in string.punctuation})
def remove_puncuations(text):
return text.translate(puncuation_table)
def text_to_sequence(text, as_token=False):
cleaner_names = [x.strip() for x in hparams.cleaners.split(',')]
return _text_to_sequence(text, cleaner_names, as_token)
def _text_to_sequence(text, cleaner_names, as_token):
'''Converts a string of text to a sequence of IDs corresponding to the symbols in the text.
The text can optionally have ARPAbet sequences enclosed in curly braces embedded
in it. For example, "Turn left on {HH AW1 S S T AH0 N} Street."
Args:
text: string to convert to a sequence
cleaner_names: names of the cleaner functions to run the text through
Returns:
List of integers corresponding to the symbols in the text
'''
sequence = []
# Check for curly braces and treat their contents as ARPAbet:
while len(text):
m = _curly_re.match(text)
if not m:
sequence += _symbols_to_sequence(_clean_text(text, cleaner_names))
break
sequence += _symbols_to_sequence(_clean_text(m.group(1), cleaner_names))
sequence += _arpabet_to_sequence(m.group(2))
text = m.group(3)
# Append EOS token
sequence.append(_symbol_to_id[EOS])
if as_token:
return sequence_to_text(sequence, combine_jamo=True)
else:
return np.array(sequence, dtype=np.int32)
def sequence_to_text(sequence, skip_eos_and_pad=False, combine_jamo=False):
'''Converts a sequence of IDs back to a string'''
result = ''
for symbol_id in sequence:
if symbol_id in _id_to_symbol:
s = _id_to_symbol[symbol_id]
# Enclose ARPAbet back in curly braces:
if len(s) > 1 and s[0] == '@':
s = '{%s}' % s[1:]
if not skip_eos_and_pad or s not in [EOS, PAD]:
result += s
result = result.replace('}{', ' ')
if combine_jamo:
return jamo_to_korean(result)
else:
return result
def _clean_text(text, cleaner_names):
for name in cleaner_names:
cleaner = getattr(cleaners, name)
if not cleaner:
raise Exception('Unknown cleaner: %s' % name)
text = cleaner(text)
return text
def _symbols_to_sequence(symbols):
return [_symbol_to_id[s] for s in symbols if _should_keep_symbol(s)]
def _arpabet_to_sequence(text):
return _symbols_to_sequence(['@' + s for s in text.split()])
def _should_keep_symbol(s):
return s in _symbol_to_id and s is not '_' and s is not '~'

80
text/cleaners.py Normal file
View file

@ -0,0 +1,80 @@
'''
Cleaners are transformations that run over the input text at both training and eval time.
Cleaners can be selected by passing a comma-delimited list of cleaner names as the "cleaners"
hyperparameter. Some cleaners are English-specific. You'll typically want to use:
1. "english_cleaners" for English text
2. "transliteration_cleaners" for non-English text that can be transliterated to ASCII using
the Unidecode library (https://pypi.python.org/pypi/Unidecode)
3. "basic_cleaners" if you do not want to transliterate (in this case, you should also update
the symbols in symbols.py to match your data).
'''
import re
from .korean import tokenize as ko_tokenize
# Regular expression matching whitespace:
_whitespace_re = re.compile(r'\s+')
def korean_cleaners(text):
'''Pipeline for Korean text, including number and abbreviation expansion.'''
text = ko_tokenize(text)
return text
# List of (regular expression, replacement) pairs for abbreviations:
_abbreviations = [(re.compile('\\b%s\\.' % x[0], re.IGNORECASE), x[1]) for x in [
('mrs', 'misess'),
('mr', 'mister'),
('dr', 'doctor'),
('st', 'saint'),
('co', 'company'),
('jr', 'junior'),
('maj', 'major'),
('gen', 'general'),
('drs', 'doctors'),
('rev', 'reverend'),
('lt', 'lieutenant'),
('hon', 'honorable'),
('sgt', 'sergeant'),
('capt', 'captain'),
('esq', 'esquire'),
('ltd', 'limited'),
('col', 'colonel'),
('ft', 'fort'),
]]
def expand_abbreviations(text):
for regex, replacement in _abbreviations:
text = re.sub(regex, replacement, text)
return text
def expand_numbers(text):
return normalize_numbers(text)
def lowercase(text):
return text.lower()
def collapse_whitespace(text):
return re.sub(_whitespace_re, ' ', text)
def basic_cleaners(text):
'''Basic pipeline that lowercases and collapses whitespace without transliteration.'''
text = lowercase(text)
text = collapse_whitespace(text)
return text
def transliteration_cleaners(text):
'''Pipeline for non-English text that transliterates to ASCII.'''
text = convert_to_ascii(text)
text = lowercase(text)
text = collapse_whitespace(text)
return text

69
text/en_numbers.py Normal file
View file

@ -0,0 +1,69 @@
import inflect
import re
_inflect = inflect.engine()
_comma_number_re = re.compile(r'([0-9][0-9\,]+[0-9])')
_decimal_number_re = re.compile(r'([0-9]+\.[0-9]+)')
_pounds_re = re.compile(r'£([0-9\,]*[0-9]+)')
_dollars_re = re.compile(r'\$([0-9\.\,]*[0-9]+)')
_ordinal_re = re.compile(r'[0-9]+(st|nd|rd|th)')
_number_re = re.compile(r'[0-9]+')
def _remove_commas(m):
return m.group(1).replace(',', '')
def _expand_decimal_point(m):
return m.group(1).replace('.', ' point ')
def _expand_dollars(m):
match = m.group(1)
parts = match.split('.')
if len(parts) > 2:
return match + ' dollars' # Unexpected format
dollars = int(parts[0]) if parts[0] else 0
cents = int(parts[1]) if len(parts) > 1 and parts[1] else 0
if dollars and cents:
dollar_unit = 'dollar' if dollars == 1 else 'dollars'
cent_unit = 'cent' if cents == 1 else 'cents'
return '%s %s, %s %s' % (dollars, dollar_unit, cents, cent_unit)
elif dollars:
dollar_unit = 'dollar' if dollars == 1 else 'dollars'
return '%s %s' % (dollars, dollar_unit)
elif cents:
cent_unit = 'cent' if cents == 1 else 'cents'
return '%s %s' % (cents, cent_unit)
else:
return 'zero dollars'
def _expand_ordinal(m):
return _inflect.number_to_words(m.group(0))
def _expand_number(m):
num = int(m.group(0))
if num > 1000 and num < 3000:
if num == 2000:
return 'two thousand'
elif num > 2000 and num < 2010:
return 'two thousand ' + _inflect.number_to_words(num % 100)
elif num % 100 == 0:
return _inflect.number_to_words(num // 100) + ' hundred'
else:
return _inflect.number_to_words(num, andword='', zero='oh', group=2).replace(', ', ' ')
else:
return _inflect.number_to_words(num, andword='')
def normalize_numbers(text):
text = re.sub(_comma_number_re, _remove_commas, text)
text = re.sub(_pounds_re, r'\1 pounds', text)
text = re.sub(_dollars_re, _expand_dollars, text)
text = re.sub(_decimal_number_re, _expand_decimal_point, text)
text = re.sub(_ordinal_re, _expand_ordinal, text)
text = re.sub(_number_re, _expand_number, text)
return text

69
text/english.py Normal file
View file

@ -0,0 +1,69 @@
# Code from https://github.com/keithito/tacotron/blob/master/util/numbers.py
import inflect
_inflect = inflect.engine()
_comma_number_re = re.compile(r'([0-9][0-9\,]+[0-9])')
_decimal_number_re = re.compile(r'([0-9]+\.[0-9]+)')
_pounds_re = re.compile(r'£([0-9\,]*[0-9]+)')
_dollars_re = re.compile(r'\$([0-9\.\,]*[0-9]+)')
_ordinal_re = re.compile(r'[0-9]+(st|nd|rd|th)')
_number_re = re.compile(r'[0-9]+')
def _remove_commas(m):
return m.group(1).replace(',', '')
def _expand_decimal_point(m):
return m.group(1).replace('.', ' point ')
def _expand_dollars(m):
match = m.group(1)
parts = match.split('.')
if len(parts) > 2:
return match + ' dollars' # Unexpected format
dollars = int(parts[0]) if parts[0] else 0
cents = int(parts[1]) if len(parts) > 1 and parts[1] else 0
if dollars and cents:
dollar_unit = 'dollar' if dollars == 1 else 'dollars'
cent_unit = 'cent' if cents == 1 else 'cents'
return '%s %s, %s %s' % (dollars, dollar_unit, cents, cent_unit)
elif dollars:
dollar_unit = 'dollar' if dollars == 1 else 'dollars'
return '%s %s' % (dollars, dollar_unit)
elif cents:
cent_unit = 'cent' if cents == 1 else 'cents'
return '%s %s' % (cents, cent_unit)
else:
return 'zero dollars'
def _expand_ordinal(m):
return _inflect.number_to_words(m.group(0))
def _expand_number(m):
num = int(m.group(0))
if num > 1000 and num < 3000:
if num == 2000:
return 'two thousand'
elif num > 2000 and num < 2010:
return 'two thousand ' + _inflect.number_to_words(num % 100)
elif num % 100 == 0:
return _inflect.number_to_words(num // 100) + ' hundred'
else:
return _inflect.number_to_words(num, andword='', zero='oh', group=2).replace(', ', ' ')
else:
return _inflect.number_to_words(num, andword='')
def normalize(text):
text = re.sub(_comma_number_re, _remove_commas, text)
text = re.sub(_pounds_re, r'\1 pounds', text)
text = re.sub(_dollars_re, _expand_dollars, text)
text = re.sub(_decimal_number_re, _expand_decimal_point, text)
text = re.sub(_ordinal_re, _expand_ordinal, text)
text = re.sub(_number_re, _expand_number, text)
return text

172
text/ko_dictionary.py Normal file
View file

@ -0,0 +1,172 @@
etc_dictionary = {
'2 30대': '이삼십대',
'20~30대': '이삼십대',
'20, 30대': '이십대 삼십대',
'1+1': '원플러스원',
'3에서 6개월인': '3개월에서 육개월인',
}
english_dictionary = {
'Devsisters': '데브시스터즈',
'track': '트랙',
# krbook
'LA': '엘에이',
'LG': '엘지',
'KOREA': '코리아',
'JSA': '제이에스에이',
'PGA': '피지에이',
'GA': '지에이',
'idol': '아이돌',
'KTX': '케이티엑스',
'AC': '에이씨',
'DVD': '디비디',
'US': '유에스',
'CNN': '씨엔엔',
'LPGA': '엘피지에이',
'P': '피',
'L': '엘',
'T': '티',
'B': '비',
'C': '씨',
'BIFF': '비아이에프에프',
'GV': '지비',
# JTBC
'IT': '아이티',
'IQ': '아이큐',
'JTBC': '제이티비씨',
'trickle down effect': '트리클 다운 이펙트',
'trickle up effect': '트리클 업 이펙트',
'down': '다운',
'up': '업',
'FCK': '에프씨케이',
'AP': '에이피',
'WHERETHEWILDTHINGSARE': '',
'Rashomon Effect': '',
'O': '오',
'OO': '오오',
'B': '비',
'GDP': '지디피',
'CIPA': '씨아이피에이',
'YS': '와이에스',
'Y': '와이',
'S': '에스',
'JTBC': '제이티비씨',
'PC': '피씨',
'bill': '빌',
'Halmuny': '하모니', #####
'X': '엑스',
'SNS': '에스엔에스',
'ability': '어빌리티',
'shy': '',
'CCTV': '씨씨티비',
'IT': '아이티',
'the tenth man': '더 텐쓰 맨', ####
'L': '엘',
'PC': '피씨',
'YSDJJPMB': '', ########
'Content Attitude Timing': '컨텐트 애티튜드 타이밍',
'CAT': '캣',
'IS': '아이에스',
'SNS': '에스엔에스',
'K': '케이',
'Y': '와이',
'KDI': '케이디아이',
'DOC': '디오씨',
'CIA': '씨아이에이',
'PBS': '피비에스',
'D': '디',
'PPropertyPositionPowerPrisonP'
'S': '에스',
'francisco': '프란시스코',
'I': '아이',
'III': '아이아이', ######
'No joke': '노 조크',
'BBK': '비비케이',
'LA': '엘에이',
'Don': '',
't worry be happy': ' 워리 비 해피',
'NO': '엔오', #####
'it was our sky': '잇 워즈 아워 스카이',
'it is our sky': '잇 이즈 아워 스카이', ####
'NEIS': '엔이아이에스', #####
'IMF': '아이엠에프',
'apology': '어폴로지',
'humble': '험블',
'M': '엠',
'Nowhere Man': '노웨어 맨',
'The Tenth Man': '더 텐쓰 맨',
'PBS': '피비에스',
'BBC': '비비씨',
'MRJ': '엠알제이',
'CCTV': '씨씨티비',
'Pick me up': '픽 미 업',
'DNA': '디엔에이',
'UN': '유엔',
'STOP': '스탑', #####
'PRESS': '프레스', #####
'not to be': '낫 투비',
'Denial': '디나이얼',
'G': '지',
'IMF': '아이엠에프',
'GDP': '지디피',
'JTBC': '제이티비씨',
'Time flies like an arrow': '타임 플라이즈 라이크 언 애로우',
'DDT': '디디티',
'AI': '에이아이',
'Z': '제트',
'OECD': '오이씨디',
'N': '앤',
'A': '에이',
'MB': '엠비',
'EH': '이에이치',
'IS': '아이에스',
'TV': '티비',
'MIT': '엠아이티',
'KBO': '케이비오',
'I love America': '아이 러브 아메리카',
'SF': '에스에프',
'Q': '큐',
'KFX': '케이에프엑스',
'PM': '피엠',
'Prime Minister': '프라임 미니스터',
'Swordline': '스워드라인',
'TBS': '티비에스',
'DDT': '디디티',
'CS': '씨에스',
'Reflecting Absence': '리플렉팅 앱센스',
'PBS': '피비에스',
'Drum being beaten by everyone': '드럼 빙 비튼 바이 에브리원',
'negative pressure': '네거티브 프레셔',
'F': '에프',
'KIA': '기아',
'FTA': '에프티에이',
'Que sais-je': '',
'UFC': '유에프씨',
'P': '피',
'DJ': '디제이',
'Chaebol': '채벌',
'BBC': '비비씨',
'OECD': '오이씨디',
'BC': '삐씨',
'C': '씨',
'B': '씨',
'KY': '케이와이',
'K': '케이',
'CEO': '씨이오',
'YH': '와이에치',
'IS': '아이에스',
'who are you': '후 얼 유',
'Y': '와이',
'The Devils Advocate': '더 데빌즈 어드보카트',
'YS': '와이에스',
'so sorry': '쏘 쏘리',
'Santa': '산타',
'Big Endian': '빅 엔디안',
'Small Endian': '스몰 엔디안',
'Oh Captain My Captain': '오 캡틴 마이 캡틴',
'AIB': '에이아이비',
'K': '케이',
'PBS': '피비에스',
}

319
text/korean.py Normal file
View file

@ -0,0 +1,319 @@
# Code based on
import re
import os
import ast
import json
from jamo import hangul_to_jamo, h2j, j2h
from .ko_dictionary import english_dictionary, etc_dictionary
PAD = '_'
EOS = '~'
PUNC = '!\'(),-.:;?'
SPACE = ' '
JAMO_LEADS = "".join([chr(_) for _ in range(0x1100, 0x1113)])
JAMO_VOWELS = "".join([chr(_) for _ in range(0x1161, 0x1176)])
JAMO_TAILS = "".join([chr(_) for _ in range(0x11A8, 0x11C3)])
VALID_CHARS = JAMO_LEADS + JAMO_VOWELS + JAMO_TAILS + PUNC + SPACE
ALL_SYMBOLS = PAD + EOS + VALID_CHARS
char_to_id = {c: i for i, c in enumerate(ALL_SYMBOLS)}
id_to_char = {i: c for i, c in enumerate(ALL_SYMBOLS)}
quote_checker = """([`"'＂“‘])(.+?)([`"'＂”’])"""
def is_lead(char):
return char in JAMO_LEADS
def is_vowel(char):
return char in JAMO_VOWELS
def is_tail(char):
return char in JAMO_TAILS
def get_mode(char):
if is_lead(char):
return 0
elif is_vowel(char):
return 1
elif is_tail(char):
return 2
else:
return -1
def _get_text_from_candidates(candidates):
if len(candidates) == 0:
return ""
elif len(candidates) == 1:
return _jamo_char_to_hcj(candidates[0])
else:
return j2h(**dict(zip(["lead", "vowel", "tail"], candidates)))
def jamo_to_korean(text):
text = h2j(text)
idx = 0
new_text = ""
candidates = []
while True:
if idx >= len(text):
new_text += _get_text_from_candidates(candidates)
break
char = text[idx]
mode = get_mode(char)
if mode == 0:
new_text += _get_text_from_candidates(candidates)
candidates = [char]
elif mode == -1:
new_text += _get_text_from_candidates(candidates)
new_text += char
candidates = []
else:
candidates.append(char)
idx += 1
return new_text
num_to_kor = {
'0': '영',
'1': '일',
'2': '이',
'3': '삼',
'4': '사',
'5': '오',
'6': '육',
'7': '칠',
'8': '팔',
'9': '구',
}
unit_to_kor1 = {
'%': '퍼센트',
'cm': '센치미터',
'mm': '밀리미터',
'km': '킬로미터',
'kg': '킬로그람',
}
unit_to_kor2 = {
'm': '미터',
}
upper_to_kor = {
'A': '에이',
'B': '비',
'C': '씨',
'D': '디',
'E': '이',
'F': '에프',
'G': '지',
'H': '에이치',
'I': '아이',
'J': '제이',
'K': '케이',
'L': '엘',
'M': '엠',
'N': '엔',
'O': '오',
'P': '피',
'Q': '큐',
'R': '알',
'S': '에스',
'T': '티',
'U': '유',
'V': '브이',
'W': '더블유',
'X': '엑스',
'Y': '와이',
'Z': '지',
}
def compare_sentence_with_jamo(text1, text2):
return h2j(text1) != h2j(text)
def tokenize(text, as_id=False):
text = normalize(text)
tokens = list(hangul_to_jamo(text))
if as_id:
return [char_to_id[token] for token in tokens] + [char_to_id[EOS]]
else:
return [token for token in tokens] + [EOS]
def tokenizer_fn(iterator):
return (token for x in iterator for token in tokenize(x, as_id=False))
def normalize(text):
text = text.strip()
text = re.sub('\(\d+일\)', '', text)
text = re.sub('\([⺀-⺙⺛-⻳⼀-⿕々〇〡-〩〸-〺〻㐀-䶵一-鿃豈-鶴侮-頻並-龎]+\)', '', text)
text = normalize_with_dictionary(text, etc_dictionary)
text = normalize_english(text)
text = re.sub('[a-zA-Z]+', normalize_upper, text)
text = normalize_quote(text)
text = normalize_number(text)
return text
def normalize_with_dictionary(text, dic):
if any(key in text for key in dic.keys()):
pattern = re.compile('|'.join(re.escape(key) for key in dic.keys()))
return pattern.sub(lambda x: dic[x.group()], text)
else:
return text
def normalize_english(text):
def fn(m):
word = m.group()
if word in english_dictionary:
return english_dictionary.get(word)
else:
return word
text = re.sub("([A-Za-z]+)", fn, text)
return text
def normalize_upper(text):
text = text.group(0)
if all([char.isupper() for char in text]):
return "".join(upper_to_kor[char] for char in text)
else:
return text
def normalize_quote(text):
def fn(found_text):
from nltk import sent_tokenize # NLTK doesn't along with multiprocessing
found_text = found_text.group()
unquoted_text = found_text[1:-1]
sentences = sent_tokenize(unquoted_text)
return " ".join(["'{}'".format(sent) for sent in sentences])
return re.sub(quote_checker, fn, text)
number_checker = "([+-]?\d[\d,]*)[\.]?\d*"
count_checker = "(시|명|가지|살|마리|포기|송이|수|톨|통|점|개|벌|척|채|다발|그루|자루|줄|켤레|그릇|잔|마디|상자|사람|곡|병|판)"
def normalize_number(text):
text = normalize_with_dictionary(text, unit_to_kor1)
text = normalize_with_dictionary(text, unit_to_kor2)
text = re.sub(number_checker + count_checker,
lambda x: number_to_korean(x, True), text)
text = re.sub(number_checker,
lambda x: number_to_korean(x, False), text)
return text
num_to_kor1 = [""] + list("일이삼사오육칠팔구")
num_to_kor2 = [""] + list("만억조경해")
num_to_kor3 = [""] + list("십백천")
#count_to_kor1 = [""] + ["하나","둘","셋","넷","다섯","여섯","일곱","여덟","아홉"]
count_to_kor1 = [""] + ["한","두","세","네","다섯","여섯","일곱","여덟","아홉"]
count_tenth_dict = {
"십": "열",
"두십": "스물",
"세십": "서른",
"네십": "마흔",
"다섯십": "쉰",
"여섯십": "예순",
"일곱십": "일흔",
"여덟십": "여든",
"아홉십": "아흔",
}
def number_to_korean(num_str, is_count=False):
if is_count:
num_str, unit_str = num_str.group(1), num_str.group(2)
else:
num_str, unit_str = num_str.group(), ""
num_str = num_str.replace(',', '')
num = ast.literal_eval(num_str)
if num == 0:
return "영"
check_float = num_str.split('.')
if len(check_float) == 2:
digit_str, float_str = check_float
elif len(check_float) >= 3:
raise Exception(" [!] Wrong number format")
else:
digit_str, float_str = check_float[0], None
if is_count and float_str is not None:
raise Exception(" [!] `is_count` and float number does not fit each other")
digit = int(digit_str)
if digit_str.startswith("-"):
digit, digit_str = abs(digit), str(abs(digit))
kor = ""
size = len(str(digit))
tmp = []
for i, v in enumerate(digit_str, start=1):
v = int(v)
if v != 0:
if is_count:
tmp += count_to_kor1[v]
else:
tmp += num_to_kor1[v]
tmp += num_to_kor3[(size - i) % 4]
if (size - i) % 4 == 0 and len(tmp) != 0:
kor += "".join(tmp)
tmp = []
kor += num_to_kor2[int((size - i) / 4)]
if is_count:
if kor.startswith("한") and len(kor) > 1:
kor = kor[1:]
if any(word in kor for word in count_tenth_dict):
kor = re.sub(
'|'.join(count_tenth_dict.keys()),
lambda x: count_tenth_dict[x.group()], kor)
if not is_count and kor.startswith("일") and len(kor) > 1:
kor = kor[1:]
if float_str is not None:
kor += "쩜 "
kor += re.sub('\d', lambda x: num_to_kor[x.group()], float_str)
if num_str.startswith("+"):
kor = "플러스 " + kor
elif num_str.startswith("-"):
kor = "마이너스 " + kor
return kor + unit_str
if __name__ == "__main__":
def test_normalize(text):
print(text)
print(normalize(text))
print("="*30)
test_normalize("JTBC는 JTBCs를 DY는 A가 Absolute")
test_normalize("오늘(13일) 101마리 강아지가")
test_normalize('"저돌"(猪突) 입니다.')
test_normalize('비대위원장이 지난 1월 이런 말을 했습니다. “난 그냥 산돼지처럼 돌파하는 스타일이다”')
test_normalize("지금은 -12.35%였고 종류는 5가지와 19가지, 그리고 55가지였다")
test_normalize("JTBC는 TH와 K 양이 2017년 9월 12일 오후 12시에 24살이 된다")

13
text/symbols.py Normal file
View file

@ -0,0 +1,13 @@
'''
Defines the set of symbols used in text input to the model.
The default is a set of ASCII characters that works well for English or text that has been run
through Unidecode. For other data, you can modify _characters. See TRAINING_DATA.md for details.
'''
from jamo import h2j, j2h
from jamo.jamo import _jamo_char_to_hcj
from .korean import ALL_SYMBOLS, PAD, EOS
#symbols = 'ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz!\'(),-.:;? '
symbols = ALL_SYMBOLS

322
train.py Normal file
View file

@ -0,0 +1,322 @@
import os
import time
import math
import argparse
import traceback
import subprocess
import numpy as np
from jamo import h2j
import tensorflow as tf
from datetime import datetime
from functools import partial
from hparams import hparams, hparams_debug_string
from models import create_model, get_most_recent_checkpoint
from utils import ValueWindow, prepare_dirs
from utils import infolog, warning, plot, load_hparams
from utils import get_git_revision_hash, get_git_diff, str2bool, parallel_run
from audio import save_audio, inv_spectrogram
from text import sequence_to_text, text_to_sequence
from datasets.datafeeder import DataFeeder, _prepare_inputs
log = infolog.log
def create_batch_inputs_from_texts(texts):
sequences = [text_to_sequence(text) for text in texts]
inputs = _prepare_inputs(sequences)
input_lengths = np.asarray([len(x) for x in inputs], dtype=np.int32)
for idx, (seq, text) in enumerate(zip(inputs, texts)):
recovered_text = sequence_to_text(seq, skip_eos_and_pad=True)
if recovered_text != h2j(text):
log(" [{}] {}".format(idx, text))
log(" [{}] {}".format(idx, recovered_text))
log("="*30)
return inputs, input_lengths
def get_git_commit():
subprocess.check_output(['git', 'diff-index', '--quiet', 'HEAD']) # Verify client is clean
commit = subprocess.check_output(['git', 'rev-parse', 'HEAD']).decode().strip()[:10]
log('Git commit: %s' % commit)
return commit
def add_stats(model, model2=None, scope_name='train'):
with tf.variable_scope(scope_name) as scope:
summaries = [
tf.summary.scalar('loss_mel', model.mel_loss),
tf.summary.scalar('loss_linear', model.linear_loss),
tf.summary.scalar('loss', model.loss_without_coeff),
]
if scope_name == 'train':
gradient_norms = [tf.norm(grad) for grad in model.gradients if grad is not None]
summaries.extend([
tf.summary.scalar('learning_rate', model.learning_rate),
tf.summary.scalar('max_gradient_norm', tf.reduce_max(gradient_norms)),
])
if model2 is not None:
with tf.variable_scope('gap_test-train') as scope:
summaries.extend([
tf.summary.scalar('loss_mel',
model.mel_loss - model2.mel_loss),
tf.summary.scalar('loss_linear',
model.linear_loss - model2.linear_loss),
tf.summary.scalar('loss',
model.loss_without_coeff - model2.loss_without_coeff),
])
return tf.summary.merge(summaries)
def save_and_plot_fn(args, log_dir, step, loss, prefix):
idx, (seq, spec, align) = args
audio_path = os.path.join(
log_dir, '{}-step-{:09d}-audio{:03d}.wav'.format(prefix, step, idx))
align_path = os.path.join(
log_dir, '{}-step-{:09d}-align{:03d}.png'.format(prefix, step, idx))
waveform = inv_spectrogram(spec.T)
save_audio(waveform, audio_path)
info_text = 'step={:d}, loss={:.5f}'.format(step, loss)
plot.plot_alignment(
align, align_path, info=info_text,
text=sequence_to_text(seq,
skip_eos_and_pad=True, combine_jamo=True))
def save_and_plot(sequences, spectrograms,
alignments, log_dir, step, loss, prefix):
fn = partial(save_and_plot_fn,
log_dir=log_dir, step=step, loss=loss, prefix=prefix)
items = list(enumerate(zip(sequences, spectrograms, alignments)))
parallel_run(fn, items, parallel=False)
log('Test finished for step {}.'.format(step))
def train(log_dir, config):
config.data_paths = config.data_paths
data_dirs = [os.path.join(data_path, "data") \
for data_path in config.data_paths]
num_speakers = len(data_dirs)
config.num_test = config.num_test_per_speaker * num_speakers
if num_speakers > 1 and hparams.model_type not in ["deepvoice", "simple"]:
raise Exception("[!] Unkown model_type for multi-speaker: {}".format(config.model_type))
commit = get_git_commit() if config.git else 'None'
checkpoint_path = os.path.join(log_dir, 'model.ckpt')
log(' [*] git recv-parse HEAD:\n%s' % get_git_revision_hash())
log('='*50)
log(' [*] dit diff:\n%s' % get_git_diff())
log('='*50)
log(' [*] Checkpoint path: %s' % checkpoint_path)
log(' [*] Loading training data from: %s' % data_dirs)
log(' [*] Using model: %s' % config.model_dir)
log(hparams_debug_string())
# Set up DataFeeder:
coord = tf.train.Coordinator()
with tf.variable_scope('datafeeder') as scope:
train_feeder = DataFeeder(
coord, data_dirs, hparams, config, 32,
data_type='train', batch_size=hparams.batch_size)
test_feeder = DataFeeder(
coord, data_dirs, hparams, config, 8,
data_type='test', batch_size=config.num_test)
# Set up model:
is_randomly_initialized = config.initialize_path is None
global_step = tf.Variable(0, name='global_step', trainable=False)
with tf.variable_scope('model') as scope:
model = create_model(hparams)
model.initialize(
train_feeder.inputs, train_feeder.input_lengths,
num_speakers, train_feeder.speaker_id,
train_feeder.mel_targets, train_feeder.linear_targets,
train_feeder.loss_coeff,
is_randomly_initialized=is_randomly_initialized)
model.add_loss()
model.add_optimizer(global_step)
train_stats = add_stats(model, scope_name='stats') # legacy
with tf.variable_scope('model', reuse=True) as scope:
test_model = create_model(hparams)
test_model.initialize(
test_feeder.inputs, test_feeder.input_lengths,
num_speakers, test_feeder.speaker_id,
test_feeder.mel_targets, test_feeder.linear_targets,
test_feeder.loss_coeff, rnn_decoder_test_mode=True,
is_randomly_initialized=is_randomly_initialized)
test_model.add_loss()
test_stats = add_stats(test_model, model, scope_name='test')
test_stats = tf.summary.merge([test_stats, train_stats])
# Bookkeeping:
step = 0
time_window = ValueWindow(100)
loss_window = ValueWindow(100)
saver = tf.train.Saver(max_to_keep=5, keep_checkpoint_every_n_hours=2)
sess_config = tf.ConfigProto(
log_device_placement=False,
allow_soft_placement=True)
sess_config.gpu_options.allow_growth=True
# Train!
#with tf.Session(config=sess_config) as sess:
with tf.Session() as sess:
try:
summary_writer = tf.summary.FileWriter(log_dir, sess.graph)
sess.run(tf.global_variables_initializer())
if config.load_path:
# Restore from a checkpoint if the user requested it.
restore_path = get_most_recent_checkpoint(config.model_dir)
saver.restore(sess, restore_path)
log('Resuming from checkpoint: %s at commit: %s' % (restore_path, commit), slack=True)
elif config.initialize_path:
restore_path = get_most_recent_checkpoint(config.initialize_path)
saver.restore(sess, restore_path)
log('Initialized from checkpoint: %s at commit: %s' % (restore_path, commit), slack=True)
zero_step_assign = tf.assign(global_step, 0)
sess.run(zero_step_assign)
start_step = sess.run(global_step)
log('='*50)
log(' [*] Global step is reset to {}'. \
format(start_step))
log('='*50)
else:
log('Starting new training run at commit: %s' % commit, slack=True)
start_step = sess.run(global_step)
train_feeder.start_in_session(sess, start_step)
test_feeder.start_in_session(sess, start_step)
while not coord.should_stop():
start_time = time.time()
step, loss, opt = sess.run(
[global_step, model.loss_without_coeff, model.optimize],
feed_dict=model.get_dummy_feed_dict())
time_window.append(time.time() - start_time)
loss_window.append(loss)
message = 'Step %-7d [%.03f sec/step, loss=%.05f, avg_loss=%.05f]' % (
step, time_window.average, loss, loss_window.average)
log(message, slack=(step % config.checkpoint_interval == 0))
if loss > 100 or math.isnan(loss):
log('Loss exploded to %.05f at step %d!' % (loss, step), slack=True)
raise Exception('Loss Exploded')
if step % config.summary_interval == 0:
log('Writing summary at step: %d' % step)
feed_dict = {
**model.get_dummy_feed_dict(),
**test_model.get_dummy_feed_dict()
}
summary_writer.add_summary(sess.run(
test_stats, feed_dict=feed_dict), step)
if step % config.checkpoint_interval == 0:
log('Saving checkpoint to: %s-%d' % (checkpoint_path, step))
saver.save(sess, checkpoint_path, global_step=step)
if step % config.test_interval == 0:
log('Saving audio and alignment...')
num_test = config.num_test
fetches = [
model.inputs[:num_test],
model.linear_outputs[:num_test],
model.alignments[:num_test],
test_model.inputs[:num_test],
test_model.linear_outputs[:num_test],
test_model.alignments[:num_test],
]
feed_dict = {
**model.get_dummy_feed_dict(),
**test_model.get_dummy_feed_dict()
}
sequences, spectrograms, alignments, \
test_sequences, test_spectrograms, test_alignments = \
sess.run(fetches, feed_dict=feed_dict)
save_and_plot(sequences[:1], spectrograms[:1], alignments[:1],
log_dir, step, loss, "train")
save_and_plot(test_sequences, test_spectrograms, test_alignments,
log_dir, step, loss, "test")
except Exception as e:
log('Exiting due to exception: %s' % e, slack=True)
traceback.print_exc()
coord.request_stop(e)
def main():
parser = argparse.ArgumentParser()
parser.add_argument('--log_dir', default='logs')
parser.add_argument('--data_paths', default='datasets/kr_example')
parser.add_argument('--load_path', default=None)
parser.add_argument('--initialize_path', default=None)
parser.add_argument('--num_test_per_speaker', type=int, default=2)
parser.add_argument('--random_seed', type=int, default=123)
parser.add_argument('--summary_interval', type=int, default=100)
parser.add_argument('--test_interval', type=int, default=500)
parser.add_argument('--checkpoint_interval', type=int, default=1000)
parser.add_argument('--skip_path_filter',
type=str2bool, default=False, help='Use only for debugging')
parser.add_argument('--slack_url',
help='Slack webhook URL to get periodic reports.')
parser.add_argument('--git', action='store_true',
help='If set, verify that the client is clean.')
config = parser.parse_args()
config.data_paths = config.data_paths.split(",")
setattr(hparams, "num_speakers", len(config.data_paths))
prepare_dirs(config, hparams)
log_path = os.path.join(config.model_dir, 'train.log')
infolog.init(log_path, config.model_dir, config.slack_url)
tf.set_random_seed(config.random_seed)
if any("krbook" not in data_path for data_path in config.data_paths) and \
hparams.sample_rate != 20000:
warning("Detect non-krbook dataset. Set sampling rate from {} to 20000".\
format(hparams.sample_rate))
if config.load_path is not None and config.initialize_path is not None:
raise Exception(" [!] Only one of load_path and initialize_path should be set")
train(config.model_dir, config)
if __name__ == '__main__':
main()

BIN
utils/NanumBarunGothic.ttf Normal file
View file

Binary file not shown.

223
utils/__init__.py Normal file
View file

@ -0,0 +1,223 @@
import os
import re
import sys
import json
import requests
import subprocess
from tqdm import tqdm
from contextlib import closing
from multiprocessing import Pool
from collections import namedtuple
from datetime import datetime, timedelta
from shutil import copyfile as copy_file
PARAMS_NAME = "params.json"
class ValueWindow():
def __init__(self, window_size=100):
self._window_size = window_size
self._values = []
def append(self, x):
self._values = self._values[-(self._window_size - 1):] + [x]
@property
def sum(self):
return sum(self._values)
@property
def count(self):
return len(self._values)
@property
def average(self):
return self.sum / max(1, self.count)
def reset(self):
self._values = []
def prepare_dirs(config, hparams):
if hasattr(config, "data_paths"):
config.datasets = [
os.path.basename(data_path) for data_path in config.data_paths]
dataset_desc = "+".join(config.datasets)
if config.load_path:
config.model_dir = config.load_path
else:
config.model_name = "{}_{}".format(dataset_desc, get_time())
config.model_dir = os.path.join(config.log_dir, config.model_name)
for path in [config.log_dir, config.model_dir]:
if not os.path.exists(path):
os.makedirs(path)
if config.load_path:
load_hparams(hparams, config.model_dir)
else:
setattr(hparams, "num_speakers", len(config.datasets))
save_hparams(config.model_dir, hparams)
copy_file("hparams.py", os.path.join(config.model_dir, "hparams.py"))
def makedirs(path):
if not os.path.exists(path):
print(" [*] Make directories : {}".format(path))
os.makedirs(path)
def remove_file(path):
if os.path.exists(path):
print(" [*] Removed: {}".format(path))
os.remove(path)
def backup_file(path):
root, ext = os.path.splitext(path)
new_path = "{}.backup_{}{}".format(root, get_time(), ext)
os.rename(path, new_path)
print(" [*] {} has backup: {}".format(path, new_path))
def get_time():
return datetime.now().strftime("%Y-%m-%d_%H-%M-%S")
def write_json(path, data):
with open(path, 'w') as f:
json.dump(data, f, indent=4, sort_keys=True, ensure_ascii=False)
def load_json(path, as_class=False):
with open(path) as f:
content = f.read()
content = re.sub(",\s*}", "}", content)
content = re.sub(",\s*]", "]", content)
if as_class:
data = json.loads(content, object_hook=\
lambda data: namedtuple('Data', data.keys())(*data.values()))
else:
data = json.loads(content)
return data
def save_hparams(model_dir, hparams):
param_path = os.path.join(model_dir, PARAMS_NAME)
info = eval(hparams.to_json(). \
replace('true', 'True').replace('false', 'False'))
write_json(param_path, info)
print(" [*] MODEL dir: {}".format(model_dir))
print(" [*] PARAM path: {}".format(param_path))
def load_hparams(hparams, load_path, skip_list=[]):
path = os.path.join(load_path, PARAMS_NAME)
new_hparams = load_json(path)
hparams_keys = vars(hparams).keys()
for key, value in new_hparams.items():
if key in skip_list or key not in hparams_keys:
print("Skip {} because it not exists".format(key))
continue
if key not in ['job_name', 'num_workers', 'display', 'is_train', 'load_path'] or \
key == "pointer_load_path":
original_value = getattr(hparams, key)
if original_value != value:
print("UPDATE {}: {} -> {}".format(key, getattr(hparams, key), value))
setattr(hparams, key, value)
def add_prefix(path, prefix):
dir_path, filename = os.path.dirname(path), os.path.basename(path)
return "{}/{}.{}".format(dir_path, prefix, filename)
def add_postfix(path, postfix):
path_without_ext, ext = path.rsplit('.', 1)
return "{}.{}.{}".format(path_without_ext, postfix, ext)
def remove_postfix(path):
items = path.rsplit('.', 2)
return items[0] + "." + items[2]
def parallel_run(fn, items, desc="", parallel=True):
results = []
if parallel:
with closing(Pool()) as pool:
for out in tqdm(pool.imap_unordered(
fn, items), total=len(items), desc=desc):
if out is not None:
results.append(out)
else:
for item in tqdm(items, total=len(items), desc=desc):
out = fn(item)
if out is not None:
results.append(out)
return results
def which(program):
if os.name == "nt" and not program.endswith(".exe"):
program += ".exe"
envdir_list = [os.curdir] + os.environ["PATH"].split(os.pathsep)
for envdir in envdir_list:
program_path = os.path.join(envdir, program)
if os.path.isfile(program_path) and os.access(program_path, os.X_OK):
return program_path
def get_encoder_name():
if which("avconv"):
return "avconv"
elif which("ffmpeg"):
return "ffmpeg"
else:
return "ffmpeg"
def download_with_url(url, dest_path, chunk_size=32*1024):
with open(dest_path, "wb") as f:
response = requests.get(url, stream=True)
total_size = int(response.headers.get('content-length', 0))
for chunk in response.iter_content(chunk_size):
if chunk: # filter out keep-alive new chunks
f.write(chunk)
return True
def str2bool(v):
return v.lower() in ('true', '1')
def get_git_revision_hash():
return subprocess.check_output(['git', 'rev-parse', 'HEAD']).decode("utf-8")
def get_git_diff():
return subprocess.check_output(['git', 'diff']).decode("utf-8")
def warning(msg):
print("="*40)
print(" [!] {}".format(msg))
print("="*40)
print()
def query_yes_no(question, default=None):
# Code from https://stackoverflow.com/a/3041990
valid = {"yes": True, "y": True, "ye": True,
"no": False, "n": False}
if default is None:
prompt = " [y/n] "
elif default == "yes":
prompt = " [Y/n] "
elif default == "no":
prompt = " [y/N] "
else:
raise ValueError("invalid default answer: '%s'" % default)
while True:
sys.stdout.write(question + prompt)
choice = input().lower()
if default is not None and choice == '':
return valid[default]
elif choice in valid:
return valid[choice]
else:
sys.stdout.write("Please respond with 'yes' or 'no' "
"(or 'y' or 'n').\n")

50
utils/infolog.py Normal file
View file

@ -0,0 +1,50 @@
import atexit
from datetime import datetime
import json
from threading import Thread
from urllib.request import Request, urlopen
_format = '%Y-%m-%d %H:%M:%S.%f'
_file = None
_run_name = None
_slack_url = None
def init(filename, run_name, slack_url=None):
global _file, _run_name, _slack_url
_close_logfile()
_file = open(filename, 'a')
_file.write('\n-----------------------------------------------------------------\n')
_file.write('Starting new training run\n')
_file.write('-----------------------------------------------------------------\n')
_run_name = run_name
_slack_url = slack_url
def log(msg, slack=False):
print(msg)
if _file is not None:
_file.write('[%s] %s\n' % (datetime.now().strftime(_format)[:-3], msg))
if slack and _slack_url is not None:
Thread(target=_send_slack, args=(msg,)).start()
def _close_logfile():
global _file
if _file is not None:
_file.close()
_file = None
def _send_slack(msg):
req = Request(_slack_url)
req.add_header('Content-Type', 'application/json')
urlopen(req, json.dumps({
'username': 'tacotron',
'icon_emoji': ':taco:',
'text': '*%s*: %s' % (_run_name, msg)
}).encode())
atexit.register(_close_logfile)

61
utils/plot.py Normal file
View file

@ -0,0 +1,61 @@
import os
import matplotlib
from jamo import h2j, j2hcj
matplotlib.use('Agg')
matplotlib.rc('font', family="NanumBarunGothic")
import matplotlib.pyplot as plt
from text import PAD, EOS
from utils import add_postfix
from text.korean import normalize
def plot(alignment, info, text):
char_len, audio_len = alignment.shape # 145, 200
fig, ax = plt.subplots(figsize=(char_len/5, 5))
im = ax.imshow(
alignment.T,
aspect='auto',
origin='lower',
interpolation='none')
xlabel = 'Encoder timestep'
ylabel = 'Decoder timestep'
if info is not None:
xlabel += '\n{}'.format(info)
plt.xlabel(xlabel)
plt.ylabel(ylabel)
if text:
jamo_text = j2hcj(h2j(normalize(text)))
pad = [PAD] * (char_len - len(jamo_text) - 1)
plt.xticks(range(char_len),
[tok for tok in jamo_text] + [EOS] + pad)
if text is not None:
while True:
if text[-1] in [EOS, PAD]:
text = text[:-1]
else:
break
plt.title(text)
plt.tight_layout()
def plot_alignment(
alignment, path, info=None, text=None):
if text:
tmp_alignment = alignment[:len(h2j(text)) + 2]
plot(tmp_alignment, info, text)
plt.savefig(path, format='png')
else:
plot(alignment, info, text)
plt.savefig(path, format='png')
print(" [*] Plot saved: {}".format(path))

60
web/static/css/main.css Normal file
View file

@ -0,0 +1,60 @@
@media screen and (min-width: 1452px) {
.container {
max-width: 1152px;
width: 1152px;
}
}
@media screen and (min-width: 1260px) {
.container {
max-width: 960px;
width: 960px;
}
}
@media screen and (min-width: 1068px) {
.container {
max-width: 768px;
width: 768px;
}
}
.container {
margin: 0 auto;
position: relative;
}
#wave {
height: 100px;
}
#waveform {
display: none;
}
#nav {
position: fixed !important;
top: 0;
left: 0;
right: 0;
z-index: 100;
}
.card {
padding: 0;
}
.columns {
margin-left: 0rem;
margin-right: 0rem;
margin-top: 0rem;
}
#text {
font-size: 1.2em;
padding: 0.7em 1em 0.7em 1em;
background: transparent;
color: white;
}
.dark {
background-color: black;
}

105
web/static/js/main.js Normal file
View file

@ -0,0 +1,105 @@
var sw;
var wavesurfer;
var defaultSpeed = 0.03;
var defaultAmplitude = 0.3;
var activeColors = [[32,133,252], [94,252,169], [253,71,103]];
var inactiveColors = [[241,243,245], [206,212,218], [222,226,230], [173,181,189]];
function generate(ip, port, text, speaker_id) {
$("#synthesize").addClass("is-loading");
var uri = 'http://' + ip + ':' + port
var url = uri + '/generate?text=' + encodeURIComponent(text) + "&speaker_id=" + speaker_id;
fetch(url, {cache: 'no-cache', mode: 'cors'})
.then(function(res) {
if (!res.ok) throw Error(response.statusText)
return res.blob()
}).then(function(blob) {
var url = URL.createObjectURL(blob);
console.log(url);
inProgress = false;
wavesurfer.load(url);
$("#synthesize").removeClass("is-loading");
}).catch(function(err) {
showWarning("에러가 발생했습니다");
inProgress = false;
$("#synthesize").removeClass("is-loading");
});
}
(function(window, document, undefined){
window.onload = init;
function setDefaultColor(sw, isActive) {
for (idx=0; idx < sw.curves.length; idx++) {
var curve = sw.curves[idx];
if (isActive) {
curve.color = activeColors[idx % activeColors.length];
} else {
curve.color = inactiveColors[idx % inactiveColors.length];
}
}
}
function init(){
sw = new SiriWave9({
amplitude: defaultAmplitude,
container: document.getElementById('wave'),
autostart: true,
speed: defaultSpeed,
style: 'ios9',
});
sw.setSpeed(defaultSpeed);
setDefaultColor(sw, false);
wavesurfer = WaveSurfer.create({
container: '#waveform',
waveColor: 'violet',
barWidth: 3,
progressColor: 'purple'
});
wavesurfer.on('ready', function () {
this.width = wavesurfer.getDuration() *
wavesurfer.params.minPxPerSec * wavesurfer.params.pixelRatio;
this.peaks = wavesurfer.backend.getPeaks(width);
wavesurfer.play();
});
wavesurfer.on('audioprocess', function () {
var percent = wavesurfer.backend.getPlayedPercents();
var height = this.peaks[parseInt(this.peaks.length * percent)];
if (height > 0) {
sw.setAmplitude(height*3);
}
});
wavesurfer.on('finish', function () {
sw.setSpeed(defaultSpeed);
sw.setAmplitude(defaultAmplitude);
setDefaultColor(sw, false);
});
$(document).on('click', "#synthesize", function() {
synthesize();
});
function synthesize() {
var text = $("#text").val().trim();
var text_length = text.length;
var speaker_id = $('input[name=id]:checked').val();
var speaker = $('input[name=id]:checked').attr("speaker");
generate('0.0.0.0', 5000, text, speaker_id);
var lowpass = wavesurfer.backend.ac.createGain();
wavesurfer.backend.setFilter(lowpass);
}
}
})(window, document, undefined);

212
web/static/js/siriwave.js Normal file
View file

@ -0,0 +1,212 @@
(function() {
////////////////////
// SiriWave9Curve //
////////////////////
function SiriWave9Curve(opt) {
opt = opt || {};
this.controller = opt.controller;
this.color = opt.color;
this.tick = 0;
this.respawn();
}
SiriWave9Curve.prototype.respawn = function() {
this.amplitude = 0.3 + Math.random() * 0.7;
this.seed = Math.random();
this.open_class = 2+(Math.random()*3)|0;
};
SiriWave9Curve.prototype.equation = function(i) {
var p = this.tick;
var y = -1 * Math.abs(Math.sin(p)) * this.controller.amplitude * this.amplitude * this.controller.MAX * Math.pow(1/(1+Math.pow(this.open_class*i,2)),2);
if (Math.abs(y) < 0.001) {
this.respawn();
}
return y;
};
SiriWave9Curve.prototype._draw = function(m) {
this.tick += this.controller.speed * (1-0.5*Math.sin(this.seed*Math.PI));
var ctx = this.controller.ctx;
ctx.beginPath();
var x_base = this.controller.width/2 + (-this.controller.width/4 + this.seed*(this.controller.width/2) );
var y_base = this.controller.height/2;
var x, y, x_init;
var i = -3;
while (i <= 3) {
x = x_base + i * this.controller.width/4;
y = y_base + (m * this.equation(i));
x_init = x_init || x;
ctx.lineTo(x, y);
i += 0.01;
}
var h = Math.abs(this.equation(0));
var gradient = ctx.createRadialGradient(x_base, y_base, h*1.15, x_base, y_base, h * 0.3 );
gradient.addColorStop(0, 'rgba(' + this.color.join(',') + ',0.4)');
gradient.addColorStop(1, 'rgba(' + this.color.join(',') + ',0.2)');
ctx.fillStyle = gradient;
ctx.lineTo(x_init, y_base);
ctx.closePath();
ctx.fill();
};
SiriWave9Curve.prototype.draw = function() {
this._draw(-1);
this._draw(1);
};
//////////////
// SiriWave //
//////////////
function SiriWave9(opt) {
opt = opt || {};
this.tick = 0;
this.run = false;
// UI vars
this.ratio = opt.ratio || window.devicePixelRatio || 1;
this.width = this.ratio * (opt.width || 320);
this.height = this.ratio * (opt.height || 100);
this.MAX = this.height/2;
this.speed = 0.1;
this.amplitude = opt.amplitude || 1;
// Interpolation
this.speedInterpolationSpeed = opt.speedInterpolationSpeed || 0.005;
this.amplitudeInterpolationSpeed = opt.amplitudeInterpolationSpeed || 0.05;
this._interpolation = {
speed: this.speed,
amplitude: this.amplitude
};
// Canvas
this.canvas = document.createElement('canvas');
this.canvas.width = this.width;
this.canvas.height = this.height;
if (opt.cover) {
this.canvas.style.width = this.canvas.style.height = '100%';
} else {
this.canvas.style.width = (this.width / this.ratio) + 'px';
this.canvas.style.height = (this.height / this.ratio) + 'px';
}
this.container = opt.container || document.body;
this.container.appendChild(this.canvas);
this.ctx = this.canvas.getContext('2d');
// Create curves
this.curves = [];
for (var i = 0; i < SiriWave9.prototype.COLORS.length; i++) {
var color = SiriWave9.prototype.COLORS[i];
for (var j = 0; j < (3 * Math.random())|0; j++) {
this.curves.push(new SiriWave9Curve({
controller: this,
color: color
}));
}
}
if (opt.autostart) {
this.start();
}
}
SiriWave9.prototype._interpolate = function(propertyStr) {
increment = this[ propertyStr + 'InterpolationSpeed' ];
if (Math.abs(this._interpolation[propertyStr] - this[propertyStr]) <= increment) {
this[propertyStr] = this._interpolation[propertyStr];
} else {
if (this._interpolation[propertyStr] > this[propertyStr]) {
this[propertyStr] += increment;
} else {
this[propertyStr] -= increment;
}
}
};
SiriWave9.prototype._clear = function() {
this.ctx.globalCompositeOperation = 'destination-out';
this.ctx.fillRect(0, 0, this.width, this.height);
this.ctx.globalCompositeOperation = 'lighter';
};
SiriWave9.prototype._draw = function() {
for (var i = 0, len = this.curves.length; i < len; i++) {
this.curves[i].draw();
}
};
SiriWave9.prototype._startDrawCycle = function() {
if (this.run === false) return;
this._clear();
// Interpolate values
this._interpolate('amplitude');
this._interpolate('speed');
this._draw();
this.phase = (this.phase + Math.PI*this.speed) % (2*Math.PI);
if (window.requestAnimationFrame) {
window.requestAnimationFrame(this._startDrawCycle.bind(this));
} else {
setTimeout(this._startDrawCycle.bind(this), 20);
}
};
SiriWave9.prototype.start = function() {
this.tick = 0;
this.run = true;
this._startDrawCycle();
};
SiriWave9.prototype.stop = function() {
this.tick = 0;
this.run = false;
};
SiriWave9.prototype.setSpeed = function(v, increment) {
this._interpolation.speed = v;
};
SiriWave9.prototype.setNoise = SiriWave9.prototype.setAmplitude = function(v) {
this._interpolation.amplitude = Math.max(Math.min(v, 1), 0);
};
SiriWave9.prototype.COLORS = [
[32,133,252],
[94,252,169],
[253,71,103]
];
if (typeof define === 'function' && define.amd) {
define(function(){ return SiriWave9; });
} else {
window.SiriWave9 = SiriWave9;
}
})();

74
web/templates/index.html Normal file
View file

@ -0,0 +1,74 @@
<!DOCTYPE html>
<html>
<head>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1">
<title>D.Voice</title>
<link rel="stylesheet" href="https://cdnjs.cloudflare.com/ajax/libs/font-awesome/4.7.0/css/font-awesome.min.css">
<link rel="stylesheet" href="https://cdnjs.cloudflare.com/ajax/libs/bulma/0.5.1/css/bulma.min.css">
<link rel="stylesheet" href="{{ url_for('static', filename='css/main.css') }}">
<script src="https://cdnjs.cloudflare.com/ajax/libs/jquery/3.2.1/jquery.min.js"></script>
<script src="https://cdnjs.cloudflare.com/ajax/libs/underscore.js/1.8.3/underscore-min.js"></script>
<script src="https://wavesurfer-js.org/dist/wavesurfer.min.js"></script>
<script src="{{ url_for('static', filename='js/siriwave.js') }}"></script>
<script src="{{ url_for('static', filename='js/main.js') }}"></script>
</head>
<body class="layout-default">
<section class="hero is-fullheight dark">
<div class="hero-body">
<div class="container">
<div class="section-body" onKeyPress="return checkSubmit(event)">
<div class="field">
<div class="control">
<div class="columns">
<div class="column"></div>
<div class="column">
<div id="wave"></div>
</div>
<div class="column"></div>
</div>
</div>
</div>
<div class="field">
<div class="control">
<div id="waveform"></div>
</div>
</div>
<div class="field">
<div class="control has-text-centered">
<label class="radio">
<input type="radio" name="id" value="0" port="5000" checked>
Speaker 1
</label>
</div>
</div>
<div class="field">
<div class="control has-icons-right">
<textarea class="textarea" id="text" placeholder="{{ text }} "></textarea>
<span class="icon is-small is-right" id="text-warning-icon" style="display:none">
<i class="fa fa-warning"></i>
</span>
</div>
<p class="help is-danger" id="text-warning" style="display:none">
Wrong sentence
</p>
</div>
<div class="field has-text-centered">
<button class="button is-white" id="synthesize">
Synthesize
</button>
</div>
</div>
</div>
</div>
</section>
</body>
</html>