DEVTOOLS: BLADERUNNER: Quick guide for transcriptions

Also minor improvements in the python scripts

Made audio links working again with LibreOffice Calc and added a subtitles version sheet in the python script that creates the Excel document and the audio links

5 files changed, 84 insertions, 13 deletions

diff --git a/devtools/create_bladerunner/subtitles/README.md b/devtools/create_bladerunner/subtitles/README.md
index e0858a264c..3f0913aa66 100644
--- a/devtools/create_bladerunner/subtitles/README.md
+++ b/devtools/create_bladerunner/subtitles/README.md
@@ -2,6 +2,29 @@
 Some tools written in __Python 2.7__ to help add support for subtitles in Westwood's point and click adventure game Blade Runner (1997) for PC.
 The official English, German, French, Italian and Spanish versions of the game should be supported.
 
+## Quick guide for volunteer transcribers
+0. Please make sure you have an audio player installed in your Operating System (such as vlc).
+1. You will need python 2.7 installed for this guide, as well as the python libraries Pillow (or PIL), xlrd, xlwt, wave and xlutils.
+2. Using git, checkout the latest revision of the ScummVM repository in github.
+3. Make sure you've created a Blade Runner game directory and added Blade Runner in ScummVM as per the instructions in the wiki (the actual required files are __1.TLK, 2.TLK, 3.TLK, A.TLK and SPCHSFX.TLK__): https://wiki.scummvm.org/index.php?title=User_Manual/Installing_a_game_for_use_with_ScummVM#Blade_Runner
+For this guide it is assumed that this folder is "C:\Westwood\BladeRunner\"
+4. Create a folder on your HDD to export all speech audio from the game. Eg. create a "C:\Westwood\Blade Runner\AUDIO_FRA\" folder.
+5. Export all speech audio from the game and create an Excel file (xls) with links to the audio files. You will need to have some free HDD space for this step (around 650MB). Using a command line interface (eg MSYS2) navigate to the ScummVM repository folder and issue a command like the following, making sure you specify the correct target language code for your requirements; supported language code are "EN_ANY" for English, "FR_FRA" for French, "DE_DEU" for German, "IT_ITA" for Italian, "ES_ESP" for Spanish. In the following example we assume we need to export the audio from the French version of the game.
+```
+python2.7 devtools/create_bladerunner/subtitles/quotesSpreadsheetCreator/quoteSpreadsheetCreator.py -op "/c/Westwood/Blade Runner/AUDIO_FRA" -ip "/c/Westwood/Blade Runner" -ian "devtools/create_bladerunner/subtitles/common/actornames.txt" -xwav -ld FR_FRA
+```
+This could take a few minutes to complete. Please refer to the [quotesSpreadsheetCreator](#quotesspreadsheetcreator) section of this document.
+6. When you've completed the above step, an Excel file should be created in your current folder, named "out-French.xls". Open this file with a spreadsheet editor app (tested with Microsoft Excel 2007 and LibreOffice Calc 6.2.4) and select the "INGQUO_#.TR#" sheet.
+    * Please keep the column A ("Filename") as is. Do not modify the contents of this column.
+	* Edit column B ("Quote") to put in the text for your transcript of the corresponding audio file
+	* Use column G ("ShortHandFileName") to play the audio file for the quote to be transcribed in the same line. You should be able to double click in MS Excel or Ctrl+Click on the specific cell; an audio player will be launched automatically to play the audio file.
+	* Note that the columns used by the tool that will create the final subtitles file are only columns A and B.
+7. The rest of the sheets in the Excel file, excluding the "SBTLVERS.TRE" which contains versioning info) are the sheets for the video cutscenes. We don't provide links to the audio cues for those subtitles, but we do provide the timings for the English version of these scenes which you can use as a guide, along with the English transcript available online here: https://docs.google.com/spreadsheets/d/17ew0YyhSwqcqZg6bXrIgz0GkA62dhgViHN15lOu5Hj8/edit?usp=sharing
+    * Note that the important columns used by the tool that will create the final subtitles file are in this case only columns C ("Subtitle"), J ("Frame Start") and K ("Frame End").
+	* Fill in column C with the subtitle text, and set the values in columns J and K to adjust when the subtitle should start and when it should stop being displayed respectively.
+8. You may edit the info in the "SBTLVERS.TRE" sheet to fill in the credits and a version number for your transcript.
+9. If you're transcribing for a non-Latin language, you will need to create a special font for the subtitles in order for the game to display your transcript properly. Please, refer to the [fontCreator](#fontcreator) section of this document for this process.
+
 ## Building and installing a SUBTITLES.MIX file with a "make" command
 You need to follow these instructions:
 1. Download the online Excel transcript and save it as "englishTranscript.xlsx" into the "devtools\create_bladerunner\subtitles\sampleInput" folder.
@@ -34,7 +57,7 @@ Syntax Notes:
 4. The "-ld" optional switch is followed by a language description for the language of the game you are exporting Text Resources from. This switch is meaningful when you also use the "-xtre" switch to export Text Resource files.
     * Valid language values are: EN_ANY, DE_DEU, FR_FRA, IT_ITA, ES_ESP, RU_RUS 
     * Default language value is: EN_ANY (English)
-5. The "-xwav" optional switch will export __ALL__ game's audio files (AUD) (that are either speech or speech-related) in a WAV format. This is expected to run for a few minutes and take up quite a lot of your HDD space (about 650MB).
+5. The "-xwav" optional switch will export __ALL__ game's audio files (AUD) (that are either speech or speech-related) in a WAV format. This is expected to run for a few minutes and take up quite a lot of your HDD space (around 650MB).
 6. The "-xtre" optional switch will add extra sheets to the output Excel with the contents of each of the game's Text Resource files (TRx) (on sheet per TRx file).
 7. The "-xdevs" optional switch will add a sheet for Developer Commentary text and some additional voice-overs from SFX.MIX.
 8. The "-xpogo" optional switch will add a sheet for the POGO text.
diff --git a/devtools/create_bladerunner/subtitles/mixResourceCreator/subtlsVersTextResource.py b/devtools/create_bladerunner/subtitles/common/subtlsVersTextResource.py
index a372bab1a5..a372bab1a5 100644
--- a/devtools/create_bladerunner/subtitles/mixResourceCreator/subtlsVersTextResource.py
+++ b/devtools/create_bladerunner/subtitles/common/subtlsVersTextResource.py
diff --git a/devtools/create_bladerunner/subtitles/fontCreator/grabberFromPNG17BR.py b/devtools/create_bladerunner/subtitles/fontCreator/grabberFromPNG17BR.py
index 172045bd0f..c50735d9bc 100644
--- a/devtools/create_bladerunner/subtitles/fontCreator/grabberFromPNG17BR.py
+++ b/devtools/create_bladerunner/subtitles/fontCreator/grabberFromPNG17BR.py
@@ -6,7 +6,7 @@
 #			   FON file for the Westwood Blade Runner PC game.
 #
 # Author:	   antoniou
-#w
+#
 # Created:	   16-05-2018
 # Copyright:   (c) antoniou 2018
 # Licence:
@@ -45,22 +45,22 @@
 #           This should get rid of semi-transparent pixels while maintaining the "aliasing" effect. There could be a better way but this should work ok.
 
 
-# DONE: A way to export game fonts to png image (in order to be able to create new expanded fonts keeping glyph consistency!)
+# DONE: A way to export game fonts to PNG image (in order to be able to create new expanded fonts keeping glyph consistency!)
 #           Also override the corruption in TAHOMA18 while exporting
 # TODO: Re-Check the order of fonts in (in-game resource font files) TAHOMA18 (stored corrupted) and TAHOMA24 (in good condition).
 # TODO: print a warning for mismatch of number of letters in encoding override (or internal) and detected fonts in ROW IMAGE (especially if we expect a double exclamation mark at the start - and we ignoring one of the two)
-# TODO: Don't crash if mismatch in detected letter (fewer) in row image vs the letters in ascii list of overrideEncoding file
+# TODO: Don't crash if mismatch in detected letter (fewer) in row image vs the letters in ASCII list of overrideEncoding file
 # TODO: A more detailed readme for this tool and how to use it
 #
 # DONE: enforce overrideEncoding.txt -- this tool should no longer work without one
-# DONE: Test greek subs too
+# DONE: Test Greek subs too
 # DONE: Letter fonts should be spaced by TAB when copied into GIMP or other app to create the image row of all character glyphs
 # DONE: First character should be repeated in the ROW file (but taken into consideration once) in order to get the pixels for the TAB space between letters (left-start column to left-start column)
 # DONE: Use the tab space pixels to calculate the KERNING for each letter (x offset)
 # DONE: update the image segment size bytes in the header after having completed populating the image segment
-# DONE: allow settin explicit kerning and width addon for cases like i and l characters
+# DONE: allow setting explicit kerning and width addon for cases like i and l characters
 # DONE: entrée (boiled dog question) - has an e like goose liver pate --> TESTED
-# DONE: Tested ok "si senor" from peruvian lady / insect dealer too!
+# DONE: Tested ok "si senor" from Peruvian lady / insect dealer too!
 # DONE: ability to manually set kerning (x-offset) for fonts by letter like a list in parameters or in overrideEncoding.txt }  i:1,j:-1,l:1	 (no space or white line characters) - POSITIVE OR NEGATIVE VALUES BOTH ADMITTED
 # DONE: a value of '-' for this means ignore
 # DONE: ability to manually set extra width (additional columns at the end of glyph, with transparent color) for fonts by letter like a list in parameters or in overrideEncoding.txt }	 i:1,j:2,l:1 - POSITIVE VALUES ONLY
diff --git a/devtools/create_bladerunner/subtitles/mixResourceCreator/packBladeRunnerMIXFromPCTLKXLS04.py b/devtools/create_bladerunner/subtitles/mixResourceCreator/packBladeRunnerMIXFromPCTLKXLS04.py
index 2febd3576c..f7d77e29a2 100644
--- a/devtools/create_bladerunner/subtitles/mixResourceCreator/packBladeRunnerMIXFromPCTLKXLS04.py
+++ b/devtools/create_bladerunner/subtitles/mixResourceCreator/packBladeRunnerMIXFromPCTLKXLS04.py
@@ -114,6 +114,9 @@ if 	(not osLibFound) \
 	sys.stdout.write("[Error] Errors were found when trying to import required python libraries\n")
 	sys.exit(1)
 
+pathToParent = os.path.join(os.path.dirname(os.path.abspath(__file__)), os.pardir)
+pathToCommon = os.path.join(pathToParent, "common")
+sys.path.append(pathToCommon)
 
 from os import path
 from xlrd import *
@@ -122,7 +125,7 @@ from struct import *
 from subtlsVersTextResource import *
 
 COMPANY_EMAIL = "classic.adventures.in.greek@gmail.com"
-APP_VERSION = "1.00"
+APP_VERSION = "1.10"
 APP_NAME = "packBladeRunnerMIXFromPCTLKXLS"
 APP_WRAPPER_NAME = "mixResourceCreator.py"
 APP_NAME_SPACED = "Blade Runner MIX Resource Creator"
@@ -176,7 +179,8 @@ SUPPORTED_TRANSLATION_SHEETS = [('OPTIONS.TR', 'KIA6PT'),
 # TODO we probably could skip importing ERRORMSG.TRx (to SUBTITLES.MIX) altogether, since translating that has no point! In that case SYSTEM.FON should be removed from here since it's currently of no use otherwise and support for it is not really required.
 SUPPORTED_OTHER_FILES_FOR_MIX = [DEFAULT_SUBTITLES_FONT_NAME, 'KIA6PT.FON', 'TAHOMA18.FON', 'TAHOMA24.FON', 'SYSTEM.FON']
 
-SUPPORTED_LANGUAGES_DESCRIPTION_CODE_TLIST = [('EN_ANY', 'E', 'English'), ('DE_DEU', 'G', 'German'), ('FR_FRA', 'F', 'French'), ('IT_ITA', 'I', 'Italian'), ('ES_ESP', 'S', 'Spanish'), ('RU_RUS', 'R', 'Russian'), ('EFIGS', '#', 'EFIGS')]
+# v1.10: Russian code (RU_RUS) now corresponds to 'E' suffix instead of 'R' since the unofficial Russian version supported uses the English resources without renaming them, and this is how the ScummVM engine handles that version currently.
+SUPPORTED_LANGUAGES_DESCRIPTION_CODE_TLIST = [('EN_ANY', 'E', 'English'), ('DE_DEU', 'G', 'German'), ('FR_FRA', 'F', 'French'), ('IT_ITA', 'I', 'Italian'), ('ES_ESP', 'S', 'Spanish'), ('RU_RUS', 'E', 'Russian'), ('EFIGS', '#', 'EFIGS')]
 DEFAULT_LANG_DESC_CODE = SUPPORTED_LANGUAGES_DESCRIPTION_CODE_TLIST[0]
 
 DEFAULT_TARGET_ENCODING_PER_FONT = [(SUBTITLES_FONT_NAME_CATEGORY, WINDOWS_1252_ENCODING), ('KIA6PT', 'cp437'), ('TAHOMA', 'cp437'), ('SYSTEM', 'latin-1')]
diff --git a/devtools/create_bladerunner/subtitles/quotesSpreadsheetCreator/sortBladeRunnerWavs02.py b/devtools/create_bladerunner/subtitles/quotesSpreadsheetCreator/sortBladeRunnerWavs02.py
index 800372b4cc..1f1646174e 100644
--- a/devtools/create_bladerunner/subtitles/quotesSpreadsheetCreator/sortBladeRunnerWavs02.py
+++ b/devtools/create_bladerunner/subtitles/quotesSpreadsheetCreator/sortBladeRunnerWavs02.py
@@ -56,19 +56,26 @@ if 	(not osLibFound) \
 	sys.stdout.write("[Error] Errors were found when trying to import required python libraries\n")
 	sys.exit(1)
 
+
 from os import walk, errno, path
+
+pathToParent = os.path.join(os.path.dirname(os.path.abspath(__file__)), os.pardir)
+pathToCommon = os.path.join(pathToParent, "common")
+sys.path.append(pathToCommon)
+
 from xlwt import *
 from audFileLib import *
 from treFileLib import *
 from pogoTextResource import *
 from devCommentaryText import *
+from subtlsVersTextResource import *
 
 # encoding=utf8
 #reload(sys)
 #sys.setdefaultencoding('utf8')
 
 COMPANY_EMAIL = "classic.adventures.in.greek@gmail.com"
-APP_VERSION = "0.80"
+APP_VERSION = "0.85"
 APP_NAME = "sortBladeRunnerWavs"
 APP_WRAPPER_NAME = "quotesSpreadsheetCreator.py"
 APP_NAME_SPACED = "Blade Runner Transcript Excel Creator (bare bones)"
@@ -133,8 +140,10 @@ SUPPORTED_PLACEHOLDER_VQA_LOCALIZED_FILES = [
 	('END06_', 'Underground getaway - Steele Ending')
 	]
 SUPPORTED_SPECIAL_POGO_FILE = 'POGO.TR'
+SUPPORTED_DIALOGUE_VERSION_SHEET = 'SBTLVERS.TRE'
 
-SUPPORTED_LANGUAGES_DESCRIPTION_CODE_TLIST = [('EN_ANY', 'E', 'English'), ('DE_DEU', 'G', 'German'), ('FR_FRA', 'F', 'French'), ('IT_ITA', 'I', 'Italian'), ('ES_ESP', 'S', 'Spanish'), ('RU_RUS', 'R', 'Russian')]
+# v0.85: Russian code (RU_RUS) now corresponds to 'E' suffix instead of 'R' since the unofficial Russian version supported uses the English resources without renaming them, and this is how the ScummVM engine handles that version currently.
+SUPPORTED_LANGUAGES_DESCRIPTION_CODE_TLIST = [('EN_ANY', 'E', 'English'), ('DE_DEU', 'G', 'German'), ('FR_FRA', 'F', 'French'), ('IT_ITA', 'I', 'Italian'), ('ES_ESP', 'S', 'Spanish'), ('RU_RUS', 'E', 'Russian')]
 DEFAULT_LANG_DESC_CODE = SUPPORTED_LANGUAGES_DESCRIPTION_CODE_TLIST[0]
 
 gTraceModeEnabled = False
@@ -531,7 +540,7 @@ def auxPopulateExtraSpeechAudioRow(sh = None, n = 0, pFilenameStr = '', pTextStr
 						if gStringReplacementForRootFolderWithExportedFiles and gNumReplaceStartingCharacters > 0:
 							realPathOfFileNameToLink = realPathOfFileNameToLink.replace(realPathOfFileNameToLink[:gNumReplaceStartingCharacters], gStringReplacementForRootFolderWithExportedFiles)
 
-						hyperlinkAudioFormula = 'HYPERLINK("file://%s","%s")' % (realPathOfFileNameToLink, shortHandFileName)
+						hyperlinkAudioFormula = 'HYPERLINK("file:///%s","%s")' % (realPathOfFileNameToLink, shortHandFileName)
 						sh.write(n, 6, Formula(hyperlinkAudioFormula))
 						break
 			foundMatch = True
@@ -816,6 +825,41 @@ def outputXLS(filename, sheet, listTlkWavs, listDevsWavs, parseTREResourcesAlso
 	global gNumReplaceStartingCharacters
 
 	book = xlwt.Workbook()
+	# first add version info sheet
+	sbtlVersTRInstance = sbtlVersTextResource(gTraceModeEnabled)
+	vrsn_sheet = book.add_sheet(SUPPORTED_DIALOGUE_VERSION_SHEET)
+	n = 0
+	col1_name = 'Subtitles Version Info'
+	vrsn_sheet.write(n, 0, col1_name)
+	# Second Row
+	n = 1
+	col1_name = 'ID'
+	col2_name = 'Value'
+	col3_name = 'Notes'
+	vrsn_sheet.write(n, 0, col1_name)
+	vrsn_sheet.write(n, 1, col2_name)
+	vrsn_sheet.write(n, 2, col3_name)
+	n += 1
+	objUTF8Unicode = None
+	for m, e1 in enumerate(sbtlVersTRInstance.getSbtlVersEntriesList(), n):
+		vrsn_sheet.write(m, 0, e1[0])
+		tmpQuoteID = int(e1[0])
+		for i1 in range(1,3):
+			objStr = e1[i1]
+			try:
+				# We assume utf-8 charset (since we get the text from a python script)
+				# populate row with quote ID == 3 and column B with the description of language used in the command's execution (or assumed)
+				if tmpQuoteID == 3 and i1 == 1:
+					objUTF8Unicode = unicode(gActiveLanguageDescriptionCodeTuple[2], 'utf-8')
+				else:
+					objUTF8Unicode = unicode(objStr, 'utf-8')
+			except Exception as e:
+				print '[Error] Failed to create unicode string: ' + str(e)
+				objUTF8Unicode = unicode("???", 'utf-8')
+			vrsn_sheet.write(m, i1, objUTF8Unicode)
+	#
+	# Add ingame quotes sheet
+	#
 	sh = book.add_sheet(sheet)
 	# First Row
 	n = 0      # keeps track of rows
@@ -887,7 +931,7 @@ def outputXLS(filename, sheet, listTlkWavs, listDevsWavs, parseTREResourcesAlso
 				# also works in Windows + LibreOffice (run from msys) -- tried something like:
 				#	python sortBladeRunnerWavs.py -op /g/WORKSPACE/BladeRunnerWorkspace/br-mixer-master/data/WAV -m "G:/WORKSPACE/BladeRunnerWorkspace/br-mixer-master/data/WAV"
 				# put real full path for each file as FILE URL, and real (or approximate shorthand file name as alias)
-				hyperlinkAudioFormula = 'HYPERLINK("file://%s","%s")' % (realPathOfFileNameToLink, shortHandFileName)
+				hyperlinkAudioFormula = 'HYPERLINK("file:///%s","%s")' % (realPathOfFileNameToLink, shortHandFileName)
 				sh.write(m, 6, Formula(hyperlinkAudioFormula))
 			else:
 				sh.write(m, 0, e1)