DEVTOOLS: Support xlsx extension for Blade Runner subtitles

If the englishTranscript.xls is not found, the englishTranscript.xlsx will be used instead

Google sheets supports download to xlsx, so this addition was needed.
Also tested the font tool with Pillows image library (PIL fork) which works ok, and updated README file.

5 files changed, 59 insertions, 36 deletions

diff --git a/devtools/create_bladerunner/subtitles/README.md b/devtools/create_bladerunner/subtitles/README.md
index 6b174a4477..04f6e19c2e 100644
--- a/devtools/create_bladerunner/subtitles/README.md
+++ b/devtools/create_bladerunner/subtitles/README.md
@@ -4,7 +4,7 @@ The official English, German, French, Italian and Spanish versions of the game s
 
 ## Building and installing a SUBTITLES.MIX file with a "make" command
 You need to follow these instructions:
-1. Download the online Excel transcript and save it as "englishTranscript.xls" into the "devtools\create_bladerunner\subtitles\sampleInput" folder.
+1. Download the online Excel transcript and save it as "englishTranscript.xlsx" into the "devtools\create_bladerunner\subtitles\sampleInput" folder.
 __The online Excel file is available here:__
 https://docs.google.com/spreadsheets/d/17ew0YyhSwqcqZg6bXrIgz0GkA62dhgViHN15lOu5Hj8/edit?usp=sharing
 2. Edit your font glyphs PNG file (or use the provided one in the sampleInput folder). This file should be stored as "subtitlesFont.png" into the "devtools\create_bladerunner\subtitles\sampleInput" folder.
@@ -14,11 +14,12 @@ https://docs.google.com/spreadsheets/d/17ew0YyhSwqcqZg6bXrIgz0GkA62dhgViHN15lOu5
 ```
 make devtools/create_bladerunner/subtitles
 ```
+5. You may need to install some required Python 2 libraries such as Pillow (or PIL), xlrd, xlwt and wave (which is part of the Python 2 Standard Library).
 6. Copy the output file "SUBTITLES.MIX", created in the ScummVM root folder, into your Blade Runner game directory.
 7. Launch the Blade Runner game using ScummVM.
  
 ## quotesSpreadsheetCreator (quoteSpreadsheetCreator.py)
-(requires python lib *xlwt*, *wave*)
+(requires python lib *xlwt* (tested with version 1.3.0), *wave* (included in the Python 2 Standard Library)
 A tool to gather all the speech audio filenames in an Excel file which will include a column with links to the audio file location on the PC. By Ctrl+MouseClick on that column's entries you should be able to listen to the corresponding wav file.
 The output Excel file *out.xls* should help with the transcription of all the spoken *in-game* quotes. It also provides extra quote information such as the corresponding actor ID and quote ID per quote.
 
@@ -48,16 +49,16 @@ The tool __requires__ a valid path to the actornames.txt file; this file is incl
 
 
 ## mixResourceCreator (mixResourceCreator.py)
-(requires python lib *xlrd*)
+(requires python lib *xlrd* (tested with version 1.2.0))
 A tool to process the aforementioned Excel file with the dialogue transcriptions and output text resource files (TRx) that will be packed along with the external font (see fontCreator tool) into a SUBTITLES.MIX file. Multiple TRx files will be created intermediately in order to fully support subtitles in the game. One TRx file includes all in-game spoken quotes and the rest of them correspond to any VQA video sequence that contain voice acting.
 Usage:
 ```
-python2.7 mixResourceCreator.py -x excelWithTranscriptSheets.xls [-ian pathToActorNamesTxt] [-cft pathToConfigureFontsTranslationTxt] [--trace]
+python2.7 mixResourceCreator.py -x excelWithTranscriptSheets.xlsx [-ian pathToActorNamesTxt] [-cft pathToConfigureFontsTranslationTxt] [--trace]
 ```
 The tool __requires__ a valid path to the actornames.txt file, which is included in the samples folder.
 
 Syntax Notes: 
-1. The "-x" switch is followed by the path to the input Excel file (xls) which should contain the transcript sheet(s).
+1. The "-x" switch is followed by the path to the input Excel file (xls or xlsx) which should contain the transcript sheet(s).
 2. The "-ian" optional switch is followed by the path to the actornames.txt file -- if this is omitted then the file is assumed to reside in the current working directory.
 3. The "-cft" optional switch is followed by the path to the text configuration file "configureFontsTranslation.txt" -- if this is omitted then the file is assumed to reside in the current working directory.
 4. The "-ld" optional switch is followed by a language description for the language of the game you are exporting Text Resources from. This switch is meaningful when you also use the "-xtre" switch to export Text Resource files.
@@ -84,7 +85,7 @@ The __text configuration file "configureFontsTranslation.txt"__ a __text file th
 	SUBTLS_E.FON, KIA6PT.FON, TAHOMA18.FON, TAHOMA24.FON and SYSTEM.FON (practically you won't be using the last one). 
 	
 ## fontCreator (fontCreator.py)
-(requires python Image library *PIL*)
+(requires python Image library *Pillow* (tested with version 5.4.1))
 A tool to support __both__ the exporting of fonts from the game (to PNG images) __and__ the creation of a font file (FON) in order to resolve various issues with the available fonts (included in the game's own resource files). These issues include alignment, kerning, corrupted format, limited charset and unsupported characters -- especially for languages with too many non-Latin symbols in their alphabet.
 This font tool's code is based off the Monkey Island Special Edition's Translator (https://github.com/ShadowNate/MISETranslator).
 Usage:
diff --git a/devtools/create_bladerunner/subtitles/fontCreator/fonFileLib.py b/devtools/create_bladerunner/subtitles/fontCreator/fonFileLib.py
index 1eef7834ab..835cdb55c0 100644
--- a/devtools/create_bladerunner/subtitles/fontCreator/fonFileLib.py
+++ b/devtools/create_bladerunner/subtitles/fontCreator/fonFileLib.py
@@ -36,7 +36,7 @@ else:
 	structLibFound = True
 	
 try:
-	import Image 
+	from PIL import Image 
 except ImportError:
 	print "[Error] Image python library (PIL) is required to be installed!" 
 else:
diff --git a/devtools/create_bladerunner/subtitles/fontCreator/grabberFromPNG17BR.py b/devtools/create_bladerunner/subtitles/fontCreator/grabberFromPNG17BR.py
index c29fe815ce..172045bd0f 100644
--- a/devtools/create_bladerunner/subtitles/fontCreator/grabberFromPNG17BR.py
+++ b/devtools/create_bladerunner/subtitles/fontCreator/grabberFromPNG17BR.py
@@ -102,7 +102,7 @@ else:
 	structLibFound = True
 	
 try:
-	import Image 
+	from PIL import Image 
 except ImportError:
 	print "[Error] Image python library (PIL) is required to be installed!" 
 else:
diff --git a/devtools/create_bladerunner/subtitles/mixResourceCreator/packBladeRunnerMIXFromPCTLKXLS04.py b/devtools/create_bladerunner/subtitles/mixResourceCreator/packBladeRunnerMIXFromPCTLKXLS04.py
index 32b2d0bf7d..763df2f0ac 100644
--- a/devtools/create_bladerunner/subtitles/mixResourceCreator/packBladeRunnerMIXFromPCTLKXLS04.py
+++ b/devtools/create_bladerunner/subtitles/mixResourceCreator/packBladeRunnerMIXFromPCTLKXLS04.py
@@ -92,7 +92,7 @@ from xlrd import *
 from struct import *
 
 COMPANY_EMAIL = "classic.adventures.in.greek@gmail.com"
-APP_VERSION = "0.80"
+APP_VERSION = "0.90"
 APP_NAME = "packBladeRunnerMIXFromPCTLKXLS"
 APP_WRAPPER_NAME = "mixResourceCreator.py"
 APP_NAME_SPACED = "Blade Runner MIX Resource Creator"
@@ -277,7 +277,6 @@ def initOverrideEncoding(pathToConfigureFontsTranslationTxt):
 		print "[Error] Could not find proper Font Translation Configuration info in: %s" % (pathToConfigureFontsTranslationTxt)
 		sys.exit(1)	# terminate if override Failed (Blade Runner)
 	#
-	# TODO ASDF fix this!!!
 	#
 	if(len(gListOfFontNamesToOutOfOrderGlyphs) == 0):
 		tmpFontType = DEFAULT_SUBTITLES_FONT_NAME[:-4] # remove the .FON extensionFromTheName
@@ -537,18 +536,16 @@ def outputMIX():
 #
 #
 
-#def inputXLS(filename)
-	#TODO extra pass once the quotes have been updated for weird unicode characters
-	#TODO some ' quotes appear as \u2019 and others appear normally as '. what's that about?
-	#DONE manually I've replaced all weird \u2019 single quotes with '''
-	#the Spanish n is \xf1 -> we put it at ASCII value: \xA5 -- font index 0xA6 ?
-	#the Spanish i is \xed -> we put it at ASCII value: \xA2 -- font index 0xA3 ?
-	#pâté
+	# TODO some ' (single quote characters) appear as \u2019 and others appear normally as '. Probably due to Excel auto-converting them.
+	# DONE manually I've replaced all weird \u2019 single quotes with '''
+	# the Spanish n is \xf1 -> we put it at ASCII value: \xA5 -- font index 0xA6 ?
+	# the Spanish i is \xed -> we put it at ASCII value: \xA2 -- font index 0xA3 ?
+	# pâté
 	#	 a actual ASCII value is 0xE2 in codepage 1252 -- put it in ASCII value 0xA6 (165) -- font index 0xA7
 	#	 e actual ASCII value is 0xE9 in codepage 1252 -- put it in ASCII value 0xA7 (166) -- font index 0xA8
-	#TODO what are other characters are special?
-	#TODO transition to ASCII chars to store in TRE file?
-	#DONE manually I've replaced all one-char '...' with three dots
+	# TODO what are other characters are special?
+	# TODO transition to ASCII chars to store in TRE file?
+	# DONE manually I've replaced all one-char '...' with three dots
 	# TODO actors TRE has 0x49 entries, (73 names), but table of ids has 73 entries BUT the offset table (first offset is calced + 0x04, so from end of the first 4 count bytes) has 74 entries. The last entry indexes the end of file (!)
 	# TODO all strings are NULL terminated in the TRE file!
 
@@ -649,23 +646,42 @@ def translateQuoteToAsciiProper(cellObj, pSheetName):
 	return newQuoteReplaceSpecialsRetStr
 	#return newQuoteReplaceSpecialsEnStr
 
+# aux function to validate an int cell value
+def parseIntCellValue(cell_obj_value, row_idx, col_idx, xl_sheet_name, xl_sheet_nrows, xl_sheet_ncols):
+	retCandValue = 0
+	try:
+		retCandValue =	int(cell_obj_value)
+	except Exception as e:
+		print "\n[Error] Invalid cell value at row: %d, column: %d in sheet: %s (rows: %d, columns: %d)" %(row_idx+1, col_idx+1, xl_sheet_name, xl_sheet_nrows, xl_sheet_ncols)
+		print "\tPlease check for redundant empty rows or invalid cell values.\n"
+		sys.exit()
+	return retCandValue
+
 
-def inputXLS(filename):
+def inputXLS(pathtoInputExcelFilename):
 	global gNumOfSpokenQuotes
 	global gTableOfStringIds
 	global gTableOfStringOffsets
 	global gTableOfStringEntries
 	# Open the workbook
 	xl_workbook = None
+	
+	pathToExcelFilenameNoExt, excelFileExtension = os.path.splitext(pathtoInputExcelFilename)
+	# Check if no extension or invalid extension
+	if excelFileExtension is None \
+			and not (excelFileExtension.lower() == "xls" or excelFileExtension.lower() == "xlsx"):
+		print "[Error] Bad file extension found for the Excel input file %s. Supported extensions are '.xls' and 'xlsx'" % (excelFileExtension, pathtoInputExcelFilename)
+		sys.exit(1) # Terminate if the input Excel had unsupported extension (or none)
+
 	try:
-		xl_workbook = xlrd.open_workbook(filename, encoding_override="utf-8")
+		xl_workbook = xlrd.open_workbook(pathtoInputExcelFilename, encoding_override="utf-8")
 		if xl_workbook is not None:
-			print "[Info] Opened Excel input file: %s" % (filename)
+			print "[Info] Opened Excel input file: %s" % (pathtoInputExcelFilename)
 	except Exception as e:
-		print '[Debug] Could not open the Excel input file::'  + str(e)
+		print '[Error] Could not open the Excel input file: '  + str(e)
 
 	if xl_workbook is None:
-		print '[Info] Could not open the Excel input file: %s' % (filename)
+		print '[Error] Giving up. Could not open the specified Excel input file.'
 		sys.exit(1) # Terminate if the input Excel was not found
 	# List sheet names, and pull a sheet by name
 	#
@@ -755,7 +771,9 @@ def inputXLS(filename):
 							if len(twoTokensfirstColSplitAtDotXLS) == 2:
 								twoTokensfirstColSplitAtDashXLS = twoTokensfirstColSplitAtDotXLS[0].split('-', 1)
 								if len(twoTokensfirstColSplitAtDashXLS) == 2:
-									tmpQuoteID = int( twoTokensfirstColSplitAtDashXLS[0]) * 10000 + int(twoTokensfirstColSplitAtDashXLS[1])
+									tmpActorPart = parseIntCellValue(twoTokensfirstColSplitAtDashXLS[0], row_idx, col_idx, xl_sheet.name, xl_sheet.nrows, xl_sheet.ncols)
+									tmpSubQuotePart = parseIntCellValue(twoTokensfirstColSplitAtDashXLS[1], row_idx, col_idx, xl_sheet.name, xl_sheet.nrows, xl_sheet.ncols)
+									tmpQuoteID = tmpActorPart * 10000 + tmpSubQuotePart
 									#if gTraceModeEnabled:
 									#	print ('[Debug] Row_idx: %d. Tag: %s, QuoteId: [%d]' % (row_idx, twoTokensfirstColSplitAtDotXLS[0], tmpQuoteID))
 									gTableOfStringIds.append(tmpQuoteID)
@@ -789,16 +807,16 @@ def inputXLS(filename):
 					# FOR VQAs -- Iterate through columns starting from col 2. We need cols: 2, 9, 10
 					#
 					elif mode == 2:
-						if(col_idx == 2): # subtitle text
+						if (col_idx == 2): # subtitle text
 							newQuoteReplaceSpecialsAscii = translateQuoteToAsciiProper(cell_obj, xl_sheet.name)
 							#print ('[Debug] length: %d: %s' % (len(newQuoteReplaceSpecialsAscii), newQuoteReplaceSpecialsAscii))
 							#print ':'.join(x.encode('hex') for x in newQuoteReplaceSpecialsAscii)	# seems to work.  new chars are non-printable but exist in string
 							# don't append to gTableOfStringEntries yet
-						elif(col_idx == 9): # startFrame
+						elif (col_idx == 9): # startFrame
 							#print "[Debug] cell: %s" % (cell_obj.value)
-							tmpStartFrame =	 int(cell_obj.value)
-						elif(col_idx == 10): # endFrame
-							tmpEndFrame = int(cell_obj.value)
+							tmpStartFrame = parseIntCellValue(cell_obj.value, row_idx, col_idx, xl_sheet.name, xl_sheet.nrows, xl_sheet.ncols)
+						elif (col_idx == 10): # endFrame
+							tmpEndFrame = parseIntCellValue(cell_obj.value, row_idx, col_idx, xl_sheet.name, xl_sheet.nrows, xl_sheet.ncols)
 							tmpQuoteID = tmpStartFrame | (tmpEndFrame << 16) # top 16 bits are end frame (up to 65536 frames which is enough) and low 16 bits are startFrame
 
 							gTableOfStringIds.append(tmpQuoteID)
@@ -816,7 +834,7 @@ def inputXLS(filename):
 					elif mode == 3:
 					   #print ('[Debug] Column: [%s] cell_obj: [%s]' % (col_idx, cell_obj))
 						if(col_idx == 0):
-							tmpQuoteID = int(cell_obj.value)
+							tmpQuoteID = parseIntCellValue(cell_obj.value, row_idx, col_idx, xl_sheet.name, xl_sheet.nrows, xl_sheet.ncols)
 							gTableOfStringIds.append(tmpQuoteID)
 						elif(col_idx == 1) :
 							#if switchFlagShowQuote == True:
@@ -925,11 +943,11 @@ def main(argsCL):
 			print "Always keep backups!"
 			print "--------------------"
 			print "Preparatory steps:"
-			print "1. Copy the transcript Excel file (eg. BladeRunnerPCTLK.xls, latest version, downloaded from Google Sheets) in some folder on your PC."
+			print "1. Copy the transcript Excel file (eg. BladeRunnerPCTLK.xlsx, latest version, downloaded from Google Sheets) in some folder on your PC."
 			print "--------------------"
 			print "%s takes 1 mandatory argument:" % (APP_WRAPPER_NAME)
 			print "Valid syntax: "
-			print "%s -x path_to_BladeRunnerPCTLK_xls [-ian path_to_actornames_txt] [-cft path_to_configureFontsTranslation_txt] [-ld gameInputLanguageDescription] [--trace]" % (APP_WRAPPER_NAME)
+			print "%s -x path_to_BladeRunnerPCTLK_xlsx [-ian path_to_actornames_txt] [-cft path_to_configureFontsTranslation_txt] [-ld gameInputLanguageDescription] [--trace]" % (APP_WRAPPER_NAME)
 			print "-x is followed by the path to the excel file with the subtitle quotes."
 			print "-ian is followed by the path to actornames.txt, if it's not in the current working directory."
 			print "-cft is followed by the path to configureFontsTranslation.txt, if it's not in the current working directory."
@@ -1008,7 +1026,7 @@ def main(argsCL):
 	if invalidSyntax == True:
 		print "[Error] Invalid syntax\n Try: \n %s --help for more info \n %s --version for version info " % (APP_WRAPPER_NAME, APP_WRAPPER_NAME)
 		print "Valid syntax: "
-		print "%s -x path_to_BladeRunnerPCTLK_xls [-ian path_to_actornames_txt] [-cft path_to_configureFontsTranslation_txt] [-ld gameInputLanguageDescription] [--trace]" % (APP_WRAPPER_NAME)
+		print "%s -x path_to_BladeRunnerPCTLK_xlsx [-ian path_to_actornames_txt] [-cft path_to_configureFontsTranslation_txt] [-ld gameInputLanguageDescription] [--trace]" % (APP_WRAPPER_NAME)
 		print "\nDetected arguments:"
 		tmpi = 0
 		for tmpArg in argsCL:
diff --git a/devtools/create_bladerunner/subtitles/module.mk b/devtools/create_bladerunner/subtitles/module.mk
index b882b098e0..acf8568fa0 100644
--- a/devtools/create_bladerunner/subtitles/module.mk
+++ b/devtools/create_bladerunner/subtitles/module.mk
@@ -10,6 +10,10 @@ BLADERUNNER_SUBTITLES_SAMPLE_INPUT_FOLDER := $(srcdir)/devtools/create_bladerunn
 INTERMEDIATE_RESOURCE_FILES_UI := "OPTIONS.TRE" "DLGMENU.TRE" "SCORERS.TRE" "VK.TRE" "CLUES.TRE" "CRIMES.TRE" "ACTORS.TRE" "HELP.TRE" "AUTOSAVE.TRE" "ERRORMSG.TRE" "SPINDEST.TRE" "KIA.TRE" "KIACRED.TRE" "CLUETYPE.TRE" "ENDCRED.TRE" "POGO.TRE"
 INTERMEDIATE_RESOURCE_FILES_SUBS := "INGQUO_E.TRE" "WSTLGO_E.TRE" "BRLOGO_E.TRE" "INTRO_E.TRE" "MW_A_E.TRE" "MW_B01_E.TRE" "MW_B02_E.TRE" "MW_B03_E.TRE" "MW_B04_E.TRE" "MW_B05_E.TRE" "INTRGT_E.TRE" "MW_D_E.TRE" "MW_C01_E.TRE" "MW_C02_E.TRE" "MW_C03_E.TRE" "END04A_E.TRE" "END04B_E.TRE" "END04C_E.TRE" "END06_E.TRE" "END01A_E.TRE" "END01B_E.TRE" "END01C_E.TRE" "END01D_E.TRE" "END01E_E.TRE" "END01F_E.TRE" "END03_E.TRE"
 INPUT_TRANSCRIPT_FILENAME := englishTranscript.xls
+ifeq (,$(wildcard $(INPUT_TRANSCRIPT_FILENAME)))
+	INPUT_TRANSCRIPT_FILENAME := englishTranscript.xlsx
+endif
+
 INPUT_TRANSCRIPT_AUX_CONF_FILENAME := configureFontsTranslation.txt
 INPUT_FONT_GLYPHS_PNG_FILENAME := subtitlesFont.png
 INPUT_FONT_GLYPHS_PNG_AUX_CONF_FILENAME := overrideEncodingSUBLTS.txt
@@ -43,7 +47,7 @@ $(TOOL_OUTPUT): $(FONT_OUTPUT) $(BLADERUNNER_SUBTITLES_SAMPLE_INPUT_FOLDER)/$(IN
 	$(info This process assumes that the folder: )
 	$(info $(BLADERUNNER_SUBTITLES_SAMPLE_INPUT_FOLDER))
 	$(info contains: )
-	$(info *. $(INPUT_TRANSCRIPT_FILENAME) - an XLS (Excel) input file with the transcript)
+	$(info *. $(INPUT_TRANSCRIPT_FILENAME) - an XLS(X) (Excel) input file with the transcript)
 	$(info *. $(INPUT_TRANSCRIPT_AUX_CONF_FILENAME) - a TXT (text) input file with configuration settings for the transcript processing)
 	$(info If successful, a $(TOOL_OUTPUT) file will be created in your working directory)
 	$(info Please, copy this $(TOOL_OUTPUT) into your Blade Runner game directory!)