Create Canon DSLR CR2 image statistics (exiftool & GNU Plot) and convert ML dual ISO CR2 files painlessly for post-processing
You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.

1498 lines
47 KiB

6 years ago
6 years ago
6 years ago
6 years ago
6 years ago
6 years ago
6 years ago
6 years ago
6 years ago
6 years ago
6 years ago
6 years ago
6 years ago
6 years ago
6 years ago
6 years ago
6 years ago
6 years ago
6 years ago
6 years ago
6 years ago
6 years ago
6 years ago
6 years ago
6 years ago
6 years ago
6 years ago
6 years ago
6 years ago
6 years ago
6 years ago
6 years ago
6 years ago
6 years ago
6 years ago
6 years ago
6 years ago
6 years ago
6 years ago
6 years ago
6 years ago
6 years ago
6 years ago
6 years ago
6 years ago
6 years ago
6 years ago
6 years ago
6 years ago
6 years ago
6 years ago
6 years ago
6 years ago
6 years ago
6 years ago
6 years ago
6 years ago
6 years ago
6 years ago
6 years ago
6 years ago
6 years ago
6 years ago
6 years ago
6 years ago
6 years ago
6 years ago
6 years ago
6 years ago
6 years ago
6 years ago
6 years ago
6 years ago
6 years ago
6 years ago
6 years ago
6 years ago
6 years ago
6 years ago
6 years ago
6 years ago
6 years ago
6 years ago
6 years ago
6 years ago
6 years ago
6 years ago
6 years ago
6 years ago
6 years ago
6 years ago
6 years ago
6 years ago
6 years ago
6 years ago
6 years ago
6 years ago
6 years ago
6 years ago
6 years ago
6 years ago
6 years ago
6 years ago
6 years ago
6 years ago
6 years ago
6 years ago
6 years ago
6 years ago
6 years ago
6 years ago
6 years ago
6 years ago
6 years ago
6 years ago
6 years ago
6 years ago
6 years ago
6 years ago
6 years ago
6 years ago
6 years ago
6 years ago
6 years ago
6 years ago
6 years ago
6 years ago
6 years ago
6 years ago
6 years ago
6 years ago
6 years ago
6 years ago
6 years ago
6 years ago
6 years ago
6 years ago
6 years ago
6 years ago
6 years ago
6 years ago
6 years ago
6 years ago
6 years ago
6 years ago
6 years ago
6 years ago
6 years ago
6 years ago
6 years ago
6 years ago
6 years ago
  1. #!/bin/env bash
  2. # Statistics of camera RAW images with GNU Plot & Exiftool
  3. # Copyright (C) 2017,2023 Pekka Helenius
  4. #
  5. # This program is free software; you can redistribute it and/or
  6. # modify it under the terms of the GNU General Public License
  7. # as published by the Free Software Foundation; either version 2
  8. # of the License, or (at your option) any later version.
  9. #
  10. # This program is distributed in the hope that it will be useful,
  11. # but WITHOUT ANY WARRANTY; without even the implied warranty of
  12. # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
  13. # GNU General Public License for more details.
  14. #
  15. # You should have received a copy of the GNU General Public License
  16. # along with this program; if not, write to the Free Software
  17. # Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA.
  18. #
  19. ###############################################
  20. # TODO list:
  21. # TODO Add message, either of these:
  22. # If not a geotagged image file:
  23. # A You haven't geotagged selected photos. Are you sure you want to continue?
  24. # B Following photos don't have altitude value: XX XX XX Are you sure you want to continue?
  25. # TODO IF INPUT IS A CSV FILE, DO NOT SUGGEST 'EXPORT ONLY' AS AN OPTION
  26. # TODO if user chooses alternative 'Export CSV' only, do not crash if errtags or errnames encountered
  27. #
  28. # TODO Be more specific in which case errtags may crash the operation. For instance, if user has chosen only Temperature and ISO to be plotted on GNU Plot, do not crash in case where Focal Length is missing. In other words: be more specific in which conditions match the crash requirement, do not crash the process for nothing.
  29. # TODO The script has only been tested on Canon 5D Mark 3 CR2 files. How does the script work with other Canon camera models? And what about Sony, Nikon?
  30. #
  31. # TODO Re-write, parse and simplify some scripts logics!
  32. ###############################################
  33. # Supported & tested camera raw pictures:
  34. # Canon EOS 5D Mark 3
  35. # TOOL REQUIREMENTS
  36. # perl-exiftool
  37. # kdialog (Qt5)
  38. # qt5-tools (qdbus)
  39. # coreutils (md5sum, echo etc..)
  40. # netpbm (pgmhist)
  41. # dcraw
  42. # gawk
  43. # gnuplot (+ Qt interface)
  44. #
  45. #
  46. # OTHER REQUIREMENTS
  47. #
  48. # Recommended: Qt5 Dolphin file manager (dolphin)
  49. #
  50. # This script is meant to be run as a Dolphin File Manager script (Qt5). It gets file arguments straight from Dolphin
  51. # You can run the script in bash (or execute Dolphin via bash) to get detailed script output (CLI messages, debug messages etc.)
  52. #
  53. #######################################################################
  54. CSV_FILECOUNT=$(printf '%s\n' "${@}" | rev | cut -f 1 -d '.' | rev | grep -i "csv" | wc -l)
  55. RAW_FILECOUNT=$(printf '%s\n' "${@}" | rev | cut -f 1 -d '.' | rev | grep -i -E "cr2|nef|dng" | wc -l)
  56. # Count of all columns must be found in CSV file.
  57. # This number must match with the number found in kdialog processing dialog below.
  58. COLUMNCOUNT=24
  59. #######################################################################
  60. # The first file check
  61. if [[ "${@}" == "" ]]
  62. then
  63. kdialog --error "Not any files selected!";
  64. exit 1
  65. elif \
  66. [[ ! "${CSV_FILECOUNT}" -eq 0 ]] && \
  67. [[ ! "${RAW_FILECOUNT}" -eq 0 ]]
  68. then
  69. kdialog --error "Select only RAW files or a single CSV file!";
  70. exit 1
  71. elif [[ "${CSV_FILECOUNT}" -gt 1 ]]
  72. then
  73. kdialog --error "Select only one CSV file!";
  74. exit 1
  75. elif [[ "${RAW_FILECOUNT}" -eq 1 ]]
  76. then
  77. kdialog --error "Please select at least 2 valid RAW files or a CSV file!";
  78. exit 1
  79. elif \
  80. [[ "${RAW_FILECOUNT}" -eq 0 ]] && \
  81. [[ "${CSV_FILECOUNT}" -eq 0 ]]
  82. then
  83. kdialog --error "Please select valid RAW files or a CSV file!";
  84. exit 1
  85. fi
  86. #######################################################################
  87. # KDialog check list formatted selection window
  88. if \
  89. [[ $RAW_FILECOUNT == 0 ]] && \
  90. [[ $CSV_FILECOUNT == 1 ]]
  91. then
  92. SELECTION=$(kdialog --checklist "Select statistics to display:" \
  93. 1 "Apertures, Exposures & ISOs" off \
  94. 2 "Focal Lengths & Lenses" off \
  95. 3 "Temperatures & ISOs" on \
  96. 4 "Shooting & Focus Modes" off \
  97. );
  98. # 1 Apertures, Exposures & ISOs
  99. # 2 Focal Lengths & Lenses
  100. # 3 Temperatures & ISOs
  101. # 4 Shooting & Focus Modes
  102. if [[ "$?" = 0 ]]; then
  103. if [[ $(expr length "$SELECTION") -ne 0 ]]
  104. then
  105. for result in $SELECTION
  106. do
  107. [[ "${result}" = '"1"' ]] && SEL1=true
  108. [[ "${result}" = '"2"' ]] && SEL2=true
  109. [[ "${result}" = '"3"' ]] && SEL3=true
  110. [[ "${result}" = '"4"' ]] && SEL4=true
  111. done
  112. else
  113. kdialog --sorry "Aborted";
  114. fi
  115. elif [[ "$?" -eq 1 ]]
  116. then
  117. exit 0
  118. else
  119. kdialog --error "Unexpected Error";
  120. fi
  121. SEL5=false
  122. fi
  123. if \
  124. [[ $RAW_FILECOUNT -ne 0 ]] && \
  125. [[ $CSV_FILECOUNT -eq 0 ]]
  126. then
  127. SELECTION=$(kdialog --checklist "Select statistics to display:" \
  128. 1 "Apertures, Exposures & ISOs" off \
  129. 2 "Focal Lengths & Lenses" off \
  130. 3 "Temperatures & ISOs" on \
  131. 4 "Shooting & Focus Modes" off \
  132. 5 "Export Only (CSV)" off \
  133. );
  134. # 1 Apertures, Exposures & ISOs
  135. # 2 Focal Lengths & Lenses
  136. # 3 Temperatures & ISOs
  137. # 4 Shooting & Focus Modes
  138. # 5 Export Only (CSV)
  139. if [[ "$?" = 0 ]]; then
  140. if [ $(expr length "$SELECTION") -ne 0 ]
  141. then
  142. for result in $SELECTION
  143. do
  144. [[ "${result}" = '"1"' ]] && SEL1=true
  145. [[ "${result}" = '"2"' ]] && SEL2=true
  146. [[ "${result}" = '"3"' ]] && SEL3=true
  147. [[ "${result}" = '"4"' ]] && SEL4=true
  148. # If checked, we force all other values to be false
  149. if [[ "${result}" = '"5"' ]]
  150. then
  151. SEL5=true
  152. SEL4=false
  153. SEL3=false
  154. SEL2=false
  155. SEL1=false
  156. fi
  157. done
  158. else
  159. kdialog --sorry "Aborted";
  160. fi
  161. elif [[ "$?" -eq 1 ]]
  162. then
  163. exit 0
  164. else
  165. kdialog --error "Unexpected Error";
  166. fi
  167. fi
  168. # SEL1 = "Apertures, Exposures & ISOs" true/false
  169. # SEL2 = "Focal Lengths & Lenses" true/false
  170. # SEL3 = "Temperatures & ISOs" true/false
  171. # SEL4 = "Shooting & Focus Modes" true/false
  172. # SEL5 = "Export Only (CSV)" true/false
  173. #######################################################################
  174. # We get the directory just from the first filename.
  175. INPUT_DIR=$(dirname "${1}")
  176. DIR_BASENAME=$(echo -n "${INPUT_DIR}" | rev | cut -d'/' -f 1 | rev)
  177. # First & Last file names (without suffixes).
  178. for last; do true; done
  179. # Name of the first file passed into the script.
  180. FIRST=$(basename "${1}" | cut -f 1 -d '.')
  181. # Name of the last file passed into the script.
  182. LAST=$(basename "${last}" | cut -f 1 -d '.')
  183. # File name is based on the folder where files exist
  184. FILENAME=$(echo "${DIR_BASENAME}-${FIRST}-${LAST}_metadata")
  185. FILE_EXT=.csv
  186. #######################################################################
  187. # 2ND FILE CHECK
  188. # Check if we are dealing with a CSV file or bunch of RAW files.
  189. if [[ "${CSV_FILECOUNT}" -eq 1 ]]; then
  190. # Without a suffix. We use valid existing CSV file name here.
  191. FILENAME=$(basename "${1}" | sed 's/\.\w*$//')
  192. CSVFOLDER="${INPUT_DIR}"
  193. elif [[ "${RAW_FILECOUNT}" -ne 0 ]]; then
  194. echo "Multiple RAW files."
  195. fi
  196. RAWDATA_TMP="${CSVFOLDER}/${FILENAME}${FILE_EXT}"
  197. #######################################################################
  198. # 1) Check CSV file validity against the script output.
  199. # 2) Get value for 'INPUT_FILES_MD5SUM' variable.
  200. # NOTE: We don't check MD5Sums, if we use CSV file as an input.
  201. # Though this file can exist in the same folder with the pictures,
  202. # we want to keep CSV files as portable as possible in general.
  203. # Thus, we don't do the following check: CSV file list MD5Sums
  204. # vs actual corresponding files in the folder (if only CSV is selected as input).
  205. # This can arise other problems such as images with equivalent names listed in
  206. # CSV file but they are actually different files.
  207. # This causes mismatch between CSV file content and folder content. So, no go.
  208. # 1) Check validity of the selected CSV file for analysis purposes. Not RAW files selected.
  209. # Referring to existing CSV file here. User input may or may not be a CSV file, so we don't check it.
  210. if [[ -e "${RAWDATA_TMP}" ]]
  211. then
  212. echo "This is a valid CSV file. Checking columns."
  213. # This *must* return only one value (equal to COLUMNCOUNT).
  214. # If many values are returned CSV file can't be used because,
  215. # therefore, there are mismatch between column numbers in rows.
  216. FILE_COLUMNCOUNT=$(echo -n $(awk -F ',' '{print NF}' "${RAWDATA_TMP}" | sort -nu))
  217. FILE_HASMD5COLUMN=$(awk -F ',' '{print $2}' "${RAWDATA_TMP}" | head -n 1)
  218. FILE_MD5_CHARNUM=$(echo -n $(awk -F ',' ' FNR > 1 {print length($2)}' "${RAWDATA_TMP}" | sort -nu)) #This *must* return only one value. Value 32.
  219. # If the input csv file has valid count of columns and the second column includes md5sums.
  220. if \
  221. [[ "${FILE_COLUMNCOUNT}" -eq "${COLUMNCOUNT}" ]] && \
  222. [[ "${FILE_HASMD5COLUMN}" == "File MD5Sum" ]] && \
  223. [[ "${FILE_MD5_CHARNUM}" -eq 32 ]]
  224. then
  225. COLUMNS_OK=true
  226. echo "Columns OK, continuing."
  227. elif [[ "${RAW_FILECOUNT}" -eq 0 ]]
  228. then
  229. echo -e "Charnum is: ${FILE_MD5_CHARNUM}"
  230. echo "Error in columns."
  231. kdialog --error "Error in CSV file columns!";
  232. exit 0
  233. else
  234. echo "Error in matching file columns. RAW files as input."
  235. # This is a case where we have detected a pattern matcing CSV file but it has invalid columns.
  236. COLUMNS_OK=false
  237. fi
  238. fi
  239. # 2) Instead of single CSV file, if multiple RAW files have been selected, then
  240. if [[ "${RAW_FILECOUNT}" -ne 0 ]]
  241. then
  242. echo "Getting MD5Sums for RAW files..."
  243. # Get md5sums for the files and print output.
  244. # Syntax: IMG_8217,IMG_8408,IMG_8544 ...
  245. # (replace these file names just with md5sums and you get the idea)
  246. INPUT_FILES_MD5SUM=$(
  247. echo -n $(printf '%s\n' $(md5sum "${@}") | \
  248. sed '$!N;s/\n/ /' | \
  249. awk -F ' ' '{print $2,$1}' | \
  250. sed -e 's/^.*\///' | \
  251. sort -n | \
  252. awk -F ' ' '{print $2}' | \
  253. tr '\n' ',' | \
  254. sed 's/,*\r*$//'
  255. )
  256. )
  257. echo "Comparing MD5Sums..."
  258. MAINCSV=$(find "${CSVFOLDER}" -maxdepth 1 -iname "${FILENAME}*${FILE_EXT}")
  259. MAINCSV_COUNT=$(find "${CSVFOLDER}" -maxdepth 1 -iname "${FILENAME}*${FILE_EXT}" | wc -l)
  260. OTHER_CSV=$(find "${CSVFOLDER}" -maxdepth 1 -iname "*${FILE_EXT}")
  261. OTHER_CSV_COUNT=$(find "${CSVFOLDER}" -maxdepth 1 -iname "*${FILE_EXT}" | wc -l)
  262. # Main CSV file
  263. if [[ "${COLUMNS_OK}" == true ]]; then
  264. COMPAREFILE_MD5SUM=$(
  265. echo -n $(
  266. awk -F ',' 'FNR> 1 {print $1,$2}' "${RAWDATA_TMP}" | \
  267. sort -n | \
  268. awk -F ' ' '{print $2}' | \
  269. tr '\n' ',' | \
  270. sed 's/,*\r*$//'
  271. )
  272. )
  273. # If md5sums match OK, then
  274. if [[ "${INPUT_FILES_MD5SUM}" == "${COMPAREFILE_MD5SUM}" ]]
  275. then
  276. echo -e "MD5Sums match OK."
  277. USEMAINCSV=true
  278. else
  279. echo -e "MD5Sums match not OK."
  280. USEMAINCSV=false
  281. fi
  282. fi
  283. # Other CSV files, including variant of the CSV "file template".
  284. if \
  285. [[ ! -e "${RAWDATA_TMP}" ]] || \
  286. [[ "${OTHER_CSV_COUNT}" -ne 0 ]] || \
  287. [[ "${USEMAINCSV}" == false ]]
  288. then
  289. # Check for CSV variants (which match the filename syntax).
  290. if [[ "${MAINCSV_COUNT}" -gt 0 ]]
  291. then
  292. for m in ${MAINCSV}
  293. do
  294. COMPAREFILE_MD5SUM=$(
  295. echo -n $(
  296. awk -F ',' 'FNR> 1 {print $1,$2}' "${m}" | \
  297. sort -n | \
  298. awk -F ' ' '{print $2}' | \
  299. tr '\n' ',' | \
  300. sed 's/,*\r*$//'
  301. )
  302. )
  303. if [[ "${INPUT_FILES_MD5SUM}" == "${COMPAREFILE_MD5SUM}" ]]
  304. then
  305. RAWDATA_TMP="${m}"
  306. # We get the existing file name template and remove extension.
  307. FILENAME=$(basename "${m}" | cut -f 1 -d '.')
  308. USEMAINCSV=true
  309. break
  310. else
  311. USEMAINCSV=false
  312. fi
  313. done
  314. fi
  315. # Check for other CSVs.
  316. if \
  317. [[ "${MAINCSV_COUNT}" -eq 0 ]] || \
  318. [[ "${OTHER_CSV_COUNT}" -ne 0 ]] && \
  319. [[ "${USEMAINCSV}" == false ]]
  320. then
  321. for f in ${OTHER_CSV}
  322. do
  323. COMPAREFILE_MD5SUM=$(
  324. echo -n $(
  325. awk -F ',' 'FNR> 1 {print $1,$2}' "${f}" | \
  326. sort -n | \
  327. awk -F ' ' '{print $2}' | \
  328. tr '\n' ',' | \
  329. sed 's/,*\r*$//'
  330. )
  331. )
  332. if [[ "${INPUT_FILES_MD5SUM}" == "${COMPAREFILE_MD5SUM}" ]]
  333. then
  334. RAWDATA_TMP="${f}"
  335. # We get the existing file name template and remove extension.
  336. FILENAME=$(basename "${f}" | cut -f 1 -d '.')
  337. USEOTHERCSV=true
  338. break
  339. else
  340. USEOTHERCSV=false
  341. fi
  342. done
  343. fi
  344. fi
  345. fi
  346. if \
  347. [[ ${USEMAINCSV} == false ]] && \
  348. [[ ${USEOTHERCSV} == false ]]
  349. then
  350. x=1
  351. while [[ -e "${CSVFOLDER}/${FILENAME}-${x}${FILE_EXT}" ]]
  352. do
  353. let x++
  354. done
  355. FILENAME="${FILENAME}-${x}"
  356. fi
  357. echo -e "MD5Sums checked.\n"
  358. if \
  359. [[ $USEMAINCSV == true ]] || \
  360. [[ $USEOTHERCSV == true ]]
  361. then
  362. echo -e "Found an existing CSV file with MD5Sums.\n"
  363. elif \
  364. [[ $USEMAINCSV == true ]] && \
  365. [[ $USEOTHERCSV == false ]]
  366. then
  367. echo -e "Using existing CSV with correct file template.\n"
  368. elif \
  369. [[ $USEMAINCSV == true ]] && \
  370. [[ $USEOTHERCSV == false ]]
  371. then
  372. echo -e "Using a custom named CSV file.\n"
  373. elif \
  374. [[ $USEMAINCSV == false ]] || \
  375. [[ $USEOTHERCSV == false ]]
  376. then
  377. echo -e "Creating a new CSV file.\n"
  378. fi
  379. #######################################################################
  380. # We need to redefine bash variables to overwrite the old values!
  381. FILENAME2="${FILENAME}-temp"
  382. FILENAME3="${FILENAME}-iso"
  383. RAWDATA_TMP="${CSVFOLDER}/${FILENAME}${FILE_EXT}"
  384. RAWDATA_TMP2="/tmp/${FILENAME2}${FILE_EXT}"
  385. RAWDATA_TMP3="/tmp/${FILENAME3}${FILE_EXT}"
  386. #################
  387. #DEBUGGING
  388. echo "We use file named $RAWDATA_TMP"
  389. if [[ "${SEL3}" == true ]]
  390. then
  391. echo "We use isofile named $RAWDATA_TMP2"
  392. echo "We use tempfile named $RAWDATA_TMP3"
  393. fi
  394. #######################################################################
  395. # PROGRESSBAR CODE - BEGIN
  396. LABELTEXT="Exporting statistics..."
  397. numargs=$# # Number of all files
  398. tics=100 # Percentage tics
  399. inc=0 # Current file number
  400. mltp=1000 # Percentage multiplier for bash
  401. # If the file already exists, we don't want overwrite it.
  402. # Instead, we skip these steps to speed up the process.
  403. if [[ ! -e "${RAWDATA_TMP}" ]]
  404. then
  405. dbusRef=$(kdialog --title "Metadata Extraction (${DIR_BASENAME}: images ${FIRST}-${LAST})" --progressbar "${LABELTEXT}" "${tics}")
  406. qdbus $dbusRef showCancelButton true
  407. # PROGRESSBAR CODE - END
  408. while \
  409. [[ $# -gt 0 ]] && \
  410. [[ $(qdbus $dbusRef wasCancelled) == "false" ]]
  411. do
  412. i="${1}"
  413. ##############################################
  414. # 1 COLUMN
  415. # ENABLE THIS IF STATEMENT ONLY IF FILE NAMES CONTAINING 'IMG_' ARE THE ONLY ONES ACCEPTED
  416. #if [[ ! $(echo $(basename "${1}" | cut -f 1 -d '.')) == *"IMG_"* ]]
  417. #then
  418. # echo $(basename "${1}" | cut -f 1 -d '.')
  419. # ERRFILE=1 #PRINT INVALID INPUT AS THE LAST COLUMN DUE TO DATE/TIME COLUMNS! SEE BELOW!
  420. #else
  421. echo $(basename "${i}" | cut -f 1 -d '.') #| sed -e 's/IMG_//g') #echo ${i##*/} | sed -e 's/.CR2//g' -e 's/.DNG//g')
  422. # ERRFILE=0
  423. #fi
  424. ##############################################
  425. # 2 COLUMN - md5sum
  426. # Write md5sum of a file for checking purposes!
  427. md5sum "${i}" | awk -F ' ' '{print $1}'
  428. ##############################################
  429. # 3 COLUMN - temperature
  430. if [[ $(exiftool "${i}" |grep --max-count=1 "Camera Temperature" | sed -e 's/[^0-9]*//g' | wc -l) -eq 1 ]]
  431. then
  432. exiftool "${i}" |grep --max-count=1 "Camera Temperature" | sed -e 's/[^0-9]*//g'
  433. else
  434. echo "errtag"
  435. fi
  436. ##############################################
  437. # 4 COLUMN - sensitivity
  438. # ISO Speed setting (yeah, we get it from "Recommended Exposure Index" tag.
  439. if [[ $(exiftool "${i}" | grep -v "Sensitivity" | grep --max-count=1 "Recommended Exposure Index" | sed 's/[^0-9]*//g' | wc -l) -eq 1 ]]
  440. then
  441. exiftool "${i}" | grep -v "Sensitivity" | grep --max-count=1 "Recommended Exposure Index" | sed 's/[^0-9]*//g'
  442. else
  443. echo "errtag"
  444. fi
  445. ##############################################
  446. # 5 COLUMN - exposure time
  447. if [[ $(exiftool "${i}" |grep --max-count=1 "Exposure Time" | sed -e 's/[A-Za-z]*//g' -e 's/.*: //g' | wc -l) -eq 1 ]]
  448. then
  449. exiftool "${i}" |grep --max-count=1 "Exposure Time" | sed -e 's/[A-Za-z]*//g' -e 's/.*: //g'
  450. else
  451. echo "errtag"
  452. fi
  453. ##############################################
  454. # 6 COLUMN - target exposure time
  455. if [[ $(exiftool "${i}" |grep --max-count=1 "Target Exposure Time" | sed -e 's/[A-Za-z]*//g' -e 's/.*: //g' | wc -l) -eq 1 ]]
  456. then
  457. exiftool "${i}" |grep --max-count=1 "Target Exposure Time" | sed -e 's/[A-Za-z]*//g' -e 's/.*: //g'
  458. else
  459. echo "errtag"
  460. fi
  461. ##############################################
  462. # 7 COLUMN - exposure compensation
  463. if [[ $(exiftool "${i}" |grep --max-count=1 "Exposure Compensation" | sed -e 's/[A-Za-z]*//g' -e 's/.*: //g' | wc -l) -eq 1 ]]
  464. then
  465. exiftool "${i}" |grep --max-count=1 "Exposure Compensation" | sed -e 's/[A-Za-z]*//g' -e 's/.*: //g'
  466. else
  467. echo "errtag"
  468. fi
  469. ##############################################
  470. # 8 COLUMN - aperture
  471. if [[ $(exiftool "${i}" |grep --max-count=1 "Aperture Value" | sed -e 's/[A-Za-z]*//g' -e 's/.*: //g' | wc -l) -eq 1 ]]
  472. then
  473. exiftool "${i}" |grep --max-count=1 "Aperture Value" | sed -e 's/[A-Za-z]*//g' -e 's/.*: //g'
  474. else
  475. echo "errtag"
  476. fi
  477. ##############################################
  478. # 9 COLUMN - target aperture
  479. if [[ $(exiftool "${i}" |grep --max-count=1 "Target Aperture" | sed -e 's/[A-Za-z]*//g' -e 's/.*: //g' | wc -l) -eq 1 ]]
  480. then
  481. exiftool "${i}" |grep --max-count=1 "Target Aperture" | sed -e 's/[A-Za-z]*//g' -e 's/.*: //g'
  482. else
  483. echo "errtag"
  484. fi
  485. ##############################################
  486. # 10 COLUMN
  487. # Average histogram value for image (brightness etc.)
  488. # For documentation, see http://netpbm.sourceforge.net/doc/pgmhist.html
  489. # we need to convert the image into grayscale with dcraw -d option
  490. # dcraw "manual" is found here: http://www.inweb.ch/foto/dcrawhelp.txt
  491. if [[ $(dcraw -d -4 -j -c "${i}" | pgmhist -median | wc -l) -eq 1 ]]
  492. then
  493. dcraw -d -4 -j -c "${i}" | pgmhist -median | sed 's/[^0-9]*//g'
  494. else
  495. echo "errtag"
  496. fi
  497. ##############################################
  498. # 11 COLUMN - focal length
  499. if [[ $(exiftool "${i}" |grep --max-count=1 "Focal Length" | sed -e 's/[A-Za-z]*//g' -e 's/.*: //g' -e 's/ //g' | wc -l) -eq 1 ]]
  500. then
  501. exiftool "${i}" |grep --max-count=1 "Focal Length" | sed -e 's/[A-Za-z]*//g' -e 's/.*: //g' -e 's/ //g'
  502. else
  503. echo "errtag"
  504. fi
  505. ##############################################
  506. # 12 COLUMN - hyperfocal distance
  507. if [[ $(exiftool "${i}" |grep --max-count=1 "Hyperfocal Distance" | sed -e 's/.*: //g' -e 's/ m//g' | wc -l) -eq 1 ]]
  508. then
  509. exiftool "${i}" |grep --max-count=1 "Hyperfocal Distance" | sed -e 's/.*: //g' -e 's/ m//g'
  510. else
  511. echo "errtag"
  512. fi
  513. ##############################################
  514. # 13 COLUMN - upper focus distance
  515. if [[ $(exiftool "${i}" |grep --max-count=1 "Focus Distance Upper" | sed -e 's/.*: //g' -e 's/ m//g' | wc -l) -eq 1 ]]
  516. then
  517. exiftool "${i}" |grep --max-count=1 "Focus Distance Upper" | sed -e 's/.*: //g' -e 's/ m//g'
  518. else
  519. echo "errtag"
  520. fi
  521. ##############################################
  522. # 14 COLUMN - lower focus distance
  523. if [[ $(exiftool "${i}" |grep --max-count=1 "Focus Distance Lower" | sed -e 's/.*: //g' -e 's/ m//g' | wc -l) -eq 1 ]]
  524. then
  525. exiftool "${i}" |grep --max-count=1 "Focus Distance Lower" | sed -e 's/.*: //g' -e 's/ m//g'
  526. else
  527. echo "errtag"
  528. fi
  529. ##############################################
  530. # 15 COLUMN - depth of field
  531. if [[ $(exiftool "${i}" |grep --max-count=1 "Depth Of Field" | sed -e 's/.*: //g' -e 's/ m//g' | wc -l) -eq 1 ]]
  532. then
  533. exiftool "${i}" |grep --max-count=1 "Depth Of Field" | sed -e 's/.*: //g' -e 's/ m//g'
  534. else
  535. echo "errtag"
  536. fi
  537. ##############################################
  538. # 16 COLUMN - camera model
  539. if [[ $(exiftool "${i}" |grep --max-count=1 "Camera Model Name" | sed 's/.*: //g' | wc -l) -eq 1 ]]
  540. then
  541. exiftool "${i}" |grep --max-count=1 "Camera Model Name" | sed 's/.*: //g'
  542. else
  543. echo "errtag"
  544. fi
  545. ##############################################
  546. # 17 COLUMN - lens type
  547. if [[ $(exiftool "${i}" |grep --max-count=1 "Lens Type" | sed 's/.*: //g' | wc -l) -eq 1 ]]
  548. then
  549. exiftool "${i}" |grep --max-count=1 "Lens Type" | sed 's/.*: //g'
  550. else
  551. echo "errtag"
  552. fi
  553. ##############################################
  554. # 18 COLUMN - focus mode
  555. if [[ $(exiftool "${i}" |grep --max-count=1 "Focus Mode" | sed 's/.*: //g' | wc -l) -eq 1 ]]
  556. then
  557. exiftool "${i}" |grep --max-count=1 "Focus Mode" | sed 's/.*: //g'
  558. else
  559. echo "errtag"
  560. fi
  561. ##############################################
  562. # 19 COLUMN - shooting mode
  563. if [[ $(exiftool "${i}" |grep --max-count=1 "Shooting Mode" | sed 's/.*: //g' | wc -l) -eq 1 ]]
  564. then
  565. exiftool "${i}" |grep --max-count=1 "Shooting Mode" | sed 's/.*: //g'
  566. else
  567. echo "errtag"
  568. fi
  569. ##############################################
  570. # 20 COLUMN - live view shooting
  571. if [[ $(exiftool "${i}" |grep --max-count=1 "Live View Shooting" | sed 's/.*: //g' | wc -l) -eq 1 ]]
  572. then
  573. exiftool "${i}" |grep --max-count=1 "Live View Shooting" | sed 's/.*: //g'
  574. else
  575. echo "errtag"
  576. fi
  577. ##############################################
  578. # 21 COLUMN - camera orientation
  579. if [[ $(exiftool "${i}" |grep --max-count=1 "Camera Orientation" | sed 's/.*: //g' | wc -l) -eq 1 ]]
  580. then
  581. exiftool "${i}" |grep --max-count=1 "Camera Orientation" | sed 's/.*: //g'
  582. else
  583. echo "errtag"
  584. fi
  585. ##############################################
  586. # 21 COLUMN - subject
  587. if [[ $(exiftool "${i}" |grep --max-count=1 "Subject" | sed -e 's/.*: //g' | wc -l) -eq 0 ]]
  588. then
  589. # If Subject tag is empty, get input file filetype (CR2 or cr2 // DNG or dng)
  590. if [[ $(echo $(basename "${i}" | cut -f 2 -d '.' | sed -e '/^\s*$/d' -e 's/\(.*\)/\U\1/')) == "CR2" ]]
  591. then
  592. echo "Single ISO CR2"
  593. elif [[ $(echo $(basename "${i}" | cut -f 2 -d '.' | sed -e '/^\s*$/d' -e 's/\(.*\)/\U\1/')) == "DNG" ]]
  594. then
  595. echo "Single ISO DNG"
  596. fi
  597. # If we have a real Subject tag, extract info from it
  598. else
  599. exiftool "${i}" |grep --max-count=1 "Subject" | sed -e 's/.*: //g'
  600. fi
  601. ##############################################
  602. # 23 COLUMN - datetime original
  603. if [[ $(exiftool "${i}" |grep --max-count=1 "Date/Time Original" | sed -e 's/.*: //g' | cut -d' ' -f1 | awk -F ":" '{print $3, $2, $1}' | sed -e 's/ /\//g' | wc -l) -eq 1 ]]
  604. then
  605. exiftool "${i}" |grep --max-count=1 "Date/Time Original" | sed -e 's/.*: //g' | cut -d' ' -f1 | awk -F ":" '{print $3, $2, $1}' | sed -e 's/ /\//g'
  606. else
  607. echo "errtag"
  608. fi
  609. ##############################################
  610. # 24 COLUMN - datetime original
  611. if [[ $(exiftool "${i}" |grep --max-count=1 "Date/Time Original" | sed -e 's/.*: //g' | cut -d' ' -f2 | wc -l) -eq 1 ]]
  612. then
  613. exiftool "${i}" |grep --max-count=1 "Date/Time Original" | sed -e 's/.*: //g' | cut -d' ' -f2
  614. else
  615. echo "errtag"
  616. fi
  617. ##############################################
  618. # ENABLE THIS FOR CHECKING FILE NAMES. IF ENABLED, ALL FILES MUST CONTAIN 'IMG_' STRING OR OTHERWISE, THEY ARE EXCLUDED FROM THE STATISTICS!
  619. # Write this line/column only, if an invalid file name has been detected
  620. # if [[ "$ERRFILE" -eq 1 ]]; then
  621. # echo "errname"
  622. # fi
  623. ##############################################
  624. # This is written just as a dummy separator for each processed files for further line separation done below.
  625. echo "newline"
  626. ##############################################
  627. # PROGRESSBAR CODE - BEGIN
  628. # This section is increasing values seen in the kdialog processing window.
  629. let inc++
  630. #Percentage needs to be calculated like this due to bash rounding limitations...
  631. PERCENT_VALUE=$(((${mltp}*${tics})/(200*${numargs}/${inc} % 2 + ${mltp}*${numargs}/${inc})))
  632. qdbus $dbusRef Set "" "value" "${PERCENT_VALUE}";
  633. qdbus $dbusRef setLabelText "${LABELTEXT} (${inc}/${numargs})";
  634. # PROGRESSBAR CODE - END
  635. shift
  636. ##############################################
  637. # Sort output:
  638. # replace newlines with commas
  639. # remove more commas
  640. # replace 'newline' with a new line
  641. # trim the first and the last commas of any line
  642. # sort lines by Date & Time (23th & 24th column)
  643. # write output
  644. done | \
  645. tr '\n' ',' | \
  646. sed -e 's/,[^,]*$//' -e 's/newline/\n/g' | \
  647. sed -e 's/^,//g' -e 's/\(.*\),/\1/' | \
  648. sort -t ',' -n -k23 -k24 | \
  649. sed 's/inf/∞/g' \
  650. > "${RAWDATA_TMP}"
  651. #| sed 's/newline/\n/g' | sort -u
  652. ##############################################
  653. # Close processing window if cancelled event has been triggered.
  654. # PROGRESSBAR CODE - BEGIN
  655. # If the process was cancelled, remove tmp file and exit the script.
  656. if [[ ! $(qdbus $dbusRef wasCancelled) == "false" ]]; then
  657. # We can delete the file because its existence has been checked before the processing
  658. # window has been opened. Thus we don't get here, if the file already exists.
  659. rm -f "${RAWDATA_TMP}"
  660. exit 0
  661. fi
  662. # PROGRESSBAR CODE - END
  663. ###########################################
  664. #Add correct titles to the first row in RAWDATA_TMP file:
  665. sed -i '1s/^/Image File,File MD5Sum,Camera Temperature,ISO,Shutter Speed,Target Exposure,Exposure Compensation,Aperture,Target Aperture,Histogram Median,Focal Length,Hyperfocal Distance,Upper Focus Distance,Lower Focus Distance,Depth Of Field,Camera Model,Lens Model,Focus Mode,Exposure Mode,Live View,Camera Orientation,ISO Type,Date,Time\n/' "${RAWDATA_TMP}"
  666. #Close processing window if not cancelled and processing finished.
  667. # PROGRESSBAR CODE - BEGIN
  668. qdbus $dbusRef close
  669. # PROGRESSBAR CODE - END
  670. # SEL1 = "Apertures, Exposures & ISOs" true/false
  671. # SEL2 = "Focal Lengths & Lenses" true/false
  672. # SEL3 = "Temperatures & ISOs" true/false
  673. # SEL4 = "Shooting & Focus Modes" true/false
  674. # SEL5 = "Export Only (CSV)" true/false
  675. if [[ "${SEL5}" == true ]]
  676. then
  677. kdialog --msgbox "EXIF data exported successfully";
  678. exit 0
  679. fi
  680. elif \
  681. [[ -e "${RAWDATA_TMP}" ]] && \
  682. [[ "${SEL5}" == true ]]
  683. then
  684. kdialog --msgbox "EXIF data exported already.\n\nFile:\n\n${RAWDATA_TMP}";
  685. exit 0
  686. fi
  687. # KDIALOG PROCESSING WINDOW - END
  688. ##############################################
  689. # Check RAWDATA_TMP for bad line outputs
  690. # 1) BADFILES: Open written (or existing) CSV file
  691. # 2) BADFILES: List all lines matching pattern "errname" or "errtag"
  692. # 3) BADFILES: Write output as a single line, using comma mark to separate the written output (file names).
  693. # Remove the last extra comma.
  694. BADFILES=$(cat "${RAWDATA_TMP}" | sed -ne '/errname/p' -ne '/errtag/p' | sed 's/,.*$//')
  695. BADFILES_COUNT=$(cat "${RAWDATA_TMP}" | sed -ne '/errname/p' -ne '/errtag/p' | sed 's/,.*$//' | wc -l)
  696. #Count lines found in the output of BADFILES. If not zero (e.g. bad strings found), then
  697. if [[ "${BADFILES_COUNT}" -ne 0 ]]
  698. then
  699. cat "${RAWDATA_TMP}" | sed -e '/errname/d' -e '/errtag/d' > "/tmp/${FILENAME}-errtags${FILE_EXT}"
  700. # We don't want to overwrite the original file.
  701. RAWDATA_TMP_ERR="/tmp/${FILENAME}-errtags${FILE_EXT}"
  702. # If not any valid output image files. Minimum count of lines is 2.
  703. if [[ $(cat "${RAWDATA_TMP_ERR}" | wc -l) == 1 ]]
  704. then
  705. kdialog --error "Could not process any input file:\n\n${BADFILES}\n\nThis can be due to missing EXIF data such as Temperature, ISO, Date or Time.\n\nPlease check CSV file (${FILENAME}${FILE_EXT}) contents to study the problem.\n\nExiting.";
  706. rm "${RAWDATA_TMP_ERR}"
  707. exit 1
  708. # If we have just a single file here. Minimum count of lines is 3.
  709. elif [[ $(cat "${RAWDATA_TMP_ERR}" | wc -l) -le 2 ]]
  710. then
  711. kdialog --error "Could not process a valid number of input files. Minimum count of valid files is 2.\n\nFiles that could not be processed:\n\n$BADFILES\n\nThis can be due to missing EXIF data such as Temperature, ISO, Date or Time.\n\nPlease check CSV file (${FILENAME}${FILE_EXT}) contents to study the problem.\n\nExiting."
  712. rm "${RAWDATA_TMP_ERR}"
  713. exit 1
  714. else
  715. mv "/tmp/${FILENAME}-errtags${FILE_EXT}" "${CSVFOLDER}/${FILENAME}-errtags${FILE_EXT}"
  716. RAWDATA_TMP="${CSVFOLDER}/${FILENAME}-errtags${FILE_EXT}"
  717. kdialog --msgbox "Could not process files:\n\n${BADFILES}\n\nThis can be due to missing EXIF data such as Temperature, ISO, Date or Time.\n\nPlease exclude these files or check CSV file (${FILENAME}${FILE_EXT}) contents to study the problem.\n\nNew CSV file written as (bad files excluded):\n${RAWDATA_TMP}";
  718. fi
  719. fi
  720. ######################################################################################
  721. FILELIST=$(
  722. echo -n $(
  723. awk -F ',' 'FNR> 1 {print $1}' "${RAWDATA_TMP}" | \
  724. sort -n | \
  725. tr ' ' '\n' | \
  726. sort -n | \
  727. tr '\n' ',' | \
  728. sed 's/,*\r*$//'
  729. )
  730. )
  731. #Total count of accepted pictures, used for further data representation in gnuplot.
  732. # We reduce it by 1 due to file header (column titles are not counted):
  733. ACCEPTED_TOTAL=$(echo -n $(($(cat "${RAWDATA_TMP}" | wc -l) - 1)))
  734. ######################################################################################
  735. # GNUPLOT CODE - BEGIN
  736. GNUPLOT_MAINWINDOW_TITLE=$(echo "${DIR_BASENAME} (${ACCEPTED_TOTAL} images, ${FILENAME})")
  737. ###########################################################
  738. # GNUPlot time values
  739. # PLOT 1
  740. # Should we use time values in the first plot?
  741. # If too many images, plot is basically rendered unreadable.
  742. # Rotate x labels if there are too many items to be shown.
  743. # Do not rotate x labels, if we have less than 10 images selected.
  744. if [[ $ACCEPTED_TOTAL -lt 10 ]]
  745. then
  746. X_ROTATELABELS=$(echo -n "")
  747. # Time values are only if max 6 images selected.
  748. if [[ $ACCEPTED_TOTAL -le 6 ]]
  749. then
  750. X2_TIMESTRINGS=$(echo -n "set x2tics offset 0,-0.5")
  751. else
  752. X2_TIMESTRINGS=$(echo -n "unset x2tics")
  753. fi
  754. else
  755. X_ROTATELABELS=$(echo -n "set xtics rotate 90")
  756. fi
  757. ############################################################
  758. # Image count/unit scales for GNUPlot plots 2 & 3
  759. if [[ "${ACCEPTED_TOTAL}" -le 10 ]]; then
  760. SCALE=1
  761. elif [[ "${ACCEPTED_TOTAL}" -gt 10 ]] && [[ "${ACCEPTED_TOTAL}" -le 20 ]]; then
  762. SCALE=2
  763. elif [[ "${ACCEPTED_TOTAL}" -gt 20 ]] && [[ "${ACCEPTED_TOTAL}" -le 40 ]]; then
  764. SCALE=4
  765. elif [[ "${ACCEPTED_TOTAL}" -gt 40 ]] && [[ "${ACCEPTED_TOTAL}" -le 60 ]]; then
  766. SCALE=6
  767. elif [[ "${ACCEPTED_TOTAL}" -gt 60 ]] && [[ "${ACCEPTED_TOTAL}" -le 80 ]]; then
  768. SCALE=8
  769. elif [[ "${ACCEPTED_TOTAL}" -gt 80 ]] && [[ "${ACCEPTED_TOTAL}" -le 200 ]]; then
  770. SCALE=10
  771. elif [[ "${ACCEPTED_TOTAL}" -gt 200 ]] && [[ "${ACCEPTED_TOTAL}" -le 400 ]]; then
  772. SCALE=20
  773. elif [[ "${ACCEPTED_TOTAL}" -ge 400 ]]; then
  774. SCALE=40
  775. fi
  776. ############################################################
  777. # GNUPlot ISO values
  778. # Do the following, if we have selected ISO related items in kdialog selection window.
  779. if \
  780. [[ ${SEL1} == true ]] || \
  781. [[ ${SEL3} == true ]]
  782. then
  783. # ISO min max values
  784. # 1) Use awk to print field 3 from RAWDATA_TMP. Ignore the first row with FNR> 1 option.
  785. # 2) awk prints equivalent numbers as output. Merge them with "|sort -n" pipe.
  786. # sort prints numbers starting from the smallest (first line) and ending to the greatest (last line).
  787. # 3) Strip the output, either first line (head -1) or the last one (tail -1).
  788. ISO_MIN_VALUE=$(echo -n $(awk -F ',' 'FNR> 1 {print $4}' "${RAWDATA_TMP}" | sort -n | head -1))
  789. ISO_MAX_VALUE=$(echo -n $(awk -F ',' 'FNR> 1 {print $4}' "${RAWDATA_TMP}" | sort -n | tail -1))
  790. ############################################################
  791. # Get percentages for ISO values usage, generate RAWDATA_TMP2 file.
  792. # OUTPUT template for RAWDATA_TMP3 is as follows:
  793. # <count of ISO values>,<ISO value>,<percentage of specific ISO value usage in data>
  794. # Explanation for the following command:
  795. # 1) Use awk to print field 3 from RAWDATA_TMP. Ignore the first row with FNR> 1 option.
  796. # 2) awk prints equivalent numbers as output. Count and merge them with "|sort -n | uniq -c" pipe
  797. # 3) Output results leading white spaces. For each line, delete them with sed.
  798. # 4) use awk as pipe (awk starting with '{b[$2]=$1;sum=sum ...) to calculate percentage
  799. # for the first column. First column has count number for each ISO value ("how many times ISO XX is used").
  800. # ISOs are defined in column 2. Print the output to a new column 3.
  801. # 5) In step 4, the output has too many decimals. As the output of this step is written to column 3,
  802. # we use another awk pipe to strip too many decimals of the current column 3.
  803. # To keep two first decimals, we use %.2f option. Print column 1 ($1) and 2 ($2) as they are,
  804. # respectively. Add % mark and start a new line (\n) after each printf function.
  805. # 6) Replace spaces with commas for the final output, and write the final output to RAWDATA_TMP2.
  806. awk -F ',' 'FNR> 1 {print $4}' "${RAWDATA_TMP}" | \
  807. sort -n | \
  808. uniq -c | \
  809. sed "s/^[ \t]*//" | \
  810. awk '{b[$2]=$1;sum=sum+$1} END{for (i in b) print b[i],i,(b[i]/sum)*100}' | \
  811. awk '{printf "%.0f %.0f %.2f'%'\n", $1,$2,$3}' | \
  812. tr ' ' ',' \
  813. > "${RAWDATA_TMP2}"
  814. ####################################################################################################################
  815. # ISO values - minimum, maximum, least used, most used, average
  816. # What is the maximum number of matches for a single ISO value?
  817. MAX_MATCH_FOR_ISO=$(echo -n $(awk -F ',' '{print $1}' "${RAWDATA_TMP2}" | sort -n | tail -1))
  818. # We store current min/max ISOvalues to a string variables
  819. # Returns column 3 value of RAWDATA_TMP E.G. 3200, based on max column 3 value of RAWDATA_TMP
  820. WHATIS_REAL_MAX_ISO=$(echo -en "Max: $ISO_MAX_VALUE")
  821. # Returns column 3 value of RAWDATA_TMP E.G. 200, based on min column 3 value of RAWDATA_TMP
  822. WHATIS_REAL_MIN_ISO=$(echo -en "Min: $ISO_MIN_VALUE")
  823. # Format: (1*400)+(1*1600)+(2*3200) ...
  824. ISO_DIVIDEND=$(
  825. echo -n $(
  826. ($(
  827. awk -F ',' '{print "("$1,$2")"}' "${RAWDATA_TMP2}" | \
  828. sort -n | \
  829. sed -e 's/ /*/g' | \
  830. tr '\n' '+' | \
  831. sed 's/+[^+]*$//'
  832. ))
  833. )
  834. )
  835. # Just a basic average calculation
  836. ISO_AVERAGE=$(
  837. echo -e $(
  838. awk 'BEGIN {print "'"$ISO_DIVIDEND"'"/"'"$ACCEPTED_TOTAL"'"}' | awk '{printf "%.0f", $1}'
  839. )
  840. )
  841. ##########################################################
  842. # ISO VALUES - CHECK FOR MIN AND MAX VALUES
  843. # 1) Get awk output of file RAWDATA_TMP3, separator for columns is comma mark, get column 3 ($3)
  844. # 2) Sort percentage values from lowest to greatest, starting from the lowest
  845. # 3) Get the match count for each percentage value (column 3 value) with 'uniq -c'
  846. # 4) Trim all leading white spaces for each printed line
  847. # 5) We have now two columns, separated by space. Get the first column with awk pipe,
  848. # use space as a column separator, and print column 1 ($1)
  849. # 6) Get the first line. Output represents the number of matches for listed percentage.
  850. # We check if it's not 1 in the following if statement.
  851. # The whole idea is that we can't give a true statement for
  852. # "What is the least/most used ISO value" if multiple ISO values equal same percentage for usage
  853. MOSTUSED_ISO_CHECK=$(
  854. echo -n $(
  855. awk -F ',' '{print $3}' "${RAWDATA_TMP2}" | sort -n | uniq -c | sed "s/^[ \t]*//" | awk -F ' ' '{print $1}' | tail -1
  856. )
  857. )
  858. LEASTUSED_ISO_CHECK=$(
  859. echo -n $(
  860. awk -F ',' '{print $3}' "${RAWDATA_TMP2}" | sort -n | uniq -c | sed "s/^[ \t]*//" | awk -F ' ' '{print $1}' | head -1
  861. )
  862. )
  863. # The following gives a correct value ONLY IF there are unique values for EACH ISOs.
  864. # Otherwise, the output is not as expected. That's why we need to check the values
  865. # of MOST/LEASTUSED_ISO_CHECK first.
  866. MOSTUSED_ISO=$(
  867. echo -n $(
  868. awk -F ',' '{print $3,$2}' "${RAWDATA_TMP2}" | sort -n | awk -F ' ' '{print $2}' | tail -1
  869. )
  870. )
  871. LEASTUSED_ISO=$(
  872. echo -n $(
  873. awk -F ',' '{print $3,$2}' "${RAWDATA_TMP2}" | sort -n | awk -F ' ' '{print $2}' | head -1
  874. )
  875. )
  876. ##########################################################
  877. # In addition, we consider that minimum of 10 pictures must be accepted as input.
  878. # Otherwise, user can read this info pretty easily just checking the gnuplot graphs.
  879. # Least used ISO
  880. # If more than one, then
  881. if [[ "$LEASTUSED_ISO_CHECK" -ne 1 ]]
  882. then
  883. # Output string, nothing to say.
  884. WHATIS_LEASTUSED_ISO=$(echo -n "")
  885. # Else if it's one, then
  886. elif [[ "$LEASTUSED_ISO_CHECK" -eq 1 ]]
  887. then
  888. # We check the number of pictures. If it's greater than 10, then print the following string.
  889. if [[ "$ACCEPTED_TOTAL" -gt 10 ]]
  890. then
  891. # Returns column 2 value of RAWDATA_TMP2 E.G. 400, based on max column 3 value of RAWDATA_TMP2
  892. WHATIS_LEASTUSED_ISO=$(echo -n ", Least used: $LEASTUSED_ISO")
  893. # We check the number of pictures. If it's equal or less than 10, we print nothing.
  894. elif [[ "$ACCEPTED_TOTAL" -le 10 ]]
  895. then
  896. # Output string, nothing to say.
  897. WHATIS_LEASTUSED_ISO=$(echo -n "")
  898. fi
  899. fi
  900. # Most used ISO
  901. # If more than one, then
  902. if [[ "$MOSTUSED_ISO_CHECK" -ne 1 ]]
  903. then
  904. # Output string, nothing to say.
  905. WHATIS_MOSTUSED_ISO=$(echo -n "")
  906. # Else if it's one, then
  907. elif [[ "$MOSTUSED_ISO_CHECK" -eq 1 ]]
  908. then
  909. # We check the number of pictures. If it's greater than 10, then print the following string.
  910. if [[ "$ACCEPTED_TOTAL" -gt 10 ]]; then
  911. # Returns column 2 value of RAWDATA_TMP2 E.G. 400, based on max column 3 value of RAWDATA_TMP2
  912. WHATIS_MOSTUSED_ISO=$(echo -n ", Most used: $MOSTUSED_ISO")
  913. elif [[ "$ACCEPTED_TOTAL" -le 10 ]]
  914. then
  915. # We check the number of pictures. If it's equal or less than 10, we print nothing.
  916. WHATIS_MOSTUSED_ISO=$(echo -n "") #Output string, nothing to say
  917. fi
  918. fi
  919. ###########################################################
  920. # ISO values - shift ISO range values for GNUPlot
  921. # We shift down minimum ISO values to get a proper scale for gnuplot.
  922. # Use "Less than" integer comparison because there can be ISO values such as 160, 250.
  923. # DO NOT CHANGE THE CHECK ORDER!
  924. if [[ "${ISO_MIN_VALUE}" -le 100 ]]; then
  925. # Just scaling down, not a true ISO value
  926. ISO_MIN_VALUE_GNU=0
  927. elif [[ "${ISO_MIN_VALUE}" -le 200 ]]; then
  928. ISO_MIN_VALUE_GNU=100
  929. elif [[ "${ISO_MIN_VALUE}" -le 400 ]]; then
  930. ISO_MIN_VALUE_GNU=200
  931. elif [[ "${ISO_MIN_VALUE}" -le 800 ]]; then
  932. ISO_MIN_VALUE_GNU=400
  933. elif [[ "${ISO_MIN_VALUE}" -le 1600 ]]; then
  934. ISO_MIN_VALUE_GNU=800
  935. elif [[ "${ISO_MIN_VALUE}" -le 3200 ]]; then
  936. ISO_MIN_VALUE_GNU=1600
  937. elif [[ "${ISO_MIN_VALUE}" -le 6400 ]]; then
  938. ISO_MIN_VALUE_GNU=3200
  939. elif [[ "${ISO_MIN_VALUE}" -le 8000 ]]; then
  940. ISO_MIN_VALUE_GNU=6400
  941. fi
  942. if [[ "${ISO_MAX_VALUE}" -ge 8000 ]]; then
  943. ISO_MAX_VALUE_GNU=12800
  944. elif [[ "${ISO_MAX_VALUE}" -ge 6400 ]]; then
  945. ISO_MAX_VALUE_GNU=8000
  946. elif [[ "${ISO_MAX_VALUE}" -ge 3200 ]]; then
  947. ISO_MAX_VALUE_GNU=6400
  948. elif [[ "${ISO_MAX_VALUE}" -ge 1600 ]]; then
  949. ISO_MAX_VALUE_GNU=3200
  950. elif [[ "${ISO_MAX_VALUE}" -ge 800 ]]; then
  951. ISO_MAX_VALUE_GNU=1600
  952. elif [[ "${ISO_MAX_VALUE}" -ge 400 ]]; then
  953. ISO_MAX_VALUE_GNU=800
  954. elif [[ "${ISO_MAX_VALUE}" -ge 200 ]]; then
  955. ISO_MAX_VALUE_GNU=400
  956. elif [[ "${ISO_MAX_VALUE}" -ge 100 ]]; then
  957. ISO_MAX_VALUE_GNU=200
  958. fi
  959. ###########################################################
  960. # Export all used ISO values
  961. GET_ISO_VALUES=$(
  962. echo -n $(
  963. awk -F ',' 'FNR> 1 {print $4}' "${RAWDATA_TMP}" | \
  964. awk '!seen[$0]++' | \
  965. sort -n | \
  966. tr '\n' ',' | \
  967. sed -e 's/,[^,]*$//'
  968. )
  969. )
  970. ISO_TICSRANGE=$(echo -n "${ISO_MIN_VALUE_GNU},${GET_ISO_VALUES,$ISO_MAX_VALUE_GNU}")
  971. ###########################################################
  972. fi
  973. ###########################################################
  974. # GNUPlot Temperature values
  975. # Do the following, if we have selected temperature related items in kdialog selection window.
  976. if [[ "${SEL3}" == true ]]; then
  977. ###########################################################
  978. # RAWDATA_TMP3
  979. # Get percentages for temperature values
  980. # OUTPUT template for RAWDATA_TMP3 is as follows:
  981. # <count for matching temperatures>,<temperature value>,<percentage of specific temperature in data>
  982. # Explanation for the following command:
  983. # 1) Use awk to print field 2 from RAWDATA_TMP. Ignore the first row with FNR> 1 option.
  984. # 2) awk prints equivalent numbers as output. Count and merge them with "|sort -n | uniq -c" pipe
  985. # 3) Output results leading white spaces. For each line, delete them with sed.
  986. # 4) use awk as pipe (awk starting with '{b[$2]=$1;sum=sum ...) to calculate percentage
  987. # for the first column. First column has count number for each temperature value
  988. # ("how many matches for XX temperature"). Temperature values are defined in column 2.
  989. # Print the output to a new column 3.
  990. # 5) In step 4, the output has too many decimals. As the output of this step is written to column 3,
  991. # we use another awk pipe to strip too many decimals of the current column 3.
  992. # To keep two first decimals, we use %.2f option. Print column 1 ($1) and 2 ($2) as they are,
  993. # respectively. Add % mark and start a new line (\n) after each printf function.
  994. # 6) Replace spaces with commas for the final output, and write the final output to RAWDATA_TMP3.
  995. awk -F ',' 'FNR> 1 {print $3}' "${RAWDATA_TMP}" | \
  996. sort -n | \
  997. uniq -c | \
  998. sed "s/^[ \t]*//" | \
  999. awk '{b[$2]=$1;sum=sum+$1} END{for (i in b) print b[i],i,(b[i]/sum)*100}' | \
  1000. awk '{printf "%.0f %.0f %.2f'%'\n", $1,$2,$3}' | \
  1001. tr ' ' ',' \
  1002. > "${RAWDATA_TMP3}"
  1003. ###########################################################
  1004. # Temperature min max values (actual values from the file)
  1005. TEMP_MIN=$(
  1006. echo -n $(
  1007. awk -F ',' 'FNR> 1 {print $3}' "${RAWDATA_TMP}" | sort -n | head -1
  1008. )
  1009. )
  1010. TEMP_MAX=$(
  1011. echo -n $(
  1012. awk -F ',' 'FNR> 1 {print $3}' "${RAWDATA_TMP}" | sort -n | tail -1
  1013. )
  1014. )
  1015. # Format: (1*31)+(1*38)+(2*39) ...
  1016. TEMP_DIVIDEND=$(
  1017. echo -n $(
  1018. ($(
  1019. awk -F ',' '{print "("$1,$2")"}' "${RAWDATA_TMP3}" | sort -n | sed -e 's/ /*/g' | tr '\n' '+' | sed 's/+[^+]*$//'
  1020. ))
  1021. )
  1022. )
  1023. # Just a basic average calculation
  1024. TEMP_AVERAGE=$(echo -e $(awk 'BEGIN {print "'"$TEMP_DIVIDEND"'"/"'"$ACCEPTED_TOTAL"'"}' | awk '{printf "%.2f", $1}'))
  1025. ###########################################################
  1026. # What is the maximum number of matches for a single temperature?
  1027. MAX_MATCH_FOR_TEMP=$(
  1028. echo -n $(awk -F ',' '{print $1}' "${RAWDATA_TMP3}" | sort -n | tail -1)
  1029. )
  1030. # Round temperature scale.
  1031. # Multiplier for temperature scale for plot 1.
  1032. TEMP_MULTP=2
  1033. # Temperature increment steps. For example, with value of 2,
  1034. # we get ...0, 2...10, 12, 14, 16...24... etc.
  1035. # TEMP_INCREMENT=2
  1036. # We set minimum temperature to <value> - 2, rounding down
  1037. UNROUNDED_MIN=$(echo -n $((${TEMP_MIN} - ${TEMP_MULTP})))
  1038. # Basic layout for the following awk stuff:
  1039. # awk '{i=int($0/4);print((i==$0||$0>0)?i:i-1)*4}'
  1040. # Ref: https://stackoverflow.com/questions/33085008/bash-round-to-nearest-multiple-of-4
  1041. MINVALUE_TEMP=$(
  1042. echo -n $UNROUNDED_MIN | \
  1043. awk '{i=int("'"$UNROUNDED_MIN"'"/"'"$TEMP_MULTP"'");print((i=="'"$UNROUNDED_MIN"'"||"'"$UNROUNDED_MIN"'">0)?i:i-1)*"'"$TEMP_MULTP"'"}'
  1044. )
  1045. # We set maximum temperature to <value> + 2, rounding up
  1046. UNROUNDED_MAX=$(echo -n $((${TEMP_MAX} + ${TEMP_MULTP})))
  1047. # Basic layout for the following awk stuff:
  1048. # awk '{print$0+(n-$0%n)%n}'
  1049. # Ref: https://stackoverflow.com/questions/33085008/bash-round-to-nearest-multiple-of-4
  1050. MAXVALUE_TEMP=$(
  1051. echo -n $UNROUNDED_MAX | \
  1052. awk '{print"'"$UNROUNDED_MAX"'"+("'"$TEMP_MULTP"'"-"'"$UNROUNDED_MAX"'"%"'"$TEMP_MULTP"'")%"'"$TEMP_MULTP"'"}'
  1053. )
  1054. fi
  1055. ###########################################################
  1056. # GNUPlot selection 3 - Temperatures & ISOs
  1057. ###########################################################
  1058. # PLOT 1 (Images & Temperatures & ISOS)
  1059. if [[ "${SEL3}" == true ]]; then
  1060. # GNUPlot program execution starts here.
  1061. gnuplot <<EOF &
  1062. reset
  1063. ###########################################################
  1064. #set title "$GNUPLOT_TITLE" font ",12" offset 0,1
  1065. #Set Window Title in Qt environment
  1066. set term qt title "$GNUPLOT_MAINWINDOW_TITLE"
  1067. set boxwidth 0.75
  1068. set style data histograms
  1069. set style fill solid border -1
  1070. #set xlabel "Image Files ($FIRST-$LAST)" noenhanced
  1071. set xlabel " " noenhanced #Just a dummy label placeholder
  1072. #set x2label "Time"
  1073. set ylabel "Temperature (°C) -- Avg: $TEMP_AVERAGE°C"
  1074. set y2label "ISO Value -- Avg: $ISO_AVERAGE"
  1075. set datafile separator ","
  1076. set grid
  1077. #set xtics font ",7"
  1078. #set x2tics
  1079. # We adapt the layout for number of files...
  1080. $X2_TIMESTRINGS
  1081. $X_ROTATELABELS
  1082. set yrange [$MINVALUE_TEMP:$MAXVALUE_TEMP]
  1083. set y2range [$ISO_MIN_VALUE_GNU:$ISO_MAX_VALUE_GNU]
  1084. #set xtics ($FILELIST)
  1085. set y2tics ($ISO_TICSRANGE)
  1086. set xrange [0:$ACCEPTED_TOTAL]
  1087. set autoscale x
  1088. ##set timefmt '%Y-%m-%dT%H:%M:%S'
  1089. ##set mouse mouseformat 3
  1090. #Write values as they are
  1091. set tics noenhanced font ",8"
  1092. #Don't consider the first line as data but as column titles?
  1093. set key autotitle columnhead
  1094. set palette model RGB
  1095. ###########################################################
  1096. plot '${RAWDATA_TMP}' using 3:xtic(1) title 'Temperature' lc rgb "orange", \
  1097. $TEMP_AVERAGE title 'Temp Avg.' lc rgb "red", \
  1098. '' using 4:x2tic(sprintf("%s\n%s", stringcolumn(23), stringcolumn(24))) title 'ISO Value' lc rgb "blue" axis x1y2, \
  1099. $ISO_AVERAGE title 'ISO Avg.' lc rgb "black" axis x1y2
  1100. pause mouse close
  1101. EOF
  1102. ###########################################################
  1103. # PLOT 2 & 3 (More temperature & ISO analysis)
  1104. # GNUPlot program execution starts here.
  1105. # This leads actually to a bug which kills qnuplot, making all zoom/grid etc.
  1106. # options unavailable. This is what we want for the following plots.
  1107. #
  1108. # The bug is discussed here: https://sourceforge.net/p/gnuplot/bugs/1419/
  1109. # And here: https://sourceforge.net/p/gnuplot/bugs/1483/
  1110. gnuplot --persist <<EOF &
  1111. reset
  1112. ###########################################################
  1113. set term qt title "$GNUPLOT_MAINWINDOW_TITLE"
  1114. set multiplot layout 2, 1 title "ISO Speed Averages ($WHATIS_REAL_MIN_ISO, $WHATIS_REAL_MAX_ISO$WHATIS_LEASTUSED_ISO$WHATIS_MOSTUSED_ISO, Avg: $ISO_AVERAGE)" font ",10"
  1115. #THIS IS A TOP LABEL FOR ISO SPEEDS!
  1116. # Max ISO string: $WHATIS_REAL_MAX_ISO
  1117. # Min ISO string: $WHATIS_REAL_MIN_ISO
  1118. # Least used ISO string: $WHATIS_LEASTUSED_ISO
  1119. # Most used ISO string: $WHATIS_MOSTUSED_ISO
  1120. #
  1121. ###set x2label "ISO Speed Averages" font ",10" offset 0,-1
  1122. set ylabel "Image Count" font ",10"
  1123. set datafile separator ","
  1124. #set style data histogram
  1125. set style fill solid border -1
  1126. set boxwidth 0.40 relative
  1127. set palette model RGB
  1128. set yrange [0:$MAX_MATCH_FOR_ISO+1]
  1129. set ytics 0,$SCALE,$MAX_MATCH_FOR_ISO+1
  1130. set xtics rotate by 45 right
  1131. set x2tics offset 0,-0.4
  1132. #set offset 0,-2.00
  1133. #set xtics
  1134. #set xrange [$ISO_MIN_VALUE_GNU:$ISO_MAX_VALUE_GNU]
  1135. unset key
  1136. plot '${RAWDATA_TMP2}' using 1:xtic(2) title '$WHATIS_REAL_MAX_ISO' lc rgb "green" with boxes, \
  1137. '' using 1:x2tic(3) title '' with boxes fill empty
  1138. #
  1139. ###unset x2label
  1140. unset xtics
  1141. set xtics
  1142. set xlabel "Temperature Averages (Avg: $TEMP_AVERAGE°C)" font ",10" #THIS IS A TOP LABEL FOR TEMPERATURES!
  1143. #set ylabel "Image Count" font ",10"
  1144. set datafile separator ","
  1145. #set style data histogram
  1146. set boxwidth 0.40 relative
  1147. set style fill solid border -1
  1148. set palette model RGB
  1149. set yrange [0:$MAX_MATCH_FOR_TEMP+1]
  1150. set ytics 0,$SCALE,$MAX_MATCH_FOR_TEMP+1
  1151. set x2tics offset 0,-0.4
  1152. unset key
  1153. plot '${RAWDATA_TMP3}' using 1:xtic(2) title '' lc rgb "red" with boxes, \
  1154. '' using 1:x2tic(3) title '' with boxes fill empty
  1155. #
  1156. unset multiplot
  1157. #
  1158. ###########################################################
  1159. EOF
  1160. sleep 5
  1161. rm "${RAWDATA_TMP2}"
  1162. rm "${RAWDATA_TMP3}"
  1163. fi
  1164. # GNUPLOT CODE - END
  1165. ###########################################################
  1166. if [[ "${KEEPSTATS}" == false ]]
  1167. then
  1168. rm "${RAWDATA_TMP}"
  1169. fi
  1170. exit 0