Synapse Log File Converter with flight metrics (Use MK_GPXTOOL/Google Earth to Analyze)

Discussion in 'Announcements' started by Andy Johnson-Laird, Nov 30, 2015.

  1. Andy Johnson-Laird

    Andy Johnson-Laird Administrator
    Staff Member

    Joined:
    Jul 31, 2012
    Messages:
    10,347
    Likes Received:
    1,159
    As a short term, interim, solution, I've created a Synapse Log File Converter, sflc, that converts a Synapse .csv file into a .gpx and a .kml file that you can feed into the Mikrokopter MK_GPXTOOL log analyzer and its ilk and Google Earth.

    Please do read the documentation to find out how to install and use the converter. And apologies in advance for the British alleged humour/humor that seems to have infested the documentation.

    Attached is the documentation and the release files for version 2.02. I'm going to be a bit hard pressed to spend large chunks of time working on this converter, but the entire source code is provided. It's written in a venerable language especially suited for working on text files (like .csv, .gpx, and .kml) called AWK. It's pronounced "awk." Which is also strikingly similar to the noise that my colleagues make when they look at software that I have written.

    Be sure to read the section of the documentation called Dirty Little Secrets to understand the limitations of the converter (and some of the limitations of the programmer).

    if you do want changes made, feel free to hack at the code....I'm sorry, carefully craft amendments...but then let me know what you changed and why. I promise not to giggle. Much.

    I reserve the right to post new versions of this if I find out more about the deeper truths contained in the Synapse log files.

    EDIT: I have deleted the attachments to this posting as there are later versions. Please scroll down to get the latest version.

    Andy
     
    MIke Magee and Steve Maller like this.
  2. MIke Magee

    MIke Magee Active Member

    Joined:
    Jun 29, 2012
    Messages:
    422
    Likes Received:
    103
    This is awesome. I've heard rumors of it.

    But AWK? You indeed have a dark side.

    Thanks again Andy for yet another contribution.
    -m
     
  3. Andy Johnson-Laird

    Andy Johnson-Laird Administrator
    Staff Member

    Joined:
    Jul 31, 2012
    Messages:
    10,347
    Likes Received:
    1,159
    Aho, Weinberger, Kernighan liked AWK. And it's named in their honor: https://en.wikipedia.org/wiki/AWK
    "AWK is an interpreted programming language designed for text processing and typically used as a data extraction and reporting tool. It is a standard feature of most Unix-like operating systems."

    It was first created in the 1970's, so they've got the bugs out now. :)

    Andy.
     
  4. Andy Johnson-Laird

    Andy Johnson-Laird Administrator
    Staff Member

    Joined:
    Jul 31, 2012
    Messages:
    10,347
    Likes Received:
    1,159
    I've been working on a new version that, in addition to creating a .gpx file (for the likes of MK_GPXTOOL) and a .kml file (for Google Earth), now creates a summary of the flight metrics (.metrics.txt) as well as a flight logbook entry (.logbook.csv) to make it easier to log files using Excel. The new software and alleged documentation are attached to this posting.

    There's an example of the metrics file below. Two versions of the documentation are attached: one shows (using Word's Track Changes) what was changed between this version and the previous version, and a version without the changes marked.

    Hope this is useful to ALTA pilots.

    Andy

    Synapse Flight Metrics created by:
    sflc Version 3.02 2015-12-11 Copyright (c) Johnson-Laird Inc.


    Flight date (UTC): 2015-10-06

    PHYSICAL METRICS
    Flight duration: from 21:32:35 to 21:48:12 (UTC) : 942.675 seconds (0:15:42.674 H:MM:SS.m)
    GPS Base Altitude = 3.6m
    Home Latitude/Longitude = 45.649418 N 122.822349 W
    Bounding box for flight from home (meters) based on IMU data:
    North = 3.33 South = 20.35 East = 12.09 West = 21.15 Up = 15.38 Down = 0.84
    Bounding box for flight from home (meters) based on GPS data:
    North = 3.35 South = 20.36 East = 12.11 West = 21.09 Up = 15.44 Down = 0.64
    Maximum horizontal distance from home: IMU = 23.33 meters GPS = 23.37 meters
    Maximum slant angle distance from home: IMU = 25.74 meters GPS = 25.69 meters
    Horizontal distance flown: IMU = 383.09 meters GPS = 378.14 meters
    Flightpath distance flown: IMU = 549.56 meters GPS = 556.46 meters

    Synapse Flight Control Data:
    Flight Controller Boot-up Temperature = 43 C at IMUTime 1.7420 seconds
    Flight Controller Max Temperature 44 C at IMUTime 261.3824 seconds

    ELECTRICAL DATA
    Maximum Battery Bus Current = 122.00 Amps at IMUTime 891.8548 seconds
    Minimum Voltage = 22.12 Volts at IMUTime 874.8956 seconds
    Maximum Power = 2706.70 Watts at IMUTime 891.8548 seconds
    Hover Power = 349.61 Watts
    Hover Battery Bus Current = 15.21 Amps
    Hover Throttle = 1249 microsecond pulse, 25.0%
    Charge used = 5.8 Amp hours
    Energy used = 134.3 Watt hours
    Energy efficiency = 0.2444 Watt hours/meter traveled

    ESC/MOTOR DATA
    Pre-flight ESC Temperatures (check for ESC/Motors already too hot):
    1 = 37 C at IMUTime 84.0391 seconds
    2 = 36 C
    3 = 39 C
    4 = 39 C
    5 = 39 C
    6 = 39 C
    Average = 38 C, Odd/CW = 38 C, Even/CCW = 38 C, Difference = 0 C

    Maximum ESC Temperatures (check for ESC/Motors running hot in flight):
    1 = 41 C at IMUTime 420.2594 seconds
    2 = 41 C at IMUTime 324.9896 seconds
    3 = 43 C at IMUTime 420.3399 seconds
    4 = 42 C at IMUTime 325.0702 seconds
    5 = 43 C at IMUTime 325.5133 seconds
    6 = 43 C at IMUTime 258.4015 seconds
    Average = 42 C, Odd/CW = 42 C, Even/CCW = 42 C, Difference = 0 C

    Maximum ESC Phase Currents (check for ESC drawing too much current):
    1 = 61.8 A at IMUTime 977.2149 seconds
    2 = 63.0 A at IMUTime 977.2149 seconds
    3 = 66.9 A at IMUTime 136.3703 seconds
    4 = 61.3 A at IMUTime 977.6580 seconds
    5 = 64.8 A at IMUTime 136.3703 seconds
    6 = 53.7 A at IMUTime 977.2552 seconds
    Average = 61.9 A, Odd/CW = 64.5 A, Even/CCW = 59.3 A, Difference = 5.2 A

    Minimum ESC Volts (check for ESC operating at too low a voltage):
    1 = 21.00 V at IMUTime 891.8548 seconds
    2 = 21.86 V at IMUTime 872.1563 seconds
    3 = 21.81 V at IMUTime 877.8765 seconds
    4 = 21.52 V at IMUTime 863.6968 seconds
    5 = 21.55 V at IMUTime 977.2552 seconds
    6 = 21.91 V at IMUTime 832.9608 seconds
    Average = 21.61 V, Odd/CW = 21.45 V, Even/CCW = 21.76 V, Difference = -0.31 V

    Maximum ESC RPM (Spds) (Check for motor spinning too slow):
    1 = 4327 RPM at IMUTime 419.8968 seconds
    2 = 4219 RPM at IMUTime 419.7357 seconds
    3 = 4342 RPM at IMUTime 419.8565 seconds
    4 = 4220 RPM at IMUTime 419.8163 seconds
    5 = 4261 RPM at IMUTime 419.7357 seconds
    6 = 4154 RPM at IMUTime 419.8163 seconds
    Average = 4253 RPM, Odd/CW = 4310 RPM, Even/CCW = 4197 RPM, Difference = 112 RPM

    Maximum ESC Power (check for motor consuming too much power):
    1 = 497.6 W at IMUTime 891.8548 seconds
    2 = 437.8 W at IMUTime 324.7076 seconds
    3 = 474.8 W at IMUTime 136.3703 seconds
    4 = 418.1 W at IMUTime 977.6580 seconds
    5 = 460.3 W at IMUTime 136.3703 seconds
    6 = 383.8 W at IMUTime 892.7813 seconds
    Average = 445.4 W, Odd/CW = 477.6 W, Even/CCW = 413.2 W, Difference = 64.3 W

    Maximum ESC Power Recovery (check power recovery operating):
    1 = 102.7 W at IMUTime 131.2131 seconds
    2 = 111.5 W at IMUTime 893.2244 seconds
    3 = 115.7 W at IMUTime 891.9354 seconds
    4 = 122.6 W at IMUTime 893.2244 seconds
    5 = 107.4 W at IMUTime 131.2131 seconds
    6 = 115.0 W at IMUTime 893.2244 seconds
    Average = 112.5 W, Odd/CW = 108.6 W, Even/CCW = 116.4 W, Difference = -7.8 W

    Maximum ESC Energy (check for ESC/motor consuming too much energy):
    1 = 23.6 Wh at IMUTime 1012.1807 seconds
    2 = 23.6 Wh at IMUTime 1010.9320 seconds
    3 = 24.3 Wh at IMUTime 1015.8062 seconds
    4 = 18.7 Wh at IMUTime 993.6505 seconds
    5 = 23.7 Wh at IMUTime 998.9276 seconds
    6 = 20.5 Wh at IMUTime 994.4561 seconds
    Average = 22.4 Wh, Odd/CW = 23.9 Wh, Even/CCW = 20.9 Wh, Difference = 2.9 Wh

    Maximum ESC Acceleration (check for excessive vibration):
    1 = 32.10 at IMUTime 475.0043 seconds
    2 = 22.30 at IMUTime 480.2814 seconds
    3 = 35.90 at IMUTime 376.1493 seconds
    4 = 24.60 at IMUTime 481.0870 seconds
    5 = 25.00 at IMUTime 433.9960 seconds
    6 = 20.80 at IMUTime 542.9217 seconds
    Average = 26.78, Odd/CW = 31.00, Even/CCW = 22.57, Difference = 8.43
     

    Attached Files:

  5. Steve Maller

    Steve Maller UAV Grief Counselor

    Joined:
    Oct 30, 2012
    Messages:
    3,979
    Likes Received:
    807
    Fabulous, Andy. Thank you!!!
     
  6. MIke Magee

    MIke Magee Active Member

    Joined:
    Jun 29, 2012
    Messages:
    422
    Likes Received:
    103
    Andy, you contribute above and beyond. Should you EVER travel to the Boston area, anticipate rock-star treatment from me.
    -m
     
  7. Andy Johnson-Laird

    Andy Johnson-Laird Administrator
    Staff Member

    Joined:
    Jul 31, 2012
    Messages:
    10,347
    Likes Received:
    1,159
    My pleasure, Steve, Mike. I appreciate the offer for rock star treatment!
    Andy
     
  8. Humberto Barchini

    Joined:
    Oct 31, 2012
    Messages:
    94
    Likes Received:
    26
    Hi Andy... I´m not very familiarised with the automator... but following your instructions is just telling me "the action Run Shell Script encountered an error... I´m doing something wrong?

    Thanks in advance
     
  9. Andy Johnson-Laird

    Andy Johnson-Laird Administrator
    Staff Member

    Joined:
    Jul 31, 2012
    Messages:
    10,347
    Likes Received:
    1,159
    Hi Humberto:
    Hmmm. Well, evidently something is wrong, but it's hard to know what it might be without more information.
    Can you confirm that the following three files are in your home directory on your Mac:
    sflc.app
    sflc.workflow
    sflc.awk
    sflc.bash

    Also can you tell me what version of Mac OS X you are running?

    And what do you double click on that causes the error message you're seeing?

    Andy.
     
  10. Humberto Barchini

    Joined:
    Oct 31, 2012
    Messages:
    94
    Likes Received:
    26
    OK... certainly I´m doing something wrong thats for sure hahahaha
    Those files are in a folder "sflc" on my desktop
    My OSX version is 10.11.3
    When I try the sflc app it ask me to choose a csv file... when I choose that it shows that error...
    When I try with the workflow it shows the att error...
     

    Attached Files:

  11. Andy Johnson-Laird

    Andy Johnson-Laird Administrator
    Staff Member

    Joined:
    Jul 31, 2012
    Messages:
    10,347
    Likes Received:
    1,159
    Could you confirm that you've followed paragraphs 79 and 80 of the alleged documentation:

    79.To install these, simply put them all in your home directory on your Mac. (Technically, this is /Users/<yourname>.) ​

    80.If it is more convenient to you, you can put sflc.app anywhere on your Mac that you want, but you must put sflc.bash and sflc.awk in your home directory otherwise it will not work.​
    That might be the problem -- for safety's sake, move all the files into your home directory at the top level (in other words don't put them into a folder that is in your home directory.
    Let me know if that fixes the problem, please.
    Thanks
    Andy
     
  12. Humberto Barchini

    Joined:
    Oct 31, 2012
    Messages:
    94
    Likes Received:
    26
    Almost there... now I have a new error... (me bad you were right I did not put the files in the correct folder), now I have a Gawk error... att the error and the Gawk -V comm
     

    Attached Files:

  13. Andy Johnson-Laird

    Andy Johnson-Laird Administrator
    Staff Member

    Joined:
    Jul 31, 2012
    Messages:
    10,347
    Likes Received:
    1,159
    The first screen shot suggests that gawk is not installed in the subdirectory /sw/bin/gawk.

    To find out where it is installed, open Terminal and type the command:

    which gawk

    That will tell you where it is installed. All I did was follow the instructions at http://macappstore.org/gawk but it sounds like it may have installed gawk somewhere other than /sw/bin/gawk.

    Let me know what the output of the which gawk is -- you can click and drag through the text in the Terminal window and just paste into your posting if that's easier than taking screen shots.

    Andy
     
  14. Humberto Barchini

    Joined:
    Oct 31, 2012
    Messages:
    94
    Likes Received:
    26
    Hi Andy... sorry about the delay.... here is the output...
    Last login: Mon Jan 25 10:48:54 on console
    MacBook-Pro:~ Humberto$ which gawk
    /usr/local/bin/gawk

    MacBook-Pro:~ Humberto$

    how can I put it in the right folder?
     
  15. Andy Johnson-Laird

    Andy Johnson-Laird Administrator
    Staff Member

    Joined:
    Jul 31, 2012
    Messages:
    10,347
    Likes Received:
    1,159
    Oh. They must have changed things. Hmm. The easiest thing to do is to use the attached script.
    For other people who read this thread, if you see the same thing as Humberto, use a text editor and edit the text file sflc.bash and change line 4, which currently starts with /sw to start with /usr/local so that it now reads /usr/local/bin/gawk

    If you're not sure how to do this, use Finder, go to your home directory, click right on the sflc.bash file and open it with TextEdit.app (which is a standard part of the Mac OS X system).

    When I get a chance, I'll modify the sflc.bash file to look for gawk in either /sw/bin or /usr/local/bin. Sorry. Shoddy programming on my part.


    Andy
     
  16. Humberto Barchini

    Joined:
    Oct 31, 2012
    Messages:
    94
    Likes Received:
    26
    Andy!!!!! You are the Man!!!! thanks now it is working....!!!!!
     
  17. Andy Johnson-Laird

    Andy Johnson-Laird Administrator
    Staff Member

    Joined:
    Jul 31, 2012
    Messages:
    10,347
    Likes Received:
    1,159
    Thanks for hanging in there with me... I didn't realize that the new default for installing gawk was to put it into the /usr/local/bin subdirectory. Sigh

    Andy.
     
    Humberto Barchini likes this.
  18. steve peters

    steve peters New Member

    Joined:
    Oct 25, 2015
    Messages:
    2
    Likes Received:
    0
    Nice one Andy, works a treat, especially after following the thread to the bottom. spot on.
     
  19. steve peters

    steve peters New Member

    Joined:
    Oct 25, 2015
    Messages:
    2
    Likes Received:
    0
    now just need to compile all the flight log CSV's into one sheet and jobs a good un.....
     
  20. Humberto Barchini

    Joined:
    Oct 31, 2012
    Messages:
    94
    Likes Received:
    26
    Hi Andy!!!! me again... I don´t know how but I think something change... now when I try to use the program it says "the action ""run shell script"" encountered an error."

    I love to use this tool sorry to bother you :)
     

Share This Page