X8 in-flight Roll to inversion and crash. Analysis/opinions appreciated

Discussion in 'Cinestar 8' started by MIke Magee, Dec 9, 2014.

  1. Andy Johnson-Laird

    Andy Johnson-Laird Administrator
    Staff Member

    Joined:
    Jul 31, 2012
    Messages:
    10,369
    Likes Received:
    1,162
    Hmmm. Dave King has had some experience with Quadro Cools and version 1.02d firmware problems I believe. There's no specific data that I've seen to say that they are the cause of the crash in this case though.

    Andy.
     
  2. MIke Magee

    MIke Magee Active Member

    Joined:
    Jun 29, 2012
    Messages:
    422
    Likes Received:
    103
    NOTE TO ALL: Our products and science is obviously imperfect. At this point, we rely on and appreciate learning from the experiences and opinions of folks like those that have opined. We are on the verge of regulation and in the infancy of our industry.

    I have over 1400 (well, maybe 1398) successful flights but still consider myself a newbie, and appreciate the folks here like Mike, Steve, Dave, and ANDY. (and others across this forum that contribute and opine).

    Thanks All.
    -m

    p.s. Keep the comments and opinions coming.
     
  3. MIke Magee

    MIke Magee Active Member

    Joined:
    Jun 29, 2012
    Messages:
    422
    Likes Received:
    103
    Andy, AFA the tanks, YES, the base of the tanks are some type of metal. However, I could show you a dozen+ shots from this specific location (Stills and video), even with the massive tanks there.

    HOWEVER, just around the moment of this incident, there was a ~40 meter relocation of a "several" stories of iron, some 300+ meters away. Would you think that this type of iron shift could broadcast a magnetic anomaly that could cause the compass confusion?
    -m


     
  4. Steve Maller

    Steve Maller UAV Grief Counselor

    Joined:
    Oct 30, 2012
    Messages:
    3,979
    Likes Received:
    807
    I am struggling at the moment with getting my copter flying straight after rebuilding and upgrading some things, and I have had some magnetic/compass issues. But AFAIK, while that category of error can certainly cause anomalous flight behavior, what I believe took your copter down was (in all likelihood) something much more catastrophic than a magnetic error. I strongly support Dave King's comment that some of these systems are likely to wear out from hundreds of flights, and we don't have a good baseline of what to look for in the category of fatigued systems. Part of the problem is that no two heavy lift systems are exactly alike, and certainly none have identical flight patterns. I know many of us here, for example, have blurred the line between "crash" and "landing" on all too many occasions. :eek:

    Flight logs are certainly helpful, but there are far too many opaque "black box" systems on these machines to inspire confidence. Over the past year we've seen many, many experienced UAV operators experience serious in-flight failures, many leading to similar total losses as you experienced. And while I have had many non-crashes where a thorough pre-flight or maintenance check uncovered a potentially fatal problem, I myself had a crash a couple months ago that I think was pilot error, but that was what I (and others) inferred after failing to find clear evidence of a failure.

    Sorry for the rant. It's not altogether off topic, but I'll get off my soap box and go back to staring blankly at GPX files... :confused:
     
    MIke Magee likes this.
  5. MIke Magee

    MIke Magee Active Member

    Joined:
    Jun 29, 2012
    Messages:
    422
    Likes Received:
    103
    Steve, not a rant, EXCELLENT point. As I said above " We are on the verge of regulation and in the infancy of our industry.". We are kind of like pioneers/cowboys, except that we have a profound sense of professionalism and conscience. We struggle with on-the-fly (relatively) components and we try to do the best we can with them. There is no standard of time since overhaul or TSOH.

    Good comment.
    -m


     
  6. Andy Johnson-Laird

    Andy Johnson-Laird Administrator
    Staff Member

    Joined:
    Jul 31, 2012
    Messages:
    10,369
    Likes Received:
    1,162

    I honestly don't know, Mike. The earth's magnetic field is not very strong so I suspect it is possible but this is baseless speculation on my part -- and as you know, lack of knowledge has never held me back before. :)

    Andy
     
  7. Jason Smoker

    Jason Smoker Active Member

    Joined:
    Oct 23, 2012
    Messages:
    849
    Likes Received:
    67
    mike sorry for your loss!! :eek:

    Talking about the compass. out of interest what did you have your compass effect set too?

    I always have mine at 0 because of my work i fly over/on all types of structures/cranes metal pipes and big steel plates mostly construction sites. I have only had a few spots where its been dangerous. Then i just fly manually
     
  8. Dave King

    Dave King Well-Known Member

    Joined:
    Dec 24, 2012
    Messages:
    2,711
    Likes Received:
    311
    Here's what I found on the BL updates. Here's really no mention of what improvement of I2C communication in non redundant setups. I see a few updates about I2C communication improvement in the redundant setups only. I know there has been improvements I just don't know what they are and to what versions.


    V1.10


    Since 23/09/2014
    • the motors run with this version a bit quiet at constant speed
    Version V1.08


    Since 07/18/2014
    • Redundancy: In case of failure of redundant communication failure is reported faster now
    • Default timing again 24 °
    • Bugfix: on some engines were commutation at high currents -> eg for a U5 (T engine) or KDE 4012/4014
    Version V1.06


    Since 30.06.2014
    • Improvement in communication for redundancy
    • small changes in I2C protocol (will not be required in the normal MK-operation)
    • small change in self-test (Subject BL-12 * Error flashing)
    • In current peaks konne "ERR: Engine Restart" to be reported. Now "BL-Limit" reported
    • in some engines it was commutation
    • since V1.06e -> default timing is now 28 ° (in V1.06d it was still 24 °)
    Version V1.04c


    Since 7/5/2014
    [​IMG]
    • better protection against power surges during operation
    • better protection against Dekommutierung (loss of synchronization)
    • Safer behavior during operation without propeller
    • Integrated watchdog that restarts the BL-controller in failure
    • Various bug fixes
    • [​IMG]this version is stable. Update erfolderlich only when problems occur
    Version V1.02d


    Since 17/03/2014
    • improved start-up behavior through self-learning startup procedure
    • first version of redundant operation ( Link )
    • integrated speed limit
      • The maximum speed is calculated as follows: 200.000UPM divided by number of magnets in the motor housing
      • eg 16.666U / min at 12 magnets
    • [​IMG] If this version is installed, but it please update!
    Version V1.00f


    Since 1.1.2014
    • first version
    • [​IMG] If this version is installed, but it please update!
     
    MIke Magee likes this.
  9. Dave King

    Dave King Well-Known Member

    Joined:
    Dec 24, 2012
    Messages:
    2,711
    Likes Received:
    311
    I think its time to test the redundancy system. My boards have the redundancy feature so they are good to go. I am going to work on this project right after the holidays. There's some really good info on the MK redundancy system here http://wiki.mikrokopter.de/Redundant
    It also shows you how to simulate an I2C failure and how to test each flight controller's redundancy backup.
    • If both FCs are connected, the green LED flashes on the redundant FC quickly. Then she gets data from BL-controller. 1 and 5
    • If the main FC is disconnected (Molex plug cap), the Green LED will continue to flash and also lights the red.
    • If the main FC is disconnected, the red light error LEDs on all BL controllers NOT!
    I2C error


    What is needed is to a free channel on the transmitter is placed on a switch.
    This channel is in the settings of the user parameter 7 set: Link .
    • If a value> 150 is (switch pressed), the master FlightCtrl the I2C bus is switched off, thus simulating the bus failure.
    • Condition: The FC only switches off the bus when redundancy maker available ('R' in the display).
    [​IMG]For the test, you should first make the Mikrokopter with engines running on the floor stand and then turn off the I2C bus. The main FC Beeps then as I2C errors and the engines do not go out. A take off and fly to the redundant FC is then possible. If this is OK can also be performed in flight test.

    Please note
    If the master FlightCtrl a NaviCtrl and GPS used and on the slave FlightCtrl not, functions are PositionHold , Coming Home or Care Free in an error not available and the UAV must be flown manually! Only when the slave FlightCtrl also a NaviCtrl features and GPS, these functions are also available in case of error.

    Here's a picture of the setup.

    Redundant-Double.jpg
     
    MIke Magee likes this.
  10. Dave King

    Dave King Well-Known Member

    Joined:
    Dec 24, 2012
    Messages:
    2,711
    Likes Received:
    311
    Ok guys I received all the redundant power boards and hardware from MK. I am going to start work on this next week. This will be interesting :)
     
    MIke Magee likes this.
  11. Andy Johnson-Laird

    Andy Johnson-Laird Administrator
    Staff Member

    Joined:
    Jul 31, 2012
    Messages:
    10,369
    Likes Received:
    1,162
    Do you also plan on having redundant GPS boards and a dual receiver like the GR-32, Dave?

    Andy.
     
  12. Dave King

    Dave King Well-Known Member

    Joined:
    Dec 24, 2012
    Messages:
    2,711
    Likes Received:
    311
    Andy I would need to confirm but I don't think they are set up for redundant GPS or NAV boards or a dual receiver.
     
  13. Andy Johnson-Laird

    Andy Johnson-Laird Administrator
    Staff Member

    Joined:
    Jul 31, 2012
    Messages:
    10,369
    Likes Received:
    1,162
    The GR32 doesn't "appear" to be a dual receiver to the MK electronics. It simply has two receivers and dual antennae.
    Holger said that the system was indeed capable of handling dual GPS.

    I don't think he would object to me posting what he wrote, so:

    Just connect a redundant NC and GPS to the redundant FC.​
    The redundant FC has all features like the main-FC if the NC is also connected.​

    The Pilot can choose between both FC+NC systems by a switch and can switch over to the redundant system.​

    He wrote this to me on 2014-10-14.
    Andy.
     
    Steve Maller likes this.
  14. Steve Maller

    Steve Maller UAV Grief Counselor

    Joined:
    Oct 30, 2012
    Messages:
    3,979
    Likes Received:
    807
    I still believe that until we also have redundant power systems (ESCs, etc.) this "redundancy" conversation is only mildly interesting. Granted, it potentially addresses one possible (not quite proven) source of catastrophic errors specific to MK copters (I2C bus failures), but even that is a bit of a band-aid. Until there's a proper "supervisory" system aboard the machine that can make command decisions in real time, we're kind of throwing darts at the problem.
     
  15. Gary Haynes

    Gary Haynes Administrator
    Staff Member

    Joined:
    Jul 6, 2012
    Messages:
    5,211
    Likes Received:
    460
    Unless of course the I2C failure is on the PDB boards.
     
  16. Andy Johnson-Laird

    Andy Johnson-Laird Administrator
    Staff Member

    Joined:
    Jul 31, 2012
    Messages:
    10,369
    Likes Received:
    1,162
  17. Dave King

    Dave King Well-Known Member

    Joined:
    Dec 24, 2012
    Messages:
    2,711
    Likes Received:
    311
    Andy is correct the new Double Quadro XL boards are redundant where that each board has a seperate I2C databus. If there is any glitch on the first buss the 2nd flight controller and 2nd buss become active. There's a test that you can do by hooking up a switch to the radio to simulate a lost or corrupted I2C signal.

    I was not aware that you could have a second navigation board, I assume you would need a 2nd GPS board. Not sure how that would work with an external compass or how all these electronics would be stacked. You would assume that the 2nd GPS and navigation board would be included in the documentation but there's no mention of this.

    As far as redundant power supplies I don't think that's going to happen any time soon. Personally I don't know of anyone that's lost more than one BL at the same time a failure occurs unless it involves water or moisture. In those incidents a cover would help greatly. I will be fabricating a cover through a 3D printer as soon as I figure out exactly how many boards I will be running to determine how high I need it to clear. I know Steve and I have talked about many potential solutions and situations that involve failure as to what happens if you lose more than 1 motor.

    I think the majority of the MK electronic issues involve the failure of the I2C signal and this is a step in the right direction.
     
  18. Ryan McMaster

    Ryan McMaster Active Member

    Joined:
    Mar 8, 2014
    Messages:
    289
    Likes Received:
    40
    This looks just like my Crash mike.

    From the GPX and the version of the BL firmware you were running it was I2C and the bus failing. We had the same thing, sent it to MK and it was confirmed to be I2C, but no reason why it failed. Upon talking with some Hardware guys I work with we THINK that the older firmware was allowing voltage spikes that could damage the bus physically. Our crash was only 20-30 hours in on the board. On the magnetic/EMI side, Mike did you braid your wires? I just finished testing a Coax Y hex (Results are a bit... less then fun) and tried the wires braided and un-braided. In the un-braided config it had issues that did not even show up when the wires were properly braided and cased.

    In regards to the concerns about durability most bearings in brushless motors start to fail around 600 hours in (from KDE) and be replaced around 400, electronics running high voltage/amp vary on the quality of the components but should be in the 2000+ hour range. With good isolation the I2C should have the same lifetime as the rest of the board other then physical damage (ALWAYS hot glue the I2c cables into the FC/PDB). Also I have got into the habit of filing ALL the edges of the CS center plate down. I recently worked on a CS rig that had frayed pigtails because the edges of the plate cut the wires. did you check all the wires from the PDB for damage? A short like that could cause an I2C fail.

    It sucks, and I know I spent a few weeks after we had that I2C fail thinking about just letting someone else fly for me. Firmware updates are SUPER important I am finding in MK. Since my crash, and update to new boards/firmware etc. I have not had any major issues with any of my rigs.
     
  19. MIke Magee

    MIke Magee Active Member

    Joined:
    Jun 29, 2012
    Messages:
    422
    Likes Received:
    103
    Ryan, looks like it. Thanks for sharing. I'm going to dedicate a second test machine to staying up to the edge on the firmware. Sometimes newer is not always better. AMHIK.

    I'm hoping that the Herkules will let the BL3 shake out a bit. On the other hand, I had hundreds of flights on that (deceased) setup.

    This I2C stuff may be too lightweight for commercial UAS platforms like this going forward. We'll have to see.

    -m

     
  20. Ryan McMaster

    Ryan McMaster Active Member

    Joined:
    Mar 8, 2014
    Messages:
    289
    Likes Received:
    40
    I feel the same way about I2C Mike. My octo is double redundant (dual FC) but the HL hex I fly more often is not. I have a test rig that always gets fresh firmware and its a wonderful idea.

    The Herk gear is great! I just finished a flat 8 using Herk and it flys perfectly. The setup can be a bit annoying but over all its a great system.
     

Share This Page