At Approximately 15:20 on Thursday 12/04/2014, A flight of our Cinestar X8 originated from a public area in Salem Ma. Approximately 3.5 minutes into the flight, the UAV rolled to the right, became inverted and impacted the ground. The GPX data is of course attached. I would appreciate any and all opinions on what may have happened and most importantly how to mitigate it. If anyone knows of a SOLID reference on interpreting the raw GPX Data, I'd love to know (Andy, I got got MKTOOLS DVD someplace, but misplaced it) This ia a learning process, Please opine so hopefully others will benefit. I have pictures of the gory outcome detail if anyone interested, but if you have seen one of these, you have seen them all. Here are some of the details about that flight in no particular order. Equipment Overview All Freefly carbon parts X8 Config, MK FC and BL3 purchased in May 14 (versions below) KDE 4014's with 18" props Graupner radio for the X8, DX6 for the traditional Radian 3 axis gimbal Single 16000 mah centrally mounted, no batteries on boom trays. When recovered showed 71% capacity. Canon 5DMKII, 24mm Prime This uav was new in Mid-may and has about 250 no-issue flights on it. It was going to be one of my last flights before taking it all apart and sending the KDE's back for shaft refurb and upgrading the BL and FC firmware. Conditions There was a light wind of approx 6mph with gusts to 12mph. Well within operating limits for our craft and for shooting stills. Light was good, and approaching dusk. All equipment was checked and operational before takeoff (battery, compass, gps, radios, video feed) Flight Area This was not a commercial flight. This was a personal projects flight. We have been documenting the removal of a local landmark - the Salem Powerplant. The power plant has been shutdown for 6 months and is being demolished. ====> The plant is NOT producing, or capable of producing electricity. All of the storage tanks and pads have been removed. We were never within several hundred feet of any structure. (Except for a small unoccupied building on the pier behind us) Flight After takeoff, we performed a standard check of systems - Fly up and out approx 30', confirm radio, battery and that positive control of uav and gimbal exists, verbally confirm flight ok status with cam-op and resume. Up until the final seconds of the flight, positive control was maintained, and the ship was performing perfectly. Altitude was constant at about200' speed was slow for photographic purposes. After initial ascent, flight was typically slow and straight . At about 3:20 into the flight, the uav rolled to the right (without any stick input - confirmed by flight data) and became inverted. This roll was not jerky or abrupt. It was "perfectly performed" at about (estimate) 30 degrees/second resulting in level inversion. Control could not be re-established and the UAV impacted the ground in a horizontal attitude. (After/during the crash I heard a bunch of kids laughing at a boatyard about 250' away, and suspected that they might have hit it with a pellet gun or something. After I dissembled the uav, I so NO indication of that.) Outcome Due to the perfectly inverted impact into a puddle with rocks and loose mud, it's pretty much a total loss. I believe one leg from the gimbal is actually intact. The area of impact was a collection of 6-10" deep puddles in an open area. There were no trees or structures in the area. The uas was submerged and the camera was sheered off, extensively damaged and submerged. All flight and gimbal electronics were submerged with power being maintained for at least a minute until I disconnected the battery. All motor booms (not the subbed ones with the battery trays) were sheered off. ALL cables (I2C buss and others) were hot glued and amazingly intact and seated. Virtually all carbon fiber is compromised in some way. Many "accessories" eg: bl cooling fins, various small electronic components, etc are missing The 5D is toast, the Body is cracked open and submerged in gritty, muddy water Lens is gone as well, filled with mud and shattered. The CF card survived and delivered the shots. VERY initial analysis of the flight data suggests that there was an I2C error, but based on the fact that it occured post inversion and after significant altitude loss, I think it may be from the impact, but am TOTALLY open to suggestions. GPX Overview - basic Flight date: 12/4/2014 3:22:25 PM Flight time: 3:22:25 PM.4 - 3:25:51 PM.4 Duration : 206 secs, 00:03:26 Batt. time : 208 secs, 00:03:28 Start Location : +42.5222841 / -70.8809393 Elevation(GPS) : 2.348 62.31 79.434 m (min/avg/max) Altitude(Barom.): 4.25 78.26 98.05 m Vertical speed : -9.33 0.36 3.25 m/s Max speed : 19.8 km/h Max target dist.: 6.8 m Max distance/LOS: 62.3 m / 114.8 m Sats : 7 7 8 Voltage : min. 22.1, max. 23.8 V Current : 0.5 56 73.1 A Wattage : 12 1292 1681.3 W Capacity: 3283 mAh Motor1: 2.5 6.4 9.7 A Temp: 5 8 12 °C Motor2: 2.5 6.6 10.2 A Temp: 6 11 16 °C Motor3: 4.6 9.0 12.9 A Temp: 10 17 22 °C Motor4: 3.6 8.0 11.9 A Temp: 10 12 17 °C Motor5: 2.2 5.1 8.4 A Temp: 7 8 11 °C Motor6: 1.7 3.5 6.3 A Temp: 12 13 17 °C Motor7: 2.3 6.0 9.7 A Temp: 7 8 13 °C Motor8: 5.5 9.4 12.9 A Temp: 8 13 18 °C Magnet Field: 106 108 112 % (ok) Magnet Inclination: 54 63 83 deg Errors / warnings: I2C errors: 010 Error "ERR: FC I2C" (17) occured 1 times! GPX Overview - extra GPX Creator: NC (2.0) [MK Version] FC HW:2.5 SW:2.06a NC HW:2.0 SW:2.06a BL:V3 MK Time: 1042min [License Data] Info : No License installed [Settings] Setting No. : 3 CompassOffset: 0 FCOrientation: 0 GeoMag : -15 MagSensor : internal MaxAltitude : 150 (meters) RotationRateLimiter: VarioFailsafe : Bytes : 6b,32,51
Aw man, that's a heartbreaking loss. So sorry. I took a look at your GPX file, and your assessment of the I2C errors seems right on. The copter was definitely still in the air at the time of the I2C errors, and there were many of them right at the end of the file. Unfortunately, it's well-documented that the Mikrokopter GPX recording system "buffers" its writes to the microSD card, and only flushes them every so often. Consequently, we often see the file ended prematurely in the case of a crash. This is what I think happened to your file. But at the end of the file (the last two lines), you see the FC_I2C_Error_Counter with values of 023 and 010. I don't know if these are decimal or hex, but either way, there were a lot of them. The appearance of the red error flag in the end of the file when viewed in the MK GPXTool app confirms that. And unfortunately we also know that the Mikrokopter I2C bus is notoriously fragile, and a serious error on the bus will usually cause the entire bus to fail. The results of that are completely random, but usually result in a crash. The question that is going to be hard to answer is "why?". In the state the copter was in after the crash, it was likely pointless to try and examine every connection, particularly those little Molex connectors that connect the PDB<->FC<->NC. But there are other reasons for the bus to fail, including BL errors such as MOSFET failures. I do see that the copter was pitched at 22° and moving at a decent speed at the end of the flight. The current draw of a couple of the motors was quite a bit higher than others, but the numbers are not particularly alarming (11A or so). In summary, this is very worrisome. Holger has built some newer versions of some of the boards ostensibly to support redundant flight controls, and they no longer shut down in the event of I2C errors. But it's not clear that this is the answer. The reliance on a digital bus to carry these critical signals is clearly a design limitation, not to mention an issue for interoperability. Hopefully others can shed additional light on this. I can say with confidence that you were operating prudently and respectfully, and IMHO this is just (very) bad luck.
Mike Sorry to hear. Very well written and detailed explaination!!! Very unfortunate outcome. I looked at the GPX file and in my opinion that the I2C error you see in the last line actually caused the crash. If you look at line 390 - You are flying normal at 70 meters in altitude with normal roll angle, on the next line the roll angle starts to go bad, and then the next line which is the last line (line 392) you are 38 meters and then the error occurs. I am not sure that a severe roll could cause a I2C error especially if you hot glued the molex connector. So what came first the chicken or the egg? You are only seeing sample data twice a second and I think an I2C "glitch" can show up a line or two later than it really does. Again this is my theory. I believe in "some" isolated cases that the stress from constant landings can cause connection issues on the I2C buss between the two power boards. The constant stress can cause fatique in the connection and that connection goes slightly intermittant causing a crash. Again these are just my own theories based on my own personal experiences and I'm sure others will chime in as well. I know that there are a lot of these boards out there and maybe my theory is flawed. Here's my question to everyone reading this, can the latest and greatest boards with redundancy prevent a possible I2C issue like mentioned above? I believe Andy thinks the new boards have full I2C redundancy but I think only Holger could answer this.
Such a tough thing to have to read...really sorry about the crash. I have a theory - maybe others can chime in on it too. I think line 391 is where your problem happens. If you look, prior to that you are recording log entries twice a second as you are supposed to. On 391 look over to the compass data. Your compass info had been running within about a few degrees of each other. A couple seconds prior (line 386) your values begin to get further apart. I have heard that 10 degrees is the tolerance - but am not 100% certain it is a hard limit. Line 387 you are 10 degrees apart, line 388 you are 11, line 389 back to 10, line 390 back to 11, and at 391 you jump to 36 degrees apart and that is when your roll angle freaks out (without your stick input - I verified this). Your FC registers error counter code 23 on this line when it happens. And, at this point you have had no deviation in your height at all. Your log then misses entries for 5 seconds (10 skipped log entries). Your next entry you must be upside down, but you have decended from 71 meters to 38 meters. Your FC records another error code there (error 17 - counter code 10). That was your last entry - and you are still 38 meters high. Note that at 38 meters you stop drawing any current as well - as if it was shut off. So there are a couple things to look at. What caused the compass problem - metal / iron / etc - or do you use the external second compass and maybe it came loose and partially rotated. Either way, it looks like a huge compass discrepancy began the problem. Also - to indicate that the motors were stopped in the air, the battery voltage difference between lines 391 and 392 is a full volt INCREASE from 22.2 up to 23.2 (indicating the load being removed from the battery). This corresponds to the amp draw going from 68.2 to .5 (probably just the LEDs). As for the firmware, you were on 2.06a and the current version is 2.09b. Lots of updates and bug fixes since 2.06a. I think 2.06 got to version F, not many on 2.07 from what I remember and 2.08 had been around for a while too. 2.09 is fairly new and corrects a problem that limits the FC data being recorded up until the last second (a fix that I think was pushed for by Andy after Dave’s crashes that lost the last few seconds of data). Not sure it would have gotten you more data since you did get an entry after the big error and before it hit the ground. I really think it turned off up there. I vaguely remember one of the updates fixing a possible motor shut down problem upon a particular error type. The more I type the more I think thats what you experienced. The error may have caused the shut down - which in turn caused the rapid drop while upside down. Others on the BL3s could probably give more detail on the updates and what they fixed. Several guys had been sending in motors to get tested on the BL3 boards and I think some updates were made because of them - especially for running the larger KDE motors with a higher pole count. Again, sorry about the crash...and glad there was no damage beyond the copter.
Michael Good catch with the compass data. I do see the copter flying close to a structure but that's much earlier in the flight. I see what you are referring to with the battery voltage going up like the load is almost completely off it but if you look at the the individual motor current on 392 its all normal which contradicts it. I think like Steve says there is a buffer for the data and I think that its very hard to tell exactly when something really occurs because we are only looking at data every half second. If we had data more like every tenth of a second that would be much easier to detemine the chain of events. Here's some of the changes for each update. 2.07 is a beta version that was primarily for the addition of a boat mode and 2.08 I believe is the actual version of 2.07 that includes the changes and improvements to the datalogging. Bugfixes NC 2.06b: bugfix: logfile stopped for about 5 seconds if an error occurred NC 2.06d: Bugfix: Waypoint Events FC 2.06c & NC 2.06f: (10/05/2014) Bugfix: the relative rotation waypoints when the camera is not aligned with Nick (Cam-Orientation) FC 2.06e: Bugfix: BL-Config the engines were disabled 9-12 Reading SW version of the BLs All version numbers to the KopterTool email NC 2.06i: <Ele_raw> added in GPX log file BL SW versions ind GPX log file Bugfix: After deleting the waypoints with "GPS-Free" it could happen that was not processed while loading the next waypoints in the event for the first item 2.06f (21/07/2014): Security Function: Prevent the heights setpoint is significantly greater than the actual value Mingas use before the start instead of " AltitudeMinGas "
I've been looking at the GPX file and there's something curious going on. Mike: I was using MK_GPXTOOL to do the analysis -- this link might be of interest to you: http://rathergoodguides.com/dvds/mk_gpxtool-and-google-earth.html The curious thing is that if you look at the map display showing the copter's ground track it's really not moving that violently, but if you look at the simulated on-screen display, it shows a fairly consistent "nick down" (boom 1 below the horizon). This is visible from row 272 onwards -- yet, with boom 1 down, you'd expect the copter to be flying in the direction of boom 1 (which was around 046 degrees), yet it does not do this -- if anything it seems to be flying in the direction of boom 7. Mike: Did you make any recent changes to the copter -- for example, dismounting and remounting any of the boards? Relating to Steve's email: While this doesn't help Mike's situation, I believe the latest beta firmware from Holger now writes every data row to the microSD card so when folks upgrade they will not experience data loss after when a crash occurs. See http://forum.mikrokopter.de/topic-post528189.html#post528189 Steve: You are correct -- any system that uses a bus structure will have a very hard time functioning if that bus becomes compromised -- basically it's like trying to talk on a telephone and suddenly getting burst of loud, prolonged static -- you cannot separate the signal from the noise. MK does have a redundant configuration where there are two I2C busses, precisely for this purpose. As to why the I2C bus failed -- that's impossible to say without more information. Dave: I would not be quite so quick to dismiss the wreckage as not having any useful clues. Mike: Do you still have the wreckage pretty much "as is" or have you disassembled it? I think there might well be some clues to be had if it is still pretty much "as is." Andy.
Thanks for the link Andy. I'll review in a bit. AFA the boards, there was absolutely no modification or adjusting of anything. Our positive control test (up and out 30 and waggle around until convinced it's OK) looked great. Your observation about the boom direction is interesting. On simple flights like this, it's not uncommon for me to fly with the nose at a 45 and not yaw to keep it straight in the direction of flight. I'd like to hear more about what your brain is saying about this. -m
Mike, I like your interpretation about the compass. MAYBE the compass board gave up the ghost, MAYBE there was some hidden iron below, and MAYBE the shifting of major iron structures during demolition contributed... I wonder of there's an easy way to see of there is any magnetic anomaly going on before flights? A whiskey compass and a gps? -m
Thanks Dave. In the software and hardware world, exactly half the times I find myself saying "I wish I had applied that update". Exactly half the time I find myself saying "What was I thinking applying that update before it was worked out" I have tended to err on the side of "If it don't seem broken, don't fix it". Going forward, I'm going to try to be religious about getting right on the updates on a test rig and flying them out. Thanks.
It's a self-serving link to one of my videos, Mike. (Hangs head in shame....) Yeah I noted that. EDIT: For ease of reference I have assumed that you're flying a flat 8 not an X8 -- sorry if this causes any confusion! Have a look at this screen capture from MK_GPXTOOL. It shows row 272 of the GPX file. (Click on the image to see a larger version.) Note the Course value: 270 -- heading west -- from right to left along the magenta line. Now look at the Compass, column. It shows 47, 40. The 47 is the "gyro-compass" value -- that is the processed value of the "raw" magnetic compass (which is the 40). So boom 1 is pointing to 047 (you can actually see the tiny little quadcopter's red boom -- that shows the orientation. Now look at the Virtual OSD window and note the compass heading is confirmed at the top center at being 047 degrees. But also note the artificial horizon showing that boom one is depressed some 15 degrees below the horizon. The question going through my brain is this: If I nick down (depress boom 1 below the horizon), the copter should fly in the direction in which boom 1 is pointed because the nick down displaces the center of lift towards boom 1. Yet, looking at the compass heading and MK_GPXTOOLS display of the copter's ground path (superimposed on Google satellite map view), shows that the aircraft was not flying towards boom 1 (at least where boom 1 was pointing) but is instead flying towards boom 7 (270 degrees) (EDIT: Actually, this is closer to where boom 6 is pointing). Yet the Nick and Roll angles (shown in their own columns numerically and graphically on the Virtual OSD) show the the aircraft is not rolling towards boom 7 (EDIT: or towards boom 6). So how come with boom 1 15 degrees down and pointing to 047 degrees is the copter's course over the ground 270 degrees? I could understand if this was a momentary phenomenon, but it's not. Normally, when I have seen this phenomenon, it is an indication of a serious compass error -- e.g. (are those big metal tanks I see in the satellite image for example?) or it is because the flight control board is not mounted with the orientation arrow pointing towards boom 1. (Not having the GPS receiver oriented correctly might do this too, but I don't recall seeing it). Anyway, that's what's going through my brain. Andy.
A whisky compass will give you some indications of anomalies, but only from the ground. And even then a compass failure is unlikely to cause the aircraft to roll inverted....the primary purpose of the compass is to stabilize the aircraft in yaw, as I understand it. Andy.
Andy: I was thinking compass too. In my post above, I noted the large gap in compass values at line 391. Can a failing compass cause something like that or would you think its more likely an external influence from metal, etc?
I wouldn't put too much stock in row 391, Michael. The I2C bus has started to exhibit errors (023 shown) -- and that means the bus is corrupting the data -- hence the gap in GPS time. Rows 391 and 392 are the result of the failure, I suspect, not the cause of it. Andy.
Andy, I've taken it all apart to look for anything interesting. What is it you would have looked for? -m
I would wash off the flight controller and the navigation controller in distilled water (which is not electrically conductive), parked them in a warm place overnight to thoroughly dry off, then using the MK USB adapter with its little jumper installed, connect up to JUST the Flight Controller -- it can be just resting on a piece of cardboard and completely disconnected from everything else. See if the MK USB Adapter can fire it up (green LED comes up). If so then try and talk to the board using MK Tool. If you can talk to it, don't be alarmed to see some errors, but try and Write the current configuration for the Easy Parameter Set (#3) out to an .mkp file and post the .mkp file on this thread. There is a chance, of course, that the flight controller has switched to paperweight mode, joined the choir eternal, and is an ex-flight controller -- and you can either use it for a paperweight or put it into the Museum of Broken Parts. Do you have any recent images of the board stack on the hub of the copter that show the flight controller so you can check the orientation of the flight controller with respect to boom #1. Hope this helps. Andy.
Regarding images, Mike. Thanks. #17 confirms all is well with FC orientation but does not explain the anomaly between nick axis and actual direction of flight. Were those large circular shapes/tanks made out of metal? I'm still trying to figure out the disparity between compass heading of boom #1 and ground track when you pushed the right hand stick forward to nick down. By the way, I can see (from image #22) you originally had version 1.02d firmware on the Quadro Cool board -- did you upgrade the BL-Ctrls to the latest firmware? There were known problems with 1.02 firmware -- these were corrected in April -- did you upgrade since then? Andy.
Pretty gruesome Andy. Thanks for looking. I, as well as others are concerned about the anomaly, as well. I got the board from QC with 1.02D as confirmed by QC after they shipped it. I had NOT upgraded it since and was planning to do it over the XMAS Holiday. Ugh. Thanks for your help Andy. -m